Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learning
Authors: Alex Chan
Mihaela van der Schaar
Venue: NeurIPS 2022 Accept (2022)
Keywords: instance-wise
ensembles

Abstract: Consider making a prediction over new test data without any opportunity to learn from a training set of labelled data - instead given access to a set of expert models and their predictions alongside some limited information about the dataset used to train them. In scenarios from finance to the medical sciences, and even consumer practice, stakeholders have developed models on private data they either cannot, or do not want to, share. Given the value and legislation surrounding personal information, it is not surprising that only the models, and not the data, will be released - the pertinent question becoming: how best to use these models? Previous work has focused on global model selection or ensembling, with the result of a single final model across the feature space. Machine learning models perform notoriously poorly on data outside their training domain however, and so we argue that when ensembling models the weightings for individual instances must reflect their respective domains - in other words models that are more likely to have seen information on that instance should have more attention paid to them. We introduce a method for such an instance-wise ensembling of models, including a novel representation learning step for handling sparse high-dimensional domains. Finally, we demonstrate the need and generalisability of our method on classical machine learning tasks as well as highlighting a real world use case in the pharmacological setting of vancomycin precision dosing.

Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learning
Alex J. Chan
Mihaela van der Schaar
University of Cambridge Cambridge, UK aj ajc340cam. ac. uk
University of Cambridge
Cambridge Centre for AI in Medicine
Cambridge, UK mv4720cam.c ac.uk
Abstract
Consider making a prediction over new test data without any opportunity to learn from training set of labelled data instead given access to set of expert models and their predictions alongside some limited information about the dataset used to consumer
train them. In scenarios from finance to the medical sciences, and practice, stakeholders have developed models on private data they either canmot, do not want to, share. Given the value and legislation surrounding personal information, is not surprising that only the models, and not the data, will be released the pertinent question becoming: how best to use these models? Previous work has focused on global model selection or ensembling, with the result of a single final model across the feature space. Machine learning models perform notoriously poorly on data outside their training domain however, and so we argue that when ensembling models the weightings for individual instances must reflect their respective domains in other words models that are more likely to have seen information on that instance should have more attention paid to them. We introduce a method for such an settaceewwise ensembling of models, including novel representation learning step for handling sparse high-dimensional domains. Finally, we demonstrate the need and generalisability of our method on classical machine learning tasks as well as highlighting a real world use case in the pharmacological setting of vancomycin precision dosing.
1 Introduction
Sharing data is often very problematic affair before we even arrive at whether stakeholder will want to, given the modern day value it may not even be allowed. particular, when the data contains identifiable and personal information it may be inappropriate, and illegal, to do so. A common solution is to provide some proxy of the true data the form a generated fully synthetic dataset (Alaa et al. 2020) one that has undergone some privatisation Chan et al. 202T), but this can often lead to low low-quality or extremely noisy data (Alaa et al. 2021 that is hard to gain insight
anonymisation process (Elliot et al. 20188
Alternatively, groups may release models that they have trained on their private data. for a given task there are multiple such models, we are left with the task of how to best use these models in combination when they have potentially conflicting predictions. This problem is known as unsupervised ensemble learning since we have no explicit signal on the task, and aim to construct combination of the models for making future predictions (Jaffe al. 2016).
Without having trained the models ourselves, it is very challenging task for us to know how well the individual models should perform, making the task of choosing the most appropriate model (or ensemble) difficult. This is compounded by the problem that the provided models could perform poorly for not one, but two main reasons: Firstly, the model itself may not have been flexible
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

enough to properly capture the underlying true function present in the data; and secondly, in the area that they are making prediction there may not have been sufficient training data used for the model to have been able to learn appropriately i.e. the model is extrapolating (potentially unreasonably) to cover a new feature point the main issue in covariate shifted problems (Bickel et al. 2009). Current practice is usually to try and select the globally optimal model (Shaham ctt al. 2016; Dror et al., 2017), that is to say of those made available, which model should be used to make predictions for any given test point in the feature space. This approach potentially addresses the first point, it completely overlooks the second and this is what we will focus on. In order to consider this problem of extrapolation amongst the individual models, the important question is: what might a solution look like? We must consider the desiderata that given no augmentation of the individual models, the ensemble weights must vary depending features and additionally, these weights should reflect the confidence that a model will be able to make appropriate prediction, and we should be able to tell generally when our confidence is low.
the test
Contributions In this work we make a three-fold contribution. First, we establish and document the need for instance-wit -wiee predictions in the setting of unsupervised ensemble learning, in doing so intro- ducing the concept of Synthetic Model Combination (SMC), shown in Figure D Second, we introduce a novel unsupervised representation learning procedure that can be incorporated into SMC allowing for more appropriate separation of models and estima- tion of ensemble weights in sparse high-dimennsonal settings. Finally, we provide practical demonstrations of both the success and failure cases of traditional methods and SMC, in synthetic examples as well real example of precision dosing- code for com/
M
M(A)
A
x
B
x C
M
which is made available at XanderJC/ wwy.thhtti-mmdeee-ooolllee along with the larger lab group codebase at https://github.. htt.p/a.....ooaaaaeeeeaaaaaaaaaa
M,
Figure I: Instance- wise Ensembles. Here represent the density of the training features for three separate models MM. M2.and M3. Given new test points A, B, and C, we need to construct predictions from these models. A is well represented by both M2 and M3 while B only has significant density under M3. C looks like none of the models will be able to make confident predictions.
2 Background
Formulation Consider having access only to N provided tuples | of models M, and associated information Ij- With each M, X' + Y being a mapping from some covariate space X' to another target space y and having been trained some dataset Dj which is not ob- served by us although is in some way summarised by Ij. We are then presented with some other test dataset Dr = M and consequently tasked with making predictions.
Goal Our aim is to construct a convex combination of models in a way so as to produce (²)x) = Σ ((x)) with En w(x); 1 and w((z) taking the jth index of the output of the function w X + A that maps features to set of weights on the probability N-simplex. We can summarise our task process as:
optimal model predictions
M Given: ((M,,Z)) and M
Obtain: w to predict (
Having established the setting we are focused on, we explore contemporary methods and compare how they relate to our work. We consider methods that are interested ultimately in the test-time distribution of labels conditional on feature covariates p xy- We discuss the differences in methods focuses on the distributional information available to them, summarised in Table For example, in standard supervised learning, samples of the feature-label joint distribution p(x,Y) are given the training-time time conditional distribution is then estimated and assumed to be equivalent test-time.

---

Table 1: (Un-)Related Fields and how they compare terms of: A) whether they make instance-wise
predictions; B) the distributional information which they require; C) the form of the information: and
D) the target quantity they are focused on. References given as: [1] Torrey and Shavlik (2010); [2]
Raftery et al. 1997); Ren et al. (2019); Ruta and Gabrys 2005 [5]
Jaffe et al. (016))
Problem
Ref.
A) Instance-wise
B) Distribution
C)
Information
D) Target
Transfer Learinng
[1]
Mrain Pl3t
Drvain
Pixi M(MID) px (nfoat)
Bayesiun Model Averaging
[2]
X
P(X,Y)
Dval
Dut-of-distriuttin Detection
[3]
X/ /
P
PX Piyi
<
Majority Voting
[4]
X
POXYY)
Unsupervised Ensemble Regression
[5]
X
P(Y)
B|J], VarlY)
Synthetic Model Combination
[Us] V
(P
wxxx)
In our case, we assume information from considerably less informative distributions, the train- ing time feature distributions 1 were the information most practically though through samples of the features or details of the first and second mo- ments. Practically, this appears to be the minimal set of information for which we can do some- thing useful since if only given set of models and no accompanying information then there is no way to determine which models may be best in general let alone for specific features.
take variety of forms,
How are models normally ensembled? The literature ensemble methods is vast, and we do not intend to provide a survey, a number of which already exist (Sagi and Rokach] (2088] Dong et al.) 2020). The focus is often on training ones own set of models that can then be ensembled for epistemic uncertainty insight (Rahaman et al. 2021 or boosted for performance (Chen and Guestrin 2016).
In terms of methods of ensembling models that are provided to practitioner (instead of ones trained by them as well) then closest is the setting of unsupervised ensemble regression which like us does not consider any joint feature label distribution. To make progress though some in- formation needs to be provided, with Dror et al. 2017) considering the marginal label distribution plazy, making the strong assumption the first two moments of the response are known. Instead of being directly provided the mean and variance, another strand of work assumes conditional independence given the label (Dawid and Skene 1979), meaning that any predictors that agree consistently will be more likely to be accurate. Platanios al. (2014) and Jaffe et al. (2016) relax this assumption through the use of graphical models and meta-learner construction respec- tively, with a Bayesian approach proposed by [Platanios et al. (2016) Recent work of [Shaham et al. (016) attempts to learn the dependence structure using restricted Boltzmann machines.
Model averaging when validation data is available. Moving from the unsupervised methods mentioned above, which at most considered information from only marginal distributions, we come to the case of when we may have some validation data from the joint distribution teat (or one assumed to be the same). This can effectively be used evaluate a ranking on the given models for more informed ensembles. This ensemble can be created in number of ways (Huang et al. 2009). including work that moves in the direction of instance-win wise predictions by dividing the space into regions before calculating per-region weights (Verikas et al., 1999). A practical and more common approach is Bayesian Model Averaging (BMA) (Raftery et al. | 1997p. Given an appropriate prior, we calculate the posterior probability that given model is the optimal one and once this is obtained the models can be marginalised out during test time predictions, creating an ensemble weighted by each model's posterior probability. The posterior being intractable, the probability is approximated using the Bayesian Information Criterion (BIC) (Neath and Cavanaugh, 2012)- which requires a likelihood estimate over some validation set and is estimated as: p(M,DD)
With this, along with all the ensemble methods previously mentioned, it is important to note the subtle difference in setup to the problem we are trying to work with. In all cases, it is assumed that there is some ordering for the models that holds across the feature space and so a global ensemble is produced with a fixed weighting i such that w(xx) = á Ya E X. This causes failure cases when there is variation in the models across the feature space, since it is a key point that BMA is not model combination (Minka 2000). This being an important distinction and one of the main reasons BMA has been shown to perform badly under covariate shifted tasks (Izmailov et al. 2021). That being said, it can be extended by
EN c
3

---

considering the set of models being averaged to be every possible combination of the provided models (Kim and Ghahramani 2012). although this becomes even more computationally infeasible.
Is this some form of unsupervised domain adaptation or transfer learning then? Given the focus models performing on some region of the feature space outside their training domain this may seem like a natural question. Unsupervised domain adaptation represents this task at an individual model level but usually considers, access to unlabelled data in the target domain as well as labelled data from a (different) source domain frasn fest (Chan et al. 2020). We Pi ((())
refer the interested reader to Kouw and Loog 2019) for detailed review given space constraints.
In very similar vein, the transfer learning (Torrey and Shavlik 2010) task also involves change in feature distribution but instead of being completely unsupervised tends to include some labels on the target set, thus requiring some information on the joint distribution both test and training time Y') (').Y) A great deal of work then involves learning a prior on the first domain that can be updated appropriately on the target domain (Raina et al. 2006 Karbalaygharch et al. 2018). In contrast to both of these areas, we do not aim to improve the performance of the individual models, but rather combine them based on how well we expect them to perform in the new domain.
3 Introducing Synthetic Model Combination
Our success hinges on the assumption that the quality of a model's prediction will depend on the context features. That is to say that the performance ordering of the models will not stay constant across the feature space. With this being the case, it should seem obvious that the weightings of individual models should depend on the features presented. As such, we in- troduce our notion of Synthetic Model -ombinatior[] a method that constructs a representa- tion space within which we can practically reason, enabling it to select weights for the mod- els based on a given feature's location within the representation space. Recalling our starting point of I,M and {x,),, we proceed in three main steps which are outlined below:
1. Estimate densities for models: From each information Zy. we generate a density vz (z). M 2. Learn low-dimensional representation space: Using { and (r)())) learn a mapping to low dimensional representation space fo X' + Z.
3. Calculate ensemble weights for predictions: Evaluate weights wx(x)) so that predictions can then be made as ÿ Σ (xx))),,(zI.
These steps highlighted again in the full Algorithm I well shown pictorially in Figure 2
3.1 From Information to Probability Densities
The first step in SMC is use the information I, to produce density estimate such that we can sam- ple from each model's 's effective support. Given the flexibility in the form of what we allow I, to take, SMC must remain relatively agnostic to this step. common example of the type of information we expect will simply be example feature samples, and in this case a simple kernel density estimate (Ter) rell and Scott 1992) or other density estimation method could be employed. On the other hand, the medical setting for example, when models are published authors will often also provide demographics information on the patients that were involved in the study, such as the mean and variance of each covariate recorded. In this case we may simply want to approximate the density using a Gaussian and moment-matee matccngg for example. When the information is provided in the form of samples it is possible to skip this step, they can be used directly in the subsequent representation learning step's 's losses.
3.2 Learning a Separable and Informative Space
The point of learning a new representation space and not simply using the original feature space is twofold. Firstly, we would like to reduce the dimensionality, leading to a more compact rep-
'Our name is a nod towards synthetic control (Abadie et al. 2010 as construct new synthetic model as convex combination of others that think will be most appropriate for a given instance.

---

resentation and is important because in higher dimensions the densities we model will often end up effectively non-zero in only very small regions, making the last step of SMC difficult. Sec- ondly, we would like to induce structure, so that distances and regions in the space better reflect how capable different models will be over the space. This should aid better selection of model weights when making predictions by allowing SMC to better understand the relationship between the models and the representation space. We should note that if the feature dimensions are low then this step is not strictly necessary- the model densities may have appropriate coverage of the space, but this does not then allow you to obtain none of the benefits of the second point.
To proceed we define a space Z on which we will work and aim learn a parameterised map- ping fo X' + Z such that the representations of features on this space is useful and aids us in
Orgaial dhmuet-
Prosuce wndi
Enaslli
our goal of constructing instance-wise ensem- bles. In each optimisation step of our algorithm we will sample model dataset DM = { J}=1 sampling one feature example 2 from each den- sity associated with model. This model dataset will play an important part in the learning pro- cess as these examples serve as proxies to the regions in the feature space that a model is con- fident on. Now, given both Dr and DDM, we aim to construct an optimisation target for represen- tation learning step that can be learnt end-to-end an autoencoder fashion. As such, we introduce a total loss that can be broken down into three parts:
Pronieel iyfermtivr
3. Caladaae cnasnk segght
2 Leave rpproratation
Exiizaat
Figure 2: Diagrammatic Model Approach. We visually represent the steps (coloured) and objects (italics) involved in the SMC approach.
Liotel = &.econstruction + Leonnection + sepperation
(1)
The first, Creconstruction is the simplest used so that the latent space can accurately represent and reconstruct the feature space:
simple regularised autoencoder loss and is
Lree Σ
(Iao(J((x)) - 2.112 + 31150(2))11
(2)
ET U D.
This introduces second function Io : 2 + X' that can be jointly optimised in order to learn the space using standard autoencoder techniques. Alternatively, to incorporate prob- abilistic element, we can substitute in a (3-) loss (Kingma and Welling 2013: Higgins et al.) 2016) by considering samples from an approximate posterior, the probabilistic nature has been shown to improve interpretability in the representation space (Burgess et al., 2018).
We next introduce the first of our more specialised components with the Lconnection loss:
Leon
Σ
(1 (y(M, x()ff((.) fo((t))
(3)
(b) DM DD4
Predictive Similarity Soore
which is designed so that models that make similar predictions have domains that map to similar areas in the space. Here, Dy |1 can be any normalised distance metric over ) and is used to calculate the Predictive Similarity Score between two models evaluated on the same feature. This loss aims to minimise the distance [[, between the domains of models i and j when the predictions between the two models are more similar. This is based on the intuition that if models are making randomly incorrect predictions for a feature they are unlikely to agree with other models (Dror et al., (0077). Thus, these points are more likely to be correctly classified suggesting some overlap in training domain of the two models.
fo(ī,) )|| which is a proxy for the distance
Given the non-negativity of distances we will need to balance the previous loss, which will always aim to reduce the distance between all the pairs in the loss, albeit up-weighting those that have more similar predictions between models. Thus, we employ Lxeparation
Leep-
Σ
1 1 (m)((m) + (m)) x (lfo(E.) - fo(2))11 2
1
(R,,2,) E DMIDD
II is possible to sample larger model dataset to provide lower variance estimate of the loss at each step, but given the pairwise distances will calculate this can become computationally challenging.
5

---

which is designed so that models are naturally moved away from each other- essentially encoding a prior that data distributions for the different models will be distinct. With 1 denoting the indi- cator function and sr() denoting the model density from which the point was sampled, this loss pushes apart points that were not sampled from the same model. These losses can be balanced with weighting hyperparameters, the potential optimisation of which is discussed in the appendix.
3.3 Weights Estimation
Once a space learnt, we can use to make predictions. Given model densities in the fea- ture space, denoted p (z), we construct a cor- responding density in the representation space pf - this can be achieved simply by sam- pling from (x)), passing through So and mod- elling the new density with a kernel density estimate. From here, we calculate weights as the relative density a feature representa-
Algorithm 1: Synthetic Model Combination
Result: Test predictions using mapping from data to model weights
Input: {[M, and Dr;
1. Use information to produce density models;
2. Sample data from models and combine with test data;
Learn representation space;
4. Re-model densities in new space;
tion has under the densities in the new space:
5. Calculate weights in new space;
(fo(z))) w(x), ((
6. Make predictions {ū. x== over test set;
(5)
Return: {
with y a regularisation hyperparameter chosen be very small such that an outlier's weights are not dominated by the closest model. The quantity Σ}= p5 of any prediction made by SMC. Particularly low values will indicate that the feature had low density under all the domains and as such it may be likely that none of the models were accurate. We note as well that assuming a hierarchical generative model for the test data where one of the models training data distributions is selected and then sampled from this can be interpreted the posterior probability that a test instance was sampled from a model's 's domain and is thus well represented by it.
(x)) can be used to inform the confidence
4 Experimental Demonstration
In this section we will use a series of experiments to make the following concrete points about our method: 1) In common scenarios, global ensembles do not work, and we must make instance- wise predictions (Section : 2) Doing so in even slightly high dimensions requires a repre- sentation learning step, and our proposed losses improves the quality of the learnt representation (Section 4.2); 3) We can make good predictions with surprisingly little information (Section 4.2 4) This is useful beyond synthetic example setups in real-world case scenarios (Section 4.3): 5) Naturally, there are setups where we will underperform, which we should understand (Section
4.4); and 6) We are completely agnostic to the type of models used (Section 4 we demonstrate across a variety of models from simple regressions to convolutional nets and differential equations).
4.1 A Simple Regression Example
In this first example, aim demonstrate why this approach is clear necessity when making future predictions, showing that we need to construct ensembles with an instance-wise approach to generate predictions that are accurate and appropriately calibrated. In our examples, simulate feature data from experiment-dependan distributions and targets from a simple noisy sin wave curve.
Instance wise ensembles are necessary under our assumptions. In our first example we con- sider two models: both simple neural networks, one trained on features sampled from a Gaus- sian centred at 5, with the other centred at 15, and standard deviation 3.5 as can be seen from Figure 3 there is very little overlap in the support of the training data for the two models, and it can be seen that Model 1 clearly makes appropriate predictions when < 10, while it is Model 2 that is accurate when x > 10. This relationship cannot be captured by a global en- semble of the two methods, as seen in Figure 3 whereas SMC is perfectly able to do so.
Our uncertainty can show when we are not confident. our previous example, SMC was able to match the target function across the feature space given the distribution of the training

---

Problem Setup
Method Predictions
Uncertainyy with low overlop
Marginal gains with high ovelup
Tapst
Tape
5MC BAA
SMC
SNLL
Makel
IMA
Nave
Madel
Navo
-1
-1
-|
Traneg
Figure 3: Regression Example. In all cases the background colour reflects the optimal model. This problem setup (a) demonstrates the key need of an instance-wise ensemble. It can be seen from the model predictions (b) that SMC is the only ensemble that can capture the target function. Altering the setup such that training domains are further away (c) shows our uncertainty (background colour concentration) can helpfully highlight where we should not be confident. Finally, when domains completely overlap (d) we can see all ensembles perform equally well and SMC loses its edge.
Coppormatos foenecttion
7
Digit
Din
Dim
High boweers and
4
AUROC:
Dim 1
Dim
Figure 4: MNIST Example. Representation spaces learnt from MNIST using just the reconstruction loss well adding the separation and connection losses. Model information consisted of feature- only samples from the model training data. Test points are plotted to show their relationship to the model densities, showing that in low-dimensional space they usually have high density under model.
features of the two models. What happens though if none of the models make accurate predic- tions on an instance because it is outside all of their domains? We explore this by moving the two Gaussians used to generate features for the two models further apart, centring them at 0 and 20 respectively resulting in region around 10 where none of the models make correct predic- tions, and consequently SMC's predictions also suffer as seen in Figure Bt. However, using the uncertainty explained in Section 3.3 (which is visualised by the background colour of the plot) we can see that the accuracy of SMC is calibrated well against its confidence. SMC can pro- vide useful information about which parts of the feature space it is relatively less confident on.
4.2 Higher Dimensions with MNIST
In order to develop further points of our method, we move to a more complicated example with the most iconic ML problem of handwritten digit classification. Using MNIST (d = 784), we construct a problem where ten different classifiers are trained to each individually identify a single digit effectively while their performances on other digits are significantly lower this is achieved by providing mostly only data of the respective single digit. The information provided per model involves a number of feature samples from the training set (number depending on exact experiment).
High dimensions need to be reduced. As discussed earlier, density estimation becomes ineffective in high dimensions. the space gets bigger, the relative proportion covered by the data reduces. We find that even in the case where the full set of features (but of course not targets) used to train the predictive models are made available to SMC, none of the test set features have non-zero density to numerical precision and hence a meaningful prediction cannot be made. On the other hand, once we incorporate representation learning step, the coverage of the training examples significantly increases. This can be seen in Figure 4 which shows two separate 2-dimensional learnt
7

---

representation spaces: on the left using only the Lree loss; and on the right including the Seep and Loon losses. Here. test features are plotted as points, while the training features used per model are represented as density estimates. It can be very evidently seen that the majority of test features have significant density under at least one of the models in both of the example representation spaces learnt.
Standard representation learning is good but we are better. We have established that some low- dimensional representation learning is necessary for problems like these the fact is that by simply in- corporating a standard representation learning step, using a standard autoencoder for example, will allow us to make reasonable predictions, indeed we can achieve OneVsRest Area Under the Receiver Oper- ating Characteristic curve (AUROC) of 0.969 on the test set. By including the extra regularisation though we can improve the ability of our algorithm. We can immediately see with a visual inspection of Figure 4 and the two learnt representation spaces that their inclusion results in more spread out and differentiated densities (both are plotted the same scale). Addi- tionally, we see and increase in the AUROC to 0.972. This appears to allow more appropriate evaluation of
AUROC as Information Increases
0.9
0.5
SMC
Unccrraiityy Weighting
0.7
Majerity Vodig
0.5
0.5
10'
10"
10'
Num, of Samples (lug scale)
Figure 5: Information Gain to Performance Relationship. The AUROC obtained by SMC against the number of training data feature samples given to the method on a log scale. points like the example '4' in Figure 4 which could reasonably be mistaken for an incomplete '9'.
Many examples are unnecessary for significant performance gain. In Figure 5 we plot the predictive AUROC of SMC against the number of feature samples from the training data passed to the method. In this setup the information provided to SMC are a number of feature examples per model that can be sub-sampled from to evaluate the regularisation losses and then used to construct density estimates in the representation space. Naturally, as the number of samples increases so does the quantity of information and so too does the performance of SMC. In the extremes, of course with no samples no meaningful prediction can be made, and when all the training features provided an AUROC of about 0.97 is achieved. Interestingly, we can see that actually not a very large number of samples required to obtain strong performance here. Notably, after only 3 samples are provided, SMC sees significant performance improvement to 0.83, compared to the baseline of 0.50. This is not significant amount of information, with just three features and none of the target details.
Further failure of global ensembles. We include some additional global ensemble baselines in Figure 5 to demonstrate that they fail in more general settings than the synthetic example we previously discussed. We include a simple majority voting ensemble as well as a method that weights predictions based on measure of the uncertainty of the ensemble member. In particular, we weight proportional to the exponential of the reciprocal of the entropy of the predicted categorical distribution.
4.3 Case Study: Vancomycin Precision Dusing
Population pharmacodynamic (PopPK) models are differential equations that model the concentration of drug in the bloodstream of a patient. Van- comycin one of the most common antibiotics
with multiple PopPK models (Broeker et al. (019)) having been developed on different patient popu- lations. For many drugs, including vancomycin, the area under the curve (AUC) of blood concen- tration over time is an important marker in estimat- ing the effectiveness of a drug intervention, and can be used to support dose individualisation by predicting drug response in the future. An exam- ple of the models are found in Figure 6 showing the predicted concentration levels in an example patient for number of different PopPK models.
Figure 6: Example Patient Blood Concen- tration. Predicted concentration levels in the blood of an example patient. Fach line repre- sents a different model with the black being an ensemble. The area under the curve (AUC) is an important target to estimate the impact the drug will have, as it reflects the total expo- sure to the patient, and one that is impossible to calculate directly without continuous mon- itoring. Graphic produced using the TDMx Wicha et al. 2015) software.
SMC can be applied to real challenges. This phar- macological setting is one where SMC would be
8

---

Table 2: Model Performance for predicting the AUC given varying numbers of blood concentration
observations. We report relative root-mean-squas enror (RMSE) note: lower scores are better.
Model
A priori
One
Two
Three
Four
Adane al. 20155
80.0 0.7%
47.3 + 0.4%
40.9 0.6%
32.5
0.3%
29.4 0/3%
Mangın et al.
(2014
83.3
0.3%
0.30
15.6 2.24
Medellin- Garblay et al.
2016
16.5=0.6%
20.9 2.25 0.2%
17.4 1..7+
Revilla et 201O
0..7%
0.2%
20.0+
15.3 + 0.2%
Roberts et al.
(2011
35.8 0.3%
0.1%
20.0
0.11
14.5 + 0.2%
Thomson et al.
2009
48.2 0.4%
30.2 0.2%
20.9 2.2%
+ +
Model Averaging
55.7 + 0.4%
28.5 0.2%
24.9 + 0.2%
19.3 + 0.2%
16.6 + 0.2%
SMC
41.8 = 0.5%
22.1 + 0.2%
19.6 + 0.2%
15.2 + 0.2%
12.8 = 0.1%
particularly appropriate method of choice. First, the private medical nature of the problem means that data on drug response in humans is not widely available, and so new models cannot be trained on the data of the previous models. Second, the datasets that the models are built on are often relatively small and focus on a specific subpopulation, such as those at particular hospital, or suffering a spe- cific comorbidity. Third, researchers usually publish the model alongside demographic information on the patients used to produce them, fulfilling SMC's need for both model and information.
Accurate Drug Response Estimation. We base our experiment around those of Uster et al. (2021) who themselves consider an ensembling approach through the application of model averaging. We use simulated patients provided by the authors evaluate the effectiveness of SMC in the accuracy of predicting the AUC across a number of settings when a number E {0 (A priori), 1, 2, 3, 4} of concentration measurements are taken in a 36-hour period. Ultimately, we have six models, each from separate subpopulation !extremely obese, critically ill post heart surgery, trauma patients, intensive care patients, septic, hospitalised patients), as well as variety of demographic infor- mation for each In our experiments we focus on the age, height, weight, sex, and creatinine clearance levels as have been shown to be strongly associated with drug response {Uster al., 2021) and are provided for each model. In Table 2 we report the relative root-mean-squane error RMSE of the predictions the lower, the hetter. We can see that SMC consistently performs well and indeed performs the best when at least two observations are made available to the methods.
4.4 Understanding Challenging Scenarios for SMC
We must accept that there's no such thing as free lunch, and SMC does not have the answer for everything there are scenarios where it may underperform for some reason other than obvious cases when model information is not available or where individual models are themselves bad.
Performance discrepancy between models leads to worse predictions. Table 2 highlights situation where SMC may underperform. We see that SMC performs worse when there is high variability in the performance of individual models. For example, in the A priori setting, there is very large range in RMSE, from 32.7 all the way up to 83.3. Since SMC does not attempt to evaluate the relative performances of the models, when there are models that just perform very badly they can severely detract from SMC's performance. This highlights that SMC performs best when all the models perform well in their respective domain, but that those domains disjoint. Could you do anything if you know (through perhaps a validation set) that some models perform very badly? simple solution would involve also calculating the weights given by BMA as well, before averaging both sets of weights This would weight models globally by some level of how confident we are that the model is good as well as locally by how well we believe the model will be able to perform on a specific feature, balancing the potential causes of poor performance.
relatively
Domain overlaps result in marginal improvement. To examine this point we shall revisit the regression example of earlier. SMC assumes that the domains of the different models are at least partially different so that the feature dependant weights allow for selecting the model(s) that most appropriate. This benefit is lost when all the model domains are the same, as we show in Figure 3 all the models are equally good across the feature space and so an ensemble will perform just as well as SMC. It is important to note SMC does not perform worse, it just does not perform better.
Further information on the setup, including models and associated information can be found in the appendix.

---

5 Discussion
Society and Ethics. The setting of our method is primarily focused on situations where the sharing of data is in some way tricky limited key example being the medical regime. As such, our method is designed to be able to perform well this area and so would hopefully have an overwhelmingly positive impact. Of course with any method there is potential for it to be misused in an application which has damaging effects, but seems unlikely that SMC poses any intrinsic danger.
Conclusions. In this paper we have introduced the framework of Synthetic Model Combina- tion an instance- wise approach to unsupervised ensemble learning, having established that there are many cases when global ensembles are simply inadequate for meaningful predictions. We additionally introduced novel unsupervised representation learning method for the sparse high-dimensional setting, and showed the use of our method in both example synthetic prob- lems and the real case study of estimating the effectiveness of vancomycin precision dosing.
Acknowledgements
AJC would like to acknowledge and thank Microsoft Research for its support through its PhD Scholarship Program with the EPSRC. This work was additionally supported by the Office of Naval Research (ONR) and the NSF (Grant number: 1722516). Thanks to Sebastian Wicha and David Uster for helpfully providing simulations from their own research for the vancomycin example, as well as Jean- -aan-Baptiste Woillard for encouraging us examine this problem. We would also like to thank all of the anonymous reviewers on OpenReview, alongside the many members of the van der Schaar lab, for their input, comments, and suggestions various stages that have ultimately improved the manuscript.
References
Abadie, A., Diamond, A., and Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect california's 's tobacco control program. Journal of the American statistical Association, 105(490):493-500
Adane, E. D., Herald, M., and Koura, (2015). Pharmacokinetics of vancomycin in extremely obese patients with suspected or confirmed s taphylococcus aureus infections. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy, 35(2):127-139.
Alaa, A., Chan, A. J., and van der Schaar, M. (2020). Generative time-series modeling with Fourier flows. In International Conference on Learning Representations.
Alaa, A. M., van Breugel, B., Saveliev, E., and van der Schaar, M. (2021). How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. arXiv preprint arXiv:2102.08921.
Beal, S. and Sheiner, I.. (1979). Nonmem user's guide i: User's basic guide. Division of Clinical Pharmacology, University of California at San Francisco.
Bickel, S., Brückner, M., and Scheffer, T. (2009). Discriminative learning under covariate shift. Journal of Machine Learning Research, 10(9).
Broeker, A., Nardecchia, M., Klinker, K., Derendorf, H., Day, R., Marriott, D., Carland, J., Stocker, S., and Wicha, S. (2019). Towards precision dosing of vancomycin: a systematic evaluation of pharma- cometric models for bayesian forecasting. Clinical Microbiology and Infection, 25(10))
Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. (2018). Understanding disentangling in beta-vae, arXiv preprint arXiv:1804.03599
Chan, A. J., Alaa, A., Qian. Z., and Van Der Schaar, M. (2020). Unlabelled data improves Bayesian uncertainty calibration under covariate shift. In International Conference on Machine Learning, pages 1392-1402, PMLR.
Chan, J., Bica, I., Hüyük, A., Jarrett, D., and van der Schaar, M. (2021). The medkit-learn (ing) environment: Medical decision modelling through simulation. Thirty-fiith Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
10

---

Chan, A. J., Curth, A., and van der Schaar, M. (2022). Inverse online learning: Understanding non-stationary and reactionary policies. International Conference on Learning Representations.
Chan, A. J. and van der Schaar, M. (2020). Scalable bayesian inverse reinforcement learning. In International Conference Learning Representations.
Chen, T. and Guestrin, C. (2016). Xgboost: scalable tree boosting system. In Proceedings of the 22nd aem sigkdd international conference knowledge discovery and data mining, pages 785-794.
Dawid, A. P. and Skene, A. M. (1979). Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20-28.
Dong, X., Yu, Z.,, Cao, W. Shi, Y., and Ma, Q. (2020). survey on ensemble learning. Frontiers of Computer Science, 14(2):241-258.
Dror, O., Nadler, B., Bilal, E., and Kluger, (2017). Unsupervised ensemble regression, arXiv preprint
Elliot, M., O'hara, K., Raab, C., O'Keefe, C. M., Mackey, E., Dibben, C., Gowans, H., Purdam, K., and McCullagh, K. (2018). Functional anonymisation: Personal data and the data environment. Computer Law & Security Review, 34(2)2204-221
Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. (2016). beta-vae: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations.
Huang, F,, Xie, G., and Xiao, R. (2009). Research ensemble learning. In 2009 International Conference on Artificial Intellignnce and Computational Intelligence, volume 3, pages 249-252 IEEE.
Izmailov, P., Nicholson, P., Lotfi, S., and Wilson, A. G. (2021). Dangers bayesian model averaging under covariate shift. Advances in Neural Information Processing Systems, 34.
Jaffe, A., Fetaya, E.,, Nadler, B., Jiang, T., and Kluger, Y. (2016). Unsupervised ensemble learning with dependent classifiers. In Artificial Intelligence and Statistics, pages 351-360, PMLR.
Karbalayghareh, A., Qian, X., and Dougherty, E. R. (2018). Optimal bayesian transfer learning. IEEE Transactions on Signal Processing, 66(14):3724-3739,
Kim, H.-C. and Ghahramani, Z. (2012). Bayesian classifier combination. In Artificial Intelligence and Statistics, pages 619-627. PMLR.
Kingma, D. P. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
Kouw, W. M. and Loog, M. (2019). A review of domain adaptation without target labels. IEEE transactions on pattern analysis and machine intelligence.
Mangin, O., Urien, S., Mainardi, J.-L., Fagon, J.-Y., and Faisy, C. (2014). Vancomycin pharmacoki- netic and pharmacodynamic models for critically ill patients with post-sternotomy mediastinitis. Clinical pharmacokinetics, -4966.19-861.
Oodellín-Gariban S. F., Ortiz-Martín, B., Rueda-Naharro, A., Gareía, B., Romano-Moreno, S., and Barcia, E. (2016). Pharmacokinetics of vancomycin and dosing recommendations for trauma patients. Journal of Antimicrobial Chemotherapy, 71(2)471-479,
Minka, T. P. (2000). Bayesian model averaging is not model combination. Available electronically at http://www. stat. edu/minka/yapprr//bmu htmll, pages 1-2.
Neath, A. A. and Cavanaugh, E, (2012). The bayesian information criterion: background, derivation, and applications. Wiley Interdisciplinary Reviews: Computational Statistics, 4(2):199-203:

---

Pace, A.. Chan, A. J., and van der Schaar, M. (2021). Poetree: Interpretable policy learning with adaptive decision trees. In International Conference on Learning Representations.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances neural information processing systems, pages 8026-8037
Platanios, E. A., Blum, A., and Mitchell, M. (2014). Estimating accuracy from unlabeled data, In UAI, volume 14, page 10.
Platanios, E. A., Dubey, A., and Mitchell, T. (2016). Estimating accuracy from unlabeled data: A bayesian approach. International Conference on Machine Learning, pages 1416-1425. PMLR.
Raftery, E., Madigan, D., and Hoeting, A. (1997). Bayesian model averaging for linear regression models. Journal of the American Statistical Association, 92(437):17--19)
Rahaman, R. et al. (2021). Uncertainty quantification and deep ensembles. Advances in Neural Information Processing Systems, 34.
Raina, R., Ng. A. Y., and Koller, D. (2006). Constructing informative priors using transfer learning. In Proceedings of the 23rd international conference on Machine learning, pages 713-720. ACM.
Ren, J., Liu, P. J., Fertig, E., Snoek, J., Poplin, R., Depristo, M.. Dillon, J., and Lakshminarayanan, B. (2019). Likelihood ratios for out-of-distrit oistiiiribttion detection. Advances Neural Information Processing Systems, 32.
Revilla, N., Martín-Suáred A., Pérez, M. P., González, F. M., and Fernández de Gatta, M. d. M.
(2010). Vancomycin dosing assessment in intensive care unit patients based on a population pharma- cokinetic/pharmaccddyaat simulation. British journal of clinical pharmacology, 70(2):201-212.
Roberts, J. A., Taccone, F. S., Udy, A. A., Vincent, J.-L., Jacobs, F., and Lipman, J. (2011). Van- comycin dosing eritically ill patients: robust methods for improved continuous-infusin regimens. Antimicrobial agents and chemotherapy, 55(6):2704-257 2709.
Ross, S. and Bagnell, D. (2010). Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings.
Ruta, D. and Gabrys, B. (2005). Classifier selection for majority voting. Information fusion, 6(1):63-11.
Sagi, O. and Rokach, L. (2018). Ensemble learning: survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1249.
Shaham, U., Cheng, X., Dror, O., Jaffe, A., Nadler, B., Chang. J., and Kluger, Y. (2016). A deep learning approach unsupervised ensemble learning. International conference on machine learning, pages 30-39. PMLR.
Terrell, G. R. and Scott, D. W. (1992). Variable kemel density estimation. The Annals of Statistics, pages 1236-1265.
Thomson, A., Staatz, C., Tobin, C., Gall, M., and Lovering, A. (2009). Development and evaluation of vancomycin dosage guidelines designed to achieve new target concentrations. Journal of antimicrobial chemotherapy, 1055)-1050
Torrey, L. and Shavlik, J. (2010). Transfer learning. In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pages 242-264. IGI Global.
Uster, D. W., Stocker, S. L.,, Carland, J. F.,, Brett, J., Marriott, J., Day, R. O., and Wicha, S. G. (2021). A model averaging/selection approach improves the predictive performance of model- informed precision dosing: vancomycin as case study. Clinical Pharmacology & Therapeutics, 109(1):175-183.
Verikas, A., Lipnickas, A., Malmqvist, K., Bacauskiene, M., and Gelzinis, A. (1999). Soft combina- tion of neural classifiers: A comparative study. Pattern recognition letters, 20(4):429 444.
12

---

Wicha, S. G., Kees, M. G., Solms, A., Minichmayr, I. K., Kratzer, A., and Kloft, C. (2015). Tdmx: novel web-based open-access support tool for optimising antimicrobial dosing regimens in clinical routine. International journal of antimicrobial agents, 45(4))442-444 444.
13


Reviews:
[{"review_id": "s7BO_Pz5E08", "reviewer": "Reviewer_NtsN", "summary": "The paper proposes synthetic model combination, an unsupervised ensemble learning approach to make instance-wise predictions by assuming access to the feature distributions the available models have perceived. Instance-wise, here, refers to choosing the weights of individual models for each test instance. The central application for such an approach is trained models on private (e.g. highly sensitive) data, where model performance is sufficiently good on the original data. The authors propose to learn a latent space on available unlabeled test data (sampled from the original data distributions), which in addition to a reconstruction loss also takes into account agreements and disagreements of the available models. The approach is evaluated for a toy example to underpin the need for the approach, MNIST and Vancomycin Precision Dosing. The results are able to show that instance-wise predictions using the learned latent space are performing well if the assumptions are met.\n", "questions": "* Which baseline methods have been tested for the MNIST use case?\n\n* Did you test the approach for more challenging image / natural language datasets?", "limitations": "The paper discusses limitations of the approach, such as varying model performances. The approach therefore works best if there are individual expert models for their respective domain. If a model varies in performance in its domain and cannot dominate other models, there might be problems. \n\nAs mentioned before, it remains unclear if the dimensionality reduction step also works sufficiently well for more complex problems.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is well-writen and thoroghly introduces/motivates the topic and problem setting. The problem is further concisely distinguished from related fields, such as transfer learning. The taken approach is well motivated too. The empirical evaluation is rather thorough and the results are sufficiently discussed, such that the available claims can be supported. While the general problem of unsupervised ensemble learning is known, the approach is sufficiently novel and original.\n\nWeaknesses: A minor weakness wrt related work is missing coverage of existing unsupervised learning approaches which already incorporate agreement and disagreement based ensemble construction. Examples are [1,2]. It would also be interesting to see the performance of more challenging image or text datasets, but the current evaluation already supports central claims.\n\n[1] Platanios, E.A., Dubey, A. and Mitchell, T., 2016, June. Estimating accuracy from unlabeled data: A bayesian approach. In International Conference on Machine Learning (pp. 1416-1425). PMLR.\n[2] Platanios, E.A., Blum, A. and Mitchell, T.M., 2014, July. Estimating Accuracy from Unlabeled Data. In UAI (Vol. 14, p. 10).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "G0JoAMPW6Um", "reviewer": "Reviewer_oUUV", "summary": "This paper studies the problem of unsupervised ensemble learning in the problem setting where a set of trained models and limited unlabelled test instances are given, but no access to the data used for training the models is available. The aim of the ensemble learning is to obtain a set of optimal weights of the models specific to each given test/new instance, to form an ensemble for predicting the label of the instance.\n\nA few demonstrating experiments, including one which is based on a real world medical problem, are presented to show the advantages of the proposed method and its limitations.\n", "questions": "1. What is the minimal requirement on the number of test samples? \n2. What is the number of features of the dataset used in the “real case study” in Section 4.3?\n3. Could you elaborate on D_Y in (3) a bit more, e.g. its role, calculation and justification on the multiplication used?\n4. In terms of related work, I’d like to confirm - do all existing instance-wise or dynamic ensemble learning methods require model training data? Are there any methods which work in the same or similar setting of the proposed method? If yes, why the evaluation does not include a comparison study involving those methods?\n\n", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper addresses a practical problem in ensemble learning.\n2. The research question is very well motivated and formulated.\n3. The overall design of the method is reasonable and sound.\n4. The experiment illustration has provided some strong evidence of the effectiveness of the proposed method.\n5. The paper is well written.\n\nWeakness:\n1. The experimental demonstration, although carefully designed, with each example intended for a justifiable purpose, overall the evaluation is not strong. In particular, I have some concerns about “real case study”:\n\na. the data used for the “real case study” is simulated patient data, not real patient data, which makes people wonder whether the proposed method would perform effectively in real world cases or not. \n\nb. the distributions of the predicted concentration by different models seem to have similar shapes/coverage, so I wonder if this is a good case study for demonstrating the expected advantage of the proposed method.\n\nc. this case does not seem to be a high-dimensional case, hence the evidence provided by this case does not demonstrate the expected strength of the proposed method for dealing with high-dimensional data.\n\n2. It is not clear as a minimal, how many unlabelled test instances are needed for learning the low-dimensional representations. For the experiment datasets, I cannot find information on the size of the test dataset either. A high requirement on test instances needed may limit the practical use of the proposed method.\n\nMinor: In formula (5), w_i(x) should be w(x)_i, to be consistent with the notation used in Section 3 (outline of the three main steps)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B_sjXTHkp6_", "reviewer": "Reviewer_6vwo", "summary": "This paper proposed a new ensemble method – Synthetic Model Combination (SMC) operating on the instance-wise level in the setting of unsupervised ensemble learning. The authors also introduced a novel unsupervised representation learning method. The experimental results on MNIST and Vancomycin Precision Dosing datasets validate the effectiveness of the proposed methods. ", "questions": "1. Some figures and tables (e.g., Figure 1, Table 1, Algorithm 1) are not mentioned in the main body. The authors are suggested to link all the figures and tables to the main body. \n2. The authors are suggested to make more clarifications for some descriptions or functions. For example, you introduced p_{j}^{\\chi}(x) in line 212. More details about it are suggested to be added.  \n", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. This paper is well-written. \n2. The problem this paper focuses on is important, and the method is interesting. \n\n\nWeaknesses:\n1. In this paper, the solid theoretical analysis (e.g., theorems) is missing. I think it is a necessary part, especially for a top machine learning conference like NeurIPS.\n2. Instance-wise level methods are expected to perform better than the methods learning the fixed weights for every instance, but at the cost of higher time complexity. The part about the time complexity analysis is suggested to be added. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8q21Epv-p2D", "reviewer": "Reviewer_gNf4", "summary": "This paper proposed an approach named Synthetic Model Combination (SMC) that make weighted ensemble predictions based on local manifold context. In addition, the authors proposed a representation learning procedure to obtain the specified weights for SMC. The author provided examples and benchmarks on synthetic and real world datasets at the end.", "questions": "-It would be interesting to see how encoding simple uncertainty measures like variance or entropy as weights compares with weights calculated from VAE. Have the author performed similar experiments as a control?\n\n-Could the author clarity \"augmentation of the individual models\" at line 46?\n\n-What is the added computation complexity of SMC?", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-This work raises an important question that the ensemble process should be localized due to the partial information received during the training process of the contributing models.\n\n-Description of the problem is mostly clear and the article is well structured.\n\nWeaknesses:\n\n-Not novel. \nThe main contribution proposed in this article is a weighted ensemble on individual samples based on local context. This idea has been explored in previous studies (Huang 2009, Weighted Average with Data Dependent weights). The proposed autoencoder is new but seems to only be a way of obtaining weights. Authors need to state the added advantages of using an VAE. If it is the best way to obtain weights the author need to compare autoencoder with other methods to justify this approach.\n\n-Results not significant/lack of benchmarks:\nThe only benchmark is on a drug response estimation with incremental improvements. To demonstrate a general performance improvements, experiments on existing larger dataset like cifar10, imagenet, and other NLP datasets is needed.\n\n-Lack of comparison with existing methods:\nSimilar to the previous point. The author should systematically compare the SMC with existing ensemble learning approaches across multiple benchmarks.\n\n-The overall writing quality is not great. There are many non-idiomatic sentences with long clauses that hinders readability.\n\n-While I appreciate the illustration with well-colored diagrams, it can be a little distracting at times. Figure components should be explained in the figure legend. (e.g. dots in Figure 3.)\n\nRef:\nHuang, Faliang, Guoqing Xie, and Ruliang Xiao. \"Research on ensemble learning.\" *2009 International Conference on Artificial Intelligence and Computational Intelligence*\n. Vol. 3. IEEE, 2009.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept