Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes synthetic model combination, an unsupervised ensemble learning approach to make instance-wise predictions by assuming access to the feature distributions the available models have perceived. Instance-wise, here, refers to choosing the weights of individual models for each test instance. The central application for such an approach is trained models on private (e.g. highly sensitive) data, where model performance is sufficiently good on the original data. The authors propose to learn a latent space on available unlabeled test data (sampled from the original data distributions), which in addition to a reconstruction loss also takes into account agreements and disagreements of the available models. The approach is evaluated for a toy example to underpin the need for the approach, MNIST and Vancomycin Precision Dosing. The results are able to show that instance-wise predictions using the learned latent space are performing well if the assumptions are met.

strengths_and_weaknesses: Strengths: The paper is well-writen and thoroghly introduces/motivates the topic and problem setting. The problem is further concisely distinguished from related fields, such as transfer learning. The taken approach is well motivated too. The empirical evaluation is rather thorough and the results are sufficiently discussed, such that the available claims can be supported. While the general problem of unsupervised ensemble learning is known, the approach is sufficiently novel and original.

Weaknesses: A minor weakness wrt related work is missing coverage of existing unsupervised learning approaches which already incorporate agreement and disagreement based ensemble construction. Examples are [1,2]. It would also be interesting to see the performance of more challenging image or text datasets, but the current evaluation already supports central claims.

[1] Platanios, E.A., Dubey, A. and Mitchell, T., 2016, June. Estimating accuracy from unlabeled data: A bayesian approach. In International Conference on Machine Learning (pp. 1416-1425). PMLR.
[2] Platanios, E.A., Blum, A. and Mitchell, T.M., 2014, July. Estimating Accuracy from Unlabeled Data. In UAI (Vol. 14, p. 10).
questions: * Which baseline methods have been tested for the MNIST use case?

* Did you test the approach for more challenging image / natural language datasets?
limitations: The paper discusses limitations of the approach, such as varying model performances. The approach therefore works best if there are individual expert models for their respective domain. If a model varies in performance in its domain and cannot dominate other models, there might be problems. 

As mentioned before, it remains unclear if the dimensionality reduction step also works sufficiently well for more complex problems.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of unsupervised ensemble learning in the problem setting where a set of trained models and limited unlabelled test instances are given, but no access to the data used for training the models is available. The aim of the ensemble learning is to obtain a set of optimal weights of the models specific to each given test/new instance, to form an ensemble for predicting the label of the instance.

A few demonstrating experiments, including one which is based on a real world medical problem, are presented to show the advantages of the proposed method and its limitations.

strengths_and_weaknesses: Strength:
1. The paper addresses a practical problem in ensemble learning.
2. The research question is very well motivated and formulated.
3. The overall design of the method is reasonable and sound.
4. The experiment illustration has provided some strong evidence of the effectiveness of the proposed method.
5. The paper is well written.

Weakness:
1. The experimental demonstration, although carefully designed, with each example intended for a justifiable purpose, overall the evaluation is not strong. In particular, I have some concerns about “real case study”:

a. the data used for the “real case study” is simulated patient data, not real patient data, which makes people wonder whether the proposed method would perform effectively in real world cases or not. 

b. the distributions of the predicted concentration by different models seem to have similar shapes/coverage, so I wonder if this is a good case study for demonstrating the expected advantage of the proposed method.

c. this case does not seem to be a high-dimensional case, hence the evidence provided by this case does not demonstrate the expected strength of the proposed method for dealing with high-dimensional data.

2. It is not clear as a minimal, how many unlabelled test instances are needed for learning the low-dimensional representations. For the experiment datasets, I cannot find information on the size of the test dataset either. A high requirement on test instances needed may limit the practical use of the proposed method.

Minor: In formula (5), w_i(x) should be w(x)_i, to be consistent with the notation used in Section 3 (outline of the three main steps)

questions: 1. What is the minimal requirement on the number of test samples? 
2. What is the number of features of the dataset used in the “real case study” in Section 4.3?
3. Could you elaborate on D_Y in (3) a bit more, e.g. its role, calculation and justification on the multiplication used?
4. In terms of related work, I’d like to confirm - do all existing instance-wise or dynamic ensemble learning methods require model training data? Are there any methods which work in the same or similar setting of the proposed method? If yes, why the evaluation does not include a comparison study involving those methods?


limitations: Yes.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed an approach named Synthetic Model Combination (SMC) that make weighted ensemble predictions based on local manifold context. In addition, the authors proposed a representation learning procedure to obtain the specified weights for SMC. The author provided examples and benchmarks on synthetic and real world datasets at the end.
strengths_and_weaknesses: Strengths:

-This work raises an important question that the ensemble process should be localized due to the partial information received during the training process of the contributing models.

-Description of the problem is mostly clear and the article is well structured.

Weaknesses:

-Not novel. 
The main contribution proposed in this article is a weighted ensemble on individual samples based on local context. This idea has been explored in previous studies (Huang 2009, Weighted Average with Data Dependent weights). The proposed autoencoder is new but seems to only be a way of obtaining weights. Authors need to state the added advantages of using an VAE. If it is the best way to obtain weights the author need to compare autoencoder with other methods to justify this approach.

-Results not significant/lack of benchmarks:
The only benchmark is on a drug response estimation with incremental improvements. To demonstrate a general performance improvements, experiments on existing larger dataset like cifar10, imagenet, and other NLP datasets is needed.

-Lack of comparison with existing methods:
Similar to the previous point. The author should systematically compare the SMC with existing ensemble learning approaches across multiple benchmarks.

-The overall writing quality is not great. There are many non-idiomatic sentences with long clauses that hinders readability.

-While I appreciate the illustration with well-colored diagrams, it can be a little distracting at times. Figure components should be explained in the figure legend. (e.g. dots in Figure 3.)

Ref:
Huang, Faliang, Guoqing Xie, and Ruliang Xiao. "Research on ensemble learning." *2009 International Conference on Artificial Intelligence and Computational Intelligence*
. Vol. 3. IEEE, 2009.
questions: -It would be interesting to see how encoding simple uncertainty measures like variance or entropy as weights compares with weights calculated from VAE. Have the author performed similar experiments as a control?

-Could the author clarity "augmentation of the individual models" at line 46?

-What is the added computation complexity of SMC?
limitations: NA
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a new ensemble method – Synthetic Model Combination (SMC) operating on the instance-wise level in the setting of unsupervised ensemble learning. The authors also introduced a novel unsupervised representation learning method. The experimental results on MNIST and Vancomycin Precision Dosing datasets validate the effectiveness of the proposed methods. 
strengths_and_weaknesses: Strengths: 
1. This paper is well-written. 
2. The problem this paper focuses on is important, and the method is interesting. 


Weaknesses:
1. In this paper, the solid theoretical analysis (e.g., theorems) is missing. I think it is a necessary part, especially for a top machine learning conference like NeurIPS.
2. Instance-wise level methods are expected to perform better than the methods learning the fixed weights for every instance, but at the cost of higher time complexity. The part about the time complexity analysis is suggested to be added. 

questions: 1. Some figures and tables (e.g., Figure 1, Table 1, Algorithm 1) are not mentioned in the main body. The authors are suggested to link all the figures and tables to the main body. 
2. The authors are suggested to make more clarifications for some descriptions or functions. For example, you introduced p_{j}^{\chi}(x) in line 212. More details about it are suggested to be added.  

limitations: Yes.
review_title: null

Meta-review: metareview: This work suggests that in cases where data is sensitive it might be easier to gain access to pre-trained models instead of to the data used for training them. However, since these models were trained on different distributions, their prediction may be better/worse depending on whether the point of interest in in the support of the distribution they trained on. Hence, the setting is a sort of learning from experts’ advice [1] where the best expert should be selected locally.

In this work it is assumed that each model (expert) is published together with some information about the distribution on which the model was trained. The assumption that such data may be provided is justified by the common practice of providing descriptive statistics of the data used in publications in the medical domain, typically in Table 1 of such papers. 

Several related problems have been studied before this work. One of the main criticism reviewers had about this work was the incomplete positioning of this work in relation to these earlier studies, especially in the first version of this work submitted for review. The clarifications were given by the authors in the rebuttal. Some differences between this work in prior art might be since some earlier studies tried to provide theoretical guarantees which forced the use of stronger (and explicitly) assumptions. It may be that the current work did not have to make such assumptions since it does not contain theoretical analysis.  

The term “synthetic” here is used as a nod to synthetic control. The idea is that both in synthetic control and here a convex combination of weak models is used. However, this is true for almost any ensemble model (bagging, random forest, adaboost…). Moreover, in synthetic control the weighting is fixed (global) as opposed to the main selling point of this work which is the local weighting. 

A key assumption made in this work is that a model will be confident in its predictions on regions that are in the support of the dataset used for training it. This is stated, for example, in lines 184-185. This assumption is not always correct since the decision boundary of a model could be a region of high density on which the model is not confident about. Another case in which this assumption might fail is when the information about the distribution used for training, I_m, fails to provide good description. For example, if the data contains two well separated clusters and the information in I_m is the mean and the variance then the samples generated are likely to be from the area of low density between the two clusters on which the model might have low confidence.

The solution provided is based on intuition and is not equipped with theoretical support. The experiments show encouraging results, but they have their own limitations. For example, in the MNIST experiment, what is the information about the underlying distribution provided to the algorithm? The reviewers made several comments about the empirical evaluation and the authors discussed this at length in their response explaining that since this is a new problem domain, there are no standard benchmarks available.

Overall, this work studies an interesting problem and presents novel ideas. However, these ideas are not fully analyzed since there is no theoretical analysis and the empirical evaluation has its own limitations. It does seem that this work may contribute to problems such as medication dosing demonstrated in Section 4.3 but if this is the main contribution then it is not clear the NeurIPS is the right venue for such work.

This puts this work as a borderline case for NeurIPS: it does present new problem and some novel ideas; however, the analysis has many loose ends.


[1] Cesa-Bianchi, Nicolo, et al. "How to use expert advice." Journal of the ACM (JACM) 44.3 (1997): 427-485.

confidence: Less certain
Decision: Accept