Dynamic Post-Hoc Neural Ensemblers

Authors: Sebastian Pineda Arango, Maciej Janowski, Lennart Purucker, Arber Zela, Frank Hutter, Josif Grabocka
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: ensemble learning, neural networks, regularization, modality agnostic, dynamic ensembling

Abstract:
Ensemble methods are known for enhancing the accuracy and robustness of machine learning models by combining multiple base learners. However, standard approaches like greedy or random ensembles often fall short, as they assume a constant weight across samples for the ensemble members. This can limit expressiveness and hinder performance when aggregating the ensemble predictions. In this study, we explore employing neural networks as ensemble methods, emphasizing the significance of dynamic ensembling to leverage diverse model predictions adaptively. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions during the training. We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities. Our experiments showcase that the dynamic neural ensemblers yield competitive results compared to strong baselines in computer vision, natural language processing, and tabular data.

Decision: reject
Reviewer scores: 3, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730258206025

thread_type:
official_review

<thread_object>
creator: reviewer_eck1

<content>
summary:
The paper introduces a novel approach to post-hoc ensembling using neural networks, termed "Neural Ensemblers." These ensemblers dynamically generate weights for each base model in the ensemble on a per-instance basis, addressing the limitations of static ensembling methods. The authors propose a regularization technique inspired by DropOut in deep learning to mitigate the risk of overfitting and diversity collapse. The method is evaluated across various data modalities, including tabular data, computer vision, and natural language processing, demonstrating competitive performance against strong baselines.

strengths:
1) The introduction of Neural Ensemblers as a dynamic post-hoc ensembling method is a significant contribution to the field. The use of neural networks to dynamically assign weights to base models on a per-instance basis is a novel and effective approach.

2) The proposed regularization technique, which involves randomly dropping base model predictions during training, is theoretically sound and empirically validated. This technique effectively mitigates the risk of overfitting and enhances ensemble diversity.

weaknesses:
While the paper mentions the use of a fixed set of hyperparameters across all experiments, it does not explore the sensitivity of the Neural Ensemblers to hyperparameter changes. A more detailed analysis of hyperparameter tuning could provide deeper insights.

Does the author carefully tune the hyper-parameters of other baselines e.g., XGBoot?

The author should also compare their method with other advanced methods, e.g., LightGBM, CatBoot, which can be directly used in the open source libraries.

Lack of references and comparisons to Ensemble NN [1] and SCARF [2]. Both of these papers explore the ensemble or pretraining in Deep neural networks on tabular datasets. 

[1] Zhang S, Liu M, Yan J. The diversified ensemble neural network[J]. Advances in Neural Information Processing Systems, 2020, 33: 16001-16011.

[2] Bahri D, Jiang H, Tay Y, et al. Scarf: Self-Supervised Contrastive Learning using Random Feature Corruption[C]//International Conference on Learning Representations.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730669376615

thread_type:
official_review

<thread_object>
creator: reviewer_62pt

<content>
summary:
A common approach to combining predictions from multiple ensemble members is to assign equal weights to each prediction, as seen in hard and soft voting methods. However, it is clear that certain ensemble members may offer better predictions for specific instances, and it leads to the concept of weighted voting. This work proposes obtaining such instance-wise ensemble weights through a separate neural network, referred to as the Neural Ensembler.

strengths:
1. The problem statement is well-motivated; it would be beneficial to adaptively determine ensemble weights for a given instance.

2. The authors propose two setups: "stacking" and "model-averaging." In the stacking setup, the Neural Ensembler directly outputs the ensembled prediction. In contrast, the model-averaging setup has the Neural Ensembler provide the weights for each ensemble member. While the stacking setup aligns with the ultimate goal of obtaining an ensembled prediction, the model-averaging setup is also valuable as it explicitly reveals the contribution of each ensemble member.

weaknesses:
1. There are concerns about fair comparison since the proposed Neural Ensemble ultimately learns from additional validation data. While baseline methods also train statistical models based on validation data, the issue is that the Neural Ensemble is significantly larger than the baseline methods. For example, methods like Greedy, Quick, and BO train statistical models with $M$ (unnormalized) ensemble weights as parameters. In contrast, NE-Stack trains a statistical model (neural network) with $D \times 32 + 32 \times 32 + 32 \times 32 + 32 \times D$ parameters, which amounts to $2688$, $8448$, and $66048$ parameters for $D=10$, $100$, and $1000$, respectively. Given the significant difference in the scale of the introduced statistical models, ranging from hundreds to tens of thousands of times larger, it is questionable whether introducing the same additional validation data could make a fair comparison.

2. I believe the assumption of having a sufficiently large amount of validation data, which can be used to train the neural network model, is quite strong. If the "Avg. samples for Validation" shown in Table 1 indicates the size of the validation dataset used to train the Neural Ensembler and the statistical models of the baseline methods for each dataset, it appears particularly large for NASBench and FTC. Conducting an ablation study on this would be valuable to assess how much validation data is necessary for the Neural Ensembler to be effective. Currently, the only ablation study presented is on the dropout probability, aimed at preventing overfitting on the given validation data.

3. Additionally, the Neural Ensembler is implemented using an MLP architecture with a hidden dimension of 32 and a depth of four. Such architectural choice also warrants an ablation study; a model that is too large may incur significant costs and increase overfitting, while one that is too small may not perform adequately. Currently, there is no discussion regarding the size of the Neural Ensembler model.

4. The key feature of the proposed Neural Ensembler is its ability to adjust the weights of each ensemble member in an instance-specific manner, either implicitly (in a stacking setup) or explicitly (in a model-averaging setup). However, there is currently no direct evidence regarding its effectiveness from this perspective. One potential approach could involve directly learning the weighting coefficients for each instance (i.e., oracle) and evaluating how closely the ensembled predictions (or ensemble weights in the model-averaging setup) generated by the Neural Ensembler align with them.

5. Above all, are the values reported in the tables the averages and standard deviations of the results for the sub-datasets within each meta dataset? The fact that nearly all results overlap within the range of standard deviations suggests that the experimental results may not be statistically significant. A clear clarification on this matter seems necessary.

questions:
1. I think the assumption of having a sufficiently large amount of validation data (that can be used to train the neural network model) is quite strong. In such cases, further exploration of existing methodologies based on well-grounded principles (such as Bayesian principles) would be valuable.

2. Specifically, one can approximate the model probability in Bayesian model averaging using the Bayesian Information Criterion (BIC), where the likelihood is estimated from the validation set.

3. Following BIC, the harmonic mean estimator and its variants come to mind. A recent relevant work is "Machine Learning Assisted Bayesian Model Comparison: Learnt Harmonic Mean Estimator" by McEwen et al. (2023). Have you explored such classical methods grounded in Bayesian principles alongside BO-based approaches?

4. I recognize the value of presenting normalized metrics for relative comparisons in the main text. However, it would be helpful to include the raw numbers in the appendix. Relying solely on relative metrics can make it challenging to assess the significance of the results for each methodology.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730686877854

thread_type:
official_review

<thread_object>
creator: reviewer_qkae

<content>
summary:
The paper proposes Dynamic Post-Hoc Neural Ensemblers for replacing the traditional ensemble method. Their main contribution is the use of two MLPs to dynamically generate weights for ensemble members. Additionally, when training these MLPs, they randomly drop base models to reduce overfitting and improve generalization capabilities. Lastly, their empirical results demonstrate that Neural Ensemblers consistently form competitive ensembles across diverse data modalities, including tabular data (classification and regression), computer vision, and natural language processing.

strengths:
**Innovative Approach:** The paper introduces Dynamic Post-Hoc Neural Ensemblers,  aiming to generate dynamic weights for ensemble members.

**Reduced Overfitting:** By randomly dropping base models during training, the method addresses overfitting and potentially improves generalization.

**Empirical Validation:** The paper provides empirical results showing the method’s outperformance and robustness across diverse datasets.

weaknesses:
1. The study lacks consideration of computational efficiency. The Neural Ensemblers’ performance depends on a strong subset of 50 base models, selected by `DivBO`, which may increase computational requirements. The author should compare the Neural Ensembler's performance with that of a single model of equivalent size to better evaluate its efficiency. For example,  to evaluate Neural Ensember, the author could provide some specific efficiency metric comparisons, such as training time, inference time, or memory usage compared to baseline methods, or analyze the computational complexity of their approach. 

2. My review of the author's code confirmed that the meta-datasets are limited to a tabular format, making the paper's description somewhat vague. A clearer explanation of the data structure would have been beneficial. Otherwise, the application scope should be limited to tabular data or metadata in a tabular format from fields like computer vision or natural language processing. 
\
In line 126, the author states that NasBench 201 is a neural architecture search (NAS) method for computer vision (Dong & Yang, 2020). However, I think the Neural Ensembler is actually trained on tabular data, as indicated in the code at SearchingOptimalEnsembles/metadatasets/nasbench201/metadataset.py, lines 93-95. The author can compare the following two different pipelines:
- The author selects some base models and computes their predictions for each example. These predictions are then stored in tabular data format as examples to generate the training set. Lastly, train the Neural Ensembler on this set.
- Freeze all base models. Then, sample a mini-batch of examples from the raw training set. Lastly, Train the Neural Ensembler to generate the weights.

I believe the second schema is worth considering because 1) the input is raw data from the dataset, not the predictions from base models, 2) data augmentation can be used to avoid overfitting, and 3) for some recognition tasks, such as object detection, the predictions of different base models cannot be aligned, making it impossible to store base models' predictions in a tabular format.

I guess that the author may have used too many base models to construct the Neural Ensembler, which made it hard to train in the second mode with limited computing resources. This is just my guess, so I provide several solutions in the questions section for the author to refute this weakness.

questions:
Overall, this paper requires some clarifications on theory and experiments. Given these clarifications, if provided in an author's response, I would consider increasing the score. 

---

For the experiments, `any of the following` should be addressed.

1. We note that the authors compare the Neural Ensembler to a single model (see **lines 321-322**). It would have been better also to explore the efficiency of the Neural Ensembler, particularly considering the **number of model parameters** or **FLOPS** (see [1]). Otherwise, model scaling is a better choice for improving model performance. For example, we have a list of models:

| Model    | # Params | Acc@1  |
| -------- | -------- | -------|
| VGG19    | 143.7M   | 72.376 |
| ResNet18 | 11.7M    | 69.758 |
| ResNet34 | 21.8M    | 73.314 |
| ResNet50 | 25.6M    | 76.13  |
| ViT-B16  | 86.6M    | 81.072 |

- If ResNet18 and ResNet34 are used as base models for the Neural Ensembler, the Neural Ensembler’s performance should exceed that of ResNet50 due to its higher parameter count of over 33.5M (11.7M + 21.8M). 
- When comparing with the VGG16 baseline, we should avoid using ResNet18, ResNet34, ResNet50, or ViT-B16 as base models for the Neural Ensembler, as each has better performance and a smaller scale.
- To demonstrate the advantages of the Neural Ensembler in computer vision and natural language processing, the author should use corresponding neural networks as base models and conduct experiments on image or text datasets.

2. It's important to note that the paper focuses on post-hoc ensembling of pre-trained models, rather than training base models from scratch. Thus, the author can clarify how their method compares the computational cost to other post-hoc ensembling methods.


[1] Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731226200395

thread_type:
official_review

<thread_object>
creator: reviewer_svxb

<content>
summary:
This paper introduces "Neural Ensemblers," a novel approach that uses neural networks to dynamically weight ensemble members for each input instance. Authors propose regularization technique of randomly dropping base model predictions during training.

strengths:
- Paper is well written.  
- Experimental results span different modalities

weaknesses:
- Benefit of the proposed method is not super clear in Table 2. Given the variance of different methods, efficacy of the proposed method is unclear

- POC example given in the paper can be misleading as their dynamic ensemblers train more parameters as compared to baseline. It remains unclear if this would be a problem in the first place if we could train base models that are sufficiently overparameterized

- It remains unclear what are the main reasons due to which the proposed methods improves on top of simple ensembling baselines. 

- Post-hoc ensembling in the name can be misleading for someone who is not aware of the literature. In particular, to obtain the ensemble we need to train the parameters of the ensemble model. 

I am not very well aware of the literature to comment on the novelty of paper, so I will leave that on other reviewers and the area chair.

questions:
- It is not clear when dynamic ensemblers would benefit? Is it the case that dynamic ensemblers improve because base models underfit the data? Does benefit of these ensemblers persist in cases when the base models are over parameterized neural networks? 

- More data is needed to train the parameters of the dynamic ensemblers  It is unclear if we can reuse the training data used for base models or do we need more training data? In case we need more training data, a favorable comparison would be using that data in the first place for base models and perform standard ensembling techniques.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732211017645

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We thank all the reviewers for their insightful feedback. We are happy that the reviewers consider our work novel, with a significant contribution to the field. As a general response, we listed here the changes to the submission PDF (highlighted in blue), including results after running experiments suggested by the reviewers:

* Extended discussion on related work: **Section 5**, as a response to reviewer  **eck1**.
* Unnormalized metrics results: **Appendix E1**, as a response to reviewer  **62Pt**.
* Analysis of computation cost: **Appendix F**, as a response to reviewers **Svxb, QKaE**.
* Proof of concept with overparametrized base models: **Appendix G**, as a response to reviewer **Svxb**.
* Critical difference diagrams: **Appendix H**, as a response to reviewers **Svxb,  62Pt**.
* Results with additional baselines (Catboost, XGBoost, LightGBM, Akaike weighting): **Appendix I**, as a response to reviewer **eck1**.
* Sensitivity analysis of neural ensemble hyperparameters: **Appendix J**,  as a response to reviewers **62Pt, eck1**.
* Ablation study of validation data size: **Appendix K**, as a response to reviewers **Svxb,  62Pt**.
* Effect of merging training data: **Appendix L**, as a response to reviewer **Svxb**.
* Results of Neural Ensembles operation on the original input space: **Appendix M**, as a response to reviewer **Svxb**.
* Insights of the significance in datasets with a lot of classes: **Appendix N**, as response to reviewer **Svxb, 62Pt**.

We believe our changes address the reviewers' concerns and reinforce the value of our work. Our replies further strengthen the validity of our research hypothesis. We also respond to each reviewer individually below.
</content>
</thread_object>

</thread>


Metareview: