SANE: Specialization-Aware Neural Network Ensemble
Authors: Ziyue Li
Kan Ren
XINYANG JIANG
Mingzhe Han
Haipeng Zhang
Dongsheng Li
Venue: ICLR 2022 Submitted (2022)

Abstract: Real-world data is often generated by some complex distribution, which can be approximated by a composition of multiple simpler distributions. Thus, it is intuitive to divide the complex model learning into training several simpler models, each of which specializes in one simple distribution. Ensemble learning is one way to realize specialization, and has been widely used in practical machine learning scenarios. Many ensemble methods propose to increase diversity of base models, which could potentially result in model specialization. However, our studies show that without explicitly enforcing specification, pursuing diversity may not be enough to achieve satisfactory ensemble performance. In this paper, we propose SANE --- an end-to-end ensemble learning method that actively enforces model specification, where base models are trained to specialize in sub-regions of a latent space representing the simple distribution composition, and aggregated based on their specialties. Experiments in several prediction tasks on both image datasets and tabular datasets demonstrate the superior performance of our proposed method over state-of-the-art ensemble methods.

Under review as a conference paper at ICLR 2022
SANE: SPECIALIZATION-AWANN NEURAL NETWORK ENSEMBLE
Anonymous authors Paper under double-blind review
ABSTRACT
Real-world data is often generated by some complex distribution, which be approximated by composition of multiple simpler distributions. Thus, it is in- tuitive divide complex model learning into training several simpler models, each of which specializes in one simple distribution, called model specialization in this paper. Ensemble learning one way realize specialization, and has been widely used in practical machine learning scenarios. Many ensemble meth- ods propose to increase diversity of base models, which could potentially result model specialization. However, our studies show that without explicitly enforcing specialization, pursuing diversity may not be enough to achieve satisfactory en- semble performance. In this paper, propose SANE learning method that actively enforces model specialization, where base models are trained to specialize in sub-regions of latent space representing the simple distribution composition, and aggregated based on their specialties. Experiments in several prediction tasks both image datasets and tabular datasets demonstrate the superior performance of our proposed method over state-of-the-ae ensemble methods.
end-to-end ensemble
INTRODUCTION
Real-world data distribution could be complex in most cases and people usually approximate it by composition of several simpler distributions (Xie et al.] 2016 Yang et al., 2016 Tsai et al.] 2020). Intuitively, to fit the complex data distribution conveniently, we can divide the complex model learning process into training several simpler models, each of which specializes one simple distribution. In this paper, call this idea model specialization.
Ensemble learning is one of the most effective methods to leverage model specialization. Originated from decades ago (Hansen & Salamon 1990), ensemble method (Schapire) 1990; [Beimman 1996.
Zhou et al.] (002) has been proven effective for practical machine learning tasks in various scenarios including computer vision (Huang et al. 2017). natural language processing (Shazeer et al. 2017)
and tabular data mining (Liu et al.) 2020).
Many works conduct ensemble learning through improving diversity across base models, which has been shown to reduce the variance of the combined predictions and improve generalization (Lee al. 20155: Zhou et al. 2018). The recent methodologies pursuing model diversity mainly through incorporating randomization in data sampling (Breiman 1996; Ho 1995) or model train- ing (Srivastava al., 2014 Lakshminarayanan
al. 2016).. Some other works propose sequential optimization such as boosting (Freund) (995][Cher & Guestrin 2016 Ke et al. 2017)) and snapshot ensemble (Huang al.) (0177) Other methods, e.g., model decorrelation (Zhou et al. 2018) and diversity enhancement (Zhang et al., 2020). explicitly encourage model diversity through incorpo- rating additional objectives. However, overvaluing diversity may hurt the ensemble effectiveness and the trade-off between the performance and diversity of base models still remains an open prob- lem [Rame& & Cord 2021 Fort et al.) 2019: Masegosa] 2019).
Moreover, without explicitly enforcing model specialization, pursuing model diversity does not pro- vide guarantees of each model's specialization on different simple distribution in the composition. The last two columns in Figure [ visualize the specialty of each model by plotting the data regions where the base model makes correct predictions. It shows that neither the ensemble (b) randomly initialized base models nor (c) diversified base models produce models with different specialties.

---

Under review as a conference paper at ICLR 2022
Specifically, although the method in (c) enforces diversity and shows low correlation among base models' outputs, their data regions with correct predictions are very similar as shown in the last two columns in Figure Il resulting in inferior performance after ensemble. Besides, simply putting specialized base models together does not necessarily bring promising ensemble performance. We need to know each base model's specialty over different samples and aggregate their outputs based on their specialty. As shown in Figure (d), although the overall accuracy of the two base models low, the ensemble model performs surprisingly well (100% accuracy in (d)) as long
each base model specializes in unique sub-regions of the latent space and 2) the ensemble model knows which sub-regions each base model specializes in, demonstrating the effectiveness of specialization in ensemble learning. More detailed analyses of Figure 1 Mare presented in Section 2
Base model
Base model 2
Encembled mocel
Specilly reglons model
Spediity regions mocel
dassss
pred ction
(t) random ensemble (sccuracy BEN)
dberstt-drisen ensembre (acouracy
(d) spac allaationdrhenn ensembte facouracy 10096)
Figure 1: Comparison of three different ensemble methods on the synthetic checkboard dataset (a). The first three columns in (b) (c) (d) are decision boundaries of two base models and the final ensem- ble, resp., while the last two columns indicate the samples by gray dots for which each base model makes correct predictions. (b) is averaging ensemble of models with randomly initialization and
trained independently. (c) introduces negative correlation (diversity) loss into each base model's loss function on the basis of (b). (d) uses the models' confidence of true labels to weight the training loss and aggregate the model predictions accordingly.
In this paper, we propose an end-to-end ensemble learning method that actively enforces model specialization by training models to specialize in specific simple distributions, and aggregates them based on their specialties. Firstly, we represent the composition of simple distributions by mapping them into unified latent space, where samples from each data distribution form locally clustered sub-region. This paper proposes to enforce each base model to specialize in one sub-region, and call this region the base model's 's corresponding specialty region. The base models' specialty gions are defined by a set of learnable anchor points, each of which corresponds to the center of the specialty region, called specially anchor. During training, base models' parameters and their corre- sponding specialty anchors are learned simultaneously. Each base model's specialty is enforced by encouraging model focus more on samples in the vicinity its specialty anchor. Finally, during prediction. based on the correlation between models' specialty anchors and the samples in the latent space, our method able to automatically estimate each base model's specialty over testing samples, and the outputs of the base models are adaptively aggregated based their specialty over different testing samples. To demonstrate the effectiveness of the proposed method, we conduct extensive experiments in prediction tasks two tabular datasets and two image classification datasets. The comprehensive empirical studies show that 1) coupling specialized base models and specialization- aware aggregation benefit ensemble learning and 2) the proposed specialization-awee neural network ensemble method can achieve superior performance compared with the of-te-of-the-a en- semble methods in various prediction tasks.
2

---

Under review as a conference paper at ICLR 2022
2 PRELIMINARIES AND SYNTHETIC ANALYSIS
2.1 PRELIMINARIES
In general, ensemble learning consists of two tasks: training multiple base models and aggregating their predictions. We use {M& JK to denote a collection of K base models. For a given instance Xn E IR and its label yo E Cl, each base model learns a mapping Mę: Rd + RC, which produces logits for C classes. Base models' predictions are then aggregated in weighted manner. The likelihood is written as p(y, x,, = => wy p(yn x,; 02)) s.t. Σ wy 1, where My is parameterized by O2 and uy denotes the weight for aggregating k-th model prediction.
Objectives of ensemble learning. To maximize the likelihood, most previous studies attempt to promote _k=1 P(yn, x,; 0x) (true confidence of Mı, for brevity), with fixed model weights on all samples. To this end, some works train models based on random initialization to reduce prediction variance and generalization errors, while others explicitly quantify diversity to reduce the correlation of errors between models. However, there is always problem about the trade-off between diversity and accuracy in this scenario, since the weighted majority correctness among selected models de- termines the correctness of the ensemble. One way avoid the conflict between adopting diverse models and boosting the overall performance is to use sample-specifie model weights to aggregate models' predictions. Intuitively, if base model performs much better than any other hase models over a specific sub-region of the dataset containing one or more samples (i.e., a base model special- izes in sub-set of the dataset), it should be assigned with higher weight in this sub-region. As result, we define the specialized model as follows.
Definition 2.1. (Specialized model) We define the specialized model Mr of sample (Xas yn) with two aspects: (1) k = arg max p(ya Xn; Ox,), and (2) yn = arg max, p(c) Xn; 0x) where e E [1, C] denotes the class label. If no model satisfies the above conditions, then Xn is not specialized by any model. That is, the specialized model is the one that works correctly and has the highest prediction confidence on a given sample among all the base models.
2.2 SYNTHETIC ANALYSIS
In this section, we conduct the experiment with three ensemble solutions including a random initial- ization based method and diversity- based ensemble method, and then illustrate how special- ization can benefit ensemble. A 5 x checkerboard synthetic dataset containing 25 Gaussian blobs shown in Figure I a) is used for analysis.
Two widely used ensemble methods are selected for this experiment. Firstly, following the classical random ensemble, base models with different weight initialization are trained independently and aggregated averagely. On the basis of the random ensemble, we further implement a diversity- driven ensemble by incorporating diversity loss (Li al.. 2012) into the error function of each base model to encourage base models to perform differently on the same sample. To illustrate how different base models and their ensembles make predictions, we draw their decision boundaries and graphs showing their individual specialty regions, which for a single model is where it makes the correct prediction. A multi-layer perceptron (MLP) with one hidden layer and 20 neurons serves as the base model of ensembles through this section. For simplicity of visualization and analysis, we maintain two base models for each ensemble method.
Random ensemble v.s. diversity-driven ensemble. The results of these two methods are, respec- tively, shown in Figure (bb) and (c). From random ensemble results in Figure the base models similar in performance and decision boundary thus resulting in their limited ability for fitting the complex distribution with ensemble. One question may be raised: will enforcing the base learners to predict differently on the same samples by incorporating diversity loss be sufficient to solve the above issue? From Figure [c), base models' specialty regions for the diversity-driven ensemble similar to that for the random ensemble, despite the fact that the prediction confidence of the base model diverges significantly across different sample regions. This suggests that diversity may not be sufficient encourage the base model towards specialized on more samples, thus failing to improve ensemble performance. We can also observe that ensemble fails on some samples correctly predicted by a base model, for such a model we define as an unexpressed specialist.
3

---

Under review as a conference paper at ICLR 2022
Definition 2.2. (Unexpressed specialist) Given the specialized model Mr on a sample (Xn, (x), we define M2 as an unexpressed specialist if yn + arg max, Ex=1 W; p(c) xni 0z) where e [1, C1.
Specializatinn-awaal ensemble. In terms of diversity- results, the pursuit of diversity does not lead to model specialization. To verify whether a specialization-drize approach better than the iiversity-driven approach, we further use true confidence of model prediction on a sample (i.e., p(c = yn x,; 0xx) to explicitly quantify its specialization and conduct specialization-ava awwre ensem- ble learning, As for training, we train the two base models while weighting the classification losses of each base model with the normalized true confidence. As a result, for model inference and ensem- ble, averaging their predictions only yields an accuracy of 71.0%, whereas an ensemble weighted by the true confidence of the base models yields an accuracy of 100.0%. It clearly illustrates that, as long as the specialization has been cautiously tackled, specialization-aware ensemble over weak base models will significantly improve the ensemble performance. As shown in Figure [d), base models' specialty regions overlap less, with more samples having a specialized model compared with the first two methods. Both specialization-awar training and output aggregation based on model specialty could contribute the result that improves the accuracy from 71.0% 100.0%. We verify that both factors are equally important by conducting an ablation study on the diversity- driven ensemble, where we assign the ensemble weights in (c) by the true confidences of each base model as we do in (d). However, the test accuracy is not improved (still 81 81%) by such design.
Therefore, ideally, our target to have an expressed specialist for each sample by training special- ized models and aggregating based on their specialty accordingly. To this end, problem with the current specialization-drinn samples they are not specialized in (outside their known distributions) (Hein et al. 2019). These overconfident yet incorrect predictions tend to induce the presence of unexpressed specialists. Thus, not only do need the models to be specialized in different specialty regions, but we also want these specialists to have a global perception to avoid overconfidence.
ensemble is that the base models tend to have overconfidence in the
As opposed to specialized, we also try to jointly train the sum of base models by directly minimizing the cross entropy between ground truth and the final ensemble result : M, Surprisingly, from our experiments in Section 4.3 and evidences in other study (Allen-Zhu & Li] 2020). the performance of the trained large model is even worse than that of the single model. shows that in this way we can actually give more loss weights in back- propppropagttion for models with higher true confidence, i.e., leads to specialization of models. However, without specialization-awaa aggregation, the averaged ensemble result turns out to be inferior.
3 SPECIALIZATION-AWANN NEURAL NETWORK ENSEMBLE
We propose Specialization-AWarr Neural Network Ensemble (SANE), an adaptive approach that learns enforce the base models to specialize in sub-regions of the latent space representing the complex data distribution. In contrast to previous work on trading-off diversity and performance, our goal is to help the base models develop their own specializations and then aggregate their predictions based on their specialty. In this section, we will first state the solution structure and then present the details of our design.
3.1 OVERALL FRAMEWORK
We aim to train an ensemble of base models {Ma) 1. each of which is parameterized by O₂ and gives its own prediction for the labels y/. given input x,, To aggregate the predictions of the base models, an ensemble network E : Rd base models given the current input. A = [a....., ax] is a set of learnall latent parameters of the ensemble network, and Σ k= (k: x A, w) = 1. As result, the likelihood is written as:
RK parameterized by & predicts the weights of the K
P (ym x A, Φ. 10x 10 = Σ p (K: xxi A, (yn Xn. lo; 6%)
(1)
Here, p (k] xn; A, 1)) can be regarded the predictive confidence of Ma being a proper model for X,, For model M₁. its prediction confidence of grounded label p (y X,,s k; 0%) can be seen as a measure of appropriateness.
4

---

Under review as a conference paper at ICLR 2022
3.2 DESIGN
The whole network consists of two parts: a pool of base models ]M& ==1 and an ensemble network E. In this paper, A in ensemble network is defined dataset distribution, which will guide the base models towards their own specialty regions in the latent space. During training. base models are enforced to be more specialized on the vicinity of their corresponding specialty anchors, and are encouraged to be better at what they might already be skilled in. Ultimately, the whole sample space can be covered by specialty regions of base models, whose specialties are well captured by the specialty anchors, enabling a specialization- aware aggregation. In this section, we detail in turn the design of training diversified base models, the evolution of model specialization, and specialization-awaal model aggregation.
of anchors in a latent space of the
3.2.1 TRAINING DIVERSIFIED BASE MODELS
For many tasks, is difficult to train a model to be excellent over the whole sample space. Therefore, would like to partition the entire sample space into multiple local regions and train each base model to be a specialist in one region. Thereafter they can be aggregated together to produce a feasible solution.
In ensemble learning, it is common to partition samples before feeding them into different base models, i.e., to partition them randomly (Breiman 1996) or according to their original feature space (Zhou et al. 2018). Either of these two ways would lead to a reduction in the number of samples for training each base model, which affects the performance of the model (especially for deep neural networks). Moreover, since neural networks can be overconfident in samples outside their known distributions, reducing samples will have further influence on the ensemble performance, as dis- cussed in Section 2 Besides, it is possible that segmenting samples through the raw feature space will not well capture valid information between model specialization and sample characteristics.
In contrast, we propose to locate base models' specialty regions in the latent space rather than in the raw feature space, since the latent space can provide meaningful representations (Gelada et al.] 2019). In addition, the specialty regions of the models should be explored and learned in adaptive manner to avoid inappropriate locking (Zhou et al. 2018). Through exploration, these base models not only focus on local regions, but also gain global perception. For these reasons, introduce specialty anchors to guide the traversal of the latent space.
Specialty anchors are learnable representatives of simpler distributions in the latent space, the com- position of which can derive a complex distribution to characterize the complexity of real-world data. Then, we can learn the base models, each of which is specialized in specialty region represented by the corresponding specialty anchor. However, one challenge arises when relying on the specialty anchors for ensemble learning: how to capture the relationship between each sample and the spe- cialty anchors to obtain the ensemble weights? To address this issue, we propose an attention- baassd method to guide the ensemble network adaptively weigh the base models. In addition, we expect the specialty regions of the base models to cover the entire sample space, which may place a high requirement on exploration guided by these specialty anchors. Therefore, we further resort to the help of explicit specialization measures, which will be detailed in Section 3.2.2]
Explicitly, the guidance is achieved by attentional weighting the model loss w.r.t. the specialty anchors on each sample. Given X.. an encoder F computes a nonlinear mapping of xn as en with the same embedding dimension as A. To capture the relations between Xn, and anchors {ax 1K I==
we compute the pairwise attention as {Softmax( ((ml))((e.)) ( where Q. K., are three projection functions each of which is implemented as one layer MLP and d, is the attention head number. These attention results are further fed into the scale-transfer layer, and then these K values are processed by softmax operation Softmax (z); = e** / e to produce the weights w [wi...., wx] E IRK corresponding to the K models. That is, in contrast to hard assignment where one sample is associated with a single model or subset of models, in our ensemble framework, one sample is associated with all models simultaneously. The loss of the base models can be written as
Vdn
K
Tb
) H (M& (xn) Ny))
L. =
Wk +
(2)
1 + log(T))
1=1
where 7 is the temperature parameter, T' is the number of iterations, and H is the cross-entropy loss.
5

---

Under review as a conference paper at ICLR 2022
3.2.2 THE EVOLUTION MODEL SPECIALIZATION
Given each base model, we propose learning- based method know its specialty region. As men- tioned above, wk is the learned confidence for leveraging the k-th base model to make predictions for each sample. Also, during training, we can observe the true confidence of each base model for each sample, denoted as for a pair (Xn; Vn). the weight on the model with higher true confidence should be greater than the weight on the model with lower true confidence. To facilitate the training of the ensemble network, we introduce the confidence loss as follows:
p (yn X,, k; 8g)) Thus, Wy should be close to Og. In other words,
Le Σ (ox: (I([ww)) + +I (I)) 1-II1 - wx))
This work is not the first to favor model specialization. Zhou al. (018) attempted to assign a subset of samples to each base model based on their current expertise. However, such design may limit the performance of the base models for reasons. First, the specialization of each base model is limited to a set of assigned samples, which prevents each base model from exploring other specialty regions. Second, it will lead to overfitting and overconfident issues, as discussed in Section[2 By incorporating specialty anchors and Le, we expect the ensemble network to gradually master predicting whether a base model is potentially specialized in certain samples. Therefore, in addition to focusing on the current specialty regions of each base model, specialty anchors and Le also help to explore other regions where each base model may potentially be specialized in.
3.2.3 OPECIALIZATION-AAAA MODEL ENSEMBLE
While the Le only regularizes the normalized true likelihood of base models, ensemble still fail all base models' true confidence is low. Thus, we further include an ensemble loss to optimize the likelihood of final target follows:
C. H Σ Softmax + + log(T) )a ME. (xn) Un
Te
(4)
where Te is the temperate. It is worth noting that L. does not update the parameters of Ma, only the ensemble network. As mentioned in Section ensemble loss gives more weight models with higher confidence in back poeck-ropaggtion This collides with the effect Le, which is intended to slowly approach the true confidence. In Section 4.3] we will perform ablation study on it.
All three losses Lo, L. and Le are combined to accomplish our goal. For each loss, we assign an independent loss weights 8b, Se and Se as syper-parametery
that the total loss is given by
L. = -bLs + sche + sehe.
4 EXPERIMENTS
In this section, we seek to answer research questions as follows. (RQ1) Is SANE superior to state- of-the-art ensemble methods, especially compared with diversity-driven methods? (RQ2) Is it better to encourage models to actively specialize at specific regions than statically assigning sample subsets with diversity regularization? (RQ3) How does each proposed module in SANE affect the final performance? (RQ4) Can SANE encourage models to be specialized over different sample regions?
4.1 DATASETS AND BASELINES
We briefly describe the benehmark datasets, baselines and the implementation in this section. For all classification tasks we use accuracy to evaluate, and in addition Area under ROC Curve (AUC) has also been adopted the evaluation metric for binary classification, A detailed description of the experimental setup can be found in Appendix A.I
4.1.1 TABULAR DATASETS
Higgs Boson Baldi et al. 2014 and Blastchar (Kaggle 2019) are two binary classification bench- marks. Higgs Boson contains 10.5M training samples and 0.5M test samples. Blastchar has 7,043

---

Under review as a conference paper at ICLR 2022
samples, from which randomly sample 20.0% of the dataset as the test data. We split 12.5% of the training samples from both datasets for validation.
Baselines. Gradient boosting decision trees (GBDT) are widely used in tabular data. Therefore, compare two state-of-the-art algorithms, XGBoost (Chen & Guestrin 2016) and LightGBM (Ke et al. 2017). We also conduct comparison with the deep neural network based model TabNet (Arik & Pfister 2020) which is proven to achieve tfate-of-the-an the-arr-t results on tabular data. Additionally, we benchmark with random ensemble, in which the base models are trained independently with random initialization and the outputs are averaged for inference.
Table 1: Evaluation of tabular datasets. and indi- cate results taken from Anghel et al. (2019) Ardk & Plisted (2020), respectively. The taken results of XG- Boost and LightGBM for Higgs Boson are evaluated on the same test set ours, with exhaustive hyper- parameter optimization.
Base models and hyper-parameters.. We use MLP. whose implementation is in Ap- pendix [A...]] as the base model. For fair com- parison, we keep the same hyper-parameters of the base models for random ensemble and our method.
Higgs Beson
Blastchar
Tabular dataset
Accuracy
AUC
Accuracy
AUC
XGBoost
83.53%
80.06%
83.06%
4.1.2
IMAGE DATASETS
LightGBM
85.73%
80.70%
84.53%
TabNet
78.84%
80.4%%
84.21%
CIFAR. CIFAR-10 and CIFAR-100 are the standard image classification benchmarks (Krizhevsky et al. 2009), containing 50,000 and
Single model
78.49%
87.04%
81.48%
85.45%
Kandom onsembbe
78.84%
87.36%
81.41%
85.57%
SANE
79.22%
87.74%
81.83%
85.78%
10,000 samples, labeled 10 and 100 classes, respectively. From the training set, we randomly select 2,500 samples compose the validation set and apply the standard data augmentation detailed in He al. (2016b).
Table 2: Classification top-1 accuracy of different en- semble methods on CIFAR-10 and CIFAR-IOK with
K 3. The indicates result is taken from the orig- inal paper, while the $ represents from Zhang ct al.
(020)
CIFAR-TO CIFAR 100
Baselines. We compare SANE with sev- eral state-of-the-art ensemble algorithms ap- plied in image datasets, including Snapshot Ensemble (Huang et al. 2017), Fast Ensem- ble (Garipov et al., 2018). Multi-Model Ensem- ble via Adversarial Learning (MEAL) (Shen) 2019). Diversified Ensemble (Zhang 2020 and Diverse ensemble evolution et al.
Backhone
Method
Accuracy [top-]]
Single model
93.21%
73.13%
Random ensemblle
93.80%%
73,81%5
MOB method
92.87%
70.46%
VGGNet-19
Fast ensembble
93.29%
74.53%
Snapshot ensemble
94.33%
74.60%
MEAL
94.45%
Diversified ensemble
94.91%
74.34%
et al.
SANE
95.04%
77.64%
Single model
95.07%
77. 16%
MOE methed
94.46%
77
(Div") [Zhou at 2018). Additionally, also implement mixture-of-experts (MOE method) with reference to (hazzer al. (2017) for a soft gating network, see the Appendix A.1. for im-
ResNet-18
Fast casemble
94.80%
71.32%
Snapshot ensemble
95.72%
79.40%
Diy
79.12%
SANE
Single model
93.17%
70.29 20%
MOE methed
92.78%
72.02%
plementation details.
ResNet-32
Fast ensemble
92.11%
71
Snapshot onsemble
94.21%
74.51%
Base models and hyper-parameters. We use VGGNet-19 (Sioonyan & Zisserman, 2014), 2016a) as the backbone networks for the CIFAR datasets,
SANE
94.92%
76.10%
ResNet- 18, and ResNet-32 (He et al. and follow the standard hyper-parametter
4.2 EVALUATION ANALYSIS
We assess the performance of SANE and baselines on tabular and image datasets. The results are shown in Tad Table
Tabular dataset results. SANE outperforms all the tree-based methods and the state-of-the-art TabNet method on both datasets (RQ1) which illustrates the superior performance of our method in tabular datasets with various scales.
Image dataset results. For the CIFAR dataset, experiments are conducted on several backbone networks. We can see that SANE is consistently superior to all the baselines (RQ1). Among all methods using VGGNet-19 19 on CIFAR-IO0 data, SANE outperforms the second one (Diverisfied Ensemble) by 2.70%. That is to say, specializationnddriven SANE is better than implicitly and ex- plicitly diversity- methods. Recall that Div? assigns a subset of samples to the base models with static expertise. By comparing its result with that of SANE, can see that SANE with active

---

Under review as a conference paper at ICLR 2022
specialization specific regions in latent space works better (RQ2). Additionally, SANE brings higher improvement on CIFAR-100 than CIFAR-10, which reveals that it may perform better when the task is relatively more difficult. The MOE-based approach performs worse than single model in most cases. We speculate the reason that MOE uses ensemble loss to update base models and the gating network, resulting in diverse base models while aggregation does not achieve the effect of specialized aggregation. This result can also correspond to our ablation setting O of SANE only using L., in Section 4.3 This suggests that Le not the main component that makes SANE work. For comparison with papers without publicly available code, here we only present results with the number of base models K 3. Results for other K values can be found in Appendix A.2
4.3 ABLATION STUDIES
To investigate the implications of different components in our model, we design ablation experiments with 8 dif- ferent settings, which are performed in CIFAR-100 using VGGNet-19. Recall that, we adopt three losses Li. L. and L. that effectively serve our ensemble mechanism, thus we first conduct ablation study over them. Moreover, as men- tioned in Section 3.2.3 we also empirically demonstrate the effect of detaching the update of base model parameters {0x) from L. 's computation graph (referred to as detach, otherwise attach) on the ensemble performance.
1 Co. Co Ce,, detach
Cs &e
1.77
Cr, Ens Les, attach
3,62
Cc. fee, attach
5 Cz. Lo
3.54
6 Co Eny fee detach
c.
1.81
Diversifed ed ensemble
The results are presented in Figure 2 from which it is clear
Snapshot ensemble
that each loss plays part (RQ3). seems that La in Eq. ( is the most necessary one, as the two settings without it per- form worse than the single model. After discarding L. and
Fast ensemble
0M
Rendom ensemble
Le. the performance is respectively degraded by 1.04% and 0.33% while still outperforming the baselines.
Single model
0.0
0,64
Train
Maf
K
Additionally, we can compare the performance the base model detachment operation in two different settings (with or without C.), as 1 and 3 in Figure 2 The en- semble loss La naturally assigns the weights based on the model's prediction confidence over the true label class in back-pne bockkpropagation. From the results, the detachment affects the performance in opposite ways for the two different set- tings. In setting ④ the current specialization of the base model is encouraged by La and L., but no better final results are achieved than that in setting D. In contrast, in the ab- sence of L.. the loss of L. in the ensemble makes the base model more specialized when trained under setting 3 which brings some performance improvement. This suggests that ensembles can benefit from specialized base models. Notably, in addition assigning weights based on the model's current specialization, Le encourages the model to explore potential specializations. By comparing the results of -.-driven model specialization (setting + with that of our adaptive specialization (setting ()), it illustrates that our approach yields superior performance (RQ2).
2,40
7), En, C,, attach
8 Ev, C.,, C,, attach
Figure 2:
Ablation studies on CIFAR- 100 using VGGNet-19. The values rep- resent the performance difference, aver- age over runs, of each method relative to the single base model.
4.4 FURTHER ANALYSIS
Analysis of base models' specialization By design, base models need to specialize in different regions. It worth noting that the regions (distributions) in which a model specializes do not correspond to data classes. Here we only provide an intuitive visualization to answer RQ4. Since inter class distribution differences are generally more significant than those of intra class, we attempt to analyze base models' specializations on CIFAR-100 superclasses, by finding expressed specialists of samples. Ideally, samples of the same class would be dominantly specialized by one model. Figure 3 shows the amount of samples (proportional to the connection width) for which the model specialized in that class. The 3 base models (A, B. and C illustrated on the right side) trained by S SANE show divergent specialization different data classes (on the left side).
8

---

Under review as a conference paper at ICLR 2022
The number of parameters added for ensemble network. The ensemble network can be viewed as a agnostie framework. For different tasks, only the feature dimension and the number of base models will affect the number of parameters. When using ResNet- 18 as the base model, the ensemble network uses only 4.0% of the number of parameters of the single model to control 3-model ensemble, which improves the performance of CIFAR-100 by 5.1%.
RELATED WORK
Ensemble method has been proven effective for practical machine learning scenarios (Hansen & Salamon] [990] Schapire 1990, Zhou al., 2002). There are mainly phases in ensemble methods, I.e., base model training and model aggregation, and many works are studying either or both of them with emphasizing diversity. In the first phase, for example, bagging (Breiman) 1996) proposed to randomly select the subsets from the whole training data for training various base models. However, literature (Lee et al. 2015) shows that the base models in bagging cannot make full use of data yet suppresses the perfor- mance of deep neural networks. The similar issue also occurs in other sample-model matching methods such as Zhou et al. (2018) [iu et al. (2020). Other approaches re- lying on random initialization (Zhang et al. [2020] Laksh- minarayanan et al. 2016) may derive similar base models lacking diversity, as shown in this paper. Our method is in
SASTI DESTISSTIN C
FEEER 1333
Figure 3: Proportion of samples special-
ized by the 3 VGGNet-19 19 models (A/B/C) trained with SANE for the 10 CIFAR- 100 superclasses (labeled numbers).
contrast to Bayesian neural networks which estimate un- certainty through prior assumption (Gal & Ghahramani 2016) and rely on randomization.
There also exist some other issues in the second phase of ensemble learning or the joint optimiza- tion methods. Most works adopt averaging over the outputs from the base models, which has not demonstrated promising performance after ensemble operation (Lee et al. 2015).. Some works combine the two phases and propose to directly learn all the baseline models under the loss upon ensemble (Zhang et al., 2020; Zhou et al. 2018). However, [Lee et al. (2015) has pointed out that ensemble-aware loss through averaging base model predictions incorporates instability.
Other methods incorporate boosting (Schapire 1990 Freund, 1995; Moghimi et al., 2016) which focuses on weighted prediction combination through a sequentially training paradigm for deriving series of base models where each of them is optimized based on the condition derived from the previous ones. Other sequential optimization methods also include snapshot ensemble (Huang et al.] 2017). however, either boosting or snapshot ensemble is hard to parallelize and can lead to long training time when applied to deep neural networks (Zhou et al. 2018).
Our work is also related to mixture-of-experts methods, which have been proposed to tackle the problems of combining some specialized experts to achieve promising ensemble performance (Ja-) cobs et al. [991] Jordan & Jacobs, 1994, (hazeeer al., (017). However, the existing MOE methods neither consider local patterns explicitly, nor pay enough attention to effective model aggregation. Our method learns the specialized base models and specialization-awee
model aggregation simul- taneously, which can achieve superior performance as demonstrated in the experiments.
6 CONCLUSION
Real-world data is often generated by some complex distribution, which can be approximated by composition several simpler distributions. Existing ensemble methods that diversify base models are not able to explicitly enforce model specialization. Therefore, this paper proposes end-to- end ensemble learning method that actively enforces model specialization, where base models are trained to specialize in sub-regions of a latent space representing the simple distribution composition, and aggregated based on their specialty. Experiments on both image datasets and tabular datasets demonstrate the superior performance of our proposed method.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Ensemble learning is general method for machine learning. To the best of our knowledge, there are no intentional ethical issues created by the proposed method.
REPRODUCIBILITY STATEMENT
The datasets used in this paper are all publicly available, and all implementation details are intro- duced either in the main file in the appendix. In addition, we will release the source code of all the experiments upon paper acceptance.
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li, Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816 2020.
Andreea Anghel. Nikolaos Papandreou, Thomas Parnell, Alessandro De Palma, and Haralampos Pozidis. Benchmarking and optimization of gradient boosting decision tree algorithms. In Work- shop on Systems for MI. and Open Source Software at NeurlPS 2018, Montreal, Canada, 2019.
Sercan O Ank and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. arXiv, 2020.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature communications, 5(1):1-9, 2014.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123 140, 1996.
Defang Chen, Jian-Ping San-Ping Mei, Can Wang. Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3430-3437, 2020.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pp. 785-794, 2016.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: arXiv:1708.045522 2017.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint 1rX1v:1912.02757,2222 2019..
Yoav Freund. Boosting weak learning algorithm by majority. Information and computation, 121 (2):256-285, 1995.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059.
PMLR, 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Proceedings of the 32nd Inter- national Conference Neural Information Processing Systems, pp- 8803-8812. 2018.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference on Machine Learning, pp. 2170-2179. PMLR, 2019.
Raphael Gontijo-Lppes, Yann Dauphin, and Ekin D. Cubuk. No one representation to rule them all: Overlapping features of training methods, 2021.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. JEEE transactions on pattern analysis and machine intelligence, 12(10)9933-101, 1990.
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645 Springer. 2016b.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 41- 50, 2019.
Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume pp. 278-282. IEEE, 1995.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train 1, get for free. arXiv preprint arXiv: 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 2018.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Michael I
Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neurul computation, 6(2):181-214, 1994.
Kaggle. Telco customer churn, 2019. URL httpss//ww..kgglee.c////oostttts
telco- www.tuutsterrrro churn
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30:3146-33 -354,, 2017.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint 16/2.116/2.014777 2016.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv: 5.16616311111 14, 2015.
Nan Li, Yang Yu, and Zhi-Hus Zhou. Diversity regularized ensemble pruning. In Joint European conference machine learning and knowledge discovery in databases, pp. 330-345 Springer, 2012.
Zhining Liu, Wei Cao, Zhifeng Gao, Jiang Bian, Hechang Chen, Yi Chang, and Tie- Yan Liu. Self- paced ensemble for highly imbalanced massive data classification. In 2020 IEEE 36th Interna- tional Conference on Data Engineering (ICDE), pp. 841-852 IEEE, 2020.
Yan Luo. Yongkang Wong, Mohan Kankanhalli, and Qi Zhao. Direction concentration learning: Enhancing congruency in machine learning. IEEE transactions on pattern analysis and machine intelligence, 2019.
Andrés R Masegosa. Learning under model misspecification: Applications variational and en- semble methods. arXiv preprint 19Xi0:191..08335, 2019.
Mohammad Moghimi, Serge J Belongie, Mohammad J Saberian, Jian Yang, Nuno Vasconcelos, and Li-Jia Li. Boosted convolutional neural networks. In BMVC, volume 5, pp. 6, 2016.
Alexandre Rame and Matthieu Cord. Dice: Diversity in deep ensembles via conditional redundancy adversarial estimation. arXiv preprint arXiv: 20..x:2001.05444 2021.

---

Under review as a conference paper at ICLR 2022
Robert E Schapire. The strength of weak learnability. Machine learning, 5(2):197-227, 1990.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The spansely-gated mixture-of-experts layer. arXiv preprini arXiv: 1701. 00603.068, 2017.
Zhiqiang Shen, Zhankui He, and Xiangyang Xue. Meal: Multi-modes ensemble via adversarial learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4886- 4893, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Iya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research, 15(1):1929-199)) 2014.
Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mice: Mixture of contrastive experts for unsupervised image clustering. In International Conference Learning Representations, 2020.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. arXiv preprint arXiv:2006. 13570, 2020.
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478-487. PMLR, 2016.
Jianwei Yang, Devi Parikh, and Dhruv Batra. Joint unsupervised learning of deep representations and image clusters. In Proceedings the IEEE conference on computer vision and pattern recog- nition, pp. 5147-5156, 2016.
Shaofeng Zhang, Meng Liu, and Junchi Yan. The diversified ensemble neural network. Advances in Neural Information Processing Systems, 33, 2020.
Tianyi Zhou, Shengjie Wang, and Jeff A Bilmes. Diverse ensemble evolution: Currieulum data- model marriage. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 5909-5920, 2018.
Zhi- Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could better than 2002.
all. Artificial intelligence,
12

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 IMPLEMENTATION DETAILS
In this section, we present the details of the base model design, hyper-parameter setup, and imple- mentation of the MOE method. For all tasks, we first tune the hyper-parameters of the base model and then fix them. The learning rate the base model is kept by hyper-parameter tuning of the base model, thus, the loss weight 85 in Eq. (5) is set to 1. For the ensemble network, we tune its learning rate, Sc. and 8c. Additionally, the embedding dimension of specialty anchors is also a typer-parameter Table lists the run a hhyer--aramette tuming with 100 search steps. In the ensemble network, the projections Q. K,, and V and the scale-transfer layer are implemented as fully connected layers, which are the same for both image and tabular datasets. The temperature parameters ты and Te are set as 0.12 and 0.01, respectively.
to tune and their search space. For all tasks,
Table 3: Syperr-paameee we tune for the ensemble network. We search learning rates using log uniform.
Hyper-parametes parameter
Search Range
learning rate of ensemble network loss weight &c loss weight se embedding dimension of specialty anchors (as) k=
[0.0001,0.1]
[1,2,3,4.5]
[1,2,3,4,5]
[25,50,75,100]
From the results, SANE works on larger datasets using larger models as well and consistently out- performs basclines.
A.1.1 IMPLEMENTATION DETAILS FOR TABULAR DATASETS
Base models. For tabular datasets, we use an MLP as the base model. For each layer of the MLP, the fully connected layer is followed by a batch normalization layer and a residual connection, using the gated linear unit (GLU) the activation function. The feature encoder F in the ensemble network is implemented as MLP of the same structure as the base model, and we take the output of the last fully connected layer the feature embedding. For the Higgs Boson dataset, structure used by Baldi et al. (2014), which is a 5-layer MLP with 300 hidden units in each layer. For Blastchar, we use 2-layer MLP with 32 hidden units in each layer.
follow the network
Slyper-paramete setup. For the Higgs Boson dataset, we follow most of the hyper-parameters in Baldi et al.] (2014), except that the learning rate is modified to 0.01. For Blastchar, we Adam with a learning rate of 0.001 and a weight decay of 0.001.
A.1.2 IMPLEMENTATION DETAILS FOR IMAGE DATASETS
Base models. For image datasets, we use several backbones, including VGGNet-199 19, ResNet- and ResNet-32. Implementations of these backbones are standard. We use pre-trained backbone corresponding to the base model and freeze it as the feature encoder J in the ensemble network. The pre-trained VGGNet-19 and ResNet-18 are taken from Vision, an open-source machine vision package for Torch. Since there is no available pre-trained ResNet-32 available in Torch Vision, we use VGGNet-19 19 replacement. All of these pre-trained model outputs -00-diimenioonal feature embedding.
Hyper-parameter setup. On the CIFAR datasets, the standard parameters were applied for these commonly used backbones. For VGGNet-19 19 and ResNet-18, we follow the hyper-parameter used in Vries & Taylor (2017). For ResNet-32, we use the same 2020). To be specific, for 1G and Resnet-18, the initial learning rate was 0.1, and the learning rate was divided by 5 for the 60th, 120th, and 160th epochs, with 200 epochs of training, batch size of 128, weight decay of 5e-4, and a Nesterov momentum of 0.9. For ResNet32, trained for 300 epochs, with all other parameters unchanged, while the learning rate was divided by 10 at the 150th and 225th epochs.
from Chen et al.
13

---

Under review as a conference paper at ICLR 2022
A.1.3 IMPLEMENTATION DETAILS OF MIXTURE-OF-EXPER METHOD
In similar fashion to our framework, MOE consists of set of expert networks {E, 1. and a gating network G. For the output of MOE is D G(x,, E; (x.). In Shazeer et al. 2017) the gating network realized as G(=) Softmax (xr () where W,, is trainable weight matrix. The training of expert networks and the gating network is done by back- propagatoinno of cross entropy loss between its outputs and ground truth, i.c., analogously to our L. in Eq. 44) To present fair comparison with Softmax Softmax(W(F(r))) (WFF))))) where F is the feature
ours, further strength the gating network G(x) encoder and W is an MLP with hidden layers.
A.2 NUMBER OF BASE MODELS K
CIFAR-10
CIFAR-100
94.5
76
Accuracy
Accuracy
75
94.0
93.5
Our method Snapshot
72
Our method Snapshot Fast ensemble
Fast ensemble
71
93.0
10
3
10
Number base models (K)
Number base models (K)
Figure 4: Accuracy comparison methods with different number of base models on CIFAR- 10 and CIFAR-100.
Figure 4 presents the accuracy of CIFAR-10/100 with different number of base models K. Here compare with snapshot ensembles and fast ensembles, which perform worse than our method at any K value. The best performance for CIFAR-10 and -100 obtained when K is equal and 7, respectively, seen in the figures.
B EXPERIMENTAL RESULTS ON TINY IMAGENET
In addition to the CIFAR dataset, we also conduct experiments on Tiny ImageNet with more complex
data distribution.
Base models. While previous papers focused on the CIFAR dataset, Snapshot Ensemble conducted experiments on Tiny ImageNet using ResNet-110 as the backbone network, which we followed. In addition, experiment EfficientNet-BOl a promising choice the dataset, is performed using the same
(Luo et al. 2019)).
We compared with Random Ensemble, Snapshot Ensemble, and Fast Ensemble, and results as follows.
Table 4: The comparison experiments on Tiny ImageNet using different backbones, with ensemble
size of 3.
EfficientNet-B0
ResNet- 110
Single model
59.34
51.18
Fast ensemble
60.66
53.70
Snapshot ensemble
64.59
54.03
Random ensemble
64.71
56.32
SANE
65.19
57.01
14

---

Under review as a conference paper at ICLR 2022
C RESULTS ON THE CIFAR DATASET COMPARED TO THE SUPPLEMENTAL BASELINES
Apart from baselines listed in Section 4.1.2 Stochastic Weight Averaging (Izmailov al. 2018) and Hyper-parameter parameter Ensemble (Wenzel et al. 2020) are also STOA methods in ensemble learning. Due to space limitation, we present the results of the comparison experiments in this section.
For Stochastic Weight Averaging, we conduct its experiments on CIFAR-101 10AAR-10/1000 00 with VGGNet-19 using setrings consistent with experiments ensure fair comparison. For the Hyper-parametee Ensemble, we re-experiment SANE on ResNet-20 to compare with the results presented in their paper. The comparison results are as follows:
Table 5: Complete comparison results CIFAR-10 and CIFAR-100 with K 3. The indicates result taken from the original paper, while the ý represents from Zhang et al. (2020).
CIFAR-10 CIFAR 100
Backbone
Method
Accuracy (top-I)
Single model
93.21%
73.13%
Random ensemble
93.80%
73.81 %65
MOE method
92.87%
70.46%
VGGNet-19
Stochastic Weight Averaging
93.42%
72.84%
Fast ensemble
93.29%
74.53%
Snapshot ensemble
94.53%
74.60%
MEAL
94.45%
Diversified ensemble
94.91%"
74.94%*
SANE
95.04%
77.64%
Table 6: The comparison experiments CIFAR-10/100 using ResNet-20 between SANE and Ensemble marked in green, with K 4. The results of Plyper-paramett En- semble are taken from the original paper (Wenzel al. 2020). The values in parentheses represent the performance improvement brought by the ensemble compared to the single model results.
Backbone
ResNet-20
Method
Hyper-paramen parameter Ensemble
SANE
CIFAR-10
94.00% (+ 1.30%)
94.60% % (+ 2.18%)
CIFAR- T00
74.20% (+ 6.00%)
74.82% (+ 6.82%)
From the results, SANE outperforms Stochastic Weight Averaging and Hyper-parameter Ensemble fair comparison setting.
D ENSEMBLE OF DIVERSE BASE MODELS
In previous experiments, focus the ensemble of base models with the same architecture, since evaluating the effectiveness and generalization of the framework on different data types is the main target. That the only type of randomness that exists in SANE is the different initialization of base models. Lopes et al. (2021) claimed that models across different hyper- parameters, archi- tectures, frameworks, and datasets could be specialized for sub-domains of data, leading to higher ensemble performance. It confirmed the power of specialization in ensemble learning, although they propose to implicitly achieve specialization rather than explicitly as our method.
To verify the validity of pursuing specialization explicitly, compare the performance of specialization-awarl ensemble on diverse base models with that specialization achieved by im- plicit manner, i.e., random ensemble of diverse architectures. Moreover, measure and visualize the specialization of the models trained by the two methods respectively.
15

---

Under review as a conference paper at ICLR 2022
0.
0A
Avar aga of meset Nocel coreet rod
Model confidence
0.0
1.00
0.75
6.35
6.23
0.00 0
0.15
0.50
0.73
1000
θ
Model confident
Model 2 confident
0.4
(b) Random ensemble
0.4
Alci ags visu of meast sarred Mppel
Nocal cornect ctd
60
0.2
QA
10
0.
Model confidence
(a)
0.
09 Model confident
0
Model 2 confident
(c)
Figure 5: (a) the confidence-coffidnnce plot with 0 as the specialization measurement defined in Lontijo-Lopes et al. (202T). (b) and (c) are specialization measurement of two base models trained by SANE and independently trained when only model 1/2 correct, respectively.
Base models and ResNet-18, and DenseNet-40. All the hyper-parameter setup the same as in Appendix AA.
We use three different architectures, i.e., VGGNet-
From the results, we fing that the SANE-trained ensemble (78.37%) outperform the random en- semble (76.83%) by 1.54%. It indicates that, under the same network architectures and model size setting, even though architectural diversity can benefit the ensemble by bringing implicit specializa- tion, explicitly encouraging specialization will benefit the ensemble further.
In Gontijo-Loppss et al. (2021), they define specialization metric 0 as the angular distance in the confidencee-contidence graph, as shown in Figure a)), and visualize it. Inspired by that, we show the specialization 0 of the two base models (ResNet-18 18 and DenseNet-40) in Figure 5) when only model 1/2 makes a correct prediction, respectively. We compare with the random ensemble with architecture and initialization diversity and no explicit specialization guidance, and the results shown in Figure ()c). In the figure, we label the average Ø in both cases. As can be seen, while both ensembles present that 'when only model makes the correct prediction, tends to be more confident', SANE's base model exhibits higher level of specialization compared to the baseline.
Additionally, though effective, introducing randomness in hyper- parameess,,,,rss, architectures, frame- works, and datasets will require a lot of expert knowledge and manual efforts. In contrast, our method is generic framework with the ability learn achieve ensemble and explicitly enforce specialization.
16


Reviews:
[{"review_id": "qSqfjT98xJX", "reviewer": "Reviewer_vevN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I don't work in this area, so I may be not familiar with the related work. From my viewpoint, using the performance of base learner to train a specialized weak learner is a quite natural idea. For the method part, I have one question:\n\n1. It is not quite reasonable to use a binary cross entropy to supervise w in Eqn 3. Think that one easy example can be correctly classified by all the base classifiers, then all the base learner will try to fit it. This is contradict to the motivation of specialization. \n\nFor the experiments, I have the following two questions:\n1. Comparisons with Boosting tree related methods are unfair. The base learner is quite different. According to Table1, even the single model is better than the boosting tree methods, which makes this comparison meaningless. If for a fair comparison, the authors should provide comparisons of boosting and various ensemble based on the same base learner.\n2. For the image experiments, The baseline and dataset is too weak to demonstrate the superiority. I suggest using the most recognized combination ResNet50 +ImageNet for this experiment. \n\n", "summary_of_the_paper": "This paper presents a novel method for ensemble learning. By introducing a anchor scheme and specialization loss , the base learner are forced to be specialized. The method is validated on both tabular and image datasets.", "summary_of_the_review": "To summarize, I think this paper does not contain many new concept from my view. According to the experiments, I think the results are good, however some experiments are not that solid. Since I don't work in this area, I rate this paper as borderline reject, and would like to participate the discussion in rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OAW3I7HIUmm", "reviewer": "Reviewer_etDH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major comments\n\nMotivation: the paper claims it is “intuitive to divide the complex model learning into training several simpler models.” However, such an approach has thus far not been shown to yield high performance [1]. In my opinion, the paper should justify why the approach presented fixes the issues that lead to [1] not finding high-performance diverse models.\n\nCorrectness: the paper claims it shows that  “without explicitly enforcing specialization, pursuing diversity may not be enough to achieve satisfactory ensemble performance,” but it only investigates a very narrow set of tasks. In particular, concurrent work [2] shows that with different enough models, one can achieve specialization (and high ensemble performance) without explicitly training for it. This points to the fact that the paper’s baselines of ensembling randomly-reinitialized models is not a good enough baseline for this claim. The paper also claims that “without explicitly enforcing model specialization, pursuing model diversity does not provide guarantees of each model’s specialization on different simple distribution in the composition”, but it is not shown what kinds of guarantees the specialization-enforcement losses presented yield.\n\nExperiments: the paper mainly focuses on tabular data and small image datasets, with small models (the largest used is ResNet 32). They compare their method only with randomly-reinitialized ensembles, and other diversity-inducing methods, never even ensembling two networks together, or using SOTA ensemble techniques such as weight-averaging/EMA, or hyper-parameter ensembling. This makes it difficult to see how the results presented actually compare to realistic settings.\n\n\n\n[1] http://www.gatsby.ucl.ac.uk/~balaji/why_arent_bootstrapped_neural_networks_better.pdf\n[2] https://openreview.net/pdf?id=BK-4qbGgIE3\n\nMinor comments\n\n“the proposed specialization-aware neural network ensemble method can achieve superior performance compared with the state-of-the-art ensemble methods in various prediction tasks.” SOTA methods include weight-averaging and hyper-parameter ensembling, so this sentence strikes me as incorrect.\n\n“However, overvaluing diversity may hurt the ensemble effectiveness and the\ntrade-off between the performance and diversity of base models still remains an open problem.”\nCitation needed?\n\nIt’s unclear what the message of Fig 3 is. It strikes me as uninformative.\n\nMany claims of what there “should” be in a design, without a lot of motivation, or ablation\n“there should be an evolutionary process in the specialization of the model. The specialty regions of the models should be explored and learned in an adaptive manner”\n\nFig 2 should include error bars (multiple runs of each experiment), especially since doing these on small datasets like cifar-100.\n", "summary_of_the_paper": "This paper proposes a method to train models that specialize in subsets of the data distribution by weighting the training loss based on how close example embeddings are to a learned “anchor” embedding. It also proposes to improve ensembling by combining multiple “specialized” models.\n", "summary_of_the_review": "In general, I think finding methods to encourage specialization in models (while still achieving high accuracies) is a very important direction to pursue. The method presented of how to train specialized models (using anchors) is intuitive, and might be useful if demonstrated that it works. Unfortunately, it’s not possible to assert this from the limited baselines/experiments presented.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9GfqBUDqEPQ", "reviewer": "Reviewer_G9Xj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Explicit incorporation of the model diversity through sub-region.\nNo theoretical justification to support the claim that it will give a better ensemble model.", "summary_of_the_paper": "This paper learns an ensemble of models where each of the base models has a specialization in some latent sub-region.", "summary_of_the_review": "The use of specialized models on sub-regions can be very interesting for shallow models but for complex models such as deep networks, it is not convincing that the models specialized in sub-regions can bring any change.\n\nHow the authors found their results different than predicting a sample using only the subregions models by maintaining a mapping for a test sample to a sub-region model.\n\n\nThe results show the only marginal differences.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8z4qeK4a5sK", "reviewer": "Reviewer_y2rX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Strengths:\n1. The paper is well-written and its idea is easy to follow\n2. This paper presents a new perspective - specialization - to enhance the ensemble learning. By the proposed approach, the problems of existing strategies can be solved.\n\n* Weakness:\n1. Although the analysis perspective is new, but the technique used in this paper are current existing one. The core for distributing samples in the latent space is based on the attention machanism. According to the description in the manuscript, this part is a direct usage of the one in transformer. This leads to my concern on the novelty of this paper.\n2. I think the number of specialization anchors is one of the most important hyper-parameter of the proposed approach. However, the analysis on this hyper-param. is very limited. According the appendix, the maximum of this param is only 10 in the experiments. I am concerned about if the setting is still valid on more complex data distribution, such as the common use one, ImageNet dataset. To make the conclusion more convincing, I think the author should provide analysis on this hyper-param. or experiments on large-scale dataset.", "summary_of_the_paper": "This paper presents a SANE model to improve the ensemble learning from the perspective of model specialization. In particular, it first gives the analysis on existing ensemble strategies, i.e., random and diversity-driven, via conducting experiments on synthetic data. By this way, it points out the weakness of these strategies lies in the lack of specialty. Motivated by this, this paper presents the specialization-aware method to improve the ensemble learning. Specifically, it introduces anchor points in the latent space for guiding the model learning towards specialty. To derive the correlation between samples and anchors, it utilizes the transformer-like attention mechanism for learning the weights of base models. This paper demonstrates its effectiveness on several image and tabular benchmarks. ", "summary_of_the_review": "In summary, I think the idea for encouraging specialization in the model ensemble is rational and novel. But I am conerned about the novelty of the specific technique adopted in this paper as well as the insufficient experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel ensemble method that enforces specification of the base models to improve accuracy. Base models are specialized on sub-regions of the latent space. To calculate the ensemble prediction for a given example base models are weighted based on how close the example embeddings are to a learned “anchor” embedding. To derive the correlation between samples and anchors,  transformer-like attention mechanism is used. 

Reviewers pointed out limitations in the experimental analysis. In turn authors added experiments on tiny-image net with additional architectures and a comparison to several additional baselines on CIFAR-10/100 supporting their findings, which improved the paper. Nevertheless, the paper remained on the borderline after the discussion period and reviewers continued to have doubts about the the significance and novelty of the proposed method, therefore the paper can not be accepted in its current form.
Decision: Reject