Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel method for ensemble learning. By introducing a anchor scheme and specialization loss , the base learner are forced to be specialized. The method is validated on both tabular and image datasets.
main_review: I don't work in this area, so I may be not familiar with the related work. From my viewpoint, using the performance of base learner to train a specialized weak learner is a quite natural idea. For the method part, I have one question:

1. It is not quite reasonable to use a binary cross entropy to supervise w in Eqn 3. Think that one easy example can be correctly classified by all the base classifiers, then all the base learner will try to fit it. This is contradict to the motivation of specialization. 

For the experiments, I have the following two questions:
1. Comparisons with Boosting tree related methods are unfair. The base learner is quite different. According to Table1, even the single model is better than the boosting tree methods, which makes this comparison meaningless. If for a fair comparison, the authors should provide comparisons of boosting and various ensemble based on the same base learner.
2. For the image experiments, The baseline and dataset is too weak to demonstrate the superiority. I suggest using the most recognized combination ResNet50 +ImageNet for this experiment. 


summary_of_the_review: To summarize, I think this paper does not contain many new concept from my view. According to the experiments, I think the results are good, however some experiments are not that solid. Since I don't work in this area, I rate this paper as borderline reject, and would like to participate the discussion in rebuttal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to train models that specialize in subsets of the data distribution by weighting the training loss based on how close example embeddings are to a learned “anchor” embedding. It also proposes to improve ensembling by combining multiple “specialized” models.

main_review: Major comments

Motivation: the paper claims it is “intuitive to divide the complex model learning into training several simpler models.” However, such an approach has thus far not been shown to yield high performance [1]. In my opinion, the paper should justify why the approach presented fixes the issues that lead to [1] not finding high-performance diverse models.

Correctness: the paper claims it shows that  “without explicitly enforcing specialization, pursuing diversity may not be enough to achieve satisfactory ensemble performance,” but it only investigates a very narrow set of tasks. In particular, concurrent work [2] shows that with different enough models, one can achieve specialization (and high ensemble performance) without explicitly training for it. This points to the fact that the paper’s baselines of ensembling randomly-reinitialized models is not a good enough baseline for this claim. The paper also claims that “without explicitly enforcing model specialization, pursuing model diversity does not provide guarantees of each model’s specialization on different simple distribution in the composition”, but it is not shown what kinds of guarantees the specialization-enforcement losses presented yield.

Experiments: the paper mainly focuses on tabular data and small image datasets, with small models (the largest used is ResNet 32). They compare their method only with randomly-reinitialized ensembles, and other diversity-inducing methods, never even ensembling two networks together, or using SOTA ensemble techniques such as weight-averaging/EMA, or hyper-parameter ensembling. This makes it difficult to see how the results presented actually compare to realistic settings.



[1] http://www.gatsby.ucl.ac.uk/~balaji/why_arent_bootstrapped_neural_networks_better.pdf
[2] https://openreview.net/pdf?id=BK-4qbGgIE3

Minor comments

“the proposed specialization-aware neural network ensemble method can achieve superior performance compared with the state-of-the-art ensemble methods in various prediction tasks.” SOTA methods include weight-averaging and hyper-parameter ensembling, so this sentence strikes me as incorrect.

“However, overvaluing diversity may hurt the ensemble effectiveness and the
trade-off between the performance and diversity of base models still remains an open problem.”
Citation needed?

It’s unclear what the message of Fig 3 is. It strikes me as uninformative.

Many claims of what there “should” be in a design, without a lot of motivation, or ablation
“there should be an evolutionary process in the specialization of the model. The specialty regions of the models should be explored and learned in an adaptive manner”

Fig 2 should include error bars (multiple runs of each experiment), especially since doing these on small datasets like cifar-100.

summary_of_the_review: In general, I think finding methods to encourage specialization in models (while still achieving high accuracies) is a very important direction to pursue. The method presented of how to train specialized models (using anchors) is intuitive, and might be useful if demonstrated that it works. Unfortunately, it’s not possible to assert this from the limited baselines/experiments presented.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper learns an ensemble of models where each of the base models has a specialization in some latent sub-region.
main_review: Explicit incorporation of the model diversity through sub-region.
No theoretical justification to support the claim that it will give a better ensemble model.
summary_of_the_review: The use of specialized models on sub-regions can be very interesting for shallow models but for complex models such as deep networks, it is not convincing that the models specialized in sub-regions can bring any change.

How the authors found their results different than predicting a sample using only the subregions models by maintaining a mapping for a test sample to a sub-region model.


The results show the only marginal differences.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a SANE model to improve the ensemble learning from the perspective of model specialization. In particular, it first gives the analysis on existing ensemble strategies, i.e., random and diversity-driven, via conducting experiments on synthetic data. By this way, it points out the weakness of these strategies lies in the lack of specialty. Motivated by this, this paper presents the specialization-aware method to improve the ensemble learning. Specifically, it introduces anchor points in the latent space for guiding the model learning towards specialty. To derive the correlation between samples and anchors, it utilizes the transformer-like attention mechanism for learning the weights of base models. This paper demonstrates its effectiveness on several image and tabular benchmarks. 
main_review: * Strengths:
1. The paper is well-written and its idea is easy to follow
2. This paper presents a new perspective - specialization - to enhance the ensemble learning. By the proposed approach, the problems of existing strategies can be solved.

* Weakness:
1. Although the analysis perspective is new, but the technique used in this paper are current existing one. The core for distributing samples in the latent space is based on the attention machanism. According to the description in the manuscript, this part is a direct usage of the one in transformer. This leads to my concern on the novelty of this paper.
2. I think the number of specialization anchors is one of the most important hyper-parameter of the proposed approach. However, the analysis on this hyper-param. is very limited. According the appendix, the maximum of this param is only 10 in the experiments. I am concerned about if the setting is still valid on more complex data distribution, such as the common use one, ImageNet dataset. To make the conclusion more convincing, I think the author should provide analysis on this hyper-param. or experiments on large-scale dataset.
summary_of_the_review: In summary, I think the idea for encouraging specialization in the model ensemble is rational and novel. But I am conerned about the novelty of the specific technique adopted in this paper as well as the insufficient experiments.
review_title: null

Meta-review: This paper proposes a novel ensemble method that enforces specification of the base models to improve accuracy. Base models are specialized on sub-regions of the latent space. To calculate the ensemble prediction for a given example base models are weighted based on how close the example embeddings are to a learned “anchor” embedding. To derive the correlation between samples and anchors,  transformer-like attention mechanism is used. 

Reviewers pointed out limitations in the experimental analysis. In turn authors added experiments on tiny-image net with additional architectures and a comparison to several additional baselines on CIFAR-10/100 supporting their findings, which improved the paper. Nevertheless, the paper remained on the borderline after the discussion period and reviewers continued to have doubts about the the significance and novelty of the proposed method, therefore the paper can not be accepted in its current form.
Decision: Reject