Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors conduct a large-scale study on when models are more likely to learn different representations, and how diversity in learned representations can help ensemble performance. They select 5 possible reasons for representation diversity: model reinitialization, hyperparameters, model architectures, model frameworks, and datasets. They define a measure of error correlation, and show that a pair of models with more uncorrelated errors results in higher ensemble performance - even if one or both of those models has "low accuracy". They show that models trained on different datasets are most likely to have uncorrelated errors, as opposed to the other sources of representation diversity. The authors study the effect of sampling and concatenating features from two different models, find that a mix of features from both models yields the best performance, and conclude that the models have learnt different features (yet likely overlapping). They also examine the categorical specializations between varying models, as well as the effect of training diversity for creating ensembles on downstream tasks.
main_review: Strengths:

The paper is well-written, and has clearly very thorough experiments. Notably, they include 82 different models (across their 5 categories of training diversity), which include recent high-performing works.

Each point in their 5 listed contributions at the end of the introduction is interesting and valuable for future research (although I have some concerns listed below). Each point provides evidence for the authors' concluding statement, that no single training methodology should dominate the field, but rather research effort on a diverse set of approaches should continue. They also raise a very interesting question for theory as well as empirical studies - why doesn't SGD learn the more diverse representations that are shown to perform better?

One part of "well-written" - the authors repeat key definitions throughout the work, instead of defining them once, which I found helpful (e.g. repeating the definition of "conversion rate" in the Figure 2 caption).

Weaknesses:

Uncorrelated errors seem like an intuitive metric to analyze representation diversity - however, there are other existing metrics which I imagine measure this more directly, such as mutual information. Why not measure representation diversity in that manner?

I am also not sure why the authors chose to use the angle distance on the confidence-confidence plot in their analysis on model specialization. It works, but seems a little odd - why not just subtract one model confidence from another?

In Table 3 in the Appendix, the authors seem to define "low accuracy" models as those below the 74% accuracy (on ImageNet) threshold they use in their main analysis. However, Section 4.5 considers the 74%-78% range as "low accuracy" - which is it? In the current form, the claim "low accuracy models can benefit high-accuracy ensembles" is misleading.

A small clarification question - why is the dataset star in Figure 2 marked with an additional cross? To highlight the best category? If so, that would be useful to mention (assuming I didn't miss the explanation somewhere).
summary_of_the_review: The authors present a thorough evaluation into the effect of various differences in training procedures on resulting representation diversity. I have not specialized in this research area and cannot speak well to the novelty of each claim, but believe that the paper deserves acceptance due to the extensive experiments, clear presentation, and interesting findings which open questions for future research.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper empirically investigates how different models, trained with different methodologies should be ensembled to maximize the accuracy in image-net classification task. By carefully designed experiments, the authors look at different aspects of the trained models and provide guidelines for selecting models to be ensembled. The main takeaway: models trained with increased divergence in training methodologies are best suited for ensembling.
main_review: I find this paper very insightful and believe that the investigations done here would be useful for the community.  Ensembling is a prevalent way of achieving desired accuracies in practice and this study provides helpful pointers to do that effectively. I have following questions for the authors though:

- It might just be me, but how is the training methodology "Frameworks" is incorporated in this work? I could not find any details or explanation about it.
- In the frameworks section, SimCLR is used. This model is trained with a self-supervised objective with no prediction head for labels. How do you attain prediction confidences on a test set for such models? or how do you incorporate predictions for such a model?
- Based on the previous question, is it possible to use different self-supervised or even unsupervised methods?
- The dataset difference is a part of the study, but is it fair to compare the accuracies on image-net for the models which were not trained on image-net? Wouldn't the methods which aren't trained on image-net suffer from domain generalization problem?
- In CLIP-S method, it is unclear to me whether it's the training dataset that introduces a huge divergence in the methodology or the contrastive objective. I think there could be a sub-categorization of the training methodology "Dataset" into "optimization objectives" and "datasets" for a more controlled study. I will expand on this in the following point.
- Figure 4, demonstrates some clear differences in the specialization of CLIP-S and resnet-50. Since the models are trained on different datasets and then tested on image-net. I wonder how much important the bias of the training dataset is. More importantly, what will happen if the ResNet is trained on the [somewhat similar] dataset used for training CLIP-S and tested on image-net? would the variation in architecture and frameworks still matter?
- In the light of the above, supervised contrastive learning [1] might be a useful model to try as it uses contrastive training objective and is trained on image-net.
- Have you quantified the conversion rate where both ensembling models predict incorrectly. I wonder if the ensembled prediction confidences might make the incorrect prediction correct. It will be interesting to see how various training methodologies influence that?
- Is there an opportunity for weighted ensembling? Based on the analysis of training methodology can one further improve the ensembling by rescaling prediction confidences of certain models for certain test datasets? One can probably use specialization criteria to do that?
- In section 4.6, do you train a linear classifier (via L-BFGS) on top of concatenated representations? How do you pick the portions of each representation to be concatenated? is it selected randomly?
- In section 4.6 (later half), do you select diverse features from both models at the same time and concatenate these features or you do that separately for both models?

[1] [https://arxiv.org/abs/2004.11362](https://arxiv.org/abs/2004.11362)
summary_of_the_review: I liked the overall study and believe that it will be useful for the community. I would appreciate it if the authors could address my concerns/ questions above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents an extensive empirical evaluation of what models learn when trained with different architectures, frameworks, and datasets. The authors discuss the effects of training methodology on the types of errors that models make, and show that ensembling models trained with diverse training methodologies can result in surprisingly strong performance (even when the models in the ensemble themselves are relatively weak).
main_review: This paper is well-written and interesting, and the empirical analysis is impressive. The authors present their empirical observations as the main contribution, but I think this actually undersells the contributions -- this paper is also a tour de force in methodologies about how to evaluate and compare different models. Section 4 is very well-written and is a golden example of strong experimental design/scientific method; each sub-section presents a concrete hypothesis, an experiment to test the hypothesis, and discussion of empirical results. I wish every paper were written this well!

My only complaint is that some of the language is imprecise. As a concrete example, the phrase "different enough" or "diverse enough" appears several times throughout the manuscript. This phrase can be more precise (What does it mean to be "different enough?" Different architecture? Different dataset?). The results speak for themselves by the end of section 4, but the takeaways could be more precise for the reader (note that this may take more analysis/experiments -- but I'm sure the authors can knock that out of the park).

I also think the paper could be improved by including some more intuition or examples of the types of divergences between different models. For example, what types of examples does CLIP get wrong but SimCLR gets right? This may be unwieldy for all 82 models, but a few spot instances for each framework would be very interesting and useful.
summary_of_the_review: This paper is interesting and well-written, and it presents an impressive empirical evaluation of how training methodology affects representations and predictions. The paper does not claim novel methods, but the methods for empirical analysis themselves are interesting, and the community can learn a lot from them.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an empirical study of how different modelling choices (eg, hyper-parameters, architecture, training algorithm, dataset) result in different and complementary data representations. The first half of the paper shows that those choices influence model predictions, leading to models that specialize to a subset of the data. The second half shows that those different representations are complementary and can be ensembled to yield better performing models. In particular, the authors show that ensembling models that are most diverse gives the largest improvement when transferring to a downstream task.
main_review: I find this paper well-motivated, with a clear focus on understanding whether different training methodologies learn similar representations, or not. The flow of the paper is well written, with each section addressing a one line of inquiry and naturally leading to the next. The (many) experiments are well designed, and the results clearly presented and interpreted. I especially enjoyed the first set of results (Figure 1), where the authors show that increasingly different training methodologies yield increasingly different representations — those results should be of interest to the ICLR community.

I list below some questions and concerns, in no particular order.

- If possible, please move Table 1 from the Appendix to the main text. I did not understand the experimental setup in Section 3.1 until I found this table.
- On p. 3, the paper introduces categories of possible changes to their base learning setup (supervised learning of Resnet-50 on ImageNet), and argues that some changes are more important than others: eg, changing the architecture to ViT is more important than doubling the weight-decay coefficient. Could the authors justify why changing the dataset is more important than changing the learning framework? It seems reasonable the procedure to infer model parameters (ie, the framework) can be as or more important than the choice of datasets, provided those datasets are large enough. If that's the case, we'd expect error inconsistency to be higher for "framework" than for "dataset" in Table 1, thus more diverse models are obtained by changing the framework (not the dataset). This begs the question: how much of the subsequent analysis depends on "dataset" being more important than "framework"? Would we see that better ensembles are built from models trained on the same (large enough) dataset but with different frameworks?
- The authors already provide experiments under many different conditions (see Table 1), and I do not wish to ask for more. Still, I question:
    - Why not include a Resnet-50 model when changing the framework and the dataset? This way, one could see how much improvement is expected when changing framework / dataset but the architecture is kept constant.
    - Are the (only) 2 different conditions enough to draw conclusions on the effect of changing the framework? One could imagine also including generative (auto-encoders, flows, etc) or few-shot training (protonet, metaoptnet, etc) as well.
- One of the main weaknesses of the paper is the lack of driving hypothesis. The experiments show that different training conditions yield different representations (and that those representations become specialized, yield better ensembles), but we're never told why. The second-to-last paragraph (p. 9) is not illuminating in that regard.
- Finally, there are a few moments of carelessness in the manuscript that need to be polished. For example, citations for WIT and JFT are never given; the "LBFGS Accuracy" of Section 4.6 is never properly defined (only mentioned in passing in Section 4.7); the section on bootstrapping (Section 4.5) doesn't discuss Freund and Schapire's work on weak learners; "Stochastic Weight Averaging" (Izmailov et al., UAI 2018) is never mentioned although it is a successful method to ensemble deep networks.

Some typos:

- p. 2: Use \citep for Andreassen et al. (2021)?
- p. 3: Is it 82 models since those in Table 4 are not included (looks more like 55-ish in Tables 1 and 2)?
- p. 4: "the the ensemble prediction".
- p. 8: "imagenet" not capitalized.
- p. 9: "CLIP-S is yields the most prediction diversity".
- p. 9: "any single models' accuracy" → "model's".
- p. 11: The reference is just "Lehman & Stanley", no "et al.".
summary_of_the_review: Overall I remain positive about this paper and vote for acceptance. My main concerns are points 2 and 3 from the main review.
review_title: null

Meta-review: This well written and well motivated paper has been independently reviewed by four expert reviewers. They all voted for the acceptance with three straight accepts and one marginal. The feedback provided to authors was constructive and the authors responded comprehensively. I recommend acceptance of this work for ICLR.
Decision: Accept (Poster)