Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to address an important drawback of Deep Ensemble, the inference cost of executing multiple models. The intuition behind the proposed method is that the outputs in the function subspace can be estimated from the modes without having to forward the actual parameters on the subspace. Based on this intuition, an additional lightweight network is trained as a bridge network to predict the outputs from the connecting subspace. 
strength_and_weaknesses: ### Strength 

* Reducing the inference cost of DE is an important topic. This paper is well-motivated. 
* The presentation of this paper is clear and easy to follow. 
* Extensive experiments demonstrate the effectiveness of the proposed two types of bridge networks on TinyImageNet and Cifar. 

### Weaknesses
* In Table 4, why do more bridge models lead to better results? 
* What's the functional difference between type I and II bridge models according to theoretical and empirical results? Is there any conclusion we can reach about how to choose types I and II?
* Are there any important/sensitive hyper-parameters during the training of bridge models? How to determine the optimal number of bridge models in practice? 
* I am curious about how the proposed methods perform on large-scale datasets like ImageNet. 
clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: I have some concerns about 
1) fundamental difference between type I and II bridge models
2) Why number of bridge models impact 
3) generalization of this method to the large-scale datasets

I am willing to raise my score if the above concerns are addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes two methods to reduce the inference cost of Deep Ensemble (a collection of ensemble models of the same model structure). Bézier Curve is fitted on the extracted features by a neural network (bridge neural network), so that "interpolated" inference results can be cheaply approximated by the bridge NN.
strength_and_weaknesses: Strength
The work is well motivated and the two proposed bridge networks method are well crafted.

Weakness
1. there is a key alternative not compared, in Learning Neural Network Subspaces a method is proposed to train models that can be linearly interpolated. Though it is not easy to compare the approaches, one is altering the training process and the other is more about adding an ancillary equipment, the two approaches can be evaluated in the same coordinate system with inference cost and accuracy on two axes.
1. the method has not been verified on large scale datasets, though it criticized literature with "these methods do not scale well for complex large-scale datasets"
clarity,_quality,_novelty_and_reproducibility: Proper illustrations has been given for the effects of the method. It would be appreciated if heatmap (loss landscape) can be given to facilitated comparisons in this line of work.
summary_of_the_review: The paper is well motivated and easy to follow, but still lacks solid grounding for its claims.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new ensembling framework to reduce inference cost and boost the performance. The proposed method build additional light-weight head as bridges to ensemble different runs. The experiments show the effectiveness of the proposed method on CIFAR-10, CIFAR-100, and tiny-ImageNet.
strength_and_weaknesses: Strengths:
(1)	The idea of building bridges between different models for ensembling is interesting. It re-uses the features and only focuses on classification heads. Therefore, it would reduce the inference cost compared with directly ensembling different models.
(2)	The experiments on several datasets show the effectiveness of the proposed framework.


Weaknesses:
(1)	I concern only considering classification would not lead to diverse solutions to ensembling. As described in the paper, the bridges are light-weight MLPs.
(2)	The paper pointed out “these methods (the existing methods) do not scale well for complex large-scale datasets or require network capacity”. However, this paper also DO NOT scale up on complex large-scale datasets, such as ImageNet-1k. Since the experimental platform is on 8 TPUs, it is expected to conduct on large-scale datasets such as ImageNet-1K, large-scale backbones (or SOTA backbones) such as transformers. The experiments on CIFAR-10, CIFAR-100 or tiny ImageNet are not convincing to me.
(3)	Minor comments, Page 1: the modes, three models, two modes? Mode or models?


clarity,_quality,_novelty_and_reproducibility: The idea of the proposed deep ensembling is novel and the clarity and originality of the work are good. However, the experiments are conducted on small-scale datasets, which is not convincing to the community. 
summary_of_the_review: (1)The idea of the proposed deep ensembling is novel and interesting.
(2) The experiments on CIFAR-10, CIFAR-100, and tiny-ImageNet shows the improvement, but do not scale up to large-scale datasets and large-scale backbones (or SOTA backbones) .
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new way of using an ensemble of neural networks to construct a new classifier that achieves better accuracy at a lower number of FLOPS (at inference time). This is achieved by creating so-called "bridge-classifiers" that approximate the output of an interpolating classifier between pairs of ensemble-base classifiers. The interpolating classifiers are picked from the low-loss Bezier curve in parameter space between the pair of ensemble members. The approximation classifiers are faster than computing more ensemble members because they are smaller classifiers trained on intermediate features that have already been computed in the ensemble. The paper provides experiments on (small) image datasets (CIFAR and Tiny-ImageNet) and (small) classifiers of the ResNet family. 
strength_and_weaknesses: **Strengths**
* The paper proposes a novel idea for approximating gains of ensembles at lower computational cost (at inference time).
* The presented results are interesting and show that the presented method works (for small data set and classifier sizes).
* The paper is generally easy to follow. 

**Weaknesses**
* The method is relatively complex in relation to the performance gains that can be achieved.
* A direct comparison to other methods in terms of the accuracy/computation frontier is missing. It seems that the numbers are reasonable, but e.g. the range of 63% to 67% on Tiny-INet (Table 4) is in the area that can be also reached by a single ResNet-18 according to [Papers with Code](https://paperswithcode.com/sota/image-classification-on-tiny-imagenet-1). Also, there are no direct comparisons to other "efficient ensemble" approaches.
* FLOPS alone are often not a sufficient measure of model efficiency, see e.g. [The Efficiency Misnomer](https://arxiv.org/pdf/2110.12894.pdf)

Further comments:

It was not clear to me after reading the paper at which point in the base networks the z_{i} are extracted. I think this should be clarified (or maybe I missed it?)

I did not see an explanation for the term "Bridge-S" in the paper. (Maybe I missed it?) I guess it refers to a "small" Bridge model, but I did not see what exact configuration "Bridge" and "Bridge-S" refer to.

I found the discussion of knowledge distillation a bit distracting from the main idea. Especially the experimental setting described in the last paragraph of Sec 5.2 has a form that is not super-convincing in my opinion.

It seems to me that the most interesting setting among those discussed is the DE-1 + n bridges. In that case it would actually be possible to try to approximate the r=1 responses of the ensemble (i.e. approximate the outputs of the other ensemble members from the intermediate feature). If this worked similarly well, the whole construction of the Bezier interpolation could be avoided. I think this might be an interesting ablation experiment, but maybe not feasible to perform during the rebuttal.

Overall, I found Sec.4 rather short, given that this area of research is currently quite active. For example, the following papers could be considered relevant here. (I'm not requesting that all of these should be cited, just to give a few examples).
* [Batch Ensembles](https://arxiv.org/pdf/2002.06715.pdf)
* [Ensembles with Shared Representations](https://arxiv.org/pdf/2103.03934.pdf)
* [Evaluating Scalable BDL](https://arxiv.org/pdf/1906.01620.pdf)

Minor points or typos not affecting the evaluation:
* It seems that Eq. (3) and (1) can be combined into a shorter version of the two equations.
* p.3 line ~5: parameters ... achieves -> achieve
* p.3 line ~5: add them to [the/an] ensemble
* p.3 line ~8: "these strategy provide" -> this strategy provides
* p.4 Eq. (9): last "i" should be a "j"
* Algorithm 1 looks like a fairly standard training loop, unless I overlooked something. Maybe removing it would enable moving parts of the Appendix into the main paper. 
* p.4 bottom: "when the bridge network does not step toward the Bezier predictions" - I did not fully understand what this means.
* Table 2: The FLOPS are given as a multiple. I assume this is as a multiple of a base network? 
* p.7 ~center: "achieve decent R^2 score" -> "a score" or "scores"
clarity,_quality,_novelty_and_reproducibility: Clarity: High; the paper is overall well-written and easy to understand.

Quality: See strengths and weaknesses above.

Novelty: OK; the main idea is novel and interesting, but it is a rather complex approach for which it is not clear to me whether it would be widely applicable in practice among the large number of approaches that aim to improve the accuracy/efficiency ratio.

Reproducibility: High; the authors provide the training code and configurations used.
summary_of_the_review: The paper contains interesting approaches and results. However, there are also some weaknesses and limitations. I therefore think that the paper in its current form is marginally below the acceptance threshold, but I would be happy to increase my score during the discussion period if some of the concerns can be addressed by the authors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper looks at the problem of efficient test-time prediction using deep ensembles. Traditional deep ensembles require multiple forward passes (one for each member of the ensemble). This paper presents a method that leverages the idea of mode connectivity and designs a "bridge network" which enables computing the prediction at any intermediate point on the path connecting two modes. Notably, computing the predictions doesn't require doing forward passes on the original networks.

The reviewers appreciated the idea. However, there remained several concerns related to the experiments, such as the complexity of the method not commensurate w.r.t. the performance gains achieved. It also remained unclear how the proposed method is positioned as compared to various other existing methods that have tried to address the accuracy vs efficiency trade-off.

Another aspect that the paper misses is comparison with recent methods that distill a deep ensemble into a single network. The paper talks about the basic knowledge distillation (KD) idea but there is a significant amount of recent work on KD for deep ensembles, such as "Ensemble Distribution Distillation" (Malinin et al, 2022) and follow-up works. The paper would be strengthened by a discussion of (and ideally a comparison with) these methods.

The paper was discussed but in the end, it was felt by all the reviewers that, in the current form, the paper does not meet the acceptance bar. The authors are advised to take into account the feedback from the reviewer, which we hope will strengthen the paper, and resubmit to another venue.
justification_for_why_not_higher_score: Reasons mentioned in the meta-review.
Decision: Reject