Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the problem of increasing performance on out of domain (OOD) classification, i.e. generalization/robustness on data distributions different from the training dataset.  In particular, this work focuses on the setting where there are several pre-trained models (PTMs) available.  The goal of the paper is then to use these PTMs in some efficient way to realize high OOD generalization.  The proposed approach breaks this into two parts: (1) ranking the PTMs by predicted generalization OOD, and (2) ensembling them.  The authors propose new approaches for both parts, the first using a ranking metric that uses inter-class discriminative capacity and inter-domain stability computed in a leave-one-out x-validation, and (2) using a feature sub-selection method accomplished using an approximate inference EM approach.  The authors rigorously compare and ablate the proposed approach and show strong empirical gains.
strengths_and_weaknesses: Strengths:
- (S1) Interesting and relevant topic: OOD generalization and robustness is always a critical topic, with growing resources and momentum recently; and increasingly there are many pre-trained models publicly accessible which together makes the setting very relevant.
- (S2) Strong empirical results with thorough methodological ablations: The proposed pipeline is somewhat complex but the component pieces and design choices are thoroughly evaluated overall, with very strong empirical results

Weaknesses:
- (W1) Minimal detail in main body: There is mention of a formal analysis, and there are several core algorithms that are fully relegated to the appendix; it would seem appropriate and possible to include more in the main body.
- (W2) Ablation/examination of pre-trained effects: The authors show strong absolute performance against prior work but don't go into detail on how differences in which pre-trained models were used may have influenced; i.e., are the effects of different/better pre-trained models properly controlled for?
- (W3) The benefits of ensembling over just using the best pre-trained model selected seem fairly minimal
questions: N/A
limitations: N/A
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Naive fine-tuning requires high computational cost for choosing the best performing model on the target task, and the best performing model may not perform well on unseen domains. A simple model also cannot fully utilize the knowledge presented in different PTMs. Motivated by such observations, the paper proposed a ranking metric to inspect the generalization ability of the PTMs on unseen data through quantifying the inter-class discriminability and inter-domain stability. They also propose an EM approach to select the informative features and discard the noisy features. Experimental results show that the proposed approach outperforms the previous SOTA methods with high efficiency.
strengths_and_weaknesses: Strength
1. The paper is well written and easy to follow.
2. The main contributions of the paper consist of a ranking metric of inspecting the generalization-ability of PTMs and a feature selection approach.
3. The paper conducted extensive experiments (35 PTMs and 7 datasets), and the experimental results outperforms the previous best approaches.


Weakness
1. Regarding the experiment results shown in Figure 2, the proposed ZooD approach does not show much better performance when compared to LogME, except on Terralncognita. However, the ZooD requires a linear model with Gaussian error for evaluation. 
2. For the experiment results in Table 2, the experiment result of the Single model outperforms the previous SOTA by a large margin. However, the Single model here indicates a fine-tuning model that is selected from the PTMs.  I am curious why such a baseline is not included in the table. It seems unfair to compare a selected top-1models from a zoo of 35 models with a single model on the 5 datasets. 
3. Regarding the ensemble results in Table 2, do you have the performance comparisons of the Ensemble approach with the LogME approach? 
4. Regarding the computational efficiency, the paper compared the proposed ZooD with brute-force finetuning. When comparing with a simple ensemble, such as LogME, does ZooD still have advantages?

questions: In Line 192, do you refer to the wrong equation?
In Table 2, Feature selection contributes more to the performance improvement compared to model Ensemble. What if the feature selection approach is directly applied on the single model? 

Line 48, there should be a full stop before “Given…’.

limitations: NA
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a pre-trained model ranking metric and an ensemble method to improve the generalization on target out-of-distribution tasks. The proposed metric evaluates inter-class discrimination and inter-domain stability in a cross-domain validation manner across a set of domains, so that the model would not fixate on domain-specific features. Besides, authors propose a variational EM algorithm to reduce noise of ensembled features. Extensive experiments of 35 PTMs on six datasets show the effectiveness of the method.
strengths_and_weaknesses: Strengths:
(1) The proposed method is technically solid and reasonable.
(2) Experiments are comprehensive and convincing.
(3) The proposed method significantly outperforms SOTA baselines on most datasets.

Weaknesses:
(1) The complexity of computing proposed metric on m domains is O(m^2), which will be inefficient when there are a large number of domains. Although it is still much faster than brute-force finetuning, how does it compare with other non-finetuning methods?
(2) Authors better put the results of finetuning the whole network in table 2 to improve readability.
(3) It would be better if authors report the finetuning results of the PTM selected by the IID ranking baselines (H-Score, LogME, LEEP and NCE) to better compare the ranking methods.

questions: (1) Why IID ranking baselines perform so bad on TerraIncognita? Is fine-tuning top linear classifiers a good metric on this dataset?
(2) ZooD uses a large set of PTMs including the newly proposed Transformer-based models, while the baselines (e.g., Gulrajani and Lopez-Paz [18]) use ResNet-50. My main concern is that the comparisons may be unfair due to differences in the performance of different architectures. So, do the different network architectures affect the results?

limitations: yes
review_title: null

Meta-review: metareview: The paper proposes a new method for leveraging a zoo of pre-trained models for improving OOD generalization in an efficient way. The reviewers agree that the results are strong.
confidence: Certain
Decision: Accept