Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates performance of different pretrained models on domain generalization and shows that there is no single best pretrained model that generalizes across all distribution shifts. Based on the finds, the authors propose a new method to learn to dispatch the best matched models for each out-of-distribution sample and predict with ensemble of the models. The method does not need to fine-tune the pretrained models and achieves obvious performance improvements.
strength_and_weaknesses: [+] Instead of fine-tuning pretrained models on source domains, the paper proposes to learn to dispatch pretrained models by matching the models to each test sample, which is efficient and interesting.

[+] The proposed method achieves significant performance improvement compared with the other state-of-the-arts. The ablation studies also demonstrate the effeciveness of the method.

[-] The method tends to select the best matching pretrained models for domain generalization. However, it is not sure whether method works because it handles the domain shifts or due to the strong capability of the pretrained models. For instance, the CLIP model in Model Pool B is trained on a large dataset, which contains a huge amount of different domains. 

[-] Since the method requires to process the inputs by all models in the Model Pool, there should be a big computations and memory usage cost in both training and test stages. While it is shown in Table 2 that the method achieves low FLOPs as shown. Can the authors explain this? The GPU memory usage is also important for reproducing the method.

[-] The method trains a shared label space adapter using Model Pool A. However, it is not clear how to train the adapter with Model Pool B, which contains pretrained models with different sizes of output spaces. If different adapters are used for different models, will the training procedure be influenced by the ensemble weights? For example, some of the adapters will not be trained well since the weights are small.

[-] The method learns model embeddings for the model-sample matching, but there is no information of the models considered in the embeddings. How can we make sure the embeddings related to different models?

clarity,_quality,_novelty_and_reproducibility: Clarity: Most parts of the paper is clear and easy to follow except for some details of the method. See the weakness above.

Quality: The paper investigates sufficient pretrained models for domain generalization, which provide good insight and motivation for the proposed method. But I still have some concerns about the method as in the weakness above.

Novelty: The paper is somewhat novel in that it learns sample specific ensemble of the pretrained models to handle distribution shifts.

Reproducibility: The paper provides enough details for reproduction. But run such many of pretrained models is not easy.

summary_of_the_review: The paper provides an interesting investigation of different pretrained models on domain generalization, which well motivates to learn to dispatch different models for different target samples. The idea is interesting, but there are still some unclear parts in the methods and experiments that need to be clarified as mentioned in the weaknesses. I would consider to raise my score if the authors could well explain or address the weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper investigates the generalization performance of pretrained models on domain generalization or data distribution shift problems, leading to the ''no free lunch hypothesis'' of pre-trained models in domain generalization settings.  The authors then propose a new DG method that aims to directly leverages pretrained models without fine-tuning. The experimental results on the mainstream benchmark show that the proposed method outperforms other strong baselines. 
strength_and_weaknesses: [Strengths]: The proposed method is relatively simple and easy to implement. The experimental results are convincing and partly support the main claims of the paper.

[Weakness]: My major concern is about the theoretical contribution of the paper. I did not have time to verify the results in Theorem 1, but even if they are all correct, I could not find the role of pre-trained models in the generalization error in a DG. Furthermore, the idea of (pre-trained) model-sample matching in the proposed method may be impractical since in DG we are not allowed to have access to the samples in the target domain.
clarity,_quality,_novelty_and_reproducibility: Clarity:

1. The term $a$ is undefined in that theorem. 
2. How to estimate $E^{matched}_g$, since in a DG framework we do not have any access to the target domain?
3. How many pre-trained models are enough to run the algorithm? How to optimize that quantity?

Quality: Though the experimental results of the paper are quite impressive, I think that the technical contribution of the paper is limited based on my comments about its weaknesses. 


Novelty and reproducibility: The idea of efficiently using pre-trained models in DG is quite novel but maybe not very practical due to the computational cost. Honestly, I did not have time to verify the code, but I think the experimental results of the paper are reproducible. 
summary_of_the_review: Overall, I personally think that this submission is below the acceptance rate for an ICLR paper. I would suggest the authors focus more on the theoretical part to make it stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes to leverage a pool of pretrained models without expensive fine-tuning. They empirically demonstrate the proposed method achieves SOTA results on several benchmarks without significant loss in inference speed. 

strength_and_weaknesses: **Strengths:**

The authors propose a creative way how to use a pool of pretrained models to achieve better generalization on distribution shifts. They demonstrate SOTA results, which significantly outperform previous models. They demonstrate the necessity of parts of their algorithm via an ablation study.

**Weaknesses and comments:**

 - The writing is a bit confusing. First, the authors say that they *“adopt a simple la- bel adapter that projects the label space of the pretrained domain to that of the target domain”*, but then they say *“with the adapter training on source domains”*. After reading several times, I understand what they meant in the beginning, but upon first reading, it was very confusing. The authors should stick to the strict terminology that they introduce in the very beginning, e.g. “pre-training domain” (the one that corresponds to ImageNet) and then “source” and “target” domains from a typical domain generalization literature. 
- No free lunch in domain generalization was demonstrated in several works before, e.g. in the DomainBed paper that the authors cite. So Takeaway 2 is an extended version of those results, but with varying architectures. It is even less surprising, given the fact that the authors are not trying to address distribution shift at all there.
- For me, it is not obvious why Theorem 1 would explicitly generalize to neural networks of fixed size.
- Also by just tuning the label on the source domain (not the same as image-net pretraining), it is not obvious that the model can even achieve minimal test risk and if all theoretical justifications would be true. An arbitrary model trained under ERM can exhibit quite arbitrary behavior under distribution shift, as known from many previous theoretical results. By only allowing to tune the last layer, the authors put quite a strong constraint on the class of models under which optimal risk can be achieved, therefore such models are not guaranteed to generalize. 
- Remark 3 is also not clear to me. Any causal model would achieve optimal risk under the assumption of covariate shift. 
-  How *“fitness of the pretrained models to the target distributions”* is evaluated?
- *“Table 1. Baseline results are from original papers with the same setup”*. Does it mean that the method’s performance for benchmarks is demonstrated on “one” model (originally proposed in the corresponding paper). So it is possible that the best architecture which would allow achieving a higher number in this table might have not been considered? My concern is that majority of the papers were focused on demonstrating the effect of the *learning method given a fixed architecture* and that with another architecture, the same learning method would achieve another number. So comparing it to the pool of > 200 models is not entirely fair. 
- Ablation study RQ2: the averaging strategy indeed looks not optimal. But why the “averaging” weights can’t be learned by backpropagation? 

clarity,_quality,_novelty_and_reproducibility: The paper is novel (to my knowledge) and original (except theorems), but not entirely clear. I think the authors should invest in more clear writing in the paper because some statements are very confusing. 
summary_of_the_review: From the way the paper is written, many aspects are not clear to me, which I stated in the section above. Overall the paper looks quite interesting and the empirical evaluation of the paper looks significant. However, I don’t know how to properly evaluate this paper when there are so many unclear things in it. In particular, the theoretical part and big parts of the paper are very confusing, as well as the model description. I am willing to change my evaluation when my questions are clarified.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Domain generalization is a challenging problem as target data is not concurrently available along with source data. One approach is to have a set of pre-trained source models and develop a strategy for domain generalization. A paper by Balaji et al., (NeurIPS 2018) suggested the approach of using a set of pre-trained models and set the problem of domain generalization as a regularization problem. This paper has taken a similar approach but assumes the availability of a large number of pre-trained models. The domain generalization problem is then solved y dispatching the pre-trained models to OOD samples based on their matching metric to the target task. Extensive evaluations show that the proposed approach has better generalization performance and significantly higher training efficiency compared to existing DG methods.
strength_and_weaknesses: Strengths:  An approach supported by theoretical analysis and empirical evidence based on extensive evaluations for domain generalization that assumes the availability of a large number of pre-trained source models. The success of domain generalization depends on how close the one or more pre-trained models are in terms of addressing the domain shift. Extensive experimental results and ablation studies are included.
Weaknesses: This approach  reminds me of a NeurIPS paper by Y. Balaji, S. Sankaranarayanan and R. Chellappa, “MetaReg: Towards Domain Generalization Using Meta-regularization”, Proc. Neural and Information Processing Systems, Montreal, Dec. 2018. There is a brute-force philosophy embedded in the approach. Even more performance gains can be achieved if one can assume the availability of 500 or 1000 or 10,000 models! 
clarity,_quality,_novelty_and_reproducibility: The approach of using a set of pre-trained models for domain generalization has been suggested before. See Balaji, et al, NeurIPS 2018. The difference, which I think is important, is how many pre-trained models are assumed to be available. It will be good to come up with a strategy for retaining the most useful pre-trained models.
The paper has ICLR quality, in that the theoretical basis is strong.
Reproducibility will be challenging given the number of pre-trained models used.
summary_of_the_review: Domain generalization is harder than domain adaptation. The proposed approach of leveraging a large set of pre-trained models to help with generalization is a good idea. One can see this approach as an extension of nearest neighbor rule, in which the more training data you have, the better the classification performance is. In the proposed approach, each model can be viewed as a training sample. This makes me wonder if some asymptotic performance results can be derived as is done with nearest neighbor rules. Discuss what happens if you assume a very very large number of pre-trained models.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper address the challenging domain generalization which the data in target domain is not available. Authors have proposed a method to leverage a large number of pre-trained models to help with generalization, and solve y dispatching the pre-trained models to OOD samples based on their matching metric to the target task. 

The method has been supported by theoretical analysis although theoretical contribution is not so significant. 

On the other hand, it has conducted extensive empirical evaluations which demonstrated that the proposed method has better generalization performance and significantly higher training efficiency compared to existing  methods.
justification_for_why_not_higher_score: Theoretical contribution is not so significant, although they provide theoretical analysis
justification_for_why_not_lower_score: It has proposed decent theoretical analysis and comprehensive empirical evaluations.
Decision: Accept: poster