- Our method requires fully knowing the training distributions of all the evaluated models, which is not directly applicable for large-scale pre-trained models with private training data. And this also requires the training methods not to significantly alter the training distribution beyond basic data augmentation, while some methods such as SLIP may alter training distributions more significantly, and it is unclear how we can precisely define the training distribution for a model with post-training processing, such as WiSE-FT (Wortsman et al., 2022b) and Model Soups (Wortsman et al., 2022a) with weight-space ensembling. Future work may study how these techniques may impact the ID performance evaluation (Section 5.4).
- We assume that the two ID test sets have relatively close distributions compare to the OOD test sets. We have not considered how the difference between the multiple ID test sets may affect the evaluation, and how the effective robustness should be compared if models are trained on highly different distributions.
- We use fixed OOD test sets to evaluate the OOD performance, following previous works (Radford et al., 2021; Fang et al., 2022; Nguyen et al., 2022; Schuhmann et al., 2021). When models are pre-trained on large-scale data, it becomes unclear if these "OOD" test sets are still OOD, or if these test sets could be less OOD for the pre-trained models compared to standard classifiers. There may also be some distribution overlap between these test sets and the pre-training datasets, even though Radford et al. (2021) mentioned that the likelihood of direct data overlapping is low.
- We focus on distribution shifts where at least "accuracy-on-the-line" from existing works is known to hold for models trained on the same data (Taori et al., 2020; Miller et al., 2021), yet there are counterexamples where "accuracy-on-the-line" does not hold (Section 6) and requires further study. We may set a threshold on the fitting quality and only adopt our method when the fitting quality is sufficiently good. And there are also other OOD test sets (Singla & Feizi, 2021; Rusak et al., 2021; Singla et al., 2022; Idrissi et al., 2022; Li et al., 2023; Moayeri et al., 2022; Vasudevan et al., 2022) that have not been studied in the effective robustness works yet.
- While we mostly focus on comparing CLIP-like models with standard image classifiers, due to the notable OOD robustness of CLIP-like models studied in prior works (Radford et al., 2021; Fang et al., 2022; Nguyen et al., 2022), this work may further be extended to cover other types of models (Goyal et al., 2022; Singh et al., 2022) as well as other modalities such as distribution shifts on language data (Miller et al., 2020; Awadalla et al., 2022).
- Our multi-ID evaluation is intended for the scenario with models trained on different data. For models trained on a single dataset, while there is often a correlation between the rankings derived from the single-ID evaluation and the multi-ID evaluation, respectively, the rankings are not necessarily consistent (see Appendix A.2), and thus our multi-ID evaluation is not intended to replace the single-ID evaluation in this case. We suggest using single-ID and multi-ID evaluation comprehensively.