Title: Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights
Venue: NeurIPS 2024
Accepted: yes
Keywords: Vision Transformer

Neural Architecture Search

Out-of-Distribution Generalization
Abstract: While Vision Transformer (ViT) have achieved success across various machine learning tasks, deploying them in real-world scenarios faces a critical challenge: generalizing under Out-of-Distribution (OoD) shifts. A crucial research gap remains in understanding how to design ViT architectures – both manually and automatically – to excel in OoD generalization. **To address this gap,** we introduce OoD-ViT-NAS, the first systematic benchmark for ViT Neural Architecture Search (NAS) focused on OoD generalization. This comprehensive benchmark includes 3,000 ViT architectures of varying model computational budgets evaluated on common large-scale OoD datasets. With this comprehensive benchmark at hand, we analyze the factors that contribute to the OoD generalization of ViT architecture. Our analysis uncovers several key insights. Firstly, we show that ViT architecture designs have a considerable impact on OoD generalization. Secondly, we observe that In-Distribution (ID) accuracy might not be a very good indicator of OoD accuracy. This underscores the risk that ViT architectures optimized for ID accuracy might not perform well under OoD shifts. Thirdly, we conduct the first study to explore NAS for ViT’s OoD robustness. Specifically, we study 9 Training-free NAS for their OoD generalization performance on our benchmark. We observe that existing Training-free NAS are largely ineffective in predicting OoD accuracy despite their effectiveness at predicting ID accuracy. Moreover, simple proxies like #Param or #Flop surprisingly outperform more complex Training-free NAS in predicting ViTs OoD accuracy. Finally, we study how ViT architectural attributes impact OoD generalization. We discover that increasing embedding dimensions of a ViT architecture generally can improve the OoD generalization. We show that ViT architectures in our benchmark exhibit a wide range of OoD accuracy, with up to 11.85% for some OoD shift, prompting the importance to study ViT architecture design for OoD. We firmly believe that our OoD-ViT-NAS benchmark and our analysis can catalyze and streamline important research on understanding how ViT architecture designs influence OoD generalization. **Our OoD-NAS-ViT benchmark and code are available at [https://hosytuyen.github.io/projects/OoD-ViT-NAS](https://hosytuyen.github.io/projects/OoD-ViT-NAS)**

Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights

Abstract

While Vision Transformer (ViT) have achieved success across various machine learning tasks, deploying them in real-world scenarios faces a critical challenge: generalizing under Out-of-Distribution (OoD) shifts. A crucial research gap remains in understanding how to design ViT architectures - both manually and automatically - to excel in OoD generalization. To address this gap , we introduce OoD-ViT-NAS, the first systematic benchmark for ViT Neural Architecture Search (NAS) focused on OoD generalization. This comprehensive benchmark includes 3 , 000 ViT architectures of varying model computational budgets evaluated on 8 common large-scale OoD datasets. With this comprehensive benchmark at hand, we analyze the factors that contribute to the OoD generalization of ViT architecture. Our analysis uncovers several key insights. Firstly, we show that ViT architecture designs have a considerable impact on OoD generalization. Secondly, we observe that In-Distribution (ID) accuracy might not be a very good indicator of OoD accuracy. This underscores the risk that ViT architectures optimized for ID accuracy might not perform well under OoD shifts. Thirdly, we conduct the first study to explore NAS for ViT's OoD robustness. Specifically, we study 9 Training-free NAS for their OoD generalization performance on our benchmark. We observe that existing Training-free NAS are largely ineffective in predicting OoD accuracy despite their effectiveness at predicting ID accuracy. Moreover, simple proxies like #Param or #Flop surprisingly outperform more complex Training-free NAS in predicting ViTs OoD accuracy. Finally, we study how ViT architectural attributes impact OoD generalization. We discover that increasing embedding dimensions of a ViT architecture generally can improve the OoD generalization. We show that ViT architectures in our benchmark exhibit a wide range of OoD accuracy, with up to 11 . 85% for some OoD shift, prompting the importance to study ViT architecture design for OoD. We firmly believe that our OoD-ViT-NAS benchmark and our analysis can catalyze and streamline important research on understanding how ViT architecture designs influence OoD generalization. Our OoD-NAS-ViT benchmark and code are available at https://hosytuyen.github.io/projects/OoD-ViT-NAS

1 Introduction

Vision Transformers (ViT) [1] have recently achieved impressive results and become a major area of research in computer vision, with significant efforts towards understanding how ViT works. These efforts have led to the proposal of both manually designed architectures [1, 2, 3, 4, 5] or automated-searched architectures [6, 7, 8, 9, 10, 11, 12] to advance ViT architectures.

(a)IDasindicatorfor

(c)OoDGeneralization

ViTOoDGeneralization

(b)Training-freeNASforViTOoDGeneralization

ViTArchitectureAttributes

- 1.00

IN-C

0.33

0.27

0.27

0.12

0.04

0.24

0.24

0.09

0.24

0.22

0.37

-0.0

0.11

0.08

0.75

IN-D

-0.04

-0.08

-0.11

-0.03

-0.05

-0.08

-0.09

-0.01

-0.07

-0.07

-0.12

-0.12

-0.01

0.02

IN-P

0.67

0.45

0.46

0.14

0.13

0.37

0.36

0.11

0.41

0.4

0.45

0.22

0.17

0.06

0.50

IN-O

0.08

0.59

0.6

0.17

0.25

0.46

0.46

0.1

0.59

0.6

0.63

0.25

0.03

0.09

0.25

IN-A

0.4

0.48

0.45

0.14

0.12

0.38

0.36

0.09

0.44

0.42

0.58

0.66

0.04

0.09

0.00

IN-R

0.45

0.43

0.41

0.14

0.09

0.36

0.34

0.09

0.39

0.36

0.57

0.27

0.13

0.01

=0.25

IN-Sketch

0.37

0.39

0.39

0.14

0.09

0.31

0.32

0.1

0.38

0.36

0.59

0.17

0.12

0.07

- 0.50

Stylized-IN

0.49

0.36

0.34

0.15

0.1

0.33

0.32

0.09

0.35

0.35

0.51

0.09

0.09

0.12

- 0.75

AverageOoD

0.34

0.36

0.35

0.12

0.10

0.30

0.29

0.08

0.34

0.33

0.45

0.19

0.09

0.07

1.00

ID ACC

#Param#Flops Grasp

CroZe

SNIP JacobianDSS AutoProx

EmbedDepth

MLP

#Head

Dim

Ratio

Research Gap. Existing research on ViT architectures focuses on maximizing In-Distribution (ID) accuracy, while studies on the impact of ViT architectures on Out-of-Distribution (OoD) generalization are limited. Initial works [13], [14], and [15] evaluate sets of 3 , 10 and 22 human-designed ViT architectures under OoD settings, respectively, and provide coarse insights into which models exhibit better OoD generalization. However, with very limited ViT architectures studied in previous works, the influence of ViT structural attributes (e.g., embedding dimension, number of heads, MLP ratio, number of layers) on OoD generalization remains unclear. Besides, in the context of ViT Neural Architecture Search (NAS), while there are various ViT NAS for ID accuracy [6, 7, 16, 8, 10, 11, 17, 18, 19, 20, 9], there is no study on ViT NAS for OoD generalization.

In this paper, we address existing research gaps by introducing OoD-ViT-NAS, the first comprehensive benchmark specifically designed for ViT's OoD generalization. Building NAS benchmarks is notoriously time-consuming and expensive due to the need to train and evaluate every candidate architecture. This challenge is particularly acute for ViT, known for its high computational demands and memory usage [21, 22, 23]. To overcome this bottleneck, we propose leveraging One-Shot NAS, specifically AutoFormer [6], a widely used ViT search space.We sample a diverse set of sub-architectures (models) to populate our benchmark. Importantly, these subnets inherit the weights from the pre-trained supernets, and their performance has been shown to be comparable to, or even superior to, that of architectures trained alone [20, 6] This approach enables us to efficiently acquire a large pool of ViT architectures for OoD generalization analysis. Using OoD-ViT-NAS, we conduct extensive OoD generalization analysis and gain several key insights. Additionally, with our benchmark, our work is the first to explore (training-free) NAS for ViT's OoD generalization. Our contributions are summarized below:

We introduce OoD-ViT-NAS, the first comprehensive benchmark designed for NAS research on ViT's OOD generalization. This benchmark includes 3 , 000 diverse ViT architectures sampled from the widely used ViT search space [6]. These architectures span a wide range of computational budgets. To thoroughly benchmark OoD generalization, these architectures are evaluated on the 8 most common and state-of-the-art (SOTA) OoD datasets: ImageNet-C [24], ImageNet-A [25], ImageNet-O [25], ImageNet-P [24], ImageNet-D [26], ImageNet-R [27], ImageNet-Sketch [28], and Stylized ImageNet [29] (Sec. 3)

Our analysis demonstrates the significant influence of ViT architectural designs on OoD accuracy. This observation encourages future research to focus more on ViT architecture research for OoD generalization (Sec. 4.1)

We show that high In-Distribution (ID) accuracy is not a very good indicator of OoD accuracy. This suggests that current architectural insights based on ID accuracy might not translate well to OoD generalization (Sec. 4.2)

We conduct the first study to explore NAS for ViT's OoD generalization. We study 9 Training-free NAS for their OOD generalization performance on our benchmark. We observe that despite their prediction accuracy for ID, their effectiveness significantly weakens when predicting OoD accuracy. Furthermore, simple proxies such as the number of parameters (#Param) or the number of floating point operations (#Flop) surprisingly outperform more complex Training-free NAS in predicting ViTs OoD accuracy (Sec. 4.3)

We study the impact of ViT architecture design on OoD generalization and demonstrate that careful design of ViT architectures can significantly improve OoD generalization. Specifically, increasing the embedding dimensions of a ViT architecture generally can improve its OoD generalization. (Sec. 4.4) We show that architectures with comparable ID accuracy (within an averaging range of 1 . 39% ) exhibit a wider range of OoD accuracy, averaging 3 . 80% and reaching highs of 11 . 85% , being comparable or even outperforming state-of-the-art (SOTA) training OoD generalization method, such as those based on domain invariant representation learning [30, 31]. For example, under the same OoD setting, the SOTA method [30] shows an improvement of 1 . 9% OoD accuracy.

2 Related Work

Out-of-Distribution (OoD) Generalization . Addressing Out-of-distribution (OoD) generalization is a challenge, particularly in computer vision. Various approaches have been proposed to tackle this issue. A common strategy focuses on learning features that remain consistent across different domains, thereby promoting generalizability [31, 32, 33, 34, 35, 36, 37]. Other directions explore distributional robustness [38, 39], model ensembles [40, 41], test-time adaptation [42, 43], data augmentation techniques [44, 45, 46, 47, 48, 49, 50], and meta learning [51, 52] for OoD generalization. From an architectural perspective, a few attempts investigate the impact of network architecture on OoD generalization. Early work [53] shows that over-parameterized networks can hinder OoD performance due to overfitting. This raises an intriguing question: can sub-networks within such architectures achieve better OoD performance? Inspired by the Lottery Ticket Hypothesis (LTH) [54], the Functional LTH has been explored and shown that over-parameterized networks harbor sub-networks with better OoD performance. Techniques like Modular Risk Minimization [55] and Debiased Contrastive Weight Pruning [56] aim to identify these winning tickets. Another direction [57, 14] leverages Neural Architecture Search (NAS) to analyze the OoD robust architectures. However, these studies primarily focus on CNNs. While ViTs have achieved success in various visual recognition, investigations into their OoD generalization are limited. Initial works [13], [14], and [15] evaluate sets of 3 , 10 , and 22 human-designed ViT architectures respectively, under OoD settings. Their results provide coarse insights into which models exhibit better OoD performance. However, the influence of ViT architectural attributes on OoD robustness remains unclear.

Neural Architecture Search (NAS) . NAS is a promising approach that has achieved remarkable success in automatically searching efficient and effective architectures for ID performance [58, 20, 59, 60]. Recently, NAS has been explored in the context of adversarial robustness for CNNs as well [61, 62, 63]. With the rise of Vision Transformers (ViTs), several NAS approaches have been applied to improve ViT architectures, including Autoformer [6], S3 [16], ViTAS [8], ElasticViT [9], DSS [10], Auto-Prox [17] and GLiT [64]. Additionally, hybrid CNN-ViT architectures like HR-NAS [18], UniNet [19], and NASViT [12] have also been explored. These efforts have shown promising results in terms of ID accuracy. However, there has not been any work on NAS for ViT architectures specifically for OoD generalization.

3 OoD-ViT-NAS: NAS Benchmark for ViT's OoD Generalization

In this section, we describe the construction of our OoD-ViT-NAS benchmark with details on the search spaces, datasets, evaluation metrics, and protocol. Our comprehensive benchmark includes

1.06

3.32(1.06)

3.98(1.06)

3.47(1.06)

2.25

7.32(2.25)

7.58(2.25)

11.85(2.25)

Base Small

0:56

3.35(0.56)

4.94(0.56)

2.16(0.56)

70

80

20

30

40

50

20

30

40

50

30

40

50

ID ACC (%)

IN-C Gaussian Noise Acc.

IN-C Impulse Noise Acc.

IN-C Pixelate Acc.

8.89(1.06)

2.93(1.06)

1.39(1.06)

1.48(1.06)

u

8.00(2.25)

3.81(2.25)

3.35(2.25)

5.21(2.25)

Base Small

5.54(0.56)

4.88(0.56)

1.02(0.56)

1.96(0.56)

40

50

60

20

30

40

60

70

0

10

20

IN-C Fog Acc.

IN-C Motion Blur Acc.

IN-P Zoom Blur Acc.

IN-A ACC.

2.02(1.06)

2.10(1.06)

9.79(1.06)

4.95(1.06)

Au!

2.71(2.25)

3.99(2.25)

9.65(2.25)

5.55(2.25)

Base Small

1.07(0.56)

3.99(0.56)

7.99(0.56)

3.64(0.56)

20

30

20

30

10

20

10

20

IN-R ACC.

IN-Sketch Acc.

IN-D Texture Acc.

IN-D Background Acc.

3 , 000 ViT architectures of varying sizes evaluated on 8 widely used large-scale, high-resolution, and SOTA OoD datasets. Our OoD-ViT-NAS benchmark is summarized in the Tab. 1

Search Space. We construct our benchmark based on Autoformer [6] search space. This search space is currently a widely used search space in the ViT NAS community for ID data [7, 65, 60, 10, 11, 17, 66, 67]. Autoformer search space is a large vision transformer search space including five architectural attributes that define the building block. Embedding Dimension: This determines the input feature representation size and is typically consistent across layers in ViT architectures. Q-K-V Dimension: This specifies the size of the query, key, and value vectors used in the attention mechanism. Number of Heads: This defines the number of parallel attention computations performed within a single attention block. MLP Ratio: This controls the dimensionality of the feed-forward network within each transformer block. Unlike embedding dimension, in Autoformer search space, Q-K-V Dimension, Number of Heads, and MLP Ratio can be varied across layers. Network Depth: This refers to the total number of transformer layers stacked in the architecture. It is important to note that Autoformer maintains a fixed ratio between the Q-K-V dimension and the number of heads in each block. This ensures that the scaling factor in the attention calculation remains constant. This helps stabilize the gradients of different heads during the training [6]. We strictly follow Autoformer search space. The details can be found in the Appx. E.1.

Dataset. Our benchmark consists of the evaluation on large-scale, high-resolution, and most SOTA OoD datasets, including ImageNet-1k [68], ImageNet-C [24], ImageNet-P [24], ImageNet-A [25], ImageNet-O [25], ImageNet-R [27], ImageNet-Sketch [28], Stylized ImageNet [29], and ImageNet-D [26]. These datasets capture a comprehensive range of OoD shifts such as common corruptions (blur, noise, digital, weather), Stable-Diffusion-based OoD shifts, and natural OoD shifts. A detailed description of these datasets can be found in the Appx. E.2.

Metrics. Following the previous OoD generalization methods [57, 30, 31, 15], we employ three metrics to construct our benchmark:

ID Classification Accuracy (ID Acc) : This metric measures the model performance on In-Distribution (ID) data, typically the data it was trained on (e.g., ImageNet). A higher ID Acc indicates the model's ability to learn training data's distribution.

OoD Classification Accuracy (OoD Acc) : This metric measures the model performance on Out-of-Distribution (OoD) data, which could differ significantly from the training data. A higher OoD Acc indicates a better generalization of the model to handle the OoD shifts.

For the specific case of ImageNet-O, [24], we use the Area Under the Precision-Recall Curve (AUPR) metric. A higher AUPR indicates a better generalization of the model to handle the OoD detection.

Protocol. Neural Architecture Search (NAS) is notorious for its computationally expensive nature, requiring the training and evaluation of numerous candidate architectures. To address this challenge and efficiently obtain the large number of architectures needed for our benchmark (i.e., 3 , 000 ), we make use of the One-Shot NAS approach [58, 20, 59, 12, 6, 7].

In One-shot NAS, a single supernet is first constructed. This supernet contains all possible architectures within the defined search space and is trained only once. Then, during evaluation, various architectures (i.e., subnets) can be efficiently extracted from the supernet. Importantly, these subnets inherit the weights from the pre-trained supernet, and their performance has been shown to be comparable or even superior to that of architectures trained alone [20, 6].

To support a wide range of model sizes, we leverage three supernets: Autoformer-Tiny/Small/Base, which were previously proposed for ID accuracy [6]. We randomly sample 1 , 000 architectures from each supernet, resulting in a total of 3 , 000 architectures in our OoD-ViT-NAS benchmark. Once obtained, these architectures are evaluated on 8 aforementioned OoD datasets.

4 Investigation on Out-of-Distribution Generalization of ViT

In this section, we provide the first comprehensive investigation of how ViT architectures affect OoD generalization using our OoD-ViT-NAS benchmark. In Sec. 4.1, we first demonstrate that ViT architectures considerably impact OoD accuracy. In Sec. 4.2, while existing works [2, 3, 6, 7, 8, 9, 12, 18] have made significant strides in improving ViT's ID accuracy, their findings could not be applicable for ViT's OoD generalization due to the not very high correlation between ViT's ID and OoD accuracy. In Sec. 4.3, we conduct the first study to explore NAS for ViT's OoD generalization. Specifically, we study 9 Training-free NAS based on their OoD generalization performance on our benchmark. Finally, in Sec. 4.4, we analyze the influence of individual ViT architectural attributes (i.e., embedding dimension, number of heads, MLP ratio, number of layers) on OoD generalization. Additional results of these analysis can be found in Appx. G, H, I, J, L

4.1 ViT architecture designs have a considerable impact on OoD generalization

In this section, we highlight that ViT architectures considerably impact OoD accuracy. This observation encourages future research to put more focus on ViT architecture research for OoD generalization.

IN-CPixelate

IN-CFog

IN-CGaussianNoise

0

0

0

1

1

1

Severity

2

2

2

3

3

3

4

4

4

5

5

5

40

60

80

60

70

80

40

60

80

ACC.

AcC.

ACC.

Experimental Setups. To show how ViT architecture designs impact OoD generalization, we compute the range of OoD accuracy for each search space on an OoD dataset. This range reflects the variation in OoD performance for different architectures within a search space. For example, in Fig. 2, each sub-plot represents the range of OoD accuracy for three different search spaces in our OoD-ViT-NAS benchmark on one OoD dataset. We compute the average OoD accuracy range across all datasets as general statistics. For reference, the range of ID accuracy is also included.

Results. The results are shown in Fig. 2. Additional results can be found in the Appx.G. The average range of OoD accuracy across the three search spaces in our benchmark is 3 . 81% / 4 . 86% / 2 . 74% , which is comparable to or even outperforming state-of-the-art (SOTA) training OoD generalization method, such as those based on domain invariant representation learning [30, 31]. For example, under a similar OoD setting, the current SOTA [30] shows an improvement of 1 . 9% OoD accuracy. This observation highlights the significant influence of ViT architectural designs on OoD accuracy. By carefully designing ViT architecture, the OoD accuracy could improve significantly.

We further explore how the severity of the OoD shift affects the range of ViT's OoD accuracy. We conduct similar experimental setups as before, analyzing 1 , 000 architectures from the AutoformerSmall search space within our OoD-ViT-NAS benchmark for 1 , 000 architectures in Autoformer-Small search space within our benchmark on IN-C. The results are visualized in Fig. 3. We observe that the range of OoD accuracy widens as the severity of the OoD shift increases. This suggests that under stronger OoD shifts, the architecture design becomes even more critical for OoD generalization.

When visualizing OoD accuracy, we observe a bimodal distribution. We figure out that the embedding dimension, as the primary ViT structural attribute, influences this bimodality. For example, among architectures from the Autoformer-Small search space of our benchmark, most architectures with a lower embedding dimension ( 320 ) fall within the lower OoD accuracy mode, while those with higher dimensions ( 384 and 448 ) tend to reside in the higher accuracy mode. This observation will be further discussed in detail in Sec. 4.4.

4.2 Can ID accuracy serve as a good indication for OoD accuracy?

While existing works [2, 3, 6, 7, 8, 9, 12, 18] study the impact of ViT architectures to ID accuracy, studies on OoD accuracy are limited. To what extent can we directly apply existing findings of ViT architecture insights for ID to OoD accuracy? To answer this question, we investigate the relationship between ViT ID and ViT OoD accuracy.

Several studies [69, 70, 71, 72] investigates the relationship between ID and OoD accuracy for the CNNs model. However, there is no work on such study particularly for ViT. Utilizing our OoD-ViTNAS benchmark, we provide the first comprehensive study on the relationship between ViT ID and OoD accuracy. Through our investigation, we find that the correlation between ViT ID and ViT OoD accuracy is not very high. This suggests that architectural insights optimized for ViT ID accuracy, as presented in previous work [2, 3, 6, 7, 8, 9, 12, 18] may not be applicable for ViT OoD generalization.

Experimental Setup. Following previous work [71], we use Kendall's τ rank correlation coefficient to compute the correlation between OoD and ID accuracy of all 1 , 000 architectures from a search space on one OoD dataset. Our examination comprehensively computes the correlations across all 8 OoD datasets, 3 search spaces, and 3 , 000 architectures within our OoD-ViT-NAS benchmark. We compute the average correlations across search spaces and datasets as general statistics.

OoDAccIN-CPixelate

AccIN-D Material

OoDAccStylized-IN

22.5

18

AcC

81

45

20.0

17

D

17.5

80

40

15.0

16

OoD

20

30

20

30

20

30

20

30

#Param

#Param

#Param

#Param

Besides the investigation of the correlation of various architectures, we further study the relationship between ViT ID and ViT OoD accuracy for Pareto architectures, representing the top-performing architectures for a certain model size. As shown in Fig. 4-a, the red dots represent Pareto architectures for ID accuracy In this study, we analyze 1 , 000 architectures from the Autofomer-Small search space within our OoD-ViT-NAS benchmark. To identify Pareto architectures for ID accuracy, we divide the total parameter budget into 30 equal intervals and select the architecture with the best ID performance within each interval.

Results. The correlation results are illustrated in Fig. 1-a. The individual correlations can be found in the Appx. H. We show that the correlation between ID and OoD accuracy is generally not very high. This suggests that current architectural insights based solely on ID accuracy might not effectively translate to OoD generalization.

Among all OoD datasets, the IN-P dataset exhibits the strongest correlation with ID accuracy. This can be attributed to its weaker OoD shift compared to other datasets (see the visualization in Appx. E). As a result, the OoD examples in IN-P are not very different from ID examples, leading to a relatively high correlation between OoD and ID performance. For the remaining seven datasets with stronger OoD shifts, the correlations remain relatively low.

The results of the Pareto architectures analysis are illustrated in Fig. 4. Additional results can be found in the Appx. I. We observe that Pareto architectures for ID accuracy generally perform sub-optimally under the OoD shift. This observation further supports our previous finding that ID accuracy might not be a very good indicator of OoD accuracy.

4.3 Explore Training-free NAS for OoD Generalization

Recently, there has been a new research focus on Training-free NAS, aimed at identifying highperforming architectures without the computational expense of training each candidate. To do so, [10, 11, 17, 62, 60, 73, 74] propose zero-cost proxies to predict the performance of candidate architectures in the initialization or the first training iteration, significantly accelerating NAS. While these works focus on ID accuracy, a few attempts have been made in searching for architectures robust against adversarial attacks [62, 75]. However, there is no work to explore Training-free NAS for ViT for OoD generalization.

Experimental Setup. To address this gap, we comprehensively explore the existing 9 Training-free NAS for OoD generalization on 3 , 000 ViT architectures within our OoD-ViT-NAS benchmark. Our study includes common and SOTA Training-free NAS originally proposed for CNNs for ID Acc (Grasp [74], SNIP [73], MeCo [60]), ViTs for ID Acc (DSS [10], AutoProx [17]), and CNNs for adversarial robustness (Jacobian [75], CroZe [62]). We complement this study on Training-free NAS to our OoD-ViT-NAS benchmark to equip the NAS research community with valuable tools to develop more effective Training-free NAS for OoD generalization.

Results. Our exploration provides several practical insights for designing a Training-free NAS for ViT for OoD generalization. The results of Kendall τ [76] ranking correlation between the OoD accuracy and Training-free NAS proxies on 8 common large OoD datasets are illustrated in Tab. 2. The average OoD accuracy is computed across OoD datasets and search spaces. Detailed results can be found in Fig. 1 and Appx. J.

We observe that existing Training-free NAS are largely ineffective in predicting OoD accuracy. Even recent Training-free NAS designed for ViT (i.e., DSS [10] and AutoProx-A [17] ) or Training-free NAS designed adversarial robustness [62] struggle with predicting OoD accuracy.

Surprisingly, simple zero-cost proxies such as #Param or #Flops outperform all existing, more complex proxies in predicting both OoD accuracy for ViTs. This finding poses a challenge to the Training-free NAS research community: to devise a Training-free NAS that surpasses #Params or #Flops in OoD Acc prediction for ViT.

From Fig. 1-b, we observe that all Training-free NAS methods consistently fail to predict IN-D performance. This is due to the IN-D dataset's unique generation process using Stable Diffusion, which creates images labelled with object names and varying nuisances like background, texture, and material variations. Only the most challenging images are retained, resulting in highly difficult examples, such as distorted images and unrealistic object-background placements (see Fig. E.2). These examples degrade ViT model performance significantly [26] and cause unpredictable behaviour.

Our investigation into ID accuracy for ViTs also reveals a surprising observation. While proposals for Training-free NAS designed for ViTs (i.e., DSS [10] and AutoProx-A [17]), improve the prediction of ID accuracy compared to counterparts designed for CNNs. Our study marks the first attempt to explore simple Training-free NAS like #Param or #Flops. The ID prediction of such simple proxies surprisingly outperforms SOTA Training-free NAS designed for predicting ID accuracy for ViT.

4.4 ViT Structural Attributes on OoD Generalization: Increasing Embedding Dimension is Generally Helpful

Our OoD-ViT-NAS benchmark with 3 , 000 ViT architectures covers diverse design choices in ViT structural attributes, including embedding dimension (Embed_Dim), network depth, number of heads (#Heads), and MLP ratio (MLP_Ratio), which allows for finding a wide range of ViT with different structures and complexities. Utilizing our comprehensive benchmark, we are the first to provide an analysis of the impact of these ViT structural attributes. We investigate which structural attributes in ViTs could lead to better OoD generalization. Through our analysis, we find that increasing the embedding dimension of a ViT architecture can generally improve OoD generalization. The additional analysis on another search space [1] further confirms our finding. The details for this additional analysis can be found in Appx. C

Experimental Setup. To verify the effectiveness of ViT architectural attributes on our OoD generalization benchmark, we present the results from two perspectives: (1) an analysis of rank correlation for our OoD-ViT-NAS benchmark and (2) a comparison of OoD accuracy across different embedding dimensions. To gain insights into the relationship between embedding dimension (Embed_Dim) and OoD performance, we created the visualizations for all architectures in our OoD-ViT-NAS benchmark. These visualizations compare the average OoD accuracies across different ViT architectures with varying embedding dimensions and depths. Examples of these visualizations are shown in Fig. 5. Additional results can be found in the App.K.

Results. In Fig. 1-c, we find that the embedding dimension generally has the highest correlation with OoD accuracy among all ViT architectural attributes. This positive correlation indicates that Em-

wooz

Blur

31.5

32.0

31.0

30.93

31.54

30.71

31.27

30.61

31.25

-NI

R

31.5

31.0

30.75

31.17

f

31.24

31.59

31.26

31.56

IN-Sketch

33.5

33.0

32.5

32.0

32.45

33.04

32.71

33.21

32.68

33.17

30.5

ACC

N

ACC

D

30.0

29.5

29.41

29.24

29.23

100

D

30.0

30.5

29.74

30.24

30.30

OoD

AcC

30.5

31.5

31.0

30.54

30.83

30.89

0

29.0

29.5

30.0

28.5

29.5

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

bed_Dim could play a crucial role in achieving OoD generalization performance. Our comprehensive OoD-ViT-NAS benchmark sheds light on a previously unknown relationship: the potential impact of embedding dimension (Embed_Dim) on OoD generalization in ViTs. This trend holds across most OoD shifts in our benchmark (Fig. 5), suggesting that among other architectural attributes, the design choice of Embed_Dim might significantly influence a model's OoD generalization.

Our experiments yield several intriguing phenomena. Based on Fig. 1-c, we observe that network depth has a slight impact on overall OoD generalization performance (correlation: 0 . 19 ). Also, as shown in Fig. 5, for a given embedding dimension (represented by a distinct colour), we report the mean OoD accuracy, showing how the mean OoD accuracy changes among ViT architectures of varying depths, which aligns with our empirical insight. Fig. 5 shows that while increasing depth can be beneficial for improving ViT's OoD generalization in some cases, there exist shallower models that tend to perform better in terms of OoD accuracy compared to those with deeper models.

It is evident from Fig. 1-c, where both the MLP ratio (0.09) and the number of heads ( 0 . 07 ) exhibit very low correlation values with overall OoD performance. These findings highlight that increasing the MLP ratio and the number of heads may not substantially enhance a model's robustness to OoD data. Due to space constraints, we defer additional experiments to the Appx. L, showing that the network depth, MLP ratio, and #Heads might have non-obvious impacts on OoD generalization.

Increasing Embedding Dimension help ViT learn more high-frequency patterns, leading to improve OOD generalization. In this section, we design a frequency study to understand our finding: why increasing ViT Embedding Dimension can generally improve ViT's OOD generalization. In the literature, the models obtain higher performance on preserving High-Frequency-Component (HFC) samples tend to learn more HFC [77, 78]. By learning more HFC, the models improve OOD generalization [77, 79]. Our hypothesis is that Increasing embedding dimension helps ViTs learn more HFC resulting in improving OOD generalization. We adapt the experiment from [77] to verify our hypothesis. The details on this experimental setup can be found in the Appx. D. In a nutshell, we filter HFC by hyper-parameter radius r, where the higher the r, the lesser HFC. As shown in Fig. 6, we observe that when increasing Embedding Dimension, the performances obtained on filtering-HFC samples are improved. This observation holds true across setups varying radius r, supporting that increasing Embedding Dimension helps ViT learn more HFC. In contrast, increasing other ViT structural attributes does not help improve ViT learn more HFC.

Robust ViT architectures designed by our finding. Our study provides significant insights for guiding the design of ViT architectures. Specifically, among ViT structural attributes, increasing embedding dimension can generally improve OoD generalisation of ViT architectures. Our insight leads to a simple method which can achieve ViT architectures that can outperform well-established human-designed. We demonstrate the superiority of ViT based on our insights in Tab. 3. Scaling up ViT architecture (e.g., from ViT-B-32 to ViT-L-32) by humans typically involves compound scaling of various ViT structural attributes. However, our findings suggest that not all ViT structural attributes need to be increased to benefit OoD generalisation. Among these attributes, increasing the embedding dimension is the most crucial factor for improving OoD generalisation. By only increasing the

r = 12

r = 16

22

22

22

22

20

20

20

20

18

18

18

18

OoDAccuracy

16

OoDAccuracy

16

OoDAccuracy

16

OoDAccuracy

16

14

14

14

14

12

12

12

10

00

10

10

8

4

320

384

448

12

13

14

3.125

3.5

3.833

5.3333

6.003

6.692

EmbeddingDimension

NumberofLayers

MLPRatio

NumberofHeads

embedding dimension, ours ViT architectures (e.g., Increasing embedding dimension of ViT-B-32) are significantly more efficient and outperform compound scaling architectures (e.g., ViT-L-32).

5 Conclusion

In this work, we introduce OoD-ViT-NAS, the first comprehensive benchmark for NAS on OoD generalization of ViT architectures. Using this benchmark, we conduct a comprehensive investigation on OoD generalization for ViT. Firstly, we show that ViT architecture design significantly impacts OoD accuracy. Secondly, we show that the architectural findings from existing works for ID performance could not apply to OoD generalization due to the low correlation between ID and OoD accuracy. Thirdly, we conduct the first study of NAS for ViT's OoD generalization and show that existing Training-free NAS methods struggle with OoD prediction. Surprisingly, simple proxies like #Param or #Flops outperform other complex Training-free NAS. Finally, we conduct the first study on the impact of ViT architectural attributes on OoD generalization. Our study reveals that increasing a ViT architecture's embedding dimensions can generally improve OoD generalization. We believe our benchmark OoD-ViT-NAS and comprehensive analysis will catalyze and streamline future research on understanding how ViT architecture design influences OoD generalization.

Acknowledgement. This research is supported by the National Research Foundation, Singapore under its AI Singapore Programmes (AISG Award No.: AISG2-TC-2022-007); The Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021). This research is supported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.

References

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European conference on computer vision , pages 459-479. Springer, 2022.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In European Conference on Computer Vision , pages 280-296. Springer, 2022.

Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, et al. Segvit: Semantic segmentation with plain vision transformers. Advances in Neural Information Processing Systems , 35:49714982, 2022.

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 12270-12280, 2021.

Haibin Wang, Ce Ge, Hesen Chen, and Xiuyu Sun. Prenas: Preferred one-shot learning towards efficient neural architecture search. In International Conference on Machine Learning , pages 35642-35654. PMLR, 2023.

Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Vitas: Vision transformer architecture search. In European Conference on Computer Vision , pages 139-157. Springer, 2022.

Chen Tang, Li Lyna Zhang, Huiqiang Jiang, Jiahang Xu, Ting Cao, Quanlu Zhang, Yuqing Yang, Zhi Wang, and Mao Yang. Elasticvit: Conflict-aware supernet training for deploying fast vision transformer on diverse mobile devices. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5829-5840, 2023.

Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free transformer architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10894-10903, 2022.

Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free transformer architecture search with zero-cost proxy guided evolution. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Chengyue Gong and Dilin Wang. Nasvit: Neural architecture search for efficient vision transformers with gradient conflict-aware supernet training. ICLR Proceedings 2022 , 2022.

Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding Robustness of Transformers for Image Classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 10231-10241, oct 2021.

Shiyu Tang, Ruihao Gong, Yan Wang, Aishan Liu, Jiakai Wang, Xinyun Chen, Fengwei Yu, Xianglong Liu, Dawn Song, Alan Yuille, et al. Robustart: Benchmarking robustness on architecture design and training techniques. arXiv preprint arXiv:2109.05211 , 2021.

Salman Rahman and Wonkwon Lee. Out of distribution performance of state of art vision model. arXiv preprint arXiv:2301.10750 , 2023.

Minghao Chen, Kan Wu, Bolin Ni, Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, and Haibin Ling. Searching the search space of vision transformer. Advances in Neural Information Processing Systems , 34:8714-8726, 2021.

Zimian Wei, Peijie Dong, Zheng Hui, Anggeng Li, Lujun Li, Menglong Lu, Hengyue Pan, and Dongsheng Li. Auto-prox: Training-free vision transformer architecture search via automatic proxy discovery. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 15814-15822, 2024.

Mingyu Ding, Xiaochen Lian, Linjie Yang, Peng Wang, Xiaojie Jin, Zhiwu Lu, and Ping Luo. Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2982-2992, 2021.

Jihao Liu, Xin Huang, Guanglu Song, Hongsheng Li, and Yu Liu. Uninet: Unified architecture search with convolution, transformer, and mlp. In European Conference on Computer Vision , pages 33-49. Springer, 2022.

Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16 , pages 702-717. Springer, 2020.

Zi-Hang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. Advances in neural information processing systems , 34:18590-18602, 2021.

Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision , pages 357-366, 2021.

Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12299-12310, 2021.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019.

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15262-15271, 2021.

Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object. arXiv preprint arXiv:2403.18775 , 2024.

Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision , pages 8340-8349, 2021.

Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Neural Information Processing Systems , 32, 2019.

Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 , 2018.

Haoyue Bai, Yifei Ming, Julian Katz-Samuels, and Yixuan Li. HYPO: Hyperspherical out-of-distribution generalization. In The Twelfth International Conference on Learning Representations , 2024.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision , pages 2551-2559, 2015.

Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International conference on machine learning , pages 7313-7324. PMLR, 2021.

Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International conference on machine learning , pages 10-18. PMLR, 2013.

Alexandre Rame, Corentin Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-ofdistribution generalization. In International Conference on Machine Learning , pages 18347-18377. PMLR, 2022.

Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research , 19(36):1-34, 2018.

Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937 , 2021.

Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 , 2019.

Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16 , pages 561-578. Springer, 2020.

Yimeng Chen, Tianyang Hu, Fengwei Zhou, Zhenguo Li, and Zhi-Ming Ma. Explore and exploit the diverse knowledge in model zoo for domain generalization. In International Conference on Machine Learning , pages 4623-4640. PMLR, 2023.

Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In International Conference on Machine Learning , pages 28656-28679. PMLR, 2023.

Jungwuk Park, Dong-Jun Han, Soyeong Kim, and Jaekyun Moon. Test-time style shifting: Handling arbitrary styles in domain generalization. In International Conference on Machine Learning , pages 27114-27131. PMLR, 2023.

Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24172-24182, 2023.

Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8690-8699, 2021.

Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang. Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 18710-18719, 2022.

Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9619-9628, 2021.

Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8886-8895, 2021.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint arXiv:2104.02008 , 2021.

Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. A fourier-based framework for domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14383-14392, 2021.

Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677 , 2020.

Yingjun Du, Xiantong Zhen, Ling Shao, and Cees GM Snoek. Metanorm: Learning to normalize few-shot batches across domains. In International Conference on Learning Representations , 2020.

Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain-augmented meta-learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9624-9633, 2021.

Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. In International Conference on Machine Learning , pages 8346-8356. PMLR, 2020.

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 , 2018.

Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key to out-of-distribution generalization? In International Conference on Machine Learning , pages 12356-12367. PMLR, 2021.

Geon Yeong Park, Sangmin Lee, Sang Wan Lee, and Jong Chul Ye. Training debiased subnetworks with contrastive weight pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7929-7938, 2023.

Haoyue Bai, Fengwei Zhou, Lanqing Hong, Nanyang Ye, S-H Gary Chan, and Zhenguo Li. Nasood: Neural architecture search for out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8320-8329, 2021.

Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra. Alphanet: Improved training of supernets with alpha-divergence. In International Conference on Machine Learning , pages 10760-10771. PMLR, 2021.

Dilin Wang, Meng Li, Chengyue Gong, and Vikas Chandra. Attentivenas: Improving neural architecture search via attentive sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6418-6427, 2021.

Tangyu Jiang, Haodi Wang, and Rongfang Bie. Meco: Zero-shot nas with one data and single forward pass via minimum eigenvalue of correlation. Advances in Neural Information Processing Systems , 36, 2024.

Steffen Jung, Jovita Lukasik, and Margret Keuper. Neural architecture design and robustness: A dataset. arXiv preprint arXiv:2306.06712 , 2023.

Hyeonjeong Ha, Minseon Kim, and Sung Ju Hwang. Generalizable lightweight proxy for robust nas against diverse perturbations. Advances in Neural Information Processing Systems , 36, 2024.

Yongtao Wu, Fanghui Liu, Carl-Johann Simon-Gabriel, Grigorios Chrysos, and Volkan Cevher. Robust nas under adversarial training: benchmark, theory, and beyond. In The Twelfth International Conference on Learning Representations .

Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12-21, 2021.

Mingyang Zhang, Xinyi Yu, Haodong Zhao, and Linlin Ou. Shiftnas: Improving one-shot nas via probability shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5919-5928, 2023.

Shiguang Wang, Tao Xie, Jian Cheng, Xingcheng Zhang, and Haijun Liu. Mdl-nas: A joint multi-domain learning framework for vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20094-20104, 2023.

Yaowei Li, Ruijie Quan, Linchao Zhu, and Yi Yang. Efficient multimodal fusion via interactive prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2604-2613, 2023.

Li Fei-Fei, Jia Deng, and Kai Li. Imagenet: Constructing a large-scale image database. Journal of vision , 9(8):1037-1037, 2009.

John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-ofdistribution and in-distribution generalization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7721-7735. PMLR, 18-24 Jul 2021.

Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 5389-5400. PMLR, 09-15 Jun 2019.

Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. Id and ood performance are sometimes inversely correlated on real-world datasets. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 71703-71722. Curran Associates, Inc., 2023.

Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, Bernhard Schölkopf, and Francesco Locatello. Assaying Out-Of-Distribution Generalization in Transfer Learning. In S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho, and A Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 7181-7198. Curran Associates, Inc., 2022.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340 , 2018.

Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376 , 2020.

Ramtin Hosseini, Xingyi Yang, and Pengtao Xie. Dsrna: Differentiable search of robust neural architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 61966205, 2021.

Maurice G Kendall. A new measure of rank correlation. Biometrika , 30(1/2):81-93, 1938.

Jiawang Bai, Li Yuan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, and Wei Liu. Improving vision transformers by revisiting high-frequency components. In European Conference on Computer Vision , pages 1-18. Springer, 2022.

Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of vision transformers. arXiv preprint arXiv:2103.15670 , 2021.

Paul Gavrikov and Janis Keuper. Can biases in imagenet models explain generalization? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22184-22194, 2024.

Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8684-8694, 2020.

Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision , pages 1501-1510, 2017.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

A Appendix Overview

This appendix provides supplementary information that are not included in the main paper due to space limitations.

Contents

B Limitations and Broader Impact

Given the extensive set of experiments presented in this work, the evaluation of training-free NAS proxies is significantly dependent on the initial robust benchmarks, which can be costly and resource intensive to create in the first place.

As this work studies the robustness of ViT architectures to OoD shift, we could demonstrate that carefully designed ViT architectures can significantly enhance OoD generalization. Our approach focuses on evaluating training-free NAS for ViT architectures, offering valuable insights that can complement research exploring the effects of robust architectural design. Finally, we publicly release our OoD-NAS-ViT and code base for future research.

C Analysis on Human-design ViT Search Space

Our findings on embedding dimensions are derived from analysing ViT architectures sampled through AutoFormer. To further validate these results, we also investigate the impact of ViT structural attributes on OoD generalisation within the human-designed ViT search space [1].

Experimental setup . We begin with the vanilla ViT-B-32 architecture [1], varying each structural attribute independently. For the altered ViT architectures, we increase these attributes to ensure that the capacities of the altered models remain comparable. Each architecture is trained on IN-100 and evaluated on IN-R.

Results. As shown in Tab. C.1, the results further confirms our findings that embedding dimension is the most important ViT structural attribute to OoD generalisation.

D Detailed description of the Frequency Analysis Setup

We adapt the experiment from [77] to verify our hypothesis. We quantify how the amount of HFC learnt in the ViTs changes if the embedding dimension of ViTs changes. Particularly, we first filter HFC from testing images of IN-R following [77, 79, 80] then evaluate the performance of 1000 ViTs in our search space on IN-R with filtering data points. Following [80] to generate HFC-preserving images, we first convert original images to FFT images. Then, we filter HFC by hyper-parameter radius r (r is set to 4, 8, 12, 16 in our experiments). In a nutshell, the higher the r, the lesser HFC.

E The details for our benchmark OoD-ViT-NAS

E.1 Detailed description of Autoformer Search Space

We strictly follow the search space in Autoformer [6] to construct our OoD-ViT-NAS. The variable factors of basic transformer blocks include the embedding dimension, Q-K-V dimension, number of heads, MLP ratio, and network depth. The detailed search spaces are illustrated in Tab. E.2

E.2 Detailed description of OoD Datasets

ImageNet

ImageNet-C

ImageNet-A

ImageNet-O

ImageNet-P

ImageNet

ImageNet-D

ImageNet-R

ImageNet-Sketch

ImageNet-Stylized

To construct OoD-ViT-NAS benchmark, we evaluate 3,000 architectures within our benchmark on common and most SOTA OoD datasets, including:

ImageNet-1k [68]: This is a large and common image dataset widely used in computer vision research. It contains over 1.3 million labeled high-resolution images belonging to 1,000 different object categories (classes). Each image is labeled with a class (e.g., "cat", "airplane", "chair").

ImageNet-C [24]: This dataset builds upon the original ImageNet test set by applied algorithmically corruptions. These corruptions simulate real-world factors that can deviate data from the training set, such as blur, noise, digital, and weather effects. ImageNet-C offers a comprehensive OoD scenarios with 15 different corruption types, each with 5 severity levels, resulting in a total of 75 unique OoD setups.

ImageNet-P [24]: This benchmark is constructed similarly to Imagnet-C. The difference is that ImageNet-P utilizes perturbation sequences generated from each ImageNet validation image.

ImageNet-A [25]: This dataset is a real-world OoD scenarios by leveraging adversarially filtered images that are highly likely to fool current image classifiers. ImageNet-A selects a 200 classes out of 1,000 classes from ImageNet-1K so that errors among these 200 classes would be considered egregious [68].

ImageNet-O [25]: Similar to ImageNet-A, this dataset includes adversarially filtered examples specifically designed to challenge OoD detectors trained on ImageNet. ImageNet-O selects a 200 classes out of 1,000 classes from ImageNet-1K.

ImageNet-R [27]: This dataset is a rendition of ImageNet, containing images with manipulated textures and local image statistics.

ImageNet-Sketch [28]: This dataset introduces a unique OoD challenge by providing blackand-white sketch images corresponding to the ImageNet-1K test set. This significant divergence in visual representation tests a model's ability to generalize beyond photographic data.

Stylized ImageNet [29]: This dataset consists of a stylized version of ImageNet generated through techniques like AdaIN [81] style transfer, resulting in variations like greyscale, silhouettes, and edges. This dataset assesses a model's ability to handle data with different artistic interpretations.

ImageNet-D [26]: This dataset is a variation of ImageNet generated through diffusion models. These models aim at creating images with rich diversity in backgrounds, textures, and materials.

ComputerMouse

OAILE

We visualize a few examples of OoD datasets in Fig. E.1. For ImageNet-C, we provide the visualization of different OoD shift severity in Fig. E.3.

0

1

2

3

4

5

Brightness

Contrast

Defocus Blur

Elastic

Transform

Fog

Frost

Noise

Impulse

Noise

Compression

MotionBlur

Pixelate

ShotNoise

Snow

F Overview of using the OoD-NAS-ViT benchmark

Our OoD-NAS-ViT benchmark consists of multiple json individual files. Each file includes the evaluation of one search space (Autoformer-Tiny/Small/Base [6]) on 8 most common and state-of-theart (SOTA) OoD datasets: ImageNet-C [24], ImageNet-A [25], ImageNet-O [25], ImageNet-P [24], ImageNet-D [26], ImageNet-R [27], ImageNet-Sketch [28], and Stylized ImageNet [29] (Sec. 3).

We further provide a merged json file for each search space. The structure of each merged json is illustrated as in Fig. F.4. Specifically, each merged json file includes 1,000 ViT architectures denoted by the index. Each architecture consists of:

net_setting: ViT structural attributes information:

-layer_num: network depth

-mlp_ration: MLP ration each layer, can be varied among layers

-num_heads: number of attention heads each layer, can be varied among layers

-embed_dim: embedding dimension heads each layer, fixed among layers

params: number of parameters

flops: number of flops

performance: performance on different dataset:

Imagenet: performance on IN-based datasets:

-clean: performance on IN

-sketch: performance on IN

-stylized-imagenet: performance on IN

-imagenet-R: performance on IN

-imagenet-O: performance on IN

-imagenet-A: performance on IN-A

-corruption: performance on IN-C

Fog: performance on one out of 15 corruption in IN-C: Fog, Gaussian Noise, Fog, Snow, Elastic Transform, Jpeg Compression, Frost, Motion Blur, Brightness, Defocus Blur, Glass Blur, Impulse Noise, Shot Noise, Zoom Blur, Constrast, Pixelate

1: performance at OoD shift severity level 1. There are total 5 level of OoD shift severity for each corruption in IN-C

-corruption_P: performance on IN-P

Brightness: performance on one out of 10 corruption in IN-P: Brightness, Motion Blur, Rotate, Scale, Shot Noise, Snow, Tilt, Translate, Zoom Blur, Gaussian Noise

-imagenet-D: performance on IN-D

background: performance on one out of 3 nuisances in IN-D: background, material, texture

With our the provided a merged json file for each search space, we can easily retrieve the OoD shift performance of various ViT architectures and their OoD performance on 8 prevelent and SOTA OoD datasets.

G Additional results on the analysis of OoD accuracy range

In the main paper, we visualize 11 OoD accuracy ranges and ID accuracy range for reference. In this Appx. section, we provide the visualization of the remaining OoD accuracy ranges. The results are illustrated in Fig. G.5. Our observation on other OoD accuracy ranges are generally consistent with our findings in Sec. 4.1.

1.06

2.40(1.06)

2.69(1.06)

2.93(1.06)

Tiny

2.25

4.74(2.25)

4.89(2.25)

3.81(2.25)

Small

0:56

2.53(0.56)

1.65(0.56)

4.88(0.56)

Base

70

80

40

50

40

50

60

20

30

40

ID ACC (%)

IN-C Snow Acc.

IN-C Frost Acc.

IN-C Motion Blur Acc.

1.32(1.06)

2.46(1.06)

1.38(1.06)

3.51(1.06)

Au!l

2.72(2.25)

3.15(2.25)

2.93(2.25)

7.00(2.25)

Small

0.90(0.56)

1.95(0.56)

2.22(0.56)

3.36(0.56)

Base

60

70

20

30

10

20

20

30

40

IN-C Brightness Acc.

IN-CDefocusBlurAcc.

IN-CGlassBlurAcc.

IN-C Shot Noise Acc.

1.50(1.06)

2.36(1.06)

2.04(1.06)

29.62(1.06)

Tiny

3.65(2.25)

4.52(2.25)

4.35(2.25)

8.91(2.25)

Small

2.15(0.56)

2.32(0.56)

2.24(0.56)

4.41(0.56)

Base

20

30

40

50

60

30

40

20

40

IN-CZoomBlurAcc.

IN-C Jpeg Compression Acc. IN-C Elastic Transform Acc.

IN-C Contrast Acc.

8.99(1.06)

1.02(1.06)

1.21(1.06)

1.00(1.06)

Ku!

8.50(2.25)

1.95(2.25)

2.50(2.25)

2.22(2.25)

Base Small

8.72(0.56)

0.56(0.56)

0.92(0.56)

0.65(0.56)

10

20

70

80

70

80

70

80

IN-D Material Acc.

IN-P Translate Acc.

IN-P Snow Acc.

IN-P Rotate Acc.

1.03(1.06)

1.38(1.06)

1.16(1.06)

1.11(1.06)

Tiny

2.08(2.25)

3.21(2.25)

2.12(2.25)

2.44(2.25)

Small

0.58(0.56)

2.65(0.56)

0.56(0.56)

0.53(0.56)

Base

70

80

60

70

70

80

70

80

IN-P Tilt AcC.

IN-P Scale AcC.

IN-PGaussianNoiseAcc.

IN-P Brightness Acc.

1.26(1.06)

1.18(1.06)

1.12(1.06)

Tiny

2.43(2.25)

2.20(2.25)

2.38(2.25)

Small

0.61(0.56)

0.61(0.56)

0.89(0.56)

Base

60

70

80

70

80

10

20

IN-P Motion Blur Acc.

IN-P Shot Noise Acc.

Stylized-IN Acc.

IN-C Snow

IN-C Frost

IN-C Motion Blur

IN-CBrightness

0

1

1

2

2

ver

Sev

3

3

3

4

4

4

5

5

5

60

80

50

60

70

80

40

60

80

70

75

80

AcC.

ACC.

AcC.

AcC.

IN-C Defocus Blur

IN-C Glass Blur

IN-CImpulse Noise

IN-C Shot Noise

0

0

0

0

1

Severity

2

2

2

3

3

3

4

4

4

4

5

5

40

60

80

20

40

60

80

40

60

80

40

60

80

ACC.

AcC.

ACC.

ACC.

IN-C Zoom Blur

IN-C Jpeg Compression

IN-C Elastic Transform

IN-C Contrast

0

0

0

0

1

Severity

2

2

2

3

3

3

3

4

4

5

5

5

40

60

80

50

60

70

80

40

60

80

40

60

80

ACC.

ACC.

ACC.

AcC.

H Additional results on the analysis of the correlation between ID and OoD accuracy

In the main paper, we provide the Kendall τ correlation between ID accuracy and 8 OoD datasets. For some OoD datasets with different OoD shift types, we average the correlation with ID accuracy of different OoD shift types to obtain average correlation with ID accuracy for that OoD dataset. In this Appx. section, we provide the detailed correlations with ID accuracy of each OoD shifts in such OoD data. Specifically, we provide the detailed correlation for IN-C, IN-D, and IN-P in Fig. H.7, Fig. H.8, and Fig. H.9, respectively.

Furthermore, we provide a comprehensive correlation between OoD accuracy and ID accuracy in Fig. H.10.

(a)IDasindicatorfor

(c)OoDGeneralization

ViTOoDGeneralization

(b)Training-freeNASforViTOoDGeneralization

ViTArchitectureAttributes

IN-C Gaussian Noise 5

0.34

0.22

0.23

0.12

0.04

0.20

0.20

0.10

0.20

0.20

0.70

-0.07

0.10

-0.06

- 1.00

IN-C Snow 5

50.42

0.36

0.34

0.15

0.02

0.35

0.34

0.10

0.32

0.28

0.76

-0.16

0.07

0.02

- 0.75

IN-C Fog 5

0.55

0.29

0.26

0.13

-0.02

0.29

0.28

0.08

0.24

0.20

0.33

0.16

0.07

0.09

IN-C Frost 5

0.34

0.36

0.38

0.15

0.09

0.30

0.31

0.11

0.34

0.32

0.35

0.19

0.03

0.23

-0.50

IN-C Motion Blur 5

0.26

0.35

0.37

0.12

0.18

0.27

0.27

0.09

0.34

0.36

0.12

-0.03

0.11

0.17

IN-C Brightness 5

0.53

0.30

0.27

0.13

0.02

0.31

0.28

0.08

0.26

0.23

0.59

-0.04

0.18

-0.02

0.25

IN-C Defocus Blur 5

0.28

0.28

0.29

0.12

0.06

0.25

0.25

0.11

0.26

0.25

0.18

0.02

0.21

0.08

IN-C Glass Blur 5 0.08

0.35

0.36

0.11

0.11

0.26

0.27

0.09

0.37

0.27

-0.11

0.02

-0.06

0.15

- 0.00

IN-C Impulse Noise 5

0.32

0.20

0.22

0.12

0.01

0.16

0.19

0.11

0.22

0.20

0.63

-0.18

0.11

-0.13

IN-C Shot Noise 5

0.28

0.40

0.38

0.13

0.06

0.34

0.33

0.10

0.35

0.31

0.58

-0.05

0.15

-0.10

- 0.25

Acc IN-C Pixelate 5

0.32

0.37

0.35

0.16

0.03

0.34

0.34

0.11

0.36

0.29

0.23

0.01

0.19

0.17

IN-C Zoom Blur 5

0.26

0.18

0.20

0.10

0.04

0.18

0.18

0.08

0.16

0.16

0.55

-0.30

0.12

-0.05

- 0.50

IN-C Jpeg Compression5

0.38

0.13

0.14

0.07

0.00

0.10

0.11

0.05

0.12

0.11

0.30

0.06

0.20

0.14

IN-C Elastic Transform 5

0.32

-0.02

-0.01

0.01

-0.00

-0.05

-0.03

0.03

0.01

0.02

0.25

-0.10

0.08

0.28

- 0.75

IN-C Contrast 5

50.30

0.20

0.20

0.14

-0.05

0.23

0.24

0.10

0.16

0.13

0.16

0.46

0.13

0.20

Average OoD

0.37

0.27

0.27

0.12

0.04

0.24

0.24

0.09

0.25

0.22

0.38

0.02

0.12

0.08

1.00

ID ACC

#Param#Flops Grasp MeCo CroZe SNIP Jacobian DSS AutoProx

Embed Depth

MLP

#Head

Dim

Ratio

(a)IDasindicatorfor

(c)OoDGeneralization

ViTOoDGeneralization

(b)Training-freeNASforViTOoDGeneralization

ViTArchitectureAttributes

1.00

IN-DTexture

-0.05

-0.12

-0.18

-0.07

-0.02

-0.11

-0.12

-0.04

-0.11

-0.10

-0.27

-0.00

0.03

-0.03

0.75

0.50

IN-DBackground

-0.16

-0.15

-0.16

-0.01

-0.14

-0.15

-0.15

-0.01

-0.14

-0.15

-0.11

-0.32

-0.10

0.06

0.25

- 0.00

IN-D Material

0.08

0.02

0.01

-0.01

0.01

0.02

0.01

0.02

0.03

0.02

0.03

-0.03

0.05

0.03

0.25

Average OoD

-0.04

-0.08

-0.11

-0.03

-0.05

-0.08

-0.09

-0.01

-0.07

-0.07

0.03

-0.01

0.04

0.02

= 0.50

0.75

IDACC

#Param#Flops GraspMeCoCroZe

SNIPJacobianDSSAutoProx

EmbedDepth

MLP

#Head

1.00

Dim

Ratio

I Additional results on the analysis of OoD Performance of Pareto architectures for ID

In the main paper, due to space constraints, we only provide the Pareto analysis results on a few representative OoD datasets. In this Appx. section, we provide additional results on this Pareto architecture analysis in Fig. M.24, M.19, M.20, M.21, M.22, M.23. In the following scatter plots, blue dots represent architectures in the search space, while red dots represent the ID Pareto architectures.

(a)IDasindicatorfor

(c)OoDGeneralization

ViTOoDGeneralization

(b)Training-freeNASforViTOoDGeneralization

ViTArchitectureAttributes

1.00

IN-P Translate

0.71

0.42

0.44

0.14

0.13

0.35

0.34

0.11

0.39

0.37

0.44

0.18

0.16

0.03

IN-P Snow

0.68

0.42

0.44

0.14

0.12

0.34

0.34

0.11

0.39

0.37

0.53

0.10

0.17

0.08

0.75

IN-P Rotate

0.70

0.40

0.41

0.14

0.13

0.32

0.31

0.11

0.37

0.35

0.42

0.18

0.17

0.06

-0.50

IN-P Zoom Blur

0.64

0.42

0.43

0.14

0.11

0.35

0.34

0.11

0.37

0.36

0.59

0.17

0.19

0.07

0.25

IN-P Tilt

0.71

0.56

0.58

0.18

0.23

0.45

0.44

0.11

0.54

0.55

0.43

0.19

0.16

0.03

IN-P Scale

0.53

0.45

0.46

0.15

0.13

0.38

0.37

0.12

0.40

0.39

0.50

0.56

0.15

0.13

- 0.00

IN-P Gaussian Noise

0.69

0.45

0.46

0.15

0.10

0.37

0.37

0.11

0.42

0.40

0.34

0.20

0.17

0.04

0.25

IN-P Brightness

0.70

0.44

0.45

0.15

0.13

0.36

0.35

0.12

0.40

0.38

0.39

0.21

0.18

0.04

IN-P Motion Blur

0.67

0.44

0.45

0.02

0.13

0.36

0.36

0.11

0.40

0.38

0.41

0.16

0.20

0.05

-0.50

IN-P Shot Noise

0.72

0.49

0.50

0.17

0.12

0.43

0.42

0.14

0.44

0.42

0.45

0.21

0.19

0.02

0.75

AverageOoD

0.70

0.45

0.46

0.14

0.13

0.37

0.37

0.11

0.41

0.40

0.45

0.22

0.17

0.05

ID ACC

#Param#Flops Grasp MeCo CroZe SNIP Jacobian DSS AutoProx

Embed Depth

MLP

#Head

1.00

Dim

Ratio

J Additional Results for Benchmarking Zero-cost Proxies

In the main submission, we provides the comparison of Kendall τ ranking correlation between the ID/OoD accuracies and the zero-cost proxy values across all OoD datasets. In this section, we provides the correlation for each dataset for a detailed observation. The results can be found in Tab. J.3, J.4, J.5, J.6, J.7, J.8, J.9, J.10. Our observations on individual OoD datasets are consistent with our findings in Sec. 4.3.

ID ACC

1.000.340.420.550.340.260.530.280.080.320.280.320.260.380.320.30-0.05-0.160.080.710.680.700.640.710.530.690.700.670.720.080.400.450.370.49

IN-C Snow

0.420.431

1.00

IN-C Fog

0.550.340.371

1.000.310.220.430.240.130.330.250.320.280.350.360.29-0.05-0.090.050.540.540.540.530.550.380.540.540.540.55-0.010.250.340.280.39

IN-C Frost 0.34 0.23 0.29 0.31

IN-C Moti0n Blur 0.260.160.200.220.101.000.180.320.210.20 0.120.240.230.250.170.19-0.01-0.010.050.250.24 0.260.330.24 0.090.250.230.310.26-0.060.07 0.14 0.160.26

IN-C Brightness0.530.370.500.430.360.181

1.00

0.260.090.380.370.320.290.390.390.23-0.08-0.050.100.630.660.630.540.650.460.620.640.590.630.030.330.540.410.54

IN-C Def0CUS Blur 0.280.250.220.240.160.320.261.000.220.260.230.400.350.410.260.27-0.01-0.060.040.330.320.350.390.330.220.340.330.370.340.000.130.220.210.24

IN-C Impulse Noise 0.320.800.430.330.180.200.380.260.111.00|0.680.230.280.240.190.12-0.07-0.000.080.330.350.330.350.330.200.300.310.320.340.060.18 0.300.280.33

IN-C Shot Noise

0.280.680.390.250.190.120.370.230.120.681.000.270.210.250.250.16-0.10-0.010.070.280.310.300.290.290.230.260.280.250.290.120.230.340.330.29

IN-C Pixelate 0.320.260.260.320.210.240.320.400.190.230.271

1.000.280.570.400.34-0.07-0.050.060.350.360.370.400.350.280.360.360.410.350.030.180.280.290.26

IN-C Z00m Blur 0.260.260.310.280.130.230.290.350.060.280.210.281.000.280.290.08-0.070.01 0.030.280.320.280.460.280.120.260.280.330.280.10 0.070.200.190.24

1.00

IN-C Jpeg Compression 0.380.260.320.350.330.250.390.41

1.000.370.39-0.06-0.070.060.440.450.460.450.450.350.450.450.480.450.010.220.340.310.33

0.75

IN-C Elastic Transform 0.320.220.290.360.300.170.390.260.240.190.25

1.00

0.15-0.080.020.100.400.430.400.390.400.250.380.400.380.370.020.180.290.290.39

0.50

IN-C Contrast 0.30 0.16 0.170.290.290.190.230.270.19 0.120.160.340.080

0.390.15

1.00

0.05-0.100.030.290.260.290.300.290.420.300.300.320.300.100.350.300.230.23

0.25

IN-DTexture-0.05-0.09-0.11-0.05-0.06-0.01-0.08-0.010.04-0.07-0.10-0.07-0.07-0.06-0.08-0.051.000.03-0.01-0.06-0.06-0.06-0.07-0.06-0.08-0.040.06-0.040.06-0.140.09-0.09-0.11-0.08

IN-D Background-0.16-0.040.05-0.09-0.040.01-0.05-0.060.08-0.00-0.01-0.050.01-0.070.02-0.100.031

1.00

00.06-0.14-0.11-0.13-0.15-0.13-0.18-0.13-0.13-0.14-0.15-0.03-0.140.06-0.04-0.06

0.00

IN-D Material 0.08 0.06 0.04 0.05 0.08 0.05 0.10 0.04 0.08 0.08 0.07 0.06 0.03 0.06 0.10 0.03-0.010.06

1.000.08 0.100.070.050.08 0.040.09 0.08 0.070.07-0.010.030.08 0.09 0.09

0.25

IN-P Translate

0.710.340.460.540.370.250

0.630.330.090.330.280.350.280.440.400.29-0.06-0.140.081.000.780.800.690.820.550.810.800.740.830.040.360.480.390.55

0.50

0.75

IN-PRotate0.700.340.470.540.370.260.630.350.130.330.300.370.280.460.400.29-0.06-0.130.070.800.751

1.00

1.00

IN-P Z00m Blur 0.640.370.440.530.310.330.540.390.080.350.290.400.460.450.390.30-0.07-0.150.050.69

0.68

1.000

0.680.500.670.670.690.700.090.380.440.400.49

IN-P Tilt 0.710.340.450.550.370.240.650.330.110.330.290.350.280.450.400.29-0.06-0.130.080.820.77

0.81

50.830.050.380.510.390.56

IN-P Scale 0.530.260.320.380.390.090.460.220.060.200.230.280.120.350.250.42-0.08-0.180.040.550.510.540.500

IN-P GaUSSian Noise 0.690.300.430.540.370.250.620.340.130.300.260.360.260.450.380.30-0.040.130.090.810.750.790.670.810.541.000.800.740.840.020.340.480.370.54

IN-P Brightness

0.700.320.430.540.390.230.640.330.100.310.280.360.280.450.400.30-0.06-0.130.080.800.760.780.670.800.560.801.000.720.800.040.350.480.400.54

IN-P Motion Blur

0.670.310.440.540.310.310.590.370.160.320.250.410.330.480.380.32-0.040.140.070.740.690.770.690.750.520.740.721.000.750.040.340.480.370.52

IN-P Shot Noise

1.000.380.160.180.07

00.480.380.35

0.48

0.47

1.00

0.480.48

10.4

0.380.400.390.380.37

0.40

10.3/

70.390.180.38

0.36

Stylized-IN

0.490.330

0.470.390.350.26

0.54

0.240.14

0.290.260.240.33

0.39

90.23-0.08-0.060.09

0.55

0.49

0.56

0.41

0.52

0.56

0.07

0.35

0.480.36

1.00

Acc

IN-CS

Fog

Blur

Zoom

rast

Sn

Rotate

Blur

TiIt

IN

.0

IN-A

Acc

IN

Stylized-

IN

OOD

Elasti

IN-D

IN-C Gaussian

IN-P

IN

K Additional Figures of ViT Structural Attributes: Embedding Dimension

L Additional Analysis of ViT Structural Attributes on OoD Generalization

L.1 Ablation Study on the Impact of ViT Architectural Attributes to OoD Generalization

In this section, we demonstrate the effectiveness of each ViT architectural attribute on OoD accuracy from the ablation study perspective. All ablation studies are based on 1,000 ViT architectures sampled from Autoformer-Small search space in our OoD-ViT-NAS benchmark. Through our general analysis in Sec. 4.4 in the main and ablation study, we show that the embedding dimension has the highest impact among ViT architectural attributes, while network depth has a slight impact on OoD generalization.

Experimental Setups. We conduct the ablation study on the impact of ViT architectural attributes on OoD generalization. Particularly, for each ablation study of one ViT architectural attribute, we vary that attribute while keeping all other attributes fixed. Then, we compute Kendall's τ rank correlation coefficient between each attribute and different OoD shifts. While we can directly adjust the depth and embedding dimension, adjusting MLP_Ration and #Head is challenging. This is because these two attributes for each ViT arch are in the form of a list with depth elements. Each element is selected among 3 choices. This results in a huge combination. To deal with this difficulty, we first compute the means of MLP_Ration/#Head for each architecture. Then, during the ablation study, we explore a range of values for MLP ratio and the number of heads (mean #Head = 6 ± 0 . 05 , mean MLP_Ratio =

3 . 5 ± 0 . 05 ) to capture the impact of these more nuanced architectural variations. We note that this range of values is small, allowing us to approximately fix these two attributes.

First, we assess the impact of embedding dimension on OoD generalization by fixing the configurations of all other ViT structural attributes (i.e., network depth = 13, mean #Head = 6 ± 0 . 05 , and mean MLP_Ratio = 3 . 5 ± 0 . 05 ). The correlation results are shown in Fig. L.11. We observe an overall positive correlation of 0.65. This further supports our observation in Sec. 4.4 that increasing the embedding dimension generally could lead to better OoD performance across most OoD shifts for these ViT architectures.

Average OoD

Stylized-IN

IN-Sketch

ID AcC

IN-D

IN-P

IN-O

IN-A

-IN-R

-NI

Embed_Dim

0.80

0.63

-0.11

0.81

0.82

0.68

0.70

0.80

0.69

0.65

0

To demonstrate how the number of layers (i.e., network depth) in a ViT architecture affects the model's OoD generalization, we fix other ViT architectural attributes (i.e., Embed_Dim = 384 , mean #Head = 6 ± 0 . 05 , and mean MLP_Ratio = 3 . 5 ± 0 . 05 ). The ranking correlation for all architectures between depth and OoD accuracy in Fig. L.12 suggests a weak correlation between the ViT network depth and OoD accuracy. This observation is consistent with our finding in Sec. 4.4 that the depth has a minimal influence on OoD generalization.

Average OoD

Stylized-IN

-IN-Sketch

AcC

-IN-C

IN-D

-IN-P

-IN-O

-IN-A

-IN-R

D

1

Depth

0.24

0.00

-0.23

0.40

0.22

0.49

0.49

0.14

0.24

0.22

0

For MLP_Ratio and #Heads, we observe that the rank correlation coefficients of MLP_Ratio and #Heads in Fig. L.13 and L.14, respectively, reveals a weak correlation between the #Heads/MLP_Ratio and OoD generalization. This suggests that within the explored range, these two architectural attributes have a non-obvious impact on the model's OoD generalization. To delve deeper into this observation, Section L.2 presents a layer-wise analysis of these two architectural attributes in ViT models.

Average OoD

Stylized-IN

IN-Sketch

ACC

IN-C

-IN-D

IN-P

IN-O

IN-A

IN-R

D

MLP Ratio

0.14

0.06

0.09

0.12

0.04

0.12

-0.01

-0.02

0.05

0.07

0

Average OoD

Stylized-IN

IN-Sketch

AcC

-IN-C

-IN-D

IN-P

IN-O

-IN-A

-IN-R

1

#Heads

0.06

0.07

0.02

0.08

0.02

0.13

0.16

0.07

0.20

0.09

0

L.2 Layer-Wise Analysis

The number of heads and MLP ratio vary across layers, which allows for searching more diverse architectures. In this section, we provide the layer-wise analysis of the influence of MLP_Ratio/#Heads in each layer on OoD generalization.

Experimental Setups: In Autoformer-Small search space, the MLP ratio at any given layer referred to as the i -th layer - can be set to 3.0, 3.5, or 4. In addition to architectures in our benchmark, we create a set of 108 sampled architectures from Autoformer-Small. These architectures are fixed with three architectural design attributes: (Depth = 12 , Embed_Dim = 320 , #Heads for all layers = 5 ). With these parameters fixed, the total number of potential MLP_Tatio configurations is still extensive (i.e., 3 12 ), making it impractical to absolutely fix the MLP_Ration in our ablation study. To specifically assess the influence of the MLP_Ration at the i -th layer, we further fix a constant MLP_Ration across all other layers. For example, Fig. M.18 depicts the nine configurations of MLP_Ration to analyze the impact of 5 -th layer.

We carry out a similar layer-wise analysis to demonstrate the effect of #Heads at a specific layer ( i -th layer). In addition to architectures in our benchmark, we create a set of 108 sampled from Autoformer-Small. These architectures are fixed with three architectural design attributes: (Depth = 12 , Embed_Dim = 320 , #MLP_Ratio for all layers = 3 . 0 ). To specifically assess the influence of the #Head at the i -th layer, we further fix a constant #Head across all other layers. For example, Fig. M.18 depicts the nine configurations of #Head to analyze the impact of 3 -th layer.

Results. The layer-wise analysis for MLP_Ratio are shown in Fig. L.15. Each sub-figure demonstrates the change in OoD Accuracy when varying MLP_Ratio at a particular layer. While increasing it improved OoD accuracy in some layers, it decreased it in others. This aligns with our observations in Sec. 4.4, suggesting no clear overall impact of MLP_Ratio on OoD generalization.

The layer-wise analysis for #Head is illustrated in Fig. L.16. Similar to our observation in the layer-wise analysis for MLP_Ratio, the impact of #Head is non-obvious, which is consistent with our finding in Sec. 4.4.

58

58

58

IN-CFog

IN-C Fog

IN-CFog

5

5

57

56

56

56

3.0

3.5

4.0

3.0

3.5

4.0

3.0

3.5

4.0

Layer 0, ID/OoD Acc Range = 0.11/0.03

Layer 1, ID/OoD Acc Range = 0.09/0.24

Layer 2, ID/OoD Acc Range = 0.07/0.16

58

58

58

IN-CFog

IN-CFog

IN-CFog

5

5

57

56

56

56

3.0

3.5

4.0

3.0

3.5

4.0

3.0

3.5

4.0

Layer 3, ID/OoD Acc Range = 0.03/1.71

Layer 4, ID/OoD Acc Range = 0.08/1.39

Layer 5, ID/OoD Acc Range = 0.07/0.45

58

58

58

IN-C Fog

IN-C Fog

IN-C Fog

57

57

57

56

56

56

3.0

3.5

4.0

3.0

3.5

4.0

3.0

3.5

4.0

Layer 6, ID/OoD Acc Range = 0.10/0.30

Layer 7, ID/OoD Acc Range = 0.17/0.15

Layer 8, ID/OoD Acc Range = 0.19/0.48

58

58

58

IN-C Fog

IN-C Fog

IN-C Fog

57

57

56

56

56

3.0

3.5

4.0

3.0

3.5

4.0

3.0

3.5

4.0

Layer 9, ID/OoD Acc Range = 0.20/0.53

Layer 10, ID/OoD Acc Range = 0.36/0.68

Layer 11, ID/OoD Acc Range = 0.18/0.18

54.6

54.6

54.6

IN-C

54.2

IN-C

54.2

IN-C

54.2

54.0

54.0

54.0

53.8

53.8

53.8

5

6

/

5

6

5

6

Layer 0, ID/OoD Acc Range = 0.03/0.03

Layer 1, ID/OoD Acc Range = 0.04/0.13

Layer 2, ID/OoD Acc Range = 0.06/0.14

54.6

54.6

54.6

54.4

IN-C

54.2

IN-C

54.2

-NI

54.2

54.0

54.0

54.0

53.8

53.8

53.8

5

6

7

5

6

5

6

7

Layer 3, ID/OoD Acc Range = 0.06/0.12

Layer 4, ID/OoD Acc Range = 0.05/0.11

Layer 5, ID/OoD Acc Range = 0.02/0.99

54.6

54.6

54.6

IN-C

54.2

IN-C

54.2

IN-C F

54.2

54.0

54.0

54.0

53.8

53.8

53.8

6

5

6

6

Layer 6, ID/OoD Acc Range = 0.05/0.17

Layer 7, ID/OoD Acc Range = 0.03/0.35

Layer 8, ID/OoD Acc Range = 0.09/0.09

54.6

54.6

54.6

O-NI

54.2

54.2

-NI

54.2

54.0

54.0

54.0

53.8

53.8

53.8

5

5

6

5

6

Layer 9, ID/OoD Acc Range = 0.12/0.11

Layer 10, ID/OoD Acc Range = 0.11/0.10

Layer 11, ID/OoD Acc Range = 0.10/0.36

M Reproducibility

0

5

5

5

5

5

5

5

5

5

5

5

5

1

5

5

6

5

5

5

5

5

5

5

5

5

2

5

5

7

5

5

5

5

5

5

5

5

5

Configuration

3

6

6

5

6

6

6

6

6

6

6

6

6

6

6

6

6

6

6

6

6

6

6

6

6

5

6

6

7

6

6

6

6

6

6

6

6

6

6

7

7

5

7

7

7

7

7

7

7

7

7

7

7

6

7

7

7

7

7

7

7

7

7

8

7

7

7

7

7

7

7

7

7

7

7

0

1

2

3

4

5

6

7

8

9

10

11

LayerNumber

1

0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.0

3.5

3.0

3.0

3.0

3.0

3.0

3.0

1

3.0

3.0

3.0

3.0

3.0

4.0

3.0

3.0

3.0

3.0

3.0

3.0

2

3

3.5

3.5

3.5

3.5

3.5

3.0

3.5

3.5

3.5

3.5

3.5

3.5

4

3.5

3.5

3.5

3.5

3.5

3.5

3.5

3.5

3.5

3.5

3.5

3.5

5

3.5

3.5

3.5

3.5

3.5

4.0

3.5

3.5

3.5

3.5

3.5

3.5

9

4.0

4.0

4.0

4.0

4.0

3.0

4.0

4.0

4.0

4.0

4.0

4.0

7

4.0

4.0

4.0

4.0

4.0

3.5

4.0

4.0

4.0

4.0

4.0

4.0

8

4.0

4.0

4.0

4.0

4.0

4.0

4.0

4.0

4.0

4.0

4.0

4.0

0

1

2

3

4

5

6

7

8

9

10

11

Layer Number

1

M.1 Hyper-parameters

To ensure reproducibility, we provide a detailed description of the hyper-parameters used for evaluating 3,000 ViT architectures in our OoD-NAS-ViT benchmarks on 8 common large-scale OoD datasets: ImageNet-C [24], ImageNet-A [25] [24], ImageNet-O [25], ImageNet-P [24], ImageNet-D [26], ImageNet-R [27], ImageNet-Sketch [28], and Stylized ImageNet [29]. The evaluation for ImageNet-D, ImageNet-O and Stylized ImageNet strictly follows previous works [26, 25, 29] to ensure consistency and comparability. The details of the evaluation are shown in Table M.11. The evaluated architectures are sampled on AutoFormer-Tiny/Small/Base Search Spaces [6].

M.2 Compute Resource

All our experiments are conducted using NVIDIA RTX A6000 GPUs. We utilized 2 GPUs for each experiment. The substantial computational resources required for these evaluations underscore the complexity and scale of our work. Detailed information on the GPU-hour consumed for all experiments to construct our OoD-ViT-NAS Benchmark can be found in Table M.12. In total, the experiments demanded a significant investment of approximately 3900 GPU-hours, reflecting the extensive computational effort involved.

Kendall's t correlation: 0.79

Kendall's T correlation:0.80

Kendall's t correlation: 0.77

Kendalls t correlation:0.77

81.5

OoD Acc IN-P Translate

79.5

OoD Acc IN-P Snow

76.0

OoD Acc IN-P Rotate

78.0

75.5

81.0

77.5

AcC

79.0

75.0

77.0

80.5

78.5

74.5

76.5

80.0

74.0

78.0

79.5

73.5

76.0

15

2025

30

15

20

.25

30

15

2025

30

15

2025

30

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.79

Kendall's t correlation: 0.81

Kendall's t correlation: 0.80

Kendall's t correlation: 0.78

OoD Acc IN-P Zoom Blur

81.5

70

OoD Acc IN-P Tilt

79.0

OoD Acc IN-P Scale

72

81.0

78.5

ID ACC

69

71

80.5

78.0

80.0

68

77.5

70

79.5

69

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

#Param

#Param

#Param

Kendall's T correlation:0.80

Kendall's t correlation: 0.82

OoD Acc IN-P Motion Blur?

Kendall's T correlation: 0.77

81.5

79.0

76.0

75.0

74.5

ID ACC

81.0

78.5

75.5

74.0

80.5

75.0

78.0

73.5

80.0

74.5

79.5

77.5

77.0

74.0

73.0

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.79

Kendall's t correlation: 0.81

Kendall's t correlation: 0.75

Kendall's t correlation: 0.62

OoD Acc IN-P Shot Noise

79.0

81.5

78.5

OoD Acc IN-Sketch

33

OoD AUPR IN-O

2.200

2.205

81.0

ID ACC

32

2.195

78.0

80.5

2.190

31

80.0

77.5

2.185

77.0

30

2.180

79.5

2.175

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.79

Kendall's T correlation: 0.69

Kendall's t correlation: 0.74

Kendall's T correlation: 0.62

81.5

14

OoD Acc Stylized-IN

18.0

31.5

OoD Acc IN-A

13

OoD Acc IN-R

17.5

81.0

31.0

ID ACC

2.1

17.0

80.5

30.5

16.5

80.0

10

30.0

16.0

29.5

79.5

9

15.5

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.13

OoD Acc IN-C Gaussian Noise

Kendall's t correlation: 0.19

Kendall's t correlation:0.16

Kendall'sT correlation:0.06

82.2

82.3

Snow

51.0

51.5

OoD Acc IN-C Fog

64

48

ACC

82.1

50.5

62

82.0

47

50.0

61

81.9

46

60

81.8

49.0

59

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.13

Kendall's T correlation:-0.04

Kendall's T correlation: 0.04

Kendall's T correlation:-0.08

Blur

OoD Acc IN-C Brightness

82.3

OoD Acc IN-C Frost

58.0

OoD Acc IN-C Motion B

37

73.0

82.2

36

72.8

ACC

82.1

57.5

35

72.6

82.0

34

81.9

57.0

72.4

81.8

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.10

Kendall's T correlation:-0.37

OoD Acc IN-C Impulse Noise

Kendall's T correlation: 0.07

OoD Acc IN-C Glass Blur

82.3

48

21.0

82.2

31.5

47

20.5

OoD Acc IN-C [

31.0

20.0

46

82.0

45

81.9

30.5

19.5

44

81.8

30.0

19.0

43

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.13

Kendall's T correlation: 0.03

Kendall's T correlation:0.04

Kendall's T correlation: 0.19

ShotNoise

Blur

82.3

44

37

82.2

OoD Acc IN-C Motion I

34.5

36

OoD Acc IN-C S

43

35

OoD Acc IN-C 2

34.0

82.0

33.5

81.9

42

34

81.8

41

33

33.0

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

OoD Acc IN-C Elastic Transfoi

Kendall's T correlation: -0.12

Kendall's t correlation: 0.25

82.3

60.0

41.0

OoD Acc IN-C Contrast

51

82.2

50

59.5

40.5

OoD Acc IN-C Jpeg (

49

59.0

40.0

82.0

8

81.9

58.5

39.5

48

81.8

58.0

39.0

47

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.13

Kendall's T correlation:-0.03

OoD Acc IN-D Background

Kendall's T correlation: -0.31

Kendall's T correlation: -0.18

82.3

Texture

13

OoD Acc IN-D Material

20

16

82.2

OoD Acc IN-D

14

12

18

16

82.0

12

11

81.9

81.8

10

10

。

：

：

12

14

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.13

Kendall's T correlation:0.03

Kendall's t correlation:0.06

82.2

82.3

80.2

OoD Acc IN-P Snow

76.4

OoD Acc IN-P Rotate

78.8

ACC

2 82.1

OoD Acc IN-P T

80.1

76.2

82.0

80.0

76.0

78.6

81.9

79.9

75.8

81.8

79.8

78.4

75.6

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.13

Kendall's T correlation: 0.31

Kendall's t correlation: 0.02

Kendall's t correlation:0.36

ZoomBlur

82.3

72.0

79.8

IN-P Scale

74.0

82.2

71.8

OoD Acc IN-P

79.7

73.5

79.6

82.0

81.9

OoD Acc IN-P Z

71.6

71.4

79.5

79.4

OoD

AcC

73.0

72.5

71.2

79.3

72.0

81.8

71.0

71.5

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's T correlation:-0.04

Kendall's t correlation: 0.01

Kendall's T correlation: 0.10

Brightness

i

82.3

80.2

77.4

OoD Acc IN-P Motion l

B

76.0

82.2

80.1

77.3

75.9

80.0

77.2

75.8

OoD Acc IN-PI

82.0

ACc IN-P

79.9

77.1

?

75.7

81.9

79.8

77.0

75.6

81.8

OoD

79.7

76.9

75.5

：

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.13

Kendall's T correlation: 0.07

Kendall's t correlation: 0.01

Kendall's T correlation: 0.50

82.3

OoD Acc IN-P Shot Noise

2.230

82.2

80.0

OoD Acc IN-Sketch

8

OoD AUPR IN-O

2.225

79.9

32

2.220

79.8

82.0

79.7

31

8.

2.215

81.9

79.6

2.210

81.8

79.5

：

30

.

2.205

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.13

Kendall's t correlation: 0.45

Kendall's T correlation: -0.00

Kendall's T correlation: 0.05

82.3

33.0

Acc Stylized-IN

20.0

82.2

OoD Acc IN-A

14.5

OoD Acc IN-R

32.8

19.8

32.6

14.0

19.6

82.0

32.4

81.9

13.5

32.2

OoD

19.4

81.8

13.0

32.0

19.2

50

60

70

50

60

70

50

60

70

50

60

70

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.47

GaussianNoise

Kendall's T correlation: 0.33

Kendall's t correlation: 0.31

Kendall's t correlation: 0.30

46

75.4

OoD Acc IN-C Snow

37.0

Acc IN-C Fog

20

36.5

44

75.2

AcC

75.0

74.8

OoD Acc IN-C (

19

36.0

35.5

OoD

42

40

福

74.6

18

35.0

38

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.47

Kendall's t correlation:0.55

OoD Acc IN-C Motion Blur

Kendall's t correlation: 0.12

Kendall'st correlation:0.55

75.4

OoD Acc IN-C Frost

43.5

."

25.0

24.5

63.00

43.0

75.2

24.0

62.75

AcC

75.0

42.5

23.5

IN-CE

62.50

74.8

42.0

23.0

Acc

62.25

41.5

74.6

22.5

：

OoD

62.00

41.0

22.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.47

Kendall's T correlation:-0.01

Kendall's T correlation:0.09

Impulse Noise

Kendall's T correlation: 0.29

10.50

20.5

ID ACC

75.4

75.2

20.0

19.5

ACC IN-C Gla

10.25

10.00

?

20

21

75.0

9.75

IN-C

74.8

74.6

19.0

18.5

9.25

9.50

OoD

AcC

19

18

18.0

9.00

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.47

Kendall's T correlation: 0.29

OoD Acc IN-C Motion Blur

25.0

Kendall's T correlation:0.12

Kendall's T correlation:-0.15

75.4

21

24.5

B

23.00

ID ACC

75.2

75.0

OoD Acc IN-C S

20

24.0

23.5

22.75

22.50

8

22.25

74.8

74.6

19

23.0

22.5

：

OoD

A

U

21.75

22.00

18

22.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.12

Kendall's T correlation: 0.29

44.0

OoD Acc IN-C Contrast

75.4

25.5

35

43.5

30

ID ACC

75.2

25.0

25

75.0

42.5

24.5

20

74.8

.

15

74.6

42.0

.ao.

24.0

10

41.5

6.0

6.5

6.0

6.5

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.47

Kendall's T correlation: 0.01

-i  o

14

Kendall's t correlation:-0.00

Kendall's T correlation: 0.11

Texture

OoD Acc IN-D Material

75.4

20

13

18

ID ACC

75.2

OoD Acc IN-D

18

12

16

75.0

16

11

14

74.8

14

10

12

74.6

12

9

10

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

OoD Acc IN-P Translate

73.2

OoD Acc IN-P Snow

68.0

OoD Acc IN-P Rotate

71.0

75.4

67.8

70.8

73.0

AcC

75.2

67.6

70.6

75.0

72.8

67.4

70.4

74.8

72.6

67.2

70.2

74.6

72.4

67.0

70.0

66.8

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

75.4

60.75

OoD Acc IN-P Tilt

72.4

Scale

63.50

75.2

60.50

72.2

OoD Acc IN-P

63.25

AcC

75.0

IN-P

60.25

72.0

63.00

60.00

74.8

Acc

71.8

62.75

74.6

59.50

59.75

...

71.6

62.50

62.25

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

OoD Acc IN-P Brightness

68.8

OoD Acc IN-P Motion Blur

67.4

72.6

75.4

68.6

67.2

72.4

ID ACC

75.2

68.4

67.0

72.2

75.0

68.2

66.8

72.0

66.6

74.8

68.0

71.8

66.4

74.6

71.6

67.8

66.2

：

6.0

6.5

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

OoD Acc IN-P Shot Noise

72.2

75.4

72.0

.

OoD Acc IN-Sketch

24.5

OOD AUPR IN-O

2.175

24.0

75.2

71.8

ID AcC

2.170

75.0

71.6

23.5

74.8

71.4

23.0

2.165

74.6

71.2

71.0

22.5

2.160

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

5.25

75.4

25.0

OoD Acc Stylized-IN

13.2

OoD Acc IN-A

5.00

OoD Acc IN-R

ID ACC

75.2

4.75

24.5

13.0

12.8

75.0

4.50

24.0

74.8

4.25

12.6

23.5

12.4

74.6

4.00

3.75

12.2

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

6.0

6.5

7.0

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.79

OoD Acc IN-C Gaussian Noise

Kendall'sT correlation:0.58

Kendall's T correlation: 0.64

Kendall's t correlation: 0.72

81.5

OoD Acc IN-C Snow

50

OoD Acc IN-C Fog

36

60

81.0

49

Acc

34

58

80.5

80.0

32

47

56

囍

79.5

30

46

54

.

15

2025

30

15

2025

30

15

20

25

30

15

2025

30

#Param

#Param

#Param

#Param

Kendall's t correlation: 0.79

Kendall's t correlation:0.54

OoD Acc IN-C Motion Blur

Brightness

Kendall'st correlation:0.73

71.5

81.5

OoD Acc IN-C Frost

55

71.0

AcC

81.0

54

32

70.5

OoD Acc IN-C I

80.5

31

70.0

80.0

2

69.5

30

79.5

51

69.0

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.47

Kendall's T correlation: 0.23

Impulse Noise

Kendall's T correlation: 0.52

OoD Acc IN-C Glass Blur

18.5

81.5

29.5

18.0

29.0

36

81.0

17.5

ID ACC

28.5

80.5

28.0

17.0

OoD Acc IN-C I

34

27.5

16.5

80.0

27.0

16.0

32

79.5

26.5

15.5

30

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.79

OoD Acc IN-C Shot Noise

Kendall's T correlation: 0.59

OoD Acc IN-C Motion Blur

Kendall's T correlation: 0.22

OoD Acc IN-C Zoom Blur

Kendall's T correlation: 0.57

32

81.5

33

34

81.0

31

ID ACC

32

32

80.5

30

31

80.0

品

30

29

79.5

28

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

Compression

#Param

Transform

#Param

#Param

Kendall's T correlation: 0.79

Kendall's T correlation: 0.61

Kendall's T correlation: 0.60

Kendall's t correlation: 0.49

81.5

54

36

OoD Acc IN-C Contrast

48

81.0

53

OoD Acc IN-C Elastic T

35

46

ID ACC

OoD Acc IN-C Jpeg C

80.5

52

34

44

80.0

51

42

79.5

50

32

40

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

#Param

#Param

#Param

Kendall's T correlation: 0.79

Kendall's T correlation: -0.33

Kendall's T correlation: -0.15

Kendall's T correlation: 0.12

AcC

81.5

81.0

OoD Acc IN-D Texture

20

14

13

15

OoD Acc IN-D Material

22

20

80.5

12

18

D

80.0

14

11

16

10

79.5

12

14

15

20

25

30

15

20

25

30

15

20

25

30

15

20

25

30

#Param

#Param

#Param

#Param

OoD Acc IN-C Gaussian Noise

36

34

32

35.

33.26

32.94

OoD Acc IN-C Snow

48

50

NN.

49.62

OoD Acc IN-C Fog

58

5

60

59.10

m..

60.09

57.

Atan,brly

59.76

Mvi

60.64

56.65

`M

60.51

54

53

52

55

52.

50

52.92

54.

52.

78

54.

30

。…

46

54

51

12

13

14

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

Depth

18.5

OoD Acc IN-C Motion Blur

OoD Acc IN-C Brightness

71.0

71.5

71.01

71.34

71.36

OoD Acc IN-C Defocus Blur

29.5

29.0

828.99

OoD Acc IN-C Glass Blur

18.0

32

31

30

Wnt'y

31

69.5

70.5

69.0

70.0

69

.59

69

78

27.0

28.0

27.5

26.5

28.5

27.94°

16.0

16.5

17.0

17.5

15.5

16.523

.03

16.65

-17.96.98

16.67

17.0%.98

3

12

13

14

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

Depth

OoD Acc IN-C Impulse Noise

36

34

MRA'!

35.27

35.88

??

35.

.78

OoD Acc IN-C Shot Noise

5 4.3.20

33.

OoD Acc IN-C Pixelate

46

44

48

47.

.72

48.40

47.86

48.51

47.

7.948.48

OoD Acc IN-C Zoom Blur

31

32

31.54

30.715.

31.27

31.

25

2

32.78

32.79

31

30.62

*.'!

40

2

42.51

29.41

29.

24

29.23

29

30

38

12

13

14

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

Depth

OoD Acc IN-C Jpeg Compression

54

53.80

36

48

20

OoD Acc IN-C Contrast

46.546.45

46.46.31

OoD Acc IN-D texture

20

53

53.25

53.

w.b,v

35

46

45.30

45.71

52

5

51.48

33

34

32.81

32.85

32.92

42

44

42.65

43.693

43

3.97!

14

16

16.

！

50

40

12

12

13

14

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

Depth

15

OoD Acc IN-D background

OoD Acc IN-D material

22

20

embedding_dim

320

1

12

10

11.84

16

18

17.24

18.07

384

448

10

14

12

13

14

12

13

14

Depth

Depth

OoD Acc IN-P Translate

79.5

79.22

79.48

79.35

79.57

79.59

OoD Acc IN-P Snow

75.5

76.0

75.41

75.75

75.50

75.80

75.53

75.81

OoD Acc IN-P Rotate

78.0

77.68

77.94

77.81,

78.03

78.02

OoD AccIN-P Zoom Blur

70.0

70.5

99.69

70.15

69.82

70.28

69.83

70.30

77.5

79.0

75.0

69.0

77.0

78.5

78.46

78.53

74.5

74.16

74.38

76.5

76.72

76.79

68.0

68.5

689

68.43

74.0

78.0

67.5

73.5

76.0

12

13

14

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

Depth

OoD Acc IN-P Tilt

79.0

78.5

78.98

78.86

79.07

OoD Acc IN-P Scale

72.0

71.5

71.84

71.85

72.14

OoD Acc IN-P Gaussian Noise

78.5

79.0

78.5

79.00

E6:8L

79.11

75.5

75.61.

75.95

76.11

76.14

71.0

78.0

77.64

70.0

70.5

66.69

78.0

77.73

77.

66

74.5

75.0

77.5

77.5

69.5

74.0

69.0

77.0

12

13

14

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

Depth

OoD Acc IN-P Motion Blur

74.5

75.0

74.41

74.68

74.83

74,60!

74.85

OoD Acc IN-P Shot Noise

78.5

79.0

78.4.3

78.72

78.63

78.83

OoD Acc IN-Sketch

33

32.

45

33.04

N.

3321

.

OoD AUPR IN-O

2.200

2.205

2.20

2.20

2.20

32

2.195

2.19

74.0

73.5

73.

73.57

77.5 77

78.0

39

77.65

31

30-54

0

4vym

30.89

Pa

2.190

2.185

2.18

2.18

73.0

77.0

30

2.180

2.175

12

13

14

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

Depth

3

14

13.53

31.5

31.17

31.24,

31.59

31.

OoD Acc Stylized-IN

17.5

18.0

17,

17.37

17.54

17.40

17.54

OoD Acc IN-A

11.10

11.22

OoD Acc IN-R

31.0

30.5

30.75

30.24

30

17.0

16.48

16.53

embedding_dim

384

320

448

10

30.0

29.74

16.5

16.30

29.5

16.0

9

15.5

12

13

14

12

13

14

12

13

14

Depth

Depth

Depth

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See Sec. 3 for our OoD-ViT-NAS benchmark, and Sec. 4 for our analysis. The summary can be found in Fig. 2, Fig. 1, and Tab. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] ,

Justification: See Appx. B

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Appx. M

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Appx. A

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Appx. M

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Tab. 2. Our benchmark and analysis are conducted across datasets and search space.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appx. M, ∼ 3900 GPU hours on a single Nvidia RTX A6000

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Appx. B

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Appx. E and Appx. M for description of dataset and experimental setups, respectively. We do not mention the license of the assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: See Appx. A for our OoD-ViT-NAS benchmark. See Sec. 3, and Appx. E for the description of the benchmark.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper shows that in-domain accuracy and Training-free NAS accuracy predictions correlate poorly with out-of-domain accuracy, while characteristics of the model such as Flops, Params and Embed-Dim (number of channels) correlate much better with out-of-domain accuracy. To do this, they train a supernet that contains all possible architectures (subnets), then they validate these subnets on 9 datasets ImageNet/C/D/P/O/A/R/Sketch/Stylaized.\n\nSTRENGTHS:\nThey explore various ways for increasing OOD accuracy, which is most important for real-world problems. They show that conventional approaches, such as Training-free NAS or relying on in-domain accuracy, do not work. While they found that Flops, Param and Embed-dim correlated well with OOD accuracy. To do this, they introduce OoD-ViT-NAS benchmark for NAS research on ViT’s OOD generalization, that includes 3000 diverse ViT architectures which are evaluated on 9 datasets. This allows us to look for new and better approaches for finding new models structures to improve OOD accuracy.\n\nWEAKNESSES:\nYou show that the embedding dimension has the highest impact among ViT architectural attributes, while network depth has a slight impact on OoD generalization. But the paper lacks an explanation that although embed-dim (number of channels) correlates with OOD accuracy better than depth, this does not necessarily mean that to increase OOD accuracy only embed-dim should be increased, since the computational cost and memory costs may be different for embed-dim and depth. Therefore, to create either the largest and most accurate model that fits in memory, or the most optimal model in terms of speed and accuracy, it may be optimal to increase the depth rather than embed-dim. There are also papers [1,2,3] that show that it is necessary to simultaneously increase several model parameters (depth, number of channels, resolution) in optimal proportions.\n\nYou present charts of OOD accuracy and number of parameters (Fig. 4, K19 - K24), but it would also be great to present charts of OOD-accuracy and latency, and OOD-accuracy and GPU memory consumption. Because these 3 model characteristics (OOD-accuracy, Latency, Memory consumption) are the most important for real-world problems.\n\n\n1. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, Mingxing Tan, Quoc V. Le, 2019\n2. EfficientNetV2: Smaller Models and Faster Training, Mingxing Tan, Quoc V. Le, 2021 \n3. EfficientDet: Scalable and Efficient Object Detection, Mingxing Tan, Ruoming Pang, Quoc V. Le, 2019\n\nQUESTIONS:\n- Have you tried plotting OOD-accuracy versus Latency or GPU memory consumption while increasing various model parameters such as: embed_dim, depth, MLP-ratio, num of heads, etc.? Or find correlations between ODE accuracy and these parameters, taking into account their impact on latency and memory consumption.\n\n- Have you tried to find optimal model scaling factors (similar to works [1, 2, 3]) to increase OOD-accuracy in optimal way with respect to latency or memory consumption?\n\n\n1. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, Mingxing Tan, Quoc V. Le, 2019\n2. EfficientNetV2: Smaller Models and Faster Training, Mingxing Tan, Quoc V. Le, 2021 \n3. EfficientDet: Scalable and Efficient Object Detection, Mingxing Tan, Ruoming Pang, Quoc V. Le, 2019\n\nLIMITATIONS:\nIt might be worth adding to the limitations:\nThe results obtained on Imagenet classification datasets may not correlate with the results of real-life problems. Because Imagenet classification requires to predict the class of only one usually large object in the low resolution image without predicting its location (box, polygon or mask), whereas in real problems it is usually necessary to predict the class and position of many objects, incl. very small in high resolution image. Moreover, there is ambiguous in the Imagenet classification task, the class of which of the many objects in the image should be predicted?\nThus, a continuation of the research in this direction may be the finding of correlations, NAS approaches, new network structures, their parameters and scaling factors to achieve the highest OOD accuracy on tasks close to real ones (such as Dense prediction tasks: Segmentation, Detection, etc. ) in an optimal way with respect to latency and memory consumption.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

comments: SUMMARY:
This paper introduces OoD-ViT-NAS, a benchmark designed for evaluating ViT architectures' ability to generalize under Out-of-Distribution (OoD) shifts. This paper reveals that ViT design significantly impacts OoD generalization, In-Distribution (ID) accuracy does not reliably predict OoD performance, and simpler metrics like parameter count can sometimes predict OoD accuracy better than complex training-free NAS methods.

STRENGTHS:
This paper propose a new benchmark for ViT NAS.

WEAKNESSES:
1. While the paper introduces a new benchmark, it doesn't sufficiently discuss the practical implications of the findings or potential applications. Including a section on how these insights can influence future ViT designs would add significant value. Furthermore, assessing whether ViTs designed based on these insights outperform those designed by humans would be meaningful to include in experiments and results.
2. The analysis of the experimental results is somewhat superficial. It presents observations without delving into deeper discussions. There should be more profound insights and detailed discussions on why certain ViT architectural attributes perform better under OoD conditions. This could involve providing theoretical justifications or proposing hypotheses that could be explored in future research.
3. The paper states that simple proxies like #Param and #Flop outperform more complex Training-free NAS methods but does not provide enough insight into why this might be the case. A deeper analysis of these findings, including possible reasons and implications, would strengthen the argument.

QUESTIONS:
Please address the weaknesses mentioned above.

LIMITATIONS:
No potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the out of distribution generalization of vision transformer architecture designs. Specifically the paper studies how in-distribution accuracies relate to OOD accuracies for a set pf 3000 architectures. In addition the paper also studies the correlation between different zero-cost proxies and OOD accuracies (on different variants of imagenet), observing that simply parameters or FLOPs are good proxies. Furthermore, the paper also studies the impact of different architectural decisions on the OOD accuracies, discovering that embedding dimension is of utmost importance for OOD generalization. The authors release code and the raw results to reproduce their experiments on AutoFormer-T,B and S space.\n\nSTRENGTHS:\n- The paper is the first one to study out-of-ditribution generalization of architectures discovered by NAS\n- The paper also studies zero cost proxies on the ViT space showing ineffectiveness of modern/newer ZCPs\n- The paper is written in a clear and easy to understand manner\n- Experiments on ZCPs are thorough and the results are insigntful\n\nWEAKNESSES:\n- Currently I am missing the potential use-cases of the benchmark. The paper is more of a study and not a method (and fits better in the datasets and benchmark track of NeurIPS). Also though the paper is posed as a benchmark,I miss the intended use-case of the benchmark. Do the authors intend on releasing surrogates based on proxies, to allow API based querying these metrics for any possible architecture?\n- The authors only study 3000 architectures, which are too few to lead to generalizable insights.\n- Analysis of architecture importance: The search space of AutoFormer is a bit biased in the sense that there is a larger (factor two) variabilities in embedding dimension and a less variability for number of layers. I wonder if this is the reason embedding dimension has a higher correlation with OOD accuracy compared to other architecture dimensions like number of heads, embedding dimensions?\n- Multi-Objective Search: Since inheriting weights of a supernet and evaluation is fast, generally one would perform multi-objective random or evolutionary search directly on the pretrained supernet. This would lead to different Pareto-Fronts for different dataset distributions. Could the authors also present the results of multi-objective search directly on the OOD datasets?\n- What is the cost/search time different in computing Pareto-Fronts based on MO-search v/s Zero-Cost-Proxies (a lot of these proxies are actually not zero cost)?\n\nQUESTIONS:\n- Do the authors train their own supernet, or train a one-shot supernet from scratch?\n- How does this benchmark contribute to aiding development of future NAS techniques for OOD generalization?\n- Check weakness\n\nLIMITATIONS:\nThe authors discuss limitations and broader impact sufficiently.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presented OoD-ViT-NAS, a comprehensive NAS benchmark with a focus on out-of-distribution generalisation of vision Transformer architectures.  The authors created a benchmark with 3000 diverse ViT architectures evaluated on 8 common large-scale OoD datasets, and provided the first comprehensive investigation of how ViT architectures affect OoD generalisation.  They also conducted the first study exploring NAS for ViT’s OoD, analysing the impact of individual ViT architectural attributes on OoD generalisation.\n\nSTRENGTHS:\nThis study introduces the first large-scale benchmark for ViT OoD generalisation, which could be a valuable resource to the NAS field.\n\nThe authors conducted a thorough investigation on the impact of ViT architectures in relation to OoD generalisation, covering multiple aspects such as architectural attributes, correlation between ID and OoD performance and the effectiveness of training-free NAS methods.\n\nThis study reported several findings, such as the low correlation between ID and OoD accuracy, the superiority of simple proxies, i.e., model params and FLOPs, over more computationally complex training-free metrics. Although the latter has been revealed in other work on computer vision tasks, the authors further validate that point on the OoD tasks.\n\nThe reproducibility of this work is good as the authors provided a substantial amount of details of their methodology, hyper-parameters and computational resources.\n\nWEAKNESSES:\nThis study is limited to the Autoformer architecture space, which may limit the generalisability of the findings to other ViT architectures.  Also it comprises the novelty of this work.\n\nAlthough the empirical studies are comprehensive, there are not many theoretical explanations for the observed phenomena.  More analysis and insights should be provided to support this work.\n\nThe presentation can be improved.  For example, Fig 1 appears on Page 2, but is first mentioned at Line 226, Page 7.\n\nQUESTIONS:\nLine 119 shows that the work is based on Autoformer.  What is the reason behind this choice? What is new?  \n\nLine 145, OoD Classification Accuracy, what is the difference to the ID Acc in Line 142?  Maybe a formula could clearly explain that.\n\nLine 154, we make use of the One-Shot NAS approach.  Why not use zero shot NAS instead?  What are the benefits that one short NAS can bring but not zero shot NAS?\n\nTable 2 shows correlations of some zero proxies.  Can SWAP be applied here as it demonstrates a much higher correlation than those on the table? SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS. Peng et. al. ICLR'24.\n\nLIMITATIONS:\nThe authors discussed the limitations of this work, but in the appendix instead of the main text.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}