Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights
Authors: Sy-Tuyen Ho
Tuan Van Vo
Somayeh Ebrahimkhani
Ngai-man Cheung
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision Transformer
Neural Architecture Search
Out-of-Distribution Generalization
Primary area: deep_learning_architectures
TL;DR: This work introduces OoD-ViT-NAS, the first systematic ViT NAS benchmark designed for out-of-distribution (OoD) generalization, and provide analytical insights on how ViT architectures impact OoD performance.

Abstract: While Vision Transformer (ViT) have achieved success across various machine learning tasks, deploying them in real-world scenarios faces a critical challenge: generalizing under Out-of-Distribution (OoD) shifts. A crucial research gap remains in understanding how to design ViT architectures – both manually and automatically – to excel in OoD generalization. **To address this gap,** we introduce OoD-ViT-NAS, the first systematic benchmark for ViT Neural Architecture Search (NAS) focused on OoD generalization. This comprehensive benchmark includes 3,000 ViT architectures of varying model computational budgets evaluated on common large-scale OoD datasets. With this comprehensive benchmark at hand, we analyze the factors that contribute to the OoD generalization of ViT architecture. Our analysis uncovers several key insights. Firstly, we show that ViT architecture designs have a considerable impact on OoD generalization. Secondly, we observe that In-Distribution (ID) accuracy might not be a very good indicator of OoD accuracy. This underscores the risk that ViT architectures optimized for ID accuracy might not perform well under OoD shifts. Thirdly, we conduct the first study to explore NAS for ViT’s OoD robustness. Specifically, we study 9 Training-free NAS for their OoD generalization performance on our benchmark. We observe that existing Training-free NAS are largely ineffective in predicting OoD accuracy despite their effectiveness at predicting ID accuracy. Moreover, simple proxies like #Param or #Flop surprisingly outperform more complex Training-free NAS in predicting ViTs OoD accuracy. Finally, we study how ViT architectural attributes impact OoD generalization. We discover that increasing embedding dimensions of a ViT architecture generally can improve the OoD generalization. We show that ViT architectures in our benchmark exhibit a wide range of OoD accuracy, with up to 11.85% for some OoD shift, prompting the importance to study ViT architecture design for OoD. We firmly believe that our OoD-ViT-NAS benchmark and our analysis can catalyze and streamline important research on understanding how ViT architecture designs influence OoD generalization. **Our OoD-NAS-ViT benchmark and code are available at [https://hosytuyen.github.io/projects/OoD-ViT-NAS](https://hosytuyen.github.io/projects/OoD-ViT-NAS)**

Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights
Sy-Tuyen Ho
Somayeh Ebrahimkhani Ngai-Man Cheung" Singapore University of Technology and Design (SUTD)
Tuan Van Vo*
aetuyen__h@@yyneg sutd. edu.sg,
soraaautuaauummmlloooooaaaaaaaaa angiamimmnaiiiiimaaann...m aegaaiaamnchhnngaaa edu. sg
Abstract
While Vision Transformer (ViT) have achieved success across various machine learning tasks, deploying them in real-world scenarios faces critical challenge: generalizing under Out-of- Out-of-Distribure Distribbution (OoD) shifts. A crucial research gap re- mains in understanding how design VIT architectures both manually and automatically to excel in OoD generalization. To address this gap, we introduce OoD- ViT-NST- the first systematic benchmark for VIT Neural Architecture Search (NAS) focused on OoD generalization. This comprehensive benchmark includes 3, 000 VIT architectures common large-scale OoD datasets. With this comprehensive benchmark at hand, we analyze the factors that contribute to the OoD generalization of ViT architecture. Our analysis uncovers several key insights. Firstly, we show that VIT architecture designs have a considerable impact on OoD generalization. Secondly, we observe that In In-Distribution (ID) accuracy might not be very good indicator of OoD accu- racy. This underscores the risk that ViT architectures optimized for ID accuracy might not perform well under OoD shifts. Thirdly, conduct the first study explore NAS for ViT's OoD robustness. Specifically, we study 9 Training-frr NAS for their OoD generalization performance isting Training-free NAS are largely ineffective in predicting OoD accuracy despite their effectiveness predicting ID accuracy. Moreover, simple proxies like #Param or #Flop surprisingly outperform more complex Training-fres NAS in predicting ViT' OoD accuracy. Finally, study how VIT architectural attributes impact OoD generalization. We discover that increasing embedding dimensions of VIT architecture generally can improve the OoD generalization. We show that ViT archi- tectures in our benchmark exhibit wide range of OoD accuracy, with to 1 11.85% for some OoD shift, prompting the importance to study VIT architecture design for OoD. We firmly believe that our SOD-ViT-NAS benchmark and our analysis can catalyze and streamline important research on understanding how ViT architecture designs influence OoD generalization. Our BOD-NAS-ViT benchmark and code are available at ahttss//lorriygggghhhoooooeoeoss
varying model computational budgets evaluated on 8
benchmark. We observe that
Introduction
Vision Transformers (ViT) [ have recently achieved impressive results and become a major area of research in computer vision. with significant efforts towards understanding how ViT works. These efforts have led to the proposal of both manually designed architectures [I] 2 3
or
cooomated-searchaen architectures 6
I0 to advance VIT architectures.
"Equal Contribution
Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) ID as indicator for VIT OoD Generalization
(c) OoD Generalization VIT Architecture Attributes
(b) Training-free NAS for VIT OoD Generalization
1.00
IN-C
0.33
0.27
0.27
0.12
0.01
0.44
0.44
0.09
0.24
0.22
0.37
-0.0
0.11
0.08
-33
-0.12
IN-P
0.67
0.45
0.46
0.14
0.13
0.37
0 36
11
0.41
0.4
0.45
0.22
0.17
0.06
1.
0.59
0.46
0.6
0.03
935
IN-A
0.4
0.48
0.45
0.14
0.32
0.38
0.36
0.00
0.44
0.42
0.58
0.66
0.04
0.09
-006
0.43
0.41
0.09
0.36
0.34
0.30
0.36
0.27
0.13
-025
IN Skotch
0.37
0.39
0.39
0.14
0.08
0.31
0.332
0.1
0.38
0.36
0.59
0.17
0.12
0.07
-0.5
Stylized
0.36
0:33
0.35
0.35
0.09
0.09
-45
Average DoD 0.34
0.36
0.35
0.12
0.10
0.30
0.29
0.08
0.34
0.33
0.45
0.19
0.09
0.07
-1.50
1D Acc
#Param&Flops Grasp
McCo
CroZe
SNIP
jacobian DSS AutoProx
Embnd
Depth
MLP
#Hoad
Dim
Ratio
Figure 1: We propose, OOD-VAT- the first comprehensive benchmark for NAS on OoD generalization of ViT architectures. Then, we comprehensively investigate OoD generalization for ViT. The detailed of OoD datasets in our investigation can be found in Tab. I In this figure. we show the Kendall ranking correlation between OoD accuracy of different datasets on the left and different quantities the bottom. Our analysis uncovers several key insights. (a) ID as an indicator for ViT OoD Generalization (Sec. 4.2) We show that the correlation between II) accuracy and Ool accuracy is not very high. This suggests that current architectural insights based on II) accuracy might not translate well to OoD generalization.
(b) Training-free NAS for ViT OoD Generalization. (Sec. 4.3) We conduct the first study of NAS for ViT's OoD generalization, showing that their effectiveness significantly weakens in predicting OoD accuracy. (c) OoD Generalization VIT Architectural Attributes. (Sec. 4.4) Our first study on the impact of ViT architectural attributes on OoD generalization shows that the embedding dimension generally has the highest correlation with OoD accuracy among ViT architectural attributes. Additional results can be found in the Appx.
Research Gap. Existing research on VIT architectures focuses maximizing In-Distribution (ID) ac- curacy. while studies on the impact of ViT architectures on tal-o--DDDrributtonn (OoD) generalization limited. Initial works [3]. 14, and [5] evaluate sets of 3. 10 and 22 human-designed ViT archi- tectures under OoD settings, respectively, and provide coarse insights into which models exhibit better OoD generalization. However, with very limited VIT architectures studied in previous works, the in- fluence of ViT structural attributes (e.g., embedding dimension, number of heads, MLP ratio, number of layers) on OoD generalization remains unclear. Besides, in the context of ViT Neural Architecture 18 19, 20 9]
Search (NAS), while there are various ViT NAS for ID accuracy [6] 7 16 g 10 there is no study on ViT NAS for OoD generalization.
In this paper, we address existing research gaps by introducing ViT-VAT- the first compre- hensive benchmark specifically designed for VIT's OoD generalization. Building NAS benchmarks is notoriously time-consuming and expensive due to the need to train and evaluate every candidate architecture. This challenge is particularly acute for ViT, known for its high computational demands and memory usage [21] 22 23 To overcome this bottleneck, we propose leveraging One-Shot NAS, specifically AutoFormer (6. a widely used ViT search space. We sample diverse set of aub-architecture (models) to populate our benchmark. Importantly, these subnets inherit the weights from the pre-trained supernets, and their performance has been shown be comparable to, or even superior to, that of architectures trained alone 220 6 This approach enables us to efficiently acquire large pool of ViT architectures for OoD generalization analysis. Using DoD-ViT-NAS we conduct extensive OoD generalization analysis and gain several key insights. Additionally, with our benchmark, our work is the first to explore training-freee NAS for ViT's OoD generalization. Our contributions are summarized below:
We introduce OoD- ViT-NAS, the first comprehensive benchmark designed for NAS research on ViT's OOD generalization. This benchmark includes 3, 000 diverse VIT architectures sampled from the widely used ViT search space [6] These architectures span wide range of computational budgets. To thoroughly benchmark OoD generalization, these architectures are evaluated on the 8 most common and state-of-the-art (SOTA) OoD datasets: ImageNet-C
24). ImageNet-A 25),, ImageNet-O 25 ImageNet-P (4), ImageNet-D (26, ImageNet-R
Imagehet-Sketch (28). and Stylized ImageNet
29 (Sec.

---

Our analysis demonstrates the significant influence of ViT architectural designs on OoD accuracy. This observation encourages future research focus more on ViT architecture research for OoD generalization (Sec. 4.1)
We show that high In-Distribution (ID) accuracy is not a very good indicator of OoD accuracy. This suggests that current architectural insights based on ID accuracy might not translate well to OoD generalization (See. (4.))
We conduct the first study to explore NAS for VIT's OoD generalization. We study 9 Training-free NAS for their OOD generalization performance on our benchmark. We observe that despite their prediction accuracy for ID, their effectiveness significantly weakens when predicting OoD accuracy. Furthermore, simple proxics such as the number of parameters (#Param) or the number of floating point operations (#Flop) surprisingly outperform more complex Training-free NAS in predicting ViT's OoD accuracy (Sec.
We study the impact of VIT architecture design OoD generalization and demonstrate that careful design of ViT architectures can significantly improve OoD generalization. Specifically, increasing the embedding dimensions of ViT architecture generally can improve its Ool generalization. (Sec. 4.4) We show that architectures with comparable ID accuracy (within an averaging range of 1.39%) exhibit wider range of OoD accuracy, averaging 3.80% and reaching highs of 11.85%, being comparable or even outperforming state-of-the-art (SOTA) training OoD generalization method, such those based domain invariant representation learning 330 31]. For example, under the same OoD setting, the SOTA method [30 shows an improvement of 1.9% OoD accuracy.
2 Related Work
cattof-Distribution (OoD) Generalization. Addressing Out-of-diitributiin (OoD) generalization is challenge, particularly in computer vision. Various approaches have been proposed to tackle this issue. A common strategy focuses on learning features that remain consistent across different domains, thereby promoting generalizability [31
32
33
35
36
37
Other directions explore
distributional robustness 38
39 model ensembles [40, 41) test- rest-time adaptation
42 43
data aug-
mentation techniques 44
45
46
47
48
49 [5], and meta learning [51 52 for OoD generalization.
From an architectural perspective. a few attempts investigate the impact of network architecture on OoD generalization. Early work [53] shows that over- wver-parameterized networks can hinder OoD performance due to overfitting. This raises an intriguing question: can sub-networks within such architectures achieve better OoD performance? Inspired by the Lottery Ticket Hypothesis (LTH) [54] the Functional LTH has been explored and shown that over-parameterized networks harbor sub-networks with better OoD performance. Techniques like Modular Risk Minimization (55) and Debiased Contrastive Weight Pruning [56) aim to identify these winning tickets. Another direction
57 14 leverages Neural Architecture Search (NAS) analyze the OoD robust architectures. How- ever, these studies primarily focus on CNNs. While ViTs have achieved success in various visual recognition, investigations into their OoD generalization are limited. Initial works [13]
[4] and [15]
evaluate sets of 3. 10, and 22 suman-designed ViT architectures respectively, under OoD settings. Their results provide coarse insights into which models exhibit better OoD performance. However, the influence of VIT architectural attributes on OoD robustness remains unclear.
Neural Architecture Search (NAS). NAS is promising approach that has achieved remarkable success in automatically searching efficient and effective architectures for ID performance [58] 20 Recently, NAS has been explored in the context of adversarial robustness for CNNs as well With the rise of Vision Transformers (ViTs), several NAS approaches have been applied
59
60)
61
62
63
to improve ViT architectures, including Autoformer (6) S3 [16], ViTAS [8], Elastic ViT 9. DSS
10
Auto-Prox [17] and GLiT [64] Additionally, hybrid CNN-VIT architectures like HR-NAS
18
UniNet [19], and NASVIT [12] have also been explored. These efforts have shown promising
results in terms of ID accuracy. However, there has not been any work on NAS for ViT architectures
specifically for OoD generalization.
3 OoD-VIT-NAS: NAS Benchmark for ViT's OoD Generalization
In this section, we describe the construction of our VIT--VT-NAS benchmark with details on the search spaces, datasets, evaluation metrics, and protocol. Our comprehensive benchmark includes
3

---

1.06
3.3211.00
3,99/1066
3.4711.061
Base Small Tiny
2.25
7.3212.25)
7.58/2.25)
31.85/2.3
0:56
3.35(0.56)
59(00.561
2. 3.0.655
70
80
20
30
40
50
20
30
40
50
30
40
50
ID Acc. (%)
IN- C Geussian Noise Acc.
IN Impulse Noise Acc.
IN C Pixelate Acc.
(.891.00))
2:93(1.061
1.39/2.06)
1.48(1.05)
Base Small Tiny
8.00(2.25)
3.81(2255)
3. 3.3512.25)
5.21(2.25)
5.54/0.5 56)
8.8600.55 56)
(.02/0:56
1. 1.96(0.561
60 IN-P Zoom Acc.
Fog Acc.
Mation
IN-A Acc.
2.0211.06)
2:1011.061
3.79:1.06)
4.9511.06)
Base Small Tiny
71((225)
3.99(2.25)
9.6512.25)
2.55(2.25)
1.07(0.561
7.9910.561
3.64/0.59
20
30
20
30 IN-Sketch Acc.
10 20
10 20
IN-R Acc.
IN-D Texture Acc.
IN-D Background Acc.
Figure 2: Our analysis of the OoD accuracy range highlights the significant influence of ViT
architectural designs on OoD accuracy. (Sec. 4.1 The numbers within each violin plot for each sub-figure (e.g.. IN-D 9.79 (1.06), 9.65 (2.25), and 7.99 (0.56) denote the corresponding OoD (ID) accuracy range of architectures sampled from Autoformer- Tiny/Small/Base search space, respectively. AutoformerTinyySmmllBaa
See Appx. G for additional plots and results on other OoD shifts. For a fair comparison, we fix the same range for the x-axis across all sub-figures. We include the ID accuracy range in the top-left sub-figure for reference. On average, the OoD accuracy across all shifts is 8%/4..6%/ 74% for the search spaces in our OoD- ViT-NSS benchmark. This range is comparable to and even surpasses the current SOTA method based on domain-invariant representation learning (30), which achieved 1.9% improvement in OoD accuracy under similar settings.
3. 000 ViT architectures of varying sizes evaluated on 8 widely used large-scale, high-resolution, and SOTA OoD datasets. Our VIT--ATN benchmark summarized in the Tab. I
Search Space. We construct our benchmark based on Autoformer (6 search space. This search space is currently widely used search space in the ViT NAS community for ID data 7 65 60 10 11 17 66 677] Autoformer search space is a large vision transformer search space including five architectural attributes that define the building block. Embedding Dimension: This determines the input feature representation size and is typically consistent across layers in VIT architectures. Q-K-V Dimension: This specifies the size of the query, key, and value vectors used in the attention mechanism. Number of Heads: This defines the number of parallel attention computations performed within a single attention block. MLP Ratio: This controls the dimensionality of the feed-forward network within each transformer block. Unlike embedding dimension, in Autoformer search space, Q-K-V Dimension, Number of Heads, and MLP Ratio can be varied across layers. Network Depth: This refers to the total number of transformer layers stacked in the architecture. It is important to note that Autoformer maintains a fixed ratio between the Q-K-V dimension and the number of heads in each block. This ensures that the scaling factor in the attention calculation remains constant. This helps stabilize the gradients of different heads during the training
6. We strictly follow Autoformer
search space. The details can be found in the Appx. E.I
Dataset. Our benchmark consists of the evaluation on large-scale, righhuesolution, and most SOTA OoD datasets, including ImageNet-IK 1k (68), ImageNet-C [24], ImageNet-P 224. ImageNet-A 25) ImageNet-O [25). ImageNet-R (27. ImageNet-
28]. Stylized ImageNet [29], and ImageNet-D 26). These datasets capture a comprehensive range of OoD shifts such common corruptions (blur, noise, digital, weather).
based OoD shifts, and natural OoD shifts. A detailed
description of these datasets can be found in the Appx. E.2
Metrics. Following the previous OoD generalization methods
57
30
1.5
we employ three
metrics to construct our benchmark:

---

Table I:
An overview of comprehensive setups to construct our OoD-ViT-NAS benchmark. We utilize the widely used ViT NAS search space, Autoformer [6] which includes three different search spaces Autoformer-Tinyl Tiny/Small/Base 3, 000 architectures from these search spaces to populate our benchmark. To ensure comprchensive- ness, evaluate these architectures across 8 of the most common SOTA OoD datasets. Following prior OoD generalization works (57 30 31l 15], we employ three metrics for our benchmark: ID
cover broad range of model sizes. We randomly sample
Accuracy, OoD Accuracy, and Area Under the PrncisionnRecall Curve (AUPR).
Search Space
Dataset
#Classes
#lmages
#OoD Shifts
Ool Shift Type
Metries
ImageNes-I (IN-K)
X
IK 50K
ImageNes-C (IN-C
1K
3.55
15
Autoformer-Ting
IrageNei-P (N--P)
1K
IM
9
Algorilhmic
ImageNei-D (N--Di Di 76
113
4.8K
3
Acdf
Autoformer Small 6
Stylized ImageNet (Stylized IN) (2)
1K
50K
OoD
ImageNer & (IN Ri
27]
200
30K
Autoformer-Rree Rrse X
Image-Nn-Sketch (IN-Stetch)
1K
50K
Natural
ImageNei-A (N-A)
25
200
7.5K
ID Acef
ImageNest- (IN-O)
200
AUPR
ID Classification Accuracy (ID) Acc): This metric measures the model performance on In-Distribution (ID) data, typically the data it was trained on (e.g., ImageNet). A higher II) Acc indicates the model's ability to learn training data's distribution.
Ool) Classification Accuracy (OoD Acc): This metric measures the model performance on Outso--Distributiio (OoD) data, which could differ significantly from the training data. A higher OoD Acc indicates a better generalization of the model to handle the OoD shifts.
For the specific case of ImageNet-O. [24], we use the Area Under the Precision- Recall Curve (AUPR) metric. A higher AUPR indicates better generalization of the model to handle the OoD) detection.
Protocol. Neural Architecture Search (NAS) is notorious for its computationally expensive nature, requiring the training and evaluation of numerous candidate architectures. To address this challenge and efficiently obtain the large number of architectures needed for our benchmark (i.e., 3, 000), we make use of the One-Shot NAS approach [38 20
59
12, 71.
In One-shot NAS, a single supernet is first constructed. This supernet contains all possible archi- tectures within the defined search space and is trained only once. Then, during evaluation, various be efficiently extracted from the supernet. Importantly, these subnets
architectures (i.e., subnets) inherit the weights from the pre-trained supernet, and their performance has been shown to be comparable even superior to that of architectures trained alone 20 6].
To support a wide range of model sizes, we leverage three supernets: Autoformer-Tin/SSmalll which were previously proposed for ID accuracy (6]. We randomly sample 1, 000 architectures from each supernet, resulting in a total of 3, 000 architectures our OoD-Vit ViT-NAS benchmark. Once obtained, these architectures evaluated on aforementioned OoD datasets.
4 Investigation on Out-of-Distrihutin Generalization of ViT
In this section, we provide the first comprehensive investigation of how ViT architectures affect OoD generalization using OoD-ViT-NAS benchmark. In Sec. we first demonstrate that VIT architectures considerably impact OoD accuracy. In Sec. 4.2 while existing works 2 6 7 g g
12 [8] have made significant strides in improving VIT's ID accuracy, their findings could not be applicable for ViT's OoD generalization due to the not very high correlation between ViT's ID and Ool accuracy. In Sec. 4.3 we conduct the first study to explore NAS for VIT's OoD generalization. Specifically, we study 9 Training-free NAS based on their Ool generalization performance on our benchmark. Finally, in Sec. [4.4] we analyze the influence of individual ViT architectural attributes (i.e., embedding dimension, number of heads, MLP ratio, number of layers) on OoD generalization. L
Additional results of these analysis can be found in Appx. G
4.1 VIT architecture designs have a considerable impact on OoD generalization
In this section, we highlight that VIT architectures considerably impact OoD accuracy. This observa- tion encourages future research to put more focus VIT architecture research for OoD generalization.
5

---

IN-C Pixelate
IN-C Fog
IN-C Gaussian Noise
Severity
2
3
4
5
70
60
ACC.
Acc.
AcC.
Figure 3: Visualization of Ool accuracy range across OoD shift severity. We conduct the analysis
1, 000 architectures in Autoformer-Sni 0 denotes the clean examples. All corruptions can be found in Fig. G.6 in Appx. G We generally observe that the range of OoD accuracy widens as the severity of the OoD shift increases.
search space within our OoD-NAS- ViT benchmarks. Level
Experimental Setups. To show how ViT architecture designs impact OoD generalization, we compute the range of Ool) accuracy for each search space on an OoD dataset. This range reflects the variation in OoD performance for different architectures within search space. For example, in Fig. 2 each sub-plot represents the range OoD accuracy for three different search spaces in our OoD-ViT-NAS benchmark one OoD dataset. We compute the average OoD accuracy range across all datasets as general statistics. For reference, the range of ID accuracy also included.
The results are shown in Fig. Additional results can be found in the Appx G The average range of OoD aceuracy across the three search spaces our benchmark is 81%74.86%/2 74%. which is comparable to or even outperforming state-of fate-of-the of-the-art (SOTA) training OoD generalization method, such as those based on domain invariant representation learning 30 For example, under similar Ool) setting, the current SOTA 30 shows an improvement of 1.9% Ool accuracy. This observation highlights the significant influence of ViT architectural designs on OoD accuracy. By carefully designing VIT architecture, the OoD accuracy could improve significantly.
Results.
We further explore how the severity of the OoD shift affects the range of VIT's OoD accuracy. We conduct similar experimental setups as before, analyzing 1, 000 architectures from the Autoformer- Small search space within OoD--VT-NASS ViT-NAS benchmark for 1, 000 architectures in Autoformer-Small search space within our benchmark on IN-C. The results are visualized in Fig. 3 We observe that the range of OoD accuracy widens as the severity of the OoD shift increases. This suggests that under stronger OoD shifts, the architecture design becomes even more critical for OoD generalization.
When visualizing OoD accuracy, observe a bimodal distribution. We figure out that the embedding dimension, as the primary ViT structural attribute, influences this bimodality. For example, among architectures from the Autoformer-Smal search space of our benchmark, most architectures with a lower embedding dimension (320) fall within the lower OoD accuracy mode, while those with higher dimensions (384 and 448) tend reside in the higher accuracy mode. This observation will be further discussed in detail in Sec.
4.2 Can ID accuracy serve as a good indication for OoD accuracy?
18 study the impact of VIT architectures to ID accuracy, studies on OoD accuracy are limited. To what extent can we directly apply existing findings of ViT architecture insights for ID OoD accuracy? To answer this question, we investigate the relationship between ViT ID and ViT OoD accuracy.
While existing works 2 3 6
8
9
Several studies 669 70 71 722 investigates the relationship between ID and OoD accuracy for the CNNs model. However, there is no work on such study particularly for VIT. Utilizing our OoD-ViT: NAS benchmark, provide the first comprehensive study on the relationship between VIT ID and OoD accuracy. Through our investigation, we find that the correlation between ViT ID and ViT OoD accuracy is not very high. This suggests that architectural insights optimized for VIT ID accuracy, as [ may not be applicable for VIT OoD generalization.
presented in previous work 2
Experimental Setup. Following previous work [7I]. we use Kendall's 7 rank correlation coefficient to compute the correlation between OoD and ID accuracy of all 1, 000 architectures from search space on one OoD dataset. Our examination comprehensively computes the correlations across all 8 OoD datasets, search spaces, and 3, 000 architectures within our OoD- VIT-NAS benchmark. We compute the average correlations across search spaces and datasets general statistics.

---

OoD Acc IN-C Pixelate
OoD Acc IN-D Material
OoD Acc Stylized- -N
22.5
Acc
81
45
20.0
17
17.5
80
40
15.0
16
20
30
20
30
20
30
20 30 #araam
Param
Paaram
Figure 4: Analysis of OoD Generalization Performance of Pareto Architectures for ID accuracy. Blue dots e represent architectures in the search space, while red dots represent the ID Pareto architectures. See Appx. | for additional results. We find that Pareto architectures for ID accuracy generally perform sub-optimally under OoD shift.
Besides the investigation of the correlation of various architectures, we further study the relationship between ViT ID and ViT OoD accuracy for Pareto architectures, representing the rop-performing architectures for a certain model size. As shown in Fig. Da. the red dots represent Pareto architec- tures for ID accuracy In this study, we analyze 1, 000 architectures from the Autofomer-Small search space within our MOD-ViT-NAS benchmark. To identify Pareto architectures for ID accuracy, we divide the total parameter budget into 30 equal intervals and select the architecture with the best ID performance within each interval.
Results. The correlation results are illustrated in Fig. a.. The individual correlations can be found in the Appx. H We show that the correlation between ID and OoD accuracy is generally not very high. This suggests that current architectural insights based solely on ID accuracy might not effectively translate OoD generalization.
Among all OoD datasets, the IN-P dataset exhibits the strongest correlation with ID accuracy. This can be attributed to its weaker OoD shift compared to other datasets (see the visualization in Appx. E As a result, the OoD examples in IN-P are not very different from ID examples, leading to a relatively high correlation between OoD and ID performance. For the remaining seven datasets with stronger OoD shifts, the correlations remain relatively low.
The results of the Pareto architectures analysis are illustrated in Fig. Additional results can be found in the Appx. [ We observe that Pareto architectures for ID accuracy generally perform sub-optimally under the Ool shift. This observation further supports our previous finding that ID accuracy might not be a very good indicator of OoD accuracy.
4.3 Explore Training-free NAS for OoD Generalization
Recently, there has been new research focus Training-free NAS, aimed at identifying high- performing architectures without the computational expense of training each candidate. To do propose zero-cost proxies to predict the performance of candidate architectures in the initialization or the first training iteration, significantly accelerating NAS. While these works focus on ID accuracy, few attempts have been made in searching for architectures robust against adversarial attacks [62 75). However, there is no work to explore Training- free NAS for ViT for OoD generalization.
[00
17
62 60
73 74
Experimental Setup. To address this gap, we comprehensively explore the existing 9 Training-free NAS for OoD generalization on 3, 000 ViT architectures within our OoD-VIT-NAS benchmark. Our study includes common and SOTA Training-free free NAS originally proposed for CNNs for ID Acc (Grasp (74), SNIP 73 McCo (60). ViTs for ID Acc (DSS [10], AutoProx (77), and CNNs for adversarial robustness (Jacobian 77),, CroZe (62)). We complement this study on Training- free NAS to our ViT--AS benchmark to equip the NAS research community with valuable tools to develop more effective Training-fres NAS for OoD generalization.
Results. Our exploration provides several practical insights for designing a Training free NAS for ViT for OoD generalization. The results of Kendall - 76 ranking correlation between the OoD accuracy and Training-free NAS proxies on 8 common large OoD datasets are illustrated in Tab. 2 The average OoD accuracy is computed across OoD datasets and search spaces. Detailed results can be found in Fig. and Appx. J
7

---

Table 2: Comparison of Kendall T ranking correlation between the OoD accuracies and the Training- free NAS proxies values on 8 common large OoD datasets using our DoD-ViT-NAS benchmark. Bold and underline stand for the best and second, respectively. We show that existing Training- free NAS's predictability in ViT OoD accuracy limited, E.g.. the very recently proposed Auto-Prox only achieves 0.3303 correlation. Furthermore, we make the first observation that simple proxies like
#Param or #Foops outperform other more complex proxies in predicting both ViT OoD/ID accuracy.
Originally Proposed For
Training-fres NAS
Correlation with ID Ace
Correlation with OoD Acc
Performance Architecture
Grasp
24
ID Acc
CNNs
0.1490 0.1951
0.1207 1 0.1575
SNIP
ID Act
CNNs
0.3750 0.3023
0.2889 0.2274
McCo
60
ID Acc
CNNs
0.1440 0.2371
0.0975 + 0.0819
CroZe
02
Adv Robustness
CNNS
0.3823 0.3046
0.2951 0.2223
Jacubian 75
Adv Robustness
CNNS
0.1053 -0.1509
0.0841 1 0.1232
D.4165 0.3461
0.3421
AutoProx-A
12
ID Acc
VITS
0.4023 0.3827
0.3303 0.2384
#Param
D.4607 0.3318
0.3600 0.2321
#Flops
0.4705 0,3391
0.3537 0.2327
We observe that existing Training-free NAS are largely ineffective in predicting OoD accuracy. Even
recent Training- -freen NAS designed for ViT (i.e.. DSS [0]] and AutoProx-A (17) ) or Training-free NAS designed adversarial robustness 62 struggle with predicting OoD accuracy.
Surprisingly, simple zero-cost proxies such as #Param or #Flops outperform all existing, more complex proxies in predicting both Ool accuracy for ViTs. This finding poses challenge the Training-free NAS research community: to devise a Training-free NAS that surpasses #Params or #Flops in OoD Acc prediction for VIT.
From Fig. bb, we observe that all Training-free NAS methods consistently fail to predict IN-D performance. This is due to the IN-D dataset's 's unique generation process using Stable Diffusion, which creates images labelled with object names and varying nuisances like background, texture, and material variations. Only the most challenging images are retained, resulting in highly difficult examples, such as distorted images and unrealistic object- jjjeettbackground placements (see Fig. (.2)). These examples degrade VIT model performance significantly [26] and cause unpredictable behaviour.
Our investigation into ID accuracy for ViT' also reveals a surprising observation. While proposals for Training-free free NAS designed for ViT's (i.e., DSS [10] and AutoProx-A (17), improve the prediction of ID accuracy compared to counterparts designed for CNNs. Our study marks the first attempt to explore simple Training-free free NAS like #Param or #Flops. The ID prediction of such simple proxies surprisingly outperforms SOTA Training-free NAS designed for predicting ID accuracy for ViT.
4.4
VIT Structural Attributes on OoD Generalization: Increasing Embedding Dimension is Generally Helpful
Our OoD-VIT-NAS. benchmark with 3, 000 VIT architectures covers diverse design choices in VIT structural attributes, including embedding dimension (Embed_Dinmg network depth, number of heads (#Heads), and MLP ratio (at__,, which allows for finding a wide range of ViT with different structures and complexities. Utilizing our comprehensive benchmark, analysis of the impact of these ViT structural attributes. We investigate which structural attributes in ViTs could lead to better OoD gemeralization. Through our analysis, find that increasing the embedding dimension of a ViT architecture can generally improve Ool generalization. The additional analysis another search space additional analysis can be found in Appx.
are the first to provide an
[ further confirms our finding. The details for this
Experimental Setup. To verify the effectiveness of ViT architectural attributes on our OoD gener- alization benchmark, we present the results from two perspectives: (1) an analysis of rank correlation for our ViT-NS benchmark and (2) a comparison of OoD accuracy across different embedding dimensions. To gain insights into the relationship between embedding dimension (Embed Dim) and OoD performance, we created the visualizations for all architectures our OoD-ViT-NAS benchmark. These visualizations compare the average OoD accuracies across different ViT architectures with varying embedding dimensions and depths. Examples of these visualizations are shown in Fig. [] Additional results can be found in the App/K]
Results. In Fig. cc we find that the embedding dimension generally has the highest correlation
with OoD accuracy among all ViT architectural attributes. This positive correlation indicates that Em-
8

---

OoD Acc IN-C IN-C Zoam Bltu
30,93:
-31.54
$1:27
31:25
31.5
31:17
31/24
511.99
31/26
31.56
33.5
33.0
32.5
32:45
32/21
33:21
32.68.
30/61:
OoD Acc IN-R
31.0
30/77
32.0
30.0
30.5
30/24
31.5
31.0
30.83
30
29.5
29:01
29124
20/23
30.0
30:34
30.5
29.0
29.5
30.0
28.5
29.3
12
13
14
Depth
Depth
Depth
Figure 5: The effect of #Embed Dim on robustness generalization of ViTs. The numbers denote the mean Ool accuracy across ViT architectures with specific colour-coded embedding dimensions and depths. The data points with blue e, orange o, and green colours represent ViT architectures with an embedding dimension of 320, 384, and 448, respectively. Generally, higher OoD accuracy is obtained when the embedding dimension of ViT architectures increases for most OoD shifts. See Fig.
and and M.26 in Appx. K for additional plots and results on other OoD shifts.
bed Dil could play crucial role in achieving OoD generalization performance. Our comprehensive OoD- VID-VVT-NAS benchmark sheds light on a previously unknown relationship: the potential impact of embedding dimension (Embed Dim) on OoD generalization in ViTs. This trend holds across most OoD shifts in our benchmark (Fig. 5). suggesting that among other architectural attributes, the design choice of Emmee_Dii might significantly influence a model's OoD generalization.
Our experiments yield several intriguing phenomena. Based on Fig. Tc. we observe that network depth has slight impact on overall OoD generalization performance (correlation: 0.19). Also, as shown in Fig. [5] for given embedding dimension (represented by distinct colour), we report the mean OoD accuracy, showing how the mean OoD accuracy changes among ViT architectures of varying depths, which aligns with our empirical insight. Fig. 5 shows that while increasing depth can be beneficial for improving VIT's OoD generalization in some cases, there exist shallower models that tend to perform better in terms of OoD accuracy compared to those with deeper models.
is evident from Fig. c where both the MLP ratio (0.09) and the number of heads (0.07) exhibit very low correlation values with overall OoD performance. These findings highlight that increasing the MI.P ratio and the number of heads may not substantially enhance a model's robustness to Ool) data. Due to space constraints, we defer additional experiments to the Appx. L showing that the network depth, MLP ratio, and #Heads might have non-obvious impacts on OoD generalization.
Increasing Embedding Dimension help VIT learn more high-frequency patterns, leading to improve OOD generalization. In this section, we design a frequency study to understand our finding: why increasing VIT Embedding Dimension can generally improve ViT's OOD generalization. In the literature, the models obtain higher performance on preserving
wwwrrrreeennc-oopppreeee (HFC) samples tend learn more HFC [77 78 By learning more HFC, the models improve OOD generalization [9]. Our hypothesis is that Increasing embedding dimension helps ViT' learn more HFC resulting in improving OOD generalization. We adapt the experiment from 777 to verify our hypothesis. The details on this experimental setup can be found in the Appx. [D] In nutshell, we filter HFC by hyper-parameter radius where the higher the r, the lesser HFC. As shown in Fig. 6 we observe that when increasing Embedding Dimension, the performances obtained on filterin--Hr' samples are improved. This observation holds true across setups varying radius r, supporting that increasing Embedding Dimension helps VIT learn more HFC. In contrast, increasing other VIT structural attributes does not help improve ViT learn more HFC.
Robust VIT architectures designed by our finding. Our study provides significant insights for guiding the design of ViT architectures. Specifically, among ViT structural attributes, increasing embedding dimension can generally improve OoD generalisation of VIT architectures. Our insight leads to a simple method which can achieve ViT architectures that can outperform well-established human-designed, We demonstrate the superiority of VIT based on our insights in Tab. B Scaling up ViT architecture (e.g., from ViT-B-32 to ViT-L-32) by humans typically involves compound scaling of various VIT structural attributes. However, our findings suggest that not all ViT structural attributes need to be increased to benefit OoD generalisation. Among these attributes, increasing the embedding dimension is the most crucial factor for improving OoD generalisation. By only increasing the

---

Table Comparison of ViT architectures designed based on our insights (Embedding Dimension)
and well-established ViT designed by humans in [3] I.
Architecture
Einbed
Depth
iflaad
Ratio
MLP
Lateney
(ms))
#Param (M)
IN-R OoD Acc
A wrt Latency
A wrt @Paaram
VIT-B-32
768
12
12
4.0
105.00
87.53
41.58
Ours
840
12
12
4.0
113.32
98.64
48.28
0.8054
0.6032
VIF-L-32 T
1024
16
24
4.0
258.83
305.61
44,33
0.0179
0.0126
Swin-T
12
32
4.0
100.31
28.29
46.22
Ours
128
12
32
4.0
165.80
49.91
48.28
0.0315
0.0954
Swin
3
96
24
32
4.0
184,48
49.60
47.77
0.0184
0.0725
r=
8
[= 12
r= 16
20
20
20
20
OtD Accuracy
16
OnD Accuracy
10
OoD Accuracy
OoD Accuracy
16
12
12
12
10
10
10
320
3,125
3.5
3,833
5,3383
6,002
Embedding Dimension
Number of Layers
MLP Ratio
Number of Heads
Figure 6: Following setting in 77 now tested on high frequency components (HFC) of OoD samples, with r as the radius for frequency filtering. The higher the OoD accuracy, the more HFC learned in the model.
80 79]
the ViTs which were trained on original ID data, are
embedding dimension, ours ViT architectures (e.g., Increasing embedding dimension of ViT-B-32) are significantly more efficient and outperform compound scaling architectures (e.g., ViT-L-32).
5 Conclusion
In this work, we introduce DoD-ViT-NAS the first comprehensive benchmark for NAS on OoD generalization of ViT architectures. Using this benchmark, we conduct a comprehensive investigation on OoD generalization for ViT. Firstly, we show that ViT architecture design significantly impacts OoD accuracy. Secondly, we show that the architectural findings from existing works for ID performance could not apply to OoD generalization due to the low correlation between ID and OoD accuracy. Thirdly, we conduct the first study of NAS for ViT's OoD generalization and show that existing Training-fre free NAS methods struggle with OoD prediction. Surprisingly, simple proxies like #Param or #Flops outperform other complex Training-free NAS. Finally, we conduct the first study on the impact of ViT architectural attributes on OoD generalization. Our study reveals that increasing ViT architecture's 's embedding dimensions can generally improve OoD generalization. We believe our benchmark DOD-VIT-NAS and comprehensive analysis will catalyze and streamline future research on understanding how ViT architecture design influences OoD generalization.
Acknowledgement. This research is supported by the National Research Foundation, Singapore The Agency for
under its AI Singapore Programmes (AISG Award No.: AISG2-TC-2022--00 Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021). This research is supported by the National Research Foundation. Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.
10

---

References
[1] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition at scale. arXiv preprint urXiv:2010.1192: 2020.
16x16 words:
[2] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pages 459 Springer, 2022.
[3] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[4] Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming Hc. Exploring plain vision transformer backbones for object detection. In European Conference Computer Vision, pages 280-296. Springer, 2022.
Bowen Zhang, Zhi Tian, Quan Tang, Xiangxiang Chu, Xiaolin Wei, Chunhua Shen, al. Segvit: Semantic segmentation with plain vision transformers. Advances Neural Information Processing Systems, 35:4971- 4982, 2022.
[6] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference computer vision, pages 12270-12280, 2021.
[7] Haibin Wang, Ce Ge, Hesen Chen, and Xiuyu Sun. Prenas: Preferred one-shot learning towards efficient neural architecture search. In International Conference on Machine Learning, pages 35642-35654. PMIR, 2023.
[8] Xiu Su, Shan You, Jiyang Xie, Mingkai Zheng, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang. and Chang Xu. Vitas: Vision transformer architecture search. In European Conference on Computer Vision, pages 139 157. Springer, 2022.
[9] Chen Tang, Li Lyna Zhang, Huiqiang Jiang, Jiahang Xu, Ting Cao. Quanlu Zhang. Yuging Yang. Zhi Wang, and Mao Yang. Elasticvit: Couflict-aware supernet training for deploying fast vision transformer on diverse mobile devices. In Proceedings of the IEEE/CVF International Conference on Compuder Vision, pages 0829-5840, 2023.
[10] Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Xing Sun, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free transformer architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[11] Qinqin Zhou, Kekai Sheng, Xiawu Zheng, Ke Li, Yonghong Tian, Jie Chen, and Rongrong Ji. Training-free transformer architecture search with zero-cost proxy guided evolution. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[12] Chengyue Gong and Dilin Wang. Nasvit: Neural architecture search for efficient vision transformers with gradient conflict-aware supernet training. ICLR Proceedings 2022, 2022.
[13] Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li. Thomas Unterthiner, and Andreas Veit. Understanding Robustness of Transformers for Image Classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10231-10241, oct 2021.
[14] Shiyu Tang, Ruihao Gong, Yan Wang, Aishan Liu, Jiakai Wang. Xinyun Chen, Fengwei Yu, Xianglong Liu, Dawn Song, Alan Yuille, et al. Robustart: Benchmarking robustness on architecture design and training techniques. arxiv preprint arXiv:2109.052.
2021.
[15] Salman Rahman and Wonkwon Lee. Out of distribution performance of state of art vision model. arXiv preprint a301v:2301. 10750, 2023.
[16] Minghao Chen, Kan Wu, Bolin Ni, Houwen Peng, Bei Liu, Jianlong Fu, Hongyang Chao, and Haibin Ling. Scarching the search space of vision transformer. Advances Neural Information Processing Systems, 34:8714-8728 2021.
[17] Zimian Wei. Peijie Dong Zheng Hui, Anggeng Li, Lujun Li, Menglong Lu, Hengyue Pan, and Dongsheng
Li. Auto-prox: Training free vision transformer architecture search via automatic proxy discovery. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 55814-15822 2024.
11

---

[18] Mingyu Ding, Xiaochen Lian, Linjie Yang. Peng Wang, Xiaojie Jin. Zhiwu Lu, and Ping Luo. Hr-nas: Searching efficient high-resolution neural architectures with lightweight transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2982-2992. 2021.
[19] Jihao Liu, Xin Huang. Guanglu Song. Hongsheng Li, and Yu Liu. Uninet: Unified architecture search with convolution, transformer, and mlp. In European Conference on Computer Vision, pages 33-49. Springer, 2022.
[20] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Picter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16, pages 702-717. Springer. 2020.
[21] Zi-Hang Jiang. Qibin Hou, Li Yuan, Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. Advances in neural information processing systems, 34:18590- 18900-88602, 2021.
[22] Chun-Fu Richard Chen, Quanfu Fan, and Ramoswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference computer vision, pages 357-366, 2021.
[23] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 12299-12310, 2021.
[24] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint (xXiv:1933.1261. 2019.
[25] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15271, 2021.
[26] Chenshuang Zhang, Fei Pan, Junmo Kim, In So Kweon, and Chengzhi Mao. Imagenet-d: Benchmarking neural network robustness on diffusion synthetic object. arXiv preprint or aix::24003. 18775, 2024.
[27] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A critical analysis of out-oo-distributton generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8340-8344 2021.
[28] Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. Advances in Newral Information Processing Systems, 32, 2019.
[29] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet- cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: /8/1 122231, 2018
[30] Haoyue Bai, Yifei Ming, Julian Katz-Samuels, and Yixuan Li. HYPO: Hyperspherical out-of-distribation generalization. In The Twelfth International Conference on Learning Representations, 2024.
[31] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arxiv: 000.99997099999 2019.
[32] Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. In Proceedings of the IEEE international conference on computer vision, pages 2551-2555 2015.
[33] Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In International conference machine learning, pages 7313-7324. PMI.R 2021.
[34] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International conference on machine learning, pages 10-18. PMLR, 2013.
[35] Alexandre Rame, Corentia Dancette, and Matthieu Cord. Fishr: Invariant gradient variances for out-of- distribution generalization. In International Conference on Machine Learning, pages 18347-18377. PMLR, 2022.
[36] Mateo CajallCarulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. Journal of Machine Learning Research, 1-34,, 2018
12

---

[37] Yuge Shi, Jeffrey Seely, Philip HS Torr. N Siddharth, Awni Hamnun, Nicolas Usunier. and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint urXiv:2104.09937, 2021.
[38] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint 11777.191..0873. 2019.
[39] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 361-578. Springer, 2020.
for domain generalization. In Computer
[40] Yimeng Chen, Tianyang Hu, Fengwei Zhou, Zhenguo Li, and Zhi-Ming Ma. Explore and exploit the diverse knowledge in model zoo for domain generalization. In International Conference on Machine Learning, pages 4623 4640. PMLR, 2023.
[41] Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. In International Conference Machine Learning, pages 28656 28679.887788 PMLR, 2023.
[42] Jungwuk Park, Dong-Jun Han, Soyeong Kim, and Jackyun Moon. Test-time style shifting: Handling arbitrary styles in domain generalization. In International Conference on Machine Learning, pages 27114-27131 PMLR, 2023.
[43] Liang Chen. Yong Zhang. Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14172-24182 2023.
[44] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8690-8699, 2021.
[45] Liang Chen, Yong Zhang. Yibing Song, Lingqiao Liu, and Jue Wang. Self superviee learning of adversarial example: Towards good generalizations for deepfake detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 18710-187 2022.
[46] Dachee Kim, Youngjun Yoo, Scunghyun Park, Jinkyu Kim, and Jackoo Lec. Selfreg: Self supersuue contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9619 9628, 2021.
[47] Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. A simple feature augmentation for domain generalization. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 8886 8895, 2021.
[48] Kaiyang Zhou, Yongxin Yang. Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. arXiv preprint
2021.
[49] Qinwei Xu, Ruipeng Zhang. Ya Zhang, Yanfeng Wang, and Qi Tian. fourier- framework for domain generalization. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 14383-14392, 2021.
[50] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint 2xXx:2001.00677 2020.
[51] Yingjun Du, Xiantong Zhen, Ling Shao, and Cees GM Snock. Metanorm: Learning normalize few-shot batches across domains. In International Conference on Learning Representations, 2020.
[52] Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with coomain-augmentn meta-learning In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9624 9633, 2021.
[53] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang, An investigation of why overparam- eterization exacerbates spurious correlations. In International Conference Machine Learning, pages 6346-8356. PMLR, 2020.
[54] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint (003.080033333 2018.
[55] Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang. and Aaron Courville. Can subnetwork structure be the key to out-of-distee of-distribution generalization? In International Conference on Machine Learning, pages 12356 12367. PMLR, 2021.
13

---

[56] Geon Yeong Park, Sangmin Lee, Sang Wan Lee, and Jong Chul Ye. Training debiased subnetworks with contrastive weight pruning. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7929-7938, 2023.
[57] Haoyue Bai, Fengwei 7hou, Lanqing Hong, Nanyang Ye, S-H Gary Chan, and Zhenguo Li. Nas- ood: Neural architecture search for out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2220-8329, 2021.
[58] Dilin Wang. Chengyue Gong, Meng Li. Qiang Liu, and Vikas Chandra. Alphanet: Improved training of supernets with alpha-divergence. In International Conference on Machine Learning, pages 10760-10771 PMI.R, 2021.
[59] Dilin Wang. Meng Li, Chengyue Gong, and Vikas Chandra. Attentivenas: Improving neural architecture search via attentive sampling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6418 6427, 2021.
[60] Tangyu Jiang, Haodi Wang, and Rongfang Bie. Meco: Zero-shot nas with one data and single forward pass via minimum eigenvalue correlation. Advances in Neural Information Prucessing Systems, 36, 2024.
[61] Steffen Jung. Jovita Lukasik, and Margret Keuper. Neural architecture design and robustness: A dataset. arXiv preprint arXiv:2.366.00 067/2, 2023.
[62] Hyeonjeong Ha, Minseon Kim, and Sung Ju Hwang. Generalizable lightweight proxy for robust nas against diverse perturbations. Advances in Neural Information Processing Systems, 36, 2024.
[63] Yongtao Wu, Fanghui Liu, Carl-Johann Simon-Gabrias Grigorios Chrysos, and Volkan Cevher. Robust nas under adversarial training: benchmark, theory, and beyond. In The Twelfth International Conference on Learning Representations.
[64] Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12-21, 2021.
[65] Mingyang Zhang, Xinyi Yu, Haodong Zhao, and Linlin Ou Shiftnas: Improving one-shot nas via probability shift. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5919-5928, 2023.
[66] Shiguang Wang. Tao Xie, Jian Cheng, Xingcheng Zhang, and Haijun Liu. Mdl-nas: A joint learning framework for vision transformer. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 20094-20104, 2023.
[67] Yaowei Li, Ruijie Quan, Linchao Zhu, and Yi Yang. Efficient multimodal fusion via interactive prompting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2604 22613, 2023.
[68] Li Fei-Fei. Jia Deng, and Kai Li. Imagenet: Constructing large-scale image database. Journal of vision, 18)100371) 2009.
[69] John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa. Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: the strong correlation between out-of- distribution and in-distribution generalization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7721-7735, PMLR, 18-24 Jul 2021.
[70] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5389-5400. PMLR, 09--5 Jun 2019.
[71] Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. Id and ood performance are sometimes inversely correlated real- wolld datasets. In A. Oh, T. Neumann, A. Globerson. K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 71702-71722 Curran Associates, Inc., 2023.
[72] Florian Wenzel, Andrea Dittadi, Peter Gehler, Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, Chris Russell, Thomas Brox, Bernt Schiele, Bernhard Schölkopf, and Francesco Locatello. Assaying Out-Or-Distribnne DiDssttribttion Generalization in Transfer Learning. In S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho, and A Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 1198. Curran Associates, Inc., 2022.
14

---

[73] Namhoon Lee. Thalaiyasingam Ajamthan, and Philip HSS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810002340, 2018.
[74] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXi::2002.07376, 2020.
[75] Ramtin Hosseini, Xingyi Yang, and Pengtao Xie. Dsrna: Differentiable search of robust neural architectures. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6196-
6205, 2021.
[76] Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-99 1938.
[77] Jiawang Bai, Li Yuan, Shu-Tao Xia, Shuicheng Yan, Zhifeng Li, and Wei Liu. Improving vision transform- by revisiting high-frequency components. In European Conference Computer Vision, pages 1-18. Springer, 2022.
[78] Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin- Yu Chen, and Cho-Jui Hsieh. On the adversarial robustness of vision transformers. arXiv preprint arXiv:2103. 156700 2021.
[79] Paul Gavrikov and Janis Keuper. Can biases in imagenet models explain generalization? Proceedings IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2184-22194, 2024.
[80] Haohan Wang, Xindi Wu, Zeyi Huang, and Eric Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8684-8694, 2020.
[81] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[82] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the JEEF/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
15

---

A Appendix Overview
This appendix provides supplementary information that included in the main paper due to space limitations.
Contents
Limitations and Broader Impact
17
Analysis on Human-design ViT Search Space
17
Detailed description of the Frequency Analysis Setup
17
The details for our benchmark OoD-ViT-NAS
18
Detailed description of Autoformer Search Space
18
Detailed description of OoD Datasets
18
Overview of using the OoD-NAS-VE VIT benchmark
21
Additional results on the analysis of OoD accuracy range
22
H Additional results the analysis of the correlation between II) and Ool accuracy
24
Additional results on the analysis of Ool) Performance of Pareto architectures for ID)
25
Additional Results for Benchmarking Zero-cost Proxies
26
K
Additional Figures of VIT Structural Attributes: Embedding Dimension
27
Additional Analysis of VIT Structural Attributes on OoD Generalization
28
Ablation Study on the Impact of ViT Architectural Attributes to OoD Generalization
28
Layer- Wise Analysis
32
M Reproducibility
34
M. Hyper-parameters
35
M.2 Compute Resource
35
16

---

B Limitations and Broader Impact
Given the extensive set of experiments presented in this work, the evaluation of training-free NAS proxies is significantly dependent on the initial robust benchmarks, which can be costly and resource intensive to create in the first place.
As this work studies the robustness of VIT architectures to OoD shift, we could demonstrate that carefully designed ViT architectures can significantly enhance OoD generalization. Our approach focuses on evaluating training-free NAS for ViT architectures, offering valuable insights that can complement research exploring the effects of robust architectural design. Finally, we publicly release our OoD-NAS ViT and code base for future research.
C Analysis on Human-design ViT Search Space
Our findings on embedding dimensions are derived from analysing ViT architectures sampled through AutoFormer. To further validate these results, we also investigate the impact of ViT structural attributes on OoD generalisation within the human-designed ViT search space [
Table C.I: We conduct additional experiments human design VIT further confirm main findings that, among ViT structural attributes, embedding dimension the most important ViT structural attribute OoD generalisation. We train each architecture from IN-100 and evaluate IN-R. The trade-off between OoD Acc and computational metries (i.e., Latency and #Param) quantifies by A, which the ratio of increase in OoD Acc and increase in computational metries. Higher A is better. Note that increasing #Head remains the same #Param but increases Latency in ViT setup [
Configuration
Latency
#Parum
Architecture
OnD Ace
A wrt Latency
A wrt #Param
Embed
MLP Ratio
#Head
Depth
(ms)
(M)
Vanila
768
12
3072
105.00
87.53
41.58
Increased
Embed-Dim
840
12
3072
113.32
98.64
48.28
0.8053
0.6433
Increased #Head
768
128
12
3072
120.44
87,53
43.47
0.1224
Increased Depth
768
12
16
3072
137.36
115.88
37.80
-0.1168
-0.1333
Increased
768
12
12
3840
120.05
101.70
42.61
0.0685
0.0728
MLP-Ratio
Experimental setup. We begin with the vanilla ViT-B-32 architecture [. varying each structural attribute independently. For the altered ViT architectures, we increase these attributes to ensure that the capacities of the altered models remain comparable. Each architecture is trained IN-100 and evaluated on IN-R.
Results. As shown in Tab. C.I the results further confirms findings that embedding dimension is the most important VIT structural attribute to OoD generalisation.
D Detailed description of the Frequency Analysis Setup
We adapt the experiment from to verify our hypothesis. We quantify how the amount of HFC leart in the ViT's changes if the embedding dimension of ViTs changes. Particularly, we first filter HFC from testing images of IN-R following [77
80 then evaluate the performance of 1000 ViTs in our search space on IN-R with filtering data points. Following [0 to generate HFC-preserving images, we first convert original images to FFT images. Then, we filter HFC by hyper-parameter radius (r is set to 4, 8, 12, 16 in experiments). In a nutshell, the higher the the lesser HFC.
17

---

E
The details for our benchmark
E.1
Detailed description of Autoformer Search Space
We strictly follow the search space in Autoformer 6 to construct our OoD-VIT-NAS, The variable factors of basic transformer blocks include the embedding dimension, Q-K-V dimension, number of heads, MLP ratio, and network depth. The detailed search spaces are illustrated Tab, E.2
Table E.2: Detailed search space used to construct our OoD-VIT-NAS We strictly follow the search
space in Autoformer [6.
Embedding Dim
Q-K-V Dim
MLP Ratio
Head Num
Depth Num
Max
192
192
3,5
3
12
Supernet-Tiny
Min
240
256
Step
24
64
0.5
Max
320
320
Supernet-Small
Min
448
448
4
7
14
Step
64
64
0.5
Max
528
512
3
14
Supernet-Bass
Min
624
640
4
10
Step
48
64
0.5
E.2
Detailed description of OoD Datasets
ImageNet
ImageNel-C
ImageNet-A
ImageNettOO
ImageNet-P
ImageNet
ImageNet-D D
ImageNet-R
ImageNet- Sketch
ImageNet-Stylized
Figure E.1: Visualization of different OoD shifts across datasets used construct our OoD-ViT NAS benchmark. The description of each dataset can be found in Sec.
To construct OoD- ViT-NSS benchmark, we evaluate 3,000 architectures within our benchmark on common and most SOTA OoD datasets, including:
ImageNet-Ik 688
This large and common image dataset widely used computer vision research. It contains over 1.3 million labeled nigh-resolution images belonging to 1,000 different object categories (classes). Each image labeled with a class (c.g., "cat", "airplane", "chair").
ImageNet-CC C (24 This dataset builds upon the original ImageNet test set by applied algo- rithmically corruptions. These corruptions simulate real-world factors that can deviate data from the training set, such as blur, noise, digital, and weather effects. ImageNet-C offers comprehensive OoD scenarios with 15 different corruption types, each with severity levels, resulting in a total 75 unique OoD setups.
ImageNet-P P 224 This benchmark is constructed similarly to Imagnet-C. The difference is that ImageNet-P utilizes perturbation sequences generated from each ImageNet validation image.
18

---

This dataset is real-world OoD scenarios by leveraging adversarially filtered images that are highly likely to fool current image classifiers. ImageNet-A selects 200 classes out of 1,000 classes from ImageNet- K so that emrors among these 200 classes would be considered egregious 68
ImageNet-A [25]
ImageNet-O O 25 Similar ImageNet-A, this dataset includes adversarially filtered exam- ples specifically designed to challenge OoD detectors trained on ImageNet. ImageNet-O selects a 200 classes out of 1,000 classes from ImageNet-IK.
ImageNer-R 227 This dataset is rendition of ImageNet, containing images with manipu- lated textures and local image statistics.
ImageNer-Sketch [28] : This dataset introduces a unique OoD challenge by providing black-
and-whit sketch images corresponding to the ImageNet-11 1K test set. This significant
divergence visual representation tests a model's ability to generalize beyond photographic
data.
Stylized ImageNer 29 This dataset consists of stylized version of ImageNet generated through techniques like AdaIN [ style transfer, resulting variations like greyscale, silhouettes, and edges. This dataset assesses model's 's ability to handle data with different artistic interpretations.
ImageNer-D [26]: This dataset is variation of ImageNet generated through diffusion models. These models aim at creating images with rich diversity in backgrounds, textures, and materials.
Banana
Bottle Cap
Bread Loaf
Computer Mouse
Desk Lamp
Figure E.2: Examples from ImageNet-D (26). These examples are generated by Stable Diffusion [82 and only hard examples are kept. These examples could be distorted or unrealistic in object- background placements.
We visualize a few examples of OoD datasets Fig. tion of different OoD shift severity in Fig. E.3
For ImageNet-C, we provide the visualiza-
19

---

1
4
Brightness
Contrast
Defocus Blur
Elastic Transform
Fag
Frost
Gaussian
Noise
Glass Blur
Impulse
Noise
Jpeg Compression
Motion Blur
Pixelate
Shot Noise
Snow
Zoom Blur
Figure E.3: Visualization of different corruptions and 5 different OoD shift severity for ImageNet-C 24
We note that level O means clean examples from ImageNer (8
20

---

Overview of using the OoD-NAS-ViT benchmark
Our MOD-NAS-ViT benchmark consists of multiple json individual files. Each file includes the evaluation of one search space /Autoformee:Tiy//Small///
6 on 8 most common and state-of-the-
art (SOTA) OoD datasets: ImageNet-C [24], ImageNet-A [25], ImageNet-O [25], ImageNet-P [24].
ImageNet-D [26], ImageNet-R [27]. ImageNet-Sketch (28). and Stylized ImageNet [29] (Sec.
We further provide merged json file for each search space. The structure of each merged json
is illustrated as in Fig. F.4 Specifically, each merged json file includes 1,000 VIT architectures
denoted by the index. Each architecture consists of:
net setting: ViT structural attributes information:
layer_num: num: network depth
mip_rationn MLP ration each layer, can be varied among layers
num heads; number of attention heads each layer, can be varied among layers
cemed__ding embedding dimension heads each layer, fixed among layers
params: number of parameters
flops: number of flops
performance: performance on different dataset:
Imagenet: performance on IN-based datasets:
clean: performance on IN
sketch: performance on IN
stylized-imagenet: performance on IN
imagenet-R: performance IN
imagenet-O: performacce on IN
imagenet-A: performacee IN-A
corruption: performance on IN-C
Fog: performance one out of 15 corruption in IN-C: Fog, Gaussian Noise, Fog. Snow, Elastic Transform, Jpeg Compression, Frost, Motion Blur, Brightness, Defocus Blur, Glass Blur, Impulse Noise, Shot Noise, Zoom Blur, Constrast, Pixelate
1: performance at OoD shift severity level 1. There are total 5 level of OoD shift severity for each corruption in IN-C
corruption P: performance IN-P
* Brightness: performance on one out of 10 corruption IN-P: Brightness, Motion Blur, Rotate, Scale, Shot Noise, Snow, Tilt, Translate, Zoom Blur, Gaussian Noise
imagenet-D; performance on IN-D
* background: performance one out of 3 nuisances in IN-D: background, material, texture
With the provided merged json file for each search space, we can easily retrieve the OoD shift performance of various VIT architectures and their OoD performance on 8 prevelent and SOTA OoD datasets.
21

---

1
2
"0":{
3
"net setting":
4
"layer_nume num": 15,
5
"mlp_ ratio": [3.5, 3.0, 3.5, 4.0, 3.5, 3.5, 3.5, 3.5, 3.5, 4.0, 3.0, 4.0, 3.01.
"nun_heads":
"num heads": [6, 5, 5, 7, 6, 7, 7, 7, 5, 6, 5, 7, 51,
7
"embed dim": [448, 448, 448, 448, 448, 448, 448, 448, 448, 448, 448, 448, 4481
}.
"params": 64.959352,
"flops": 5952.001599
"performance:: {
12
'Imagenet": {
13
"clean": 61.518800022197265,
14
"sketch": 33. 33.04971507080507,
15
"stylized-imagenet" 17 07.466900002441406
16
'imagenet-R": 31.59999999923766,
17
imagenet-0", 2.1998259527964194,
imagenet-A" 13.61333322342276.
imagenet-C":
"Fog": :{
"1": 71.72,
"2": 68, 422490006611322,
"3": 65. 88099002553476,,
24
"4": 65.5780000024414,
25
"5": 60.92200080854492
26
27
}.
28
"Pixelate":
"1": 288800000000944
"2": 72. 3444440004444444
"3": 67.53900080366211,
57.36600000666211, "5": 47.82799999938965
36
37
'imagenet-"": "Brightness": 76.014
38
39
49
"Gaussian Noise": 79.008
41
imagenet-D": {
"background:: 11.87739138824836
"naterial": 16.189767331561975
"texture": 14.961653801332637
49
50
51
"999":
Figure F.4: The structure of the merged json file for our VoD-VIT-NAS benchmark for Autoformer-
Small search space. The structures of the merged json files for Autoformer-Ting Tiny/Base similars.
G Additional results on the analysis of OoD accuracy range
In the main paper, we visualize OoD accuracy ranges and ID accuracy range for reference. this Appx. section, we provide the visualization of the remaining OoD accuracy ranges. The results are illustrated Fig. G.5 Our observation on other OoD accuracy ranges are generally consistent with findings in Sec.
22

---

1.06
2.4011.06)
2.69(1.06)
2.93(1.06)
Base Small Tiny
2.25
4.74(2.25)
4.89(2.25)
3.81(2.25)
0.56
5653(0.56)
1.65(0.56)
70
80
40
50
40
50
60
20
30
40
Acc (%)
IN-C Snow ACC.
IN-C Frost ACC.
IN-C Motion Blur ACC.
1.32(1.06)
2.4611.06)
1.38(1.06)
3.51(1.06)
Tiny
2.72(2.25)
3,15(2255)
2.93(2.25)
7.0012.29 25)
Base Small
0.90:0.56)
1.95(d.56)
2.22(0.56)
3.36(0.56)
70
20
30
20 IN-C Glass Blur Acc.
30
IN-C Brightness Acc.
IN-C Defocus Blur Acc.
IN-C Shot Noise Acc.
1.50(1.06)
2.36(1.06)
2.04(1.06)
29.62(1.06)
Base Small Tiny
3.65(2.25)
4.52(2.25)
4.35(2.25)
8:91(2.25)
2.15().56)
2.32(0.56)
2.24(0.56)
4.41(0.56)
20
30
50
60
30
40 IN-C Contrast Acc.
20
IN-C Zoom Blur Acc.
IN-C Jpeg Compression Acc. IN-C Elastic Transform Acc.
8.99(1.06)
1.02(1.06)
1.21(1.06)
1.00(1.06)
Base Small Tiny
8.5012.25)
1.95/2.25 25)
2.50(2 25)
2.22(2.25)
8.72(0.56)
0.56(0.56)
0.92(0.56)
0. 0.65(0.36)
20 IN-D Material Acc.
10
70
80
70
80
70
80
IN-P Translate Acc.
IN-P Snow Acc.
IN-P Rotate
1.03/1.06)
1.38(1.06)
1.16(1.06)
1.11(1.06)
Base Small Tiny
2. 2.0822225 25)
3.21(2.25)
2.12(2.25)
2.4412.25)
0.5810.56)
2.65(0.56)
0.56(0.56 56)
0.53(0.56)
70
80
60
70 IN-P Scale Acc.
80 IN-P Gaussian Noise Acc.
70
70
80
IN-P Till Acc.
IN-P Brightness Acc.
1.26(1.06)
1.18(1.06)
1.12(1.06)
Tiny
2.43(2.25)
22/22 25)
2.38(2.25)
Base Small
0.61(0.56)
0.6110.56)
0.89().56) 56)
60
80
70
20
IN-P Motion Blur Acc.
IN-P Shot Noise Acc.
Stylized-IN IN Acc.
Figure G.5: As in Figure influence of ViT architectural designs on OoD accuracy. The numbers within each violin plot for each sub-figures (e.g., IN-D Material 8.99 (1.06), 8.50 (2.25), and 8.72 (0.56) denote the corresponding OoD(ID) accuracy range of architectures sampled from AutoFormer- AutoFormen-TTny/SSal/aaaa Tiny/Small/Base search space, respectively.
our analysis on the OoD accuracy range highlights the significant
23

---

IN Snow
Frost
IN Motion Blur
IN C. Brightness
Severity
4
AcC.
IN-C Defocus Blur
IN-C Glass Blur
IN-C impuise Noise
IN-C Shot Noise
Severity
2
2
4
3
3
4
4
60
20
40
60
80
80
60 IN-C Contrast
IN-C Zoom Blur
IN-C Jpeg Compression
IN-C Elastic Transform
D
0
1
1
Severity
2
2
2
3
3
4
4
40
60 80
50
60
70
80 40
60
80
40
60
80
Acc.
Acc.
ACC.
Figure G.6: Visualization of OoD accuracy range across IN-C OoD shift severity. The experiments are conducted on 1, 000 architectures in Autoformer-Smie Small search space within our OoD-NAS-Vis benchmarks. Level 0 denotes the clean examples. We generally observe that the range of OoD accuracy widens as the severity of the OoD shift increases.
H Additional results on the analysis of the correlation between ID and OoD accuracy
In the main paper, we provide the Kendall 7 correlation between accuracy and 8 OoD datasets. For some OoD datasets with different OoD shift types, average the correlation with ID accuracy of different OoD shift types to obtain average correlation with ID accuracy for that Ool dataset. In this Appx. section, provide the detailed correlations with ID accuracy of each OoD shifts in such OoD data. Specifically, we provide the detailed correlation for IN-C, IN-D, and IN-P in Fig. H.7 Fig. H.8 and Fig. H.9 respectively.
Furthermore, we provide a comprehensive correlation between OoD accuracy and ID accuracy in Fig. H.10
24

---

(a) ID Indicator for VIT OoD Generalization
(c) OoD Generalization VIT Architecture Attributes
(b) Training-free NAS for VIT OoD Generalizatton
IN-C Gaussian Noise 5
0.30
0.22
0.23
0.12
0.04
20
0.20
0.10
0.20
0.20
0.70
0.07
0.10
0.06
IN-C Snow
0.42
0.36
0.34
0.15
0.02
0.3
0.34
0.10
0.32
0.28
0.76
0.16
0.07
0.02
CN-C Frost
0.34
0.36
0.38
0.15
0.09
0.30
0.31
0.11
0.34
0.32
0.35
0.19
0.08
0.23
IN-C Motion Blur
0.25
0.35
0.37
0.12
0.18
0.27
0.27
0.09
0.34
0.36
0.12
-0.03
0.31
0.17
Brightness
0.35
IN-C Defocus Blur
0.28
0.28
0.23
0.12
0.06
0.25
0.25
0.11
0.26
0.25
0.18
0.02
0.21
0.08
IN-C Glass Blur
0.08
0.35
0.35
0.11
0.26
00.27
0.09
0.37
0.27
-0.11
0.02
-0.06
0.11
Impulse
IN-C Shot Noise
0.28
0.40
0.38
0.13
0.06
0.34
0.33
0.10
0.35
0.31
0.58
-0.05
0.15
-0.10
-0.23
IN-C Pixelate
0.32
0.37
0.35
0.16
0.03
0.34
0.33
0.11
0.36
0.29
0.01
0.19
0.11
IN-C IN-C Compression 5
0.38
0.13
0.14
0.07
0.00
0.10
0.11
0.45
0.12
0.11
0.30
0.06
0.20
0.14
IN-C tlastic transform
0.32
-0.02
-0.03
0.33
-0.10
0.08
0.20
Average OoD 0.37
0.27
0.27
0.12
0.04
0.24
0.24
0.09
0.25
0.22
0.38
0.02
0.32
0.00
-1.00
1D Acc
#Param&Flops
Grasp
MeCo
CroZe
SNIP Jacoblan
DSS Autofrox
Embed Dim
Depth
MLP Rotio
#Head
Figure H.7: Kendall rank correlation coefficient between ID and OoD accuracies computed all 3000 architectures our OoD-ViT-NE ViT-NAS benchmark. Mcasurements are computed on different corruptions of IN-C.
indicator
Generalization
Training-free
0.73
033
IN-D Background .016
-0.15
0.16
0..01
00.4
0.15
0.15
..00
-0.14
0.15
.0.11
-0.3
0.11
0.06
6.25
IN-D Material
0.01
0.05
Average OoD
-0.05
-0.09
.0.01
-0.07
.0.07
0.03
0.99
-007
Acc
#Param#Flops
MeCo
CroZe
SNIP Jacobian
DSS AutoProx
Embed Dim
Depth
MLP Ratio
#Hlead
Figure H.8: Kendall T rank correlation coefficient between ID and OoD accuracies computed on all 3000 architectures in our OoD-ViT-NE ViT-NAS benchmark. Measurements are computed on various IN-D OoD shifts.
I Additional results on the analysis of OoD Performance of Pareto architectures for ID
In the main paper, due to space constraints, we only provide the Pareto analysis results on a few representative OoD datasets. In this Appx. section, provide additional results this Pareto architecture analysis in Fig, M.24 MM19 M.20 M.2II.M.2. M.23 In the following scatter plots, blue dots represent architectures in the search space, while red dots represent the ID Pareto architectures.
25

---

(a) ai as Indicator for VIT OoD Generalization
(c) OoD Generalization VIT Architecture Attributes
(b) Training-free NAS for VIT OoD Generalization
Transiate
0.21
0.35
IN-P Snow
0.60
0.42
0.44
0.14
0.12
0.34
0.34
0.11
0.39
0.37
0.53
0.10
0.17
0.06
035
IN-P Rotate
0.70
0.41
0.14
0.32
0.31
0.37
0.35
0.10
0.17
0.06
IN-P Til
0.71
0.55
0.58
0.18
0.23
0.45
0.44
0.11
0.54
0.55
0.43
0.19
0.16
0.03
IN Scale
0.53
0.45
0.46
0.15
0.13
0.38
0.37
0.12
0.40
0.39
0.50
0.56
0.15
0.13
Gaussian Naise
0.69
0.20
-0.03
IN-P Brightness
0.70
IN-P Motion Blur
0.67
0.44
0.45
0.02
0.13
0.36
0.36
0.11
0.40
0.38
0.41
0.16
0.20
0.05
-0.58
IN-P Shot Noise
0.72
0.49
0.50
0.17
0.12
0.43
0.42
0.14
0.44
0.42
0.45
0.21
0.19
0.02
1D AcC
#ParameFlops
Grasp
MeCo
CroZe
SNIP Jecobian
DSS
AutoProx
Embed
Depth
MLP Rabio
#Hteed
Dim
Figure H.9: Kendall rank correlation coefficient between ID and OoD accuracies computed all 3000 architectures in our OoD- ViT-NAS benchmark. Measurements are computed on various IN-P OoD shifts.
J Additional Results for Benchmarking Zero-cost Proxies
In the main submission, we provides the comparison of Kendall ranking correlation between the ID/OoD accuracies and the zero-cost proxy values across all OoD datascts. In this section, provides the correlation for each dataset for a detailed observation. The results can found in
Tab. J.4 J.5
J.8 J.9 J.10 Our observations on individual OoD datasets are consistent
with our findings in Sec. 4.3
Table J.3: Comparison of Kendall ranking correlation between the ID/OoD accuracies and the zero-cost proxy values on ImageNet-C datasets in the Autoformer search space. Bold and underline stands for the best and second, respectively.
Originally Proposed For
Training-free NAS
Correlation with ID Ace Correlation with OoD Acc
Performance
Architecture
Grasp 24
ID Ace
CNNs
0.1490 + 0.1951
0.1179 + 0.1713
SNIP
73
ID Ace
CNNs
0.3750 + 0.3023
0.2371 = 0.2579
MeCo
60
ID Acc
CNNs
0.1440 +: 0.2371
0.0392 1 0.1206
CroZe
Adv Robustness
CNNs
0.3823 0.3046
0.2356 = 0.2593
Jacobian
76
Adv Robustness
CNNs
0.1053 1 0.1509
0.0894 : 0.1369
DSS 10
ID Ace
VITs
0.4165 + 0.3461
0.2454 + 0.2630
AutoProx-A
17
ID Acc
VITs
0.4023 = 0.3827
0.2271 = 0.2758
#Param
0.4607 + 0.3318
0.2651 = 0.2546
#Flops
0.4705 0.3391
0.2656 0.2572
26

---

Acc L
255
bud
0.52 EL28
0.52
U.26
004 160
100
IN-C Snow 0.42443 6400
350
000
0:42 0.75
0.26
0-44
0.32
0.45
0.440
IN-C Fog 14
IN-C
IN C Motion Blur 0.000 200.0000222200
220222 MULS
0.4
0.44 2510 0
00.01
1.55 0.25
0.000
IN-C Drighiness
IN-C Defocus Blu
IN-C Glass Blur 0.00 0.00
0.06
13:3
3:2
2.00
2:22
Rub 0.3 0:2
3.04 0.00 0.00 0.09
0.0
0.13
0:03
0.32
0.00 0.33
0.100.00.011110 050 01
D.54
Impulse Noise
IN-C Shot Noise 026 0.00
5:37
0.21
8.25
0.25
164
IN-C Pixe alo 0.52 0,20
0.20-
5252222
3:2
50
0
0.25
0.2
0.00
0.40 4 0 100
0.5 0-43
0:50 0.28
0.20
0.00
0.43
0.50
0505
Zoom Uhr
IN-C [eC] Compression 0.28 01800200.33 0:30033
0.17
0.24 0.25
0.201
060
0-48 0:43
0:40
01
IN-C Elastic Transform 62320222000
36 11
0.25
15-6
N Texture 6:65 0:0+0.33 0 060 01000000000000000
0.00 0.000.
0.07 io 0:07 buo bud6- 1006--06-.0
0.00 OL.
0V
09
06
IN-D (ec Beckground
4.28-.244
090
04-
030 000.000000000 01-00.0-..000
00-0
IN-D Metertal
IN-P Thanslate
17. 0.24 0.66
0.37 a25
0.2 oos 0.33 0.28 0.15 2. de 0.40 0.0000000000
060 360 000000000000
de 2. 3:48
N Snow
1:32
IN-P Rotate
0.34
6.47
0:37 3.
out 0.1 6.17 6.30 0.37 0.76 46 0.24 oin
IN-P Zoom Blur
0.37
535
31
3:33
0.3 000
0.35 0:22 044 0.46 0.45 0.59 0 4 010 150 0
5:43
N.P Scale
0.260:32
300
3.00
44
2:22
000
0.20 0.2
022 0.22 0 0.42 060 100
IN-P Gaussian Nosse
0.300443
0.1
0. 0.36 0
25 a ten e 0en
IR- Rrightmess
IN Nobion Blu
0.33 0.66
33 9:31
5.37 0.1
0.32 0255 0.350.600.300 0.060 160
70
IN-P shot Noise
3220
IN-O
IN-A 8.60
0.20
25 31
3.07
3:33
0.1
0.0
0.22
0.2
0.0 0. 0.22 022.0.022
000 360 03 2.30
0.34
0:37
0:33 0:32 0.24 0.50 30 5.00 5:33 bas
OOD R
A-Sketch 027 0.22 02 200
22 0.1 0.28
0 02202222220 DeG 0-12 0.38 0-12 0.30 0.30
Stylized-IN IN
4000 0.23 0:400 30000000000 222
22 0.24 0.22 022 0.2 0.24 00330 2022-- DOU DeG
00
9.30 2.00 0.30 upp
B&C Gaussan Noiss
1D Acc
INCC Seawe
INco Motion Blur
BiC Froot
Feg
iB.C Brigtemet
it-c tifoccus Blar
WiC Glass Blar
Bec Impulta Aoise
IN Houoe House
PI- protitte
B&C mostoo
BCC. Zoom Blor
In. Elastic Bansform
BKC Contreet
16-0 Texting
w-- Back ground
16-0 Material
In-P transiate
It-P Snow
IN-P Zeain Blur
IR-F Gausoian Nitiie
Wi- Tit
04-P Scaale
16-9 arranteens
He-P ne Shatt Aroete
Mastion Blur
Wi-
Wh-8
Sketch
Stritzed
Wi-F
000
Figure 10 Kendall rank correlation coefficient between ID and OoD performance computed on all 3000 architectures in our tasks including IN-C, IN-D, IN-P, IN-A, IN-R, IN-O, Stylized-IN, and IN-Sketch.
benchmark. Measurements are computed on different OoD
K Additional Figures of ViT Structural Attributes: Embedding Dimension
27

---

Table J.4: Comparison of Kendall ranking correlation between the ID/OoD accuracies and the zero-cost proxy values on ImageNet-P datasets in the Autoformer search space. Bold and underline stands for the best and second, respectively.
Originally Proposed For
Training-free NAS
Correlation with Acc
Correlation with OoD Acc
Performance
Architecture
Grasp 74
Acc
CNNs
0.1490 0.1951
0.1373 : 0.1907
SNIP
Acc
CNNs
0,3750 0,3023
0.3652 + 0.3306
McCo 60
ID Acc
CNNs
0.1440 = 0.2371
0.1324 = 0.2240
CroZe 02
Adv Robustness
CNNs
0.3823 + 0.3046
0.3698 = 0.3307
Jacobian
75
Adv Robustness
CNNs
0.1053 + 0.1509
0.1142 + 0.1590
DSS 10
ID Acc
ViTs
0.4165 0.3461
0.4128 + 0.3621
AutoProx-A
Acc
0.4023 0.3827
= 0.3975
#Param
0.4607 + 0.3318
0.4487 = 0.3475
#Flops
0.4705 + 0.3391
0.4592 + 0.3547
Table J.5: Comparison of Kendall ranking correlation between the ID/OoD accuracies and the zero-cost proxy values ImageNet-D datasets in the Autoformer search space. Bold and underline stands for the best and second, respectively.
Originally Proposed For
Training-free NAS
Correlation with Acc
Correlation with OoD Acc
Performance
Architecture
Grasp
741
CNNs
0.1490 0.1951
-0.0306 0.0307
SNIP
Ace
CNNS
0.3750 0.3023
-0.0863 0.0887
MeCo
60
ID Ace
CNNs
0.1440 = 0.2371
-0.0518 0.0486
CroZe
02
Adv Robustness
CNNs
0.3823 = 0.3046
-0.0818 0.0906
Jacobian
75
Adv Robustness
CNNs
0.1053 = 0.1509
-0.0090 0.0318
DSS
VITs
0.4165 0.3461
-0.0743 0.1131
AutoProx-A
17
ID Ace
VITs
0,4023 + 0,3827
-0.0742 0. 1066
#Param
0.4607 + 0.3318
-0.0837 0.1117
#Flops
0.4705 = 0.3391
-0.0827 0.1133
Additional Analysis of VIT Structural Attributes on OoD Generalization
L.1 Ablation Study on the Impact of ViT Architectural Attributes to OoD Generalization
In this section, we demonstrate the effectiveness of each VIT architectural attribute on OoD accuracy from the ablation study perspective. All ablation studies are based on 1,000 VIT architectures sampled from Autoformer-Small search space in our DoD-VIT-NAS benchmark. Through our general analysis in Sec. 4.4 in the main and ablation study, we show that the embedding dimension highest impact among VIT architectural attributes, while network depth has a slight impact on OoD generalization.
Experimental Setups. We conduct the ablation study on the impact of VIT architectural attributes on OoD generalization. Particularly, for each ablation study of one ViT architectural attribute, we vary that attribute while keeping all other attributes fixed. Then, we compute Kendall's rank correlation coefficient between each attribute and different OoD shifts. While we can directly adjust the depth and embedding dimension, adjusting MLP Ration and #Head is challenging. This is because these two attributes for each ViT arch are in the form of list with depth elements. Each element selected among 3 choices. This results in a huge combination. To deal with this difficulty, we first compute the means of MI.P Head for each architecture. Then, during the ablation study, we explore range of values for MLP ratio and the number of heads (mean #Head = + 0.05, mean MLP Ratio
28

---

Table J.6: Comparison of Kendall ranking correlation between the ID/OoD accuracies and the datasets in the Autoformer search space. Bold and
zero-cost proxy values on Stylized-Imagee underline stands for the best and second, respectively.
Originally Proposed For
Training-free NAS
Correlation with Acc Correlation with Acc
Performance
Architecture
Grasp 74
CNNs
0.1490 0.1951
0.1413 + 0. 1814
SNIP
Acc
CNNs
0,3750 0,3023
0.3175 + 0.2658
McCo
60
ID Acc
CNNs
0.1440 = 0.2371
0.0937 = 0.1739
CroZe 02
Adv Robustness
CNNs
0.3823 + 0.3046
0.3091 = 0.2620
Jacobian
75
Adv Robustness
CNNs
0.1053 + 0.1509
0.1017 + 0.1442
DSS 10
ID Acc
VITs
0.4165 0.3461
0.3800 + 0.3294
AutoProx-A
Acc
0.4023 0.3827
0.3619 = 0.3573
#Param
0.4607 + 0.3318
0.3899 = 0.3004
#Flops
0.4705 + 0,3391
0.3905 + 0.3094
Table J.7: Comparison of Kendall ranking correlation between the ID/OoD accuracies and the zero-cost proxy values on ImageNet-Sketch datasets the Autoformer search space. Bold and underline stands for the best and second, respectively.
Originally Proposed For
Training-fre free NAS
Correlation with ID Acc
Correlation with OoD Acc
Performance
Architecture
Grasp 2
ID Acc
CNNs
0.1490 = 0.1951
0.1395 = 0.2299
SNIP
7
ID Acc
CNNs
0.3750 +: 0.3023
0.3434 : 0.3696
MeCo
60
Acc
CNNs
0.1440 0.2371
0.0896 0.1112
CroZe
Adv Rohustness
CNNs
0,3823 0,3046
0.3610 + 0.3616
Jacobian
25
Adv Robustness
CNNs
0.1053 + 0.1509
0.0871 = 0.1872
DSS 110
ID Acc
VITs
0.4165 + 0.3461
0.3885 = 0.4021
AutoProx-A
17
ID Acc
ViT'
0.4023 + 0.3827
0.3599 + 0.3939
#Param
0.4607 0.3318
0.4317 + 0.4013
#Flops
0.4705 0.3391
0.4060 + 0.3917
3.5 = 0.05) to capture the impact of these more nuanced architectural variations. We note that this range of values is small, allowing to approximately fix these two attributes.
First, we assess the impact of embedding dimension on OoD generalization by fixing the configu- rations of all other ViT structural attributes (i.e., network depth 13, mean #Head = 6 + 0.05, and mean MLP Ratio
MLP_Ratio = 3.5 + 0.05). The correlation results are shown in Fig. L. We observe an
overall positive correlation of 0.65. This further supports our observation in Sec. 4.4 that increasing
the embedding dimension generally could lead better OoD performance across most OoD shifts for these ViT architectures.
29

---

Table J.8: Comparison of Kendall ranking correlation between the ID/OoD accuracies and the zero-cost proxy values on ImageNet-R datasets in the Autoformer search space. Bold and underline stands for the best and second, respectively.
Originally Proposed For
Training-free NAS
Correlation with Acc
Correlation with Acc
Performance
Architecture
Grasp 74
CNNs
0.1490 0.1951
0.1435 + 0.1956
SNIP
Acc
CNNS
0,3750 0,3023
0.3576 + 0.3416
McCo
60
ID Acc
CNNs
0.1440 = 0.2371
0.1224 = 0.1820
CroZe 02
Adv Robustness
CNNs
0.3823 + 0.3046
0,3751 = 0.3342
Jacobian
75
Adv Robustness
CNNs
0.1053 + 0.1509
0.0935 + 0.1518
DSS 10
ID Acc
ViTs
0.4165 0.3461
0.4364 = 0.4047
AutoProx-A
Acc
0.4023 0.3827
0.4164 = 0.4027
#Param
0.4607 + 0.3318
0.4773 = 0.4179
#Flops
0.4705 + 0.3391
0.4507 + 0.4028
Table J.9: Comparison of Kendall ranking correlation between the ID/OoD accuracies and the zero-cost proxy values on ImageNet-A datasets in the Autoformer search space. Bold and underline stands for the best and second, respectively.
Originally Proposed For
Training-free NAS
Correlation with ID Acc
Correlation with OoD Acc
Performance
Architecture
Grasp 74
ID Acc
CNNs
0.1490 + 0.1951
0.1663 = 0.1483
SNIP
Acc
CNNs
0.3750 0.3023
0.4588 = 0.1477
MeCo
60
ID Ace
CNNs
0.1440 1 0.2371
0.2495 = 0.0180
CroZe
02
Adv Rohustness
CNNs
0.3823 + 0.3046
0.4642 + 0.1415
Jacobian
75
Adv Robustness
CNNs
0.1053 = 0.1509
0.1050 = 0.1172
DSS 10
ID Acc
VITs
0.4165 + 0.3461
0.5930 = 0.1021
AutoProx-A
17
Acc
VITs
0.4023 0.3827
0.6048 : 0.5923 + 0.1288
#Param
0.4607 0.3318
#Flops
0.4705 + 0.3391
0.5959 + 0.1230
Average OoD
-Stylized-IN
IN-Sketch
ID Acc
IN-C
IN-D
IN-P
IN-O
IN-A
IN-R
Embed_Dine
0.80
0.63
-0.11
0.81
0.82
0.68
0.70
0.80
0.69
0.65
Figure L.11: Kendall's rank correlation coefficient between varying Embed_ and OoD accuracy.
To demonstrate how the number of layers (i.e., network depth) in a ViT architecture affects the model's 's OoD generalization, we fix other ViT architectural attributes (i.e., Embed Dim 384, mean #Heed 6 + 0.05, and mean MLP Ratio 3.5 0.05). The ranking correlation for all architectures between depth and OoD accuracy in Fig. L.12 suggests weak correlation between the VIT network depth and OoD accuracy. This observation consistent with our finding in Sec. 4.4 that the depth has minimal influence OoD generalization.
30

---

Table J.10: Comparison of Kendall ranking correlation between the ID/OoD accuracies and the zero-cost proxy values on ImageNet-O datasets in the Autoformer search space. Bold and underline stands for the best and second, respectively.
Originally Proposed For
Training-free NAS
Correlation with Acc
Correlation with Acc
Performance
Architecture
Grasp 74
CNNs
0.1490 0.1951
0.1503 + 0.1638
SNIP
Acc
CNNS
0,3750 0,3023
0.3178 + 0.3219
McCo
60
ID Acc
CNNs
0.1440 = 0.2371
0.1049 = 0.2860
CroZe 02
Adv Robustness
CNNs
0.3823 + 0.3046
0.3277 = 0.3307
Jacobian
75
Adv Robustness
CNNs
0.1053 + 0.1509
0.0911 + 0.1242
DSS 10
ID Acc
VITs
0.4165 0.3461
0.3546 + 0.3478
AutoProx-A
Acc
0.4023 0.3827
= 0.3490
#Param
0.4607 + 0.3318
0.3583 = 0.3559
#Flops
0.4705 + 0,3391
0.3445 + 0.3887
Average OoD
-N-Skeech
-Stllize-IIN
ID Acc
-NN-C
-N--
IN-P
IN--O
IN-A
IN-R
Depth
0.24
0.00
-0.23
0.40
0.22
0.49
0.49
0.14
0.24
0.22
Figure L.12: Kendall's 7 rank correlation coefficient between varying network depth and OoD accuracy.
For MLP Ratio and #Heads, we observe that the rank correlation coefficients of MLP Ratio and #Heads in Fig. L.14 respectively, reveals weak correlation between the #Heads/MLP Ratio and OoD generalization. This suggests that within the explored range, these two architectural attributes have non-obvious impact on the model's OoD generalization. To delve deeper into this observation, Section L.2 presents layer-wise analysis of these two architectural attributes in VIT models.
Average OoD
IN Sketch
Stylized-IN
ID Acc
IN-C
IN-D
IN-P
-NN-O
IN-A
IN-R
MLP Ratio
0.14
0.06
0.09
0.12
0.04
0.12
-0.01
-0.02
0.05
0.07
Figure L.13: Kendall's 7 rank correlation coefficient between mean MLP ratio and all OoD accuracy. We fix Embed Dim 384, Depth 13, and mean #Head 6 + 0.05.
31

---

Average OoD
-Stylized-IN
IN-Sketch
ID Acc
-NN-C
-IN-D
INN-P
-IN-O
-NN-A
-IN-R
#Heads 0.06
0.07
0.02
0.08
0.02
0.13
0.16
0.07
0.20
0.09
Figure L.14: Kendall's 7 rank correlation coefficient between the mean number of heads and all OoD accuracy. We fix Embed Dim 384, Depth 13, and mean MLP__atio 3.5 = 0.05.
L.2 Layer-Wiss Wise Analysis
The number of heads and MLP ratio vary across layers, which allows for searching more diverse ar- chitectures. In this section, we provide the layer-wise analysis of the influence of MLP Heads in each layer on OoD generalization.
Autoformer-Smmin search space, the MLP ratio at any given layer referred the i-th layer- be 3.0, 3.5, In addition architectures in benchmark, we create a set of 108 sampled architectures from Autoformer-Small! These architectures are fixed with three architectural design attributes: (Depth 12, Embed Dim 320, #Heads for all layers = 5). With these parameters fixed, the total number of potential MLP Tatio configurations is still extensive (i.e., 312), making it impractical
Experimental Setups:
absolutely fix the MLP Ration in our ablation study, To specifically assess the influence of the MLP RatRaoiio at the i-th layer, we further fix a constant MLP_Ration Ration across all other layers. For example, Fig. M.18 depicts the nine configurations of to analyze the impact of 5-th layer.
We carry similar layer- layer-wise analysis to demonstrate the effect of #Heads (i-th layer). In addition to architectures in our benchmark, we create a set of 108 sampled from Autoformer-Small. These architectures
specific layer
fixed with three architectural design attributes: (Depth = 12, Embed_ Dim =320, #MLP _atioo for all layers =3.0). To specifically assess the influence of the #Head at the i-th layer, we further fix a constant #Head across all other layers. For example, Fig. M.18 depicts the nine configurations of #Head to analyze the impact of 3-th layer.
Results. The layer-wise analysis for MLP Ratio are shown in Fig. the change in OoD Accuracy when varying MLP Ratio at a particular layer. While increasing improved OoD accuracy in some layers, it decreased it in others. This aligns with observations in Sec. 4.4 suggesting no clear overall impact MLP Ratio on OoD generalization.
Each sub- lab-figure demonstrates
The layer-wise analysis for #Head is illustrated in Fig. L.16 Similar to our observation in the layer-wise analysis for MI.P MLP__Ratio, Ratio, the impact of #Head is non-obvious, which consistent with our finding in Sec. 4.4
32

---

50
IN.C Fog
IB-C
1.0
15
40
100
15
4.0
10
11
Layer 9, 1D/00 Acc Bange 0.13/0.23
Layer 1, IDV00D Acc fango - 0.00y0.24
Layer Dr0s0 Acc flngg 0.0700:66
IN.C 100
3.0
15
4.
1.0
15
4.0
30
1+
Layer 3, 10/000 Acc lange 0.03/1.71
Layer 5, ID/00D Acc fange - 0.08/1.30
Layer 100000 Acc llngg 0.07/0.55
N4C 100
36
30
15
40
3.0
1.
4.
.00
xt
4.
Layer ID/DoD Acc Range 0.10/0.30
Layer 7. ID/00D Acc Rango - 0.17/.115
Layer 13/000 Ace Ronge 0.130.48
14C 100
INC.
3.0
1. Layer 9, ID/DoD Acc Range - 0,200.53
4.0
3.0
3.
4,0
1.0
xt
40
Layer 10, 1D/000 Acc Range- 0.36/0.00
Layer 11. IDDoDD Aoc Range - 2.18-.18
Figure L.15: A visualization architectures sampled from Autoformer-Smal
the effect of changing MLP ratio per layer to OoD accuracy in 108 the layer-wise study. To evaluate the effect of layer i-th, we fix the MLP ratio 4.0 for the remaining layers. We observe that slightly higher OoD accuracy range can be obtained by changing the MI.P ratio at layers and 5.
54.2
54.2
56.2
54.0
54.0
54.3
Layer D. ID:000 Acc Range 6.02/0.02
Layor ID/CKO Acc Range 0,04/C.13
Layer 2,/0/Oeb Acc Range 0.06/0.14
54.8
54.6
54.5
54.4
g 54.4
54.4
542
54.2
56.2
Layer 3. 10:000 Acc Raane 0.06/0.12
Layer4 IQJC6O Acc flane 5.08/0.11
Layer 5,/0/ Acc Range 0.02/00.0
$4.8
54.6
54.5
Layer Range 6.05/0.13
Lager IDICLO Rango 0.03/0.35
Layer 8.DDOOeD Range 0.03/0.05
11.
33.E
53.3
Hange
Layer
Bange 0.1100110
Layen Hange
Figure 1. visualization on the effect of changing #Heads per layer to OoD accuracy in 108 architectures sampled from Autoformer-Snn Small in the layer-wise study. To evaluate the effect of layer ë-th, we fix the #Heads = 6 for the remaining layers.
33

---

M Reproducibility
Configuration
Layer Number
Figure M.17: The possible configurations of #Heads at 3rd layer of 108 architectures sampled from Autoformer-Snen Small layer-wise analysis.
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.5
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
3.0
Configuration
3.5
3.5
3.5
3.5
3.5
3.0
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
3.5
4,0
4.0
4.0
3.0
4.0
4.0
4.0
10
11
Layer Number
Figure M.18: The possible configurations of MLP ratio at 5-th layer of 108 architectures sampled from Autoformer-Sman in the layer- wise study.
34

---

M.1 Hyper-parameters
Table M.11: Hyper-parameters for Evaluation on common large-scale OoD datasets. In total, we evaluate 3,000 diverse ViT architectures in our OoD-NAS- VIT benchmark sampled from Autoformer- Tiny/Small/Bar search spaces [6]. Input resolution is set to 224 224 pixels, with mean and standard deviation normalization applied using ImageNet statistics (mean [0.485, 0.456, 0.406], std = [0. (0.229, 0.224, 0.225)). Transformations follow the Standard ImageNet preprocessing, including resize and center crop.
IN-C
IN-P
IN-A
IN-O
IN-R
IN-Sketch
Stylized-IN
IN-D
Batch Size
256
256
256
64
256
256
256
100
Number of workers
10
10
10
10
10
10
To ensure reproducibility, we provide a detailed description the hyper-parameters used for eval-
uating 3,000 ViT architectures in OoD-NAS- -iTT benchmarks 8 common large-scale Ool
datasets: ImageNet-C [24], Iet-e (25 24 ImageNet-O [25]. ImageNet-P [2].. ImageNet-D
[26]. ImageNet-R 27 ImagehNt-Skktch (28). and Stylized ImageNet 29) The evaluation for
ImageNet-D, ImageNet-O and Stylized ImageNet strictly follows previous works [26 25 29 to
ensure consistency and comparability. The details of the evaluation are shown in Table M.11 The
evaluated architectures are sampled on AutoFormer- aiooFormer-TinySSmlllBB.e.. Search Spaces 6
M.2 Compute Resource
All experiments are conducted using NVIDIA RTX A6000 GPUs. We utilized 2 GPUs for each experiment. The substantial computational resources required for these evaluations underscore the complexity and scale of work. Detailed information the GPU-hour consumed for all experiments to construct our ViT--ASS Benchmark can be found in Mable|M.12) In total, the experiments demanded significant investment of approximately 3900 GPU-hours, reflecting the extensive computational effort involved.
Table M.12: GPU-Hour for Computational Resources. In total, we evaluate 3,000 diverse VIT architectures in our DoD-NAS- ViT benchmark sampled from Autoformer AutofommrrTiny/SSaallla Tiny/Small/Base search spaces 6
IN-C
IN-P
IN-A
IN-O
IN-R
IN-Sketch
Stylized-IN
IN-D
Total
GPU-Hour
2958 672
93
28
53
37
50
3903
35

---

Kendall's r correlation: 0.79
Kendall's 1 correlation: 0.80
Kendall's correlation: 0.77
Kendall's correlation: 0.77
81.5
OnD Acc IN P Iranslate
29.5
OoD AcE IN-F IN-P Snow
OnD Acc IN-P Rotate
78.0
25.5
81.0
77.5
75.0
77.0
78.5
74.0
73.5
76.0
15
25
30
15
20
15
20
25
15
20
25
30
#Param
#Param
Param
Param
Kendall's correlation 0.79
Kendall's correlation: 0.81
Kendall's correlation: 0.80
Kendall's correlation: 0.78
81.5
OnD Acc Zoom
70
OoD Acc IN-P Tilt
79.0
OnD Acc IN-P Scale
72
81.0
78.5
80.5
78.0
80.0
88
15
15
20
25
25
30
#Param Kendall's correlation: 0.79
#Param
#Param
Param
OnD Acc IN.P Gaussian Noise
Kendall's correlation:
Kendall's correlation:
0.82
Kendall's correlation: 0.77
OoD Ace IN-P Brightnesss
OoD Acc IN-P Motion Blur
75.0
29.0
74.5
75.9
74.0
80.5
73.5
80.0
74.5
73.0
70.5
74.0
77.0
#Param
#Param
#Param
Param Kendell's t correlation: 0.62
Kendall's t correlation:
0.79
Kendall's t correlation: 0.81
Kendall's correlation: 0.75
OnD Acc IN P Shot Noise
79.0
81.5
OoD IN- Skettch
8
2,205
28.5
2,200
10
AUPR
2.195
2.190
2.185
30
2,180
79.5
2.175
15
25
30
15
20
25
15
20
25
30
15
20
25
30
Paraa
#Param
#aram
# Param
Kendall's correlation: 0.79
Kendall's correlation: 0.69
Kendall's carrelation: 0.74
Kendall's correlation: 0.62
14
18.0
81.5
13
OoD Acc IN
OoD Acc Stylizzd--IN
17.5
81.0
31.0
12
17.0
00.5
30.5
OnD
36.5
16.0
15.5
15
20
25
30
15
20
30
15
20
25 #Poram
30
15
20
25
30
#Param
# Param
Param
Figure M.19: As in Figure we show that lower OoD accuracy can be obtained for the higher ID accuracy in the Pareto architectures of Autoformer-Small The left panels show the ID accuracy, and each panel on columns to 4 shows results from the OoD accuracy of IN-P, IN-A, IN-R, IN- IN-Sketch, Stylized-IN, and AUPR of IN-O.
36

---

Kendall's r correlation: 0.13
Acc IN-C Gaussian Noise
Kendall's r correlation: 0.19
Kandall's correlation: 0.15
Kendall's correlation: 0.06
51.5
82.3
OoD Acc IN- C Snow
82.2
48
51.0
OoD Acc IN-C C. Fog
82.1
50.5
47
50.0
49.5
46
50
60
70
30
60
70
50
50
70
30
60
70
#Param
#Param
# Param
# #Param
Kendall's correationn 0.13
Kendall's corre ation:
0004
Kendall's correlation: 0.04
Kendall's correlation: -0.06
OoD C- Motion Blur
OoD Acc IN-C Brightness
OoD Ace IN- Frost
$8.0
73.0
82.2
72.8
82.1
18
82.0
72.6
81.9
72.4
81.0
72.2
#Param Kendall's t correlation: 0.13
#Param
#arrm
Param
OoD Acc IN-C Defocus Blu
Kendall's correlation: 0.10
Kendall's correlation:
-0.37
OnD Acc IN-C Impulse Noise
Kendall's correlation: 0.07
82.3
OoD Acc IN-C Glass Blur
21.0
47
20.5
30.5
14
83
50
80
60
70
90
70
#Param
#
#aarrm
#Param
Kendall's r correlation: 0.13
Kendall's correlation: 0.03
Kandall's correlation: 0.04
Kendall's correlation: 0.19
OoD IN-C Motion Blur
OoD Acc IN-C C Zoom Blur
82.3
44
82.2
34.5
82.1
34.0
82.0
42
33.5
81.9
81.0
OoD
15
#Param Kendall's correlation:
OoD Acc IN Jpeg Compression
#
0.0
OoD Ace IN-C Elastic Transform
Kendall's conrelation:
#Param
correlation:
60.0
$2.0
OoD Acc IN-C Contrast
29.5
58.0
50
60
50
70
#Param
#Param
#Param
# Param
Kendall's t correlation:
0.13
Kendall's corre arreaationnooo -0.03
OoD IN-D Backgroudd
Kendall's correlation -0.31
Kendall's correlation: -0.18
82.3
OoD IN-D Texture
OoD Acc IN-D Material
16
82.2
82.1
82.0
12
81.9
20
50
60 Parrm
50
60
80
70
50
70
# Paaam
#Param
Param
Figure M.20: Visualization Pareto architectures in Autoformer-Base. The left panels show the ID
accuracy, and each panel on columns 2 to shows results from the OoD accuracy of IN-C common corruptions and IN-D.
37

---

Kendall's r correlation: 0.13
Kondall's correlation: 0.03
Kendall's correlation: 0.06
Kendall's correlation: 0.05
82.3
OnD Acc IN P. Itanslate
80.3
OoD Act IN-F Snow
76.4
OoD Acc IN-P Rotate
02.2
80.2
80.1
80.0
78.6
79.9
75.8
79.8
78.4
75.6
50
60
50
60
60
70
50
70
#Param
#Param
#
# +Paaa
Kendall's comeeation:
0.13
Kendall's correlation: 0.31
Kendall's correlation: 0.02
Kendall's correlation: 0.36
82.3
OnD Acc IN Zoom Blur
70.8
OoD Acc IN-P Scale
74.0
82.2
21.8
79.7
OoD Acc IN-P
73.5
02.1
71.6
73.0
82.0
79.5
81.9
79.4
72.5
01.0
79.3
71.0
71.5
50
60
70
50
70
#Param Kendall's Lcoreaatoon:
#Param Kendall's correlation:
Kendall's correlation:
OnD Acc P Gaussian Noise
0.01
Kendail's correlation:
OoD Ace IN-P Brightness
OoD Acc Motion Blur
80.2
80.1
77.3
75.9
77.2
75.8
82.0
77.1
81.9
77.0
75.6
81.8
75.5
#Param
#Param
#Param
#Param
Kendall's correlation: 0.13
Kendall's correlation: 0.07
Kendall's conrelatio: 0.01
Kendell's correlation: 0.50
82.3
OnD Are IN P Shot Noise
2:230
OoD IN. Sketch
2:255
79.9
2:200
79.8
2:155
79.6
2,200
30
2.205
50
80
60
50
70
50
70
#Param
#Param
#Param
Param
Kendall's correlation: 0.13
Kendall's correlation: 0.45
Kendall's correlation: 0.00
Kendall's correlation: 0.05
82.3
33.0
OoD Acc Stylized-IN
20.0
82.2
OoO Acc INA
24.5
OoD Acc IN-R
32.0
19.8
82.1
32.6
14.0
29.6
82.0
32.4
81.9
13.5
32.2
29.4
50
60
70
50
60
70
80
70
50
70
Param
# Param
#Parm
Param
Figure M.21: Visualization Pareto architectures in Autoformer The left panels show the ID accuracy, and each panel on columns 2 to 4 shows results from the OoD accuracy of IN-P, IN-A, IN-R, Sketch, Stylized-IN, and AUPR of IN-O.
38

---

Kendall's r correlation: 0.47
OoD IN-C Gaussian Noise
Kendall's correlation: 0.33
Kondall's correlation: 0.31
Kendall's correlation: 0.30
OoD Acc IN C Snow
75.4
OoD Acc IN- Fog
20
36.5
44
36.0
-5
35.5
74.6
18
35.0
6.0
6.5
7.0
6.0
63
7.0
6.0
0.5
7.0
6.0
6.5
7.0
#Param
#Param
# #Param
# #Param
Kendall's correlation: 0.47
Kendall's correlation:
0.55
Kendall's correlation:
0.12
Kendell's correlation: 0.55
OoD Arc IN-C Motion Blun
OoD IN Brigntness
OnD Ace IN Frost
43.5
24.5
43.0
75.2
24.0
62.75
42.5
25.0
23.5
62.50
42.0
74.8
62.25
74.6
22.5
22.0
2a
#Param Kendall's t correlation:
#Param
#Pora
Param
0.47
OoD Acc IN-C Defocus Blur
Kendall's come ation:
-0.01
Co0 Acc IN-C Glass Blur
10.50
Kendall's correlation: 0.09
OnD Acc IN-C Impulse Noise
Kendall's correlation: 0.29
75.4
20.25
21
20.0
25.2
20.00
9.75
19.0
9.50
9.25
18
28.0
9.00
6.0
6.5
6.0
6.5
7.0
6.0
6.5
7.0
7.0
#Param
#
#
# Param
Kendall's t correlation: 0.47
Kendail's correlation:
0.29
Kandall's correlation:
0.12
Kandall's t correlation -0.15
OnD Acc IN-C (otion Blur
22
24.5
22.75
24.0
22.50
75.0
OoD IN-C
23.5
74.8
23.0
22.00
74.6
18
22.5
22.0
21.75
7.0
6.5
7.0
#Param Kendall's correlation:
OGD Acc IN C Jpeg Compression
#Param
0.1
OoD Act IN-C Elastic Transform
#Param
correlation:
OnD Acc IN-C C. Contrast
75.4
75.2
43.0
25.0
25
75.0
24.5
20
42.0
24.0
10
41.5
6.0
6.0
6.5
#Param
#Param
#Param
Param
Kendall's t correlation: 0.47
Kendall's correlation:
0.01
Kendall's correlation: -0.00
Kendall's correlation: 0.11
OoD IN-D Background
OoD Ace IN-D Material
20
OoD D
E
/4.8
14
16
74.6
12
10
6.0
6.5 #aarm
7.0
6.0
6.5
7.0
6.0
6.5
7.0
6.0
6.5
7.0
#aaram
Param
# Param
Figure M.22: Visualization of Pareto architectures in Autoformer-Ting The left panels show the ID accuracy, and each panel on columns 2 to shows results from the OoD accuracy of IN-C common corruptions and IN-D.
39

---

71.0
OoD Acc IN- P Iranslate
73.2
OgD Acc IN-FR Snow
68.0
OoD Acc IN-P Rotate
15.4
67.8
70.8
23.0
67.6
70.6
72.6
67.2
70.2
72.4
67.0
70.0
6.0
6.5
7.0
6.0
6.5
7.0
6.0
6.5
7.0
6.0
6.5
7.0
# #Poram
#Param
#Param
Param
-OoD Acc IN Zoom blur
60.75
OoD Acc IN-P Tilt
72.4
63.50
60.50
72.2
63.25
75.0
60.25
72.0
63.00
74.8
60.00
62.75
59.75
74.6
71.6
62.50
29.50
62.25
6.5
7.0
6.0
6.5
7.0
6.0
6.5
#Param
#Param
#Param
Param
OnD Arc P Gaussian Noise
OoO Ace IN-P Brightnesss
OoD Acc Motion Blur
67.4
75.4
67.2
72.4
75.2
67.0
75.0
72.2
68.2
66.8
06.6
74.8
66.4
71.6
67.0
#Param
#Param
#Param
#Param
OnD Acc IN P Shot Noise
72.2
75.4
72.0
OoD Acc IN- Skeethh
AUPR IN-O
2.175
75.2
24.0
2.170
71.6
23.5
71.4
23.0
21.0
6.0
6.5
7.0
6.0
6.5
7.0
6.0
6.5
6.0
7.0
#Param
#Param
#Param
Param
5.25
25.0
OoD Acc Stylzzed-IN
33.2
Oel Acc IN
OoD Acc IN-R
13.0
4.75
4.50
24.0
12.6
74.8
23.5
12.4
74.6
27
3.75
12.2
6.0
6.5
7.0
6.0
6.5
7,0
6.0
6.5
7.0
6.0
6.5
7.0
Param
# Param
#Poram
# Param
Figure M.23: Visualization of Pareto architectures in Autoformer- The left panels show the ID accuracy, and each panel on columns 2 to 4 shows results from the OoD accuracy of IN-P, IN-A, IN-R, Sketch, Stylized-IN, and AUPR of IN-O.
40

---

Kendall's r correlation: 0.79
OoD Acc Gaussian Noise
Kendall's r correlation: 0.58
Kendall's correlation: 0.64
Kendall's correlation: 0.72
81.5
-6
OoD C Snow
30
OoD Acc IN- C. Fog
80
81.0
48
-4
8.
46
-0
15
20
25
30
15
20
25
15
20
25
30
13
20
25
30
#Param
#Faram
#Faarm
#Param
Kendall's correlation 0.79
Kendall's correlation: 0.54
Kendall's correlation: 0.22
Kendell's correlation: 0.73
01.5
OnD Frost.
-5
OoD IN- N-C Motion Blur
OoD Acc IN-C Brightness
72.5
71.0
81.0
34
70.5
1D
80.5
70.0
80.0
69.5
79.5
Param
#Param Kendall's correlation:
#Poram
Param
Kendall's t correlation:
0.79
OoO Acr IN-C Defocus Blui
20.5
OoD Acc IN-C Glass Blur
18.5
Kendall's correlation: 0.23
OnD Acc IN-C Impulte Noise
Kendall's correlation: 0.52
18.0
28.5
17.0
344
27.5
16.5
16.0
26.5
16
30
15
25
15
20
25
30
15
20
25
30
Paaaam
#
# +Para
Kendall's t correlation: 0.79
Kendail's correlation: 0.59
Kandall's correlation: 0,22
Kendail's correlation: 0.57
OoD Acc IN-C C. Jpeg Compression OoD IN-C Woise
OoD Acc IN-C Mation Blur
OoD Acc IN C. Zoom Blur
81.5
-4
81.0
32
22
60.5
-22
50
00.0
30
29
30
Kendall's correlation:
#Param
#aram
correlation:
C Elastic Transform 8
#Param
correlation:
OoD Acc IN-C IN-. Contrast
t
88
%
53
1D
80.0
78.5
30
8
OoD
15
15
25
20
25
#Param
Param
#Param
Param
Kendall's t correlation: 0.79
Kendall's corre ation: -0.33
Kendall's correlation -0.15
Kendall's t correlation: 0.12
81.5
OoD Texture
20
OoD IN-D Background
t
OoD Acc IN-D Material
22
81.0
30
16
=:
16
16
12
14
15 20
25
30
15
20
25
15
20
25 # Param
30
15
20
25
30
#Param
#aaram
Param
Figure M.24: As in Figure we show that lower OoD accuracy can be obtained for the higher ID accuracy in the Pareto architectures of Autoformer-Small The left panels show the ID accuracy, and each panel in columns IN-D. We observe that architectural designs have greater effect on OoD accuracy than ID accuracy. especially when OoD shifts become more severe.
4 shows results from OoD accuracy of IN-C common corruptions and
41

---

36
16607
25:37
Oet Acc IN-C Snite
46000
44842
Onl Auc INC Fay
60.04
sa/ca
$4.87
30
4B
57322
ACC
2204
13
13
47
4000
%
30
13
Degth
18.5
Motion
IN arighners
72
71/07)
0n0 Ace the Defocus Blus
28.0
29.5
30.09
OnD Azc IN Glosa Blur
18.00
17.5
12
31
20.5
28.5
137//
27193
17.0
12
20.0
6977
27.5
16.5
38.22
26.33
60.5
20
27.0
15.0
15.5
14
12
14
13 Degth
14
13
Degtth
Degtt
Depth
68:40
36
115.00
33,20
OnD Acc IN-C Shott Nose
2201
33.00
33,03
11.00
Oob Act IN-C Poseltte
Zoom
12
23.27
22
30
30
12
XX33
32.99
42
30
12
30
20/3
5
Ono
15
30
38
Depth
On0 MoCE-BEE Joeg Compresaio
OnD Acc IN-C Elestic Transform
55.17
OoD Acc IN-C Contrast
46 4kcen
46
3470
12
28
17.77
34
16:30
12
$1301
16
83
52.3
33.05
42 po
14.00
1661
17
50
12
13 Dopch
14
12
14
12
13
14
12
13 Depth
24
Depth
Death
15
Acc WN- backgmun
OnB Acc IN-D material
13
11209
12:28
20
19.9
49
17.7
16.07
arbbccing.on
12
13:30
26/7
16
Depth
Figure M.25: As in Figure 5| we show the potential impact of embedding dimension (Embed Eimb on OoD generalization in ViT' architectures sampled from Autoformer-Small The numbers denote the average OoD performance, and The data points with blue e, orange e, and green colours represent ViT architectures with the embedding dimension of 320, 384, and 448, respectively. Each panel shows results from the OoD accuracy of IN-C common corruptions and IN-D.
42

---

79.5
79/22"
P1/48
75.59
73.5
75:00
75.00
75.31
70.0
7//8s
78/02
70.5
10.0
PGL1S
coja2
exjes
70:30
27.5
73.0
78:60
78
14.5
7438
Onb IN-
23.0
78/72
70
69.0
68.5
80.95
73
7636
65.0
67.5
73.5
Deptt
Depth
79.0
70,40
79.07
70
09
72.0
73,84
72:14
72.18
29.0
73:00
78/01
7% 09
76:11
Coo IN Brighnee
75.03
76:44
IN-0 Canoeian
75.63
23.5
71.0
78.0
77300
70,00
20.0
77.99
13.0
76
1475
74.5
27.5
69.5
74.0
12
14
12
13
12
23
12
13
Depth
Death
Deipth
Depth
AZE Blur
74/41
74/00
74.63
74.00
74.05
Shot Nose
70/43
reles!
OxD Acc Stethh
32/46
25.21
38.17
2,205
2,200
2.195
26.0
78.0
75387
77.5 77.99
77885
50.34
30.23
30.00
2.140
2.30
Qc0
17.0
30
2,180
2.175
12
14
12
14
12
13
12
13 Depth
Depth
Depth
Depth
11142
OnD Acc Nev
11
132.40
31.5
31.24
21.30
On0 Styliedt.t
17.5
17.32
12
17.0
11.00
20:30
16.5
30
29.5
Depth
Deprth
Deptth
Figure M.26: As in Figure 5 we show the potential impact of embedding dimension OoD generalization in ViTs architectures sampled from Autoformer-Small. The numbers denote the average OoD performance, and The data points with blue e, orange e, and green colours represent ViT architectures with the embedding dimension of 320, 384, and 448, respectively. Each panel shows results from the OoD accuracy of IN-P. IN-A, IN-R, IN- IN-Sketch, Stylized-IN, and AUPR of IN-O.
on
43

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sec. 3 for our OoD- ViT-NAS benchmark, and Sec. 4 for analysis. The summary can be found Fig. Fig. and Tab.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes],
Justification: See Appx.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
44

---

Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: See Appx.
M
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
45

---

Answer: [Yes]
Justification:
See Appx.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee cttppsscccpbbbccpppecccccc http///bbb////piiieoooee/eaaaaaa casssnnnnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See Appx. M
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Tab. search space.
Our benchmark and analysis are conducted across datasets and
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
46

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appx. M 3900 GPU hours on a single Nvidia RTX A6000
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aaaraa...cppaaaaa//aaaaaa
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appx. B
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
47

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Appx. and Appx. M for description of dataset and experimental setups, respectively. We do not mention the license of the assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
48

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: See Appx. for for OoD-ViT-NAS benchmark. See Sec. and Appx. Eoro the description of the benchmark.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooell as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
49


Reviews:
[{"review_id": "DYBwPCmdv6", "reviewer": "Reviewer_HQRy", "summary": "This work presented OoD-ViT-NAS, a comprehensive NAS benchmark with a focus on out-of-distribution generalisation of vision Transformer architectures.  The authors created a benchmark with 3000 diverse ViT architectures evaluated on 8 common large-scale OoD datasets, and provided the first comprehensive investigation of how ViT architectures affect OoD generalisation.  They also conducted the first study exploring NAS for ViT’s OoD, analysing the impact of individual ViT architectural attributes on OoD generalisation.", "questions": "Line 119 shows that the work is based on Autoformer.  What is the reason behind this choice? What is new?  \n\nLine 145, OoD Classification Accuracy, what is the difference to the ID Acc in Line 142?  Maybe a formula could clearly explain that.\n\nLine 154, we make use of the One-Shot NAS approach.  Why not use zero shot NAS instead?  What are the benefits that one short NAS can bring but not zero shot NAS?\n\nTable 2 shows correlations of some zero proxies.  Can SWAP be applied here as it demonstrates a much higher correlation than those on the table? SWAP-NAS: Sample-Wise Activation Patterns for Ultra-fast NAS. Peng et. al. ICLR'24.", "limitations": "The authors discussed the limitations of this work, but in the appendix instead of the main text.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This study introduces the first large-scale benchmark for ViT OoD generalisation, which could be a valuable resource to the NAS field.\n\nThe authors conducted a thorough investigation on the impact of ViT architectures in relation to OoD generalisation, covering multiple aspects such as architectural attributes, correlation between ID and OoD performance and the effectiveness of training-free NAS methods.\n\nThis study reported several findings, such as the low correlation between ID and OoD accuracy, the superiority of simple proxies, i.e., model params and FLOPs, over more computationally complex training-free metrics. Although the latter has been revealed in other work on computer vision tasks, the authors further validate that point on the OoD tasks.\n\nThe reproducibility of this work is good as the authors provided a substantial amount of details of their methodology, hyper-parameters and computational resources.", "weaknesses": "This study is limited to the Autoformer architecture space, which may limit the generalisability of the findings to other ViT architectures.  Also it comprises the novelty of this work.\n\nAlthough the empirical studies are comprehensive, there are not many theoretical explanations for the observed phenomena.  More analysis and insights should be provided to support this work.\n\nThe presentation can be improved.  For example, Fig 1 appears on Page 2, but is first mentioned at Line 226, Page 7.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate positive feedback and the valuable comments of the Reviewer.\n\n>**Q1:** This study is limited to the Autoformer architecture space, which may limit the generalisability of the findings to other ViT architectures.\n\n**A1:** Autoformer is the most widely used search space in many recent ViT works [7, 65, 60, 10, 11, 17, 66, 67]. In addition, a comparison of other ViT search spaces is summarized in Table A1 below. In summary, AutoFormer allows us to effectively and efficiently pool a large number of ViTs for our OoD-ViT-NAS benchmark and enables us to deeply study ViT architecture attributes for OoD generalization.\n\n**Table. A1.** Compare Autoformer with other spaces.\n|Criteria|Autoformer [6]|S3 [16]|BossNAS [a1]|NAS-ViT [12]|PiT [a6]|\n|-|:-:|:-:|:-:|:-:|:-:|\n|One-shot NAS w/o the need of further finetuning/retraining|✓|✓|X|✓|X|\n|Availability of range of ViT architectural attributes  in the sampled ViT|✓|✓|X|X|✓|\n|Accessible source code/models|✓|X|✓|✓|✓|\n|\n\nOur findings could be applicable to numerous ViT works utilizing this search space. Additionally, the Autoformer search space is accessible and comprehensive, including important ViT structural attributes for model capacities [6] and accommodating a wide range of model sizes.\n\nFurthermore, we further confirm our main findings on human design ViT architecture. As shown in Table R.1 in the rebuttal PDF, the observation is consistent with our findings in the main paper, where embedding dimension is the most important ViT structural attribute to OoD generalization.\n\n>**Q2:** More analysis and insights should be provided to support this work.\n\n**A2:** Following Reviewer’s suggestion, in this rebuttal, we provide an additional frequency analysis on our main finding showing that increasing embedding dimensions helps ViT learn more High-Frequency-Component (HFC), leading to better OoD generalization.  \n\nParticularly, in the literature, a model achieves higher accuracy on HFC of the test samples indicates that the model has learned more HFC [a2, a5]. By learning more HFC, the model improves OOD generalization [a2, a3, a4]. \n\nOur hypothesis is that increasing embedding dimension helps ViTs learn more HFC resulting in improving OoD generalization. We strictly follow the experiment setups in [a4] for the HFC filtering process. As shown in Figure R.1 in the rebuttal PDF, when increasing embedding dimensions, the performances obtained on HFC of test samples are improved. This observation holds true across setups varying radius r when filtering HFC. According to [a2, a5], our observation supports that increasing embedding dimension helps ViT learn more HFC. This leads to better OoD generalization following [a2, a3, a4].\n\n>**Q3:** Fig 1 appears on Page 2, but is first mentioned at Line 226, Page 7.\n\n**A3:** We will improve the presentation as suggested in the revision.\n\n>**Q4:** What is the reason behind the choice of Autofomer? What is new?\n\n**A4:** Our reason for selecting AutoFormer is that: AutoFormer is the most widely used [7, 65, 60, 10, 11, 17, 66, 67]; In addition,  AutoFormer allows us to effectively and efficiently pool a large amount of ViT for our OoD-ViT-NAS benchmark and study a range of ViT structural attributes to OoD generalization.\n\nPlease refer to **Q1** for the details.\n\n>**Q5:** What is the difference between OoD and ID Acc.\n\n**A5:** The OoD Acc and ID Acc metrics are similar, except for computing on different evaluation data (i.e., ID dataset and OoD dataset for ID Acc and OoD Acc, respectively).\n\nSpecifically, the classification accuracy (Acc) is the number of correct predictions divided by the total number of data points: $Acc = \\frac{1}{N}\\sum_{i=1}^N\\mathbf{1}(T(x_i) = y_i)$\n\nWhere $T(.)$ is the classifier. $A = {(x_i​,y_i​)}_{i=1}^N$​ is the evaluation dataset, where $x_i$​ is the input and $y_i$​ is the label for the i-th data point. $N$ is the number of data points in $A$. Here, the difference between ID Acc and OoD Acc are Acc computed on different dataset $A$.\n\n>**Q6:** What are the benefits that one short NAS can bring but not zero shot NAS to construct OoD-ViT-NAS?\n\n**A6:** One-Shot NAS allows us to efficiently produce a large number of ViT (i.e., 3,000) with trained weights, by sampling from the supernet [20, 6]. In contrast, the Zero-Shot NAS approach does not provide ViT with trained weights. Therefore, Zero-Shot NAS is unable to construct the benchmark necessary for our investigation.\n\n>**Q7:** Result for SWAP proxy\n\n**A7:** We provide additional results for SWAP. It achieves promising results for CNN search spaces. However, as shown in Table A2 below, when dealing with ViT-based search spaces, SWAP becomes less effective than ViT-design proxies and simple proxies such as #Params or FLOPs, which is consistent with our observations.\n\n**Table. A2.** Experimental results for SWAP proxy\n|Training-free NAS|Originally Proposed For Performance|Originally Proposed For Architecture|Correlation with ID Acc|Correlation with OoD Acc|\n|:-:|:-:|:-:|:-:|:-:|\n|SWAP|ID Acc|CNNs|0.2651 ± 0.3381|0.1201 ± 0.1069|\n|\n\n>**Q8:** The authors discussed the limitations of this work, but in the appendix instead of the main text.\n\n**A8:** Thank you, we will bring limitation to main text if possible. \n\nWe sincerely hope that reviewer could consider increasing the ratings if our responses have addressed all the questions.\n\n[a1] Li et al. \"Bossnas: Exploring hybrid cnn-transformers with block-wisely self-supervised neural architecture search.\" ICCV 2021.\n\n[a2] Bai et al. \"Improving vision transformers by revisiting high-frequency components.\" ECCV 2022.\n\n[a3] Gavrikov et al. \"Can Biases in ImageNet Models Explain Generalization?.\" CVPR 2024.\n\n[a4] Wang et al. \"High-frequency component helps explain the generalization of convolutional neural networks.\" CVPR 2020.\n\n[a5] Shao et al. \"On the adversarial robustness of vision transformers.\" arXiv 2021.\n\n[a6] Heo et al. \"Rethinking spatial dimensions of vision transformers.\" ICCV 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fkaA0umYFa", "reviewer": "Reviewer_pt1z", "summary": "This paper studies the out of distribution generalization of vision transformer architecture designs. Specifically the paper studies how in-distribution accuracies relate to OOD accuracies for a set pf 3000 architectures. In addition the paper also studies the correlation between different zero-cost proxies and OOD accuracies (on different variants of imagenet), observing that simply parameters or FLOPs are good proxies. Furthermore, the paper also studies the impact of different architectural decisions on the OOD accuracies, discovering that embedding dimension is of utmost importance for OOD generalization. The authors release code and the raw results to reproduce their experiments on AutoFormer-T,B and S space.", "questions": "- Do the authors train their own supernet, or train a one-shot supernet from scratch?\n- How does this benchmark contribute to aiding development of future NAS techniques for OOD generalization?\n- Check weakness", "limitations": "The authors discuss limitations and broader impact sufficiently.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is the first one to study out-of-ditribution generalization of architectures discovered by NAS\n- The paper also studies zero cost proxies on the ViT space showing ineffectiveness of modern/newer ZCPs\n- The paper is written in a clear and easy to understand manner\n- Experiments on ZCPs are thorough and the results are insigntful", "weaknesses": "- Currently I am missing the potential use-cases of the benchmark. The paper is more of a study and not a method (and fits better in the datasets and benchmark track of NeurIPS). Also though the paper is posed as a benchmark,I miss the intended use-case of the benchmark. Do the authors intend on releasing surrogates based on proxies, to allow API based querying these metrics for any possible architecture?\n- The authors only study 3000 architectures, which are too few to lead to generalizable insights.\n- Analysis of architecture importance: The search space of AutoFormer is a bit biased in the sense that there is a larger (factor two) variabilities in embedding dimension and a less variability for number of layers. I wonder if this is the reason embedding dimension has a higher correlation with OOD accuracy compared to other architecture dimensions like number of heads, embedding dimensions?\n- Multi-Objective Search: Since inheriting weights of a supernet and evaluation is fast, generally one would perform multi-objective random or evolutionary search directly on the pretrained supernet. This would lead to different Pareto-Fronts for different dataset distributions. Could the authors also present the results of multi-objective search directly on the OOD datasets?\n- What is the cost/search time different in computing Pareto-Fronts based on MO-search v/s Zero-Cost-Proxies (a lot of these proxies are actually not zero cost)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the positive feedback and valuable comments. \n\n>**Q1:** The paper is more of a study and not a method (and fits better in the datasets and benchmark track of NeurIPS). \n\n**A1:**  We justify that our work is suitable for the NeurIPS main track as we develop novel insights leading to a simple yet effective method to improve OoD generalization for ViT. Specifically, our insight of increasing embedding dimensions improves OoD generalization of ViT leading to a simple yet effective method can produce high OoD performance ViTs architectures. This method can achieve ViT architectures that outperform well established human-designed ViTs (See Table R.2 in the rebuttal PDF)\n\nWe create the benchmark due to lack of comprehensive data for our in-depth analysis of ViT architecture attributes for OoD generalization. Subsequently, this benchmark is also valuable for future research and other use-cases.\n\nIn contrast, papers (e.g [b2, b3]) in benchmark and dataset track usually merely construct datasets, propose benchmark frameworks for specific tasks, or discuss improvements in dataset development.\n\n>**Q2:** Also though the paper is posed as a benchmark, I miss the intended use-case of the benchmark. \n\n**A2:** Our OoD-ViT-NAS serves as a comprehensive benchmark for training-free NAS, akin to other benchmarks [10, 11, 17, 61]. It allows us to assess the effectiveness of new training-free NAS. Compared to other benchmarks, our OoD-ViT-NAS emphasizes high-resolution input and ViT search spaces, addressing not only ID performance but also the first for OoD generalization.\n\nAdditionally, our benchmark allows us to analyze and discover new and improved approaches for finding ViT architectures that improve OoD accuracy.\n\n>**Q3:** Do the authors intend on releasing surrogates based on proxies, to allow API based querying these metrics for any possible architecture?\n\n**A3:** Thank you. We will consider your suggestion when releasing our benchmark publicly.\n\n>**Q4:** The authors only study 3000 architectures, which are too few to lead to generalizable insights.\n\n**A4:**  We study 3000 ViT architectures, which is significantly more than existing research on ViT architectures on OoD generalization (3 ViT architectures in [13], 10 ViT architectures in [14] and 22 ViT architectures in [15]). \n\nOur architectures consider important ViT structural attributes for model capacities [6] and support a wide range of model sizes.\n\n>**Q5:** If the bias in Autoformer search space is the reason embedding dimension has a higher correlation with OOD accuracy compared to other architecture dimensions like number of heads, embedding dimensions?\n\t\n**A5:** We argue that the bias of ViT structural attributes in Autoformer does not affect our main finding on embedding dimensions.\n\nReviewer is correct and there is a bias for the variability for embedding dimension than other structural attributes within Autofomer search space as we utilize pre-trained supernets Autofomer to efficiently and effectively produce a large number of ViTs enabling us to deeply study ViT architecture attributes for OoD generalization.\n\nTo address this bias, we conduct experiments on human design ViT. To achieve large variabilities in depth, MLP ratio and #Heads, we increase these attributes to the extent that the capacities of the altered architectures based on these attributes are larger than that based on embedding dimensions. As shown in Table R.1 in the rebuttal PDF, the results further confirms our findings that embedding dimension is the most important ViT structural attribute to OoD generalization. \n\n>**Q6:** Could the authors also present the results of multi-objective search directly on the OOD datasets?\n\n**A6:** Following the reviewer's suggestion, we provide additional Evolutionary Search (ES) results directly on OoD datasets in Table. 1B\n\nTo clarify, our main purpose is to produce a large number of ViTs with trained weights to analyze the impact of ViT structural attributes to OoD generalization. Therefore, we do not conduct multi-objective random or ES. In contrast, other NAS papers conduct such search algorithms to search for the best ID Acc architecture.\n\nIn response, we provide ES results directly on OoD datasets, conducting ES on the Autoformer-Small search space. We strictly follow the ES setting in [6], except using OoD dataset instead of ID dataset during ES.\n\n**Table 1B.** Our ES experiment.\n|Metrics during ES|IN-R OoD Acc|IN-Sketch OoD Acc|\n|:-:|:-:|:-:|\n|OoD Acc|32.09|33.81|\n|DSS|29.85|30.44|\n|#Param|31.65|33.28|\n\n>**Q7:** What is the cost/search time different in computing Pareto-Fronts based on MO-search v/s Zero-Cost-Proxies (a lot of these proxies are actually not zero cost)?\n\n**A7:** While the setup for MO-search particular for Autoformer is faster than other setups such as [b1], evaluating each candidate still takes longer time than computing “Zero-Cost-Proxies” for that candidate\n\nWe apologize for any confusion on the naming. To clarify, we consistently use the term “Zero-Cost-Proxies” as in previous works [10, 17, 60, 62].\n\n>**Q8:** Do the authors train their own supernet, or train a one-shot supernet from scratch?\n\n**A8:** To clarify, we did not train the supernet ourselves; instead, we used pre-trained supernets from Autoformer [6] to efficiently and effectively produce a large number of ViTs with trained weights\n\n> **Q9:** How does this benchmark contribute to aiding development of future NAS techniques for OOD generalization?\n\n**A9:** Please refer to **A1**\n\nWe hope the reviewer could consider increasing the ratings if our responses have addressed all questions.\n\n[b1] Real et al. \"Regularized evolution for image classifier architecture search.\" AAAI 2019.\n\n[b2] Zheng et al. \"Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning.\" NeurIPS Benchmark & Dataset 2021\n\n[b3] Croce et al. \"Robustbench: a standardized adversarial robustness benchmark.\" NeurIPS Benchmark & Dataset 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9PNQQhJhHj", "reviewer": "Reviewer_Lm5n", "summary": "This paper introduces OoD-ViT-NAS, a benchmark designed for evaluating ViT architectures' ability to generalize under Out-of-Distribution (OoD) shifts. This paper reveals that ViT design significantly impacts OoD generalization, In-Distribution (ID) accuracy does not reliably predict OoD performance, and simpler metrics like parameter count can sometimes predict OoD accuracy better than complex training-free NAS methods.", "questions": "Please address the weaknesses mentioned above.", "limitations": "No potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper propose a new benchmark for ViT NAS.", "weaknesses": "1. While the paper introduces a new benchmark, it doesn't sufficiently discuss the practical implications of the findings or potential applications. Including a section on how these insights can influence future ViT designs would add significant value. Furthermore, assessing whether ViTs designed based on these insights outperform those designed by humans would be meaningful to include in experiments and results.\n2. The analysis of the experimental results is somewhat superficial. It presents observations without delving into deeper discussions. There should be more profound insights and detailed discussions on why certain ViT architectural attributes perform better under OoD conditions. This could involve providing theoretical justifications or proposing hypotheses that could be explored in future research.\n3. The paper states that simple proxies like #Param and #Flop outperform more complex Training-free NAS methods but does not provide enough insight into why this might be the case. A deeper analysis of these findings, including possible reasons and implications, would strengthen the argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate positive feedback and the valuable comments of the Reviewer. \n\n>**Q1:** While the paper introduces a new benchmark, it doesn't sufficiently discuss the practical implications of the findings or potential applications. Including a section on how these insights can influence future ViT designs would add significant value. \n\n**A1:** Thank you for your suggestion. We clarify that in addition to the new benchmark, our study produces significant insights for guiding the design of ViT architectures. Specifically, among ViT structural attributes, increasing embedding dimension can generally improve OoD generalization of ViT architectures. Our insight leads to a simple method, which can achieve ViT that can outperform well-established human-designed ViT architectures. Please refer to **Q2** for the details.\n\nBesides, our investigation also provides valuable ViT architectural guidelines for OoD generalization:\n\n- First, departing from existing OoD generalization methods, our study is the first to show that ViT architecture design has a considerable influence on OoD generalization of ViT. This observation encourages future research to put more focus on ViT architecture research for OoD generalization.\n\n- Second, our study suggests that current architectural insights based on ID accuracy might not translate well to OoD generalization.\n\n- Third, when utilizing training-free NAS to search for an OoD generalization ViT, the state-of-the-art zero-cost proxies methods are ineffective. Instead, simple proxies like #Param or #FLOPs could be the best options available so far.\n\n> **Q2:** Furthermore, assessing whether ViTs designed based on these insights outperform those designed by humans would be meaningful to include in experiments and results.\n\n**A2:** Thank you for the insightful feedback. Following the reviewer's suggestion, we include a comparison between ViT architectures designed based on our insights and well-established human-designed ViT architectures, as shown Table. R.2 in the rebuttal PDF. The results demonstrate that architectures based on our insights outperform those designed by humans.\n\nFor example, scaling up ViT architecture (e.g., from ViT-B-32 to ViT-L-32) by humans typically involves compound scaling of multiple ViT structural attributes. However, our findings suggest that not all ViT structural attributes need to be increased to benefit OoD generalization. Among these attributes, increasing the embedding dimension is the most crucial factor for improving OoD generalization. By only increasing the embedding dimension, our ViT architectures (e.g., Increasing embedding dimension of ViT-B-32) are significantly more efficient and outperform compound scaling architectures (e.g., ViT-L-32).\n\n>**Q3:** Providing theoretical justifications or proposing hypotheses that could be explored in future research.\n\n**A3:** Following Reviewer’s suggestion, in this rebuttal, we provide an additional frequency analysis on our main finding showing that increasing embedding dimensions helps ViT learn more High-Frequency-Component (HFC), leading to better OoD generalization.\n\nParticularly, in the literature, a model achieves higher accuracy on HFC of the test samples indicates that the model has learned more HFC [c1, c4]. By learning more HFC, the model improves OOD generalization [c1, c2, c3].\n\nOur hypothesis is that increasing embedding dimension helps ViTs learn more HFC resulting in improving OOD generalization. We strictly follow the experiment setups in [c3] for the HFC filtering process. As shown in Figure R.1 in the rebuttal PDF, when increasing embedding dimensions, the performances obtained on HFC of test samples are improved. According to [c1, c4], our observation supports that increasing embedding dimension helps ViT learn more HFC. This leads to better OoD generalization following [c1, c2, c3].\n\n>**Q4:** Simple proxies like #Param and #Flop outperform more complex Training-free NAS methods but does not provide enough insight into why this might be the case. A deeper analysis of these findings, including possible reasons and implications, would strengthen the argument.\n\n**A4:** Thank you for your feedback. Our first study on NAS for ViT on OoD generalization initially show that  #Param or #Flop outperforms more complex Training-free NAS methods. A possible reason is that existing Training-Free NAS are specifically designed for ID performance, not for OoD generalization. In the future work, we aim to further understand this observation.\n\nWe sincerely hope that reviewers could consider increasing the ratings if our responses have addressed all the questions.\n\n[c1] Bai et al. \"Improving vision transformers by revisiting high-frequency components.\" ECCV 2022.\n\n[c2] Gavrikov et al. \"Can Biases in ImageNet Models Explain Generalization?.\" CVPR 2024.\n\n[c3] Wang et al. \"High-frequency component helps explain the generalization of convolutional neural networks.\" CVPR 2020.\n\n[c4] Shao et al. \"On the adversarial robustness of vision transformers.\" arXiv 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BAYkVuLtaW", "reviewer": "Reviewer_f7Da", "summary": "The paper shows that in-domain accuracy and Training-free NAS accuracy predictions correlate poorly with out-of-domain accuracy, while characteristics of the model such as Flops, Params and Embed-Dim (number of channels) correlate much better with out-of-domain accuracy. To do this, they train a supernet that contains all possible architectures (subnets), then they validate these subnets on 9 datasets ImageNet/C/D/P/O/A/R/Sketch/Stylaized.", "questions": "- Have you tried plotting OOD-accuracy versus Latency or GPU memory consumption while increasing various model parameters such as: embed_dim, depth, MLP-ratio, num of heads, etc.? Or find correlations between ODE accuracy and these parameters, taking into account their impact on latency and memory consumption.\n\n- Have you tried to find optimal model scaling factors (similar to works [1, 2, 3]) to increase OOD-accuracy in optimal way with respect to latency or memory consumption?\n\n\n1. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, Mingxing Tan, Quoc V. Le, 2019\n2. EfficientNetV2: Smaller Models and Faster Training, Mingxing Tan, Quoc V. Le, 2021 \n3. EfficientDet: Scalable and Efficient Object Detection, Mingxing Tan, Ruoming Pang, Quoc V. Le, 2019", "limitations": "It might be worth adding to the limitations:\nThe results obtained on Imagenet classification datasets may not correlate with the results of real-life problems. Because Imagenet classification requires to predict the class of only one usually large object in the low resolution image without predicting its location (box, polygon or mask), whereas in real problems it is usually necessary to predict the class and position of many objects, incl. very small in high resolution image. Moreover, there is ambiguous in the Imagenet classification task, the class of which of the many objects in the image should be predicted?\nThus, a continuation of the research in this direction may be the finding of correlations, NAS approaches, new network structures, their parameters and scaling factors to achieve the highest OOD accuracy on tasks close to real ones (such as Dense prediction tasks: Segmentation, Detection, etc. ) in an optimal way with respect to latency and memory consumption.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "They explore various ways for increasing OOD accuracy, which is most important for real-world problems. They show that conventional approaches, such as Training-free NAS or relying on in-domain accuracy, do not work. While they found that Flops, Param and Embed-dim correlated well with OOD accuracy. To do this, they introduce OoD-ViT-NAS benchmark for NAS research on ViT’s OOD generalization, that includes 3000 diverse ViT architectures which are evaluated on 9 datasets. This allows us to look for new and better approaches for finding new models structures to improve OOD accuracy.", "weaknesses": "You show that the embedding dimension has the highest impact among ViT architectural attributes, while network depth has a slight impact on OoD generalization. But the paper lacks an explanation that although embed-dim (number of channels) correlates with OOD accuracy better than depth, this does not necessarily mean that to increase OOD accuracy only embed-dim should be increased, since the computational cost and memory costs may be different for embed-dim and depth. Therefore, to create either the largest and most accurate model that fits in memory, or the most optimal model in terms of speed and accuracy, it may be optimal to increase the depth rather than embed-dim. There are also papers [1,2,3] that show that it is necessary to simultaneously increase several model parameters (depth, number of channels, resolution) in optimal proportions.\n\nYou present charts of OOD accuracy and number of parameters (Fig. 4, K19 - K24), but it would also be great to present charts of OOD-accuracy and latency, and OOD-accuracy and GPU memory consumption. Because these 3 model characteristics (OOD-accuracy, Latency, Memory consumption) are the most important for real-world problems.\n\n\n1. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks, Mingxing Tan, Quoc V. Le, 2019\n2. EfficientNetV2: Smaller Models and Faster Training, Mingxing Tan, Quoc V. Le, 2021 \n3. EfficientDet: Scalable and Efficient Object Detection, Mingxing Tan, Ruoming Pang, Quoc V. Le, 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate positive feedback and the valuable comments of the Reviewer. \n\n> **Q1:** You show that the embedding dimension has the highest impact among ViT architectural attributes, while network depth has a slight impact on OoD generalization. But the paper lacks an explanation that although embed-dim (number of channels) correlates with OOD accuracy better than depth, this does not necessarily mean that to increase OOD accuracy only embed-dim should be increased, since the computational cost and memory costs may be different for embed-dim and depth. Therefore, to create either the largest and most accurate model that fits in memory, or the most optimal model in terms of speed and accuracy, it may be optimal to increase the depth rather than embed-dim. There are also papers [1,2,3] that show that it is necessary to simultaneously increase several model parameters (depth, number of channels, resolution) in optimal proportions.\n\t\n**A1:** Thank you for the insightful comments. Taking into account the constraint on memory, we design an experiment based on human design ViT architecture that provides variability to respond to reviewer’s concerns. The results can be found in Table R.1 in the rebuttal PDF. The results are generally consistent with our findings, where increasing embedding dimension is the most effective solution among ViT architectural attributes to improve OoD generalization.\n\nIncreasing multiple ViT structural attributes optimally is very interesting; we will study this direction in the future.\n\n>**Q2:** You present charts of OOD accuracy and number of parameters (Fig. 4, K19 - K24), but it would also be great to present charts of OOD-accuracy and latency, and OOD-accuracy and GPU memory consumption. Because these 3 model characteristics (OOD-accuracy, Latency, Memory consumption) are the most important for real-world problems.\n\n**A2:** Following your suggestion, we provide charts of OOD-accuracy and latency, and OOD-accuracy and GPU memory consumption in Figure R.3 and Figure R.4 in the rebuttal PDF.\n\n> **Q3:** Have you tried plotting OOD-accuracy versus Latency or GPU memory consumption while increasing various model parameters such as: embed_dim, depth, MLP-ratio, num of heads, etc.? Or find correlations between ODE accuracy and these parameters, taking into account their impact on latency and memory consumption.\n\n**A3:** Following your suggestion, we provide charts of plotting OOD-accuracy versus FLOPs while increasing various ViT structural attributes in Figure R.2 in the rebuttal PDF.\n\n> **Q4:** Have you tried to find optimal model scaling factors (similar to works [1, 2, 3]) to increase OOD-accuracy in optimal way with respect to latency or memory consumption?\n\n**A4:**  We have not tried this setup yet. In this study, we focus on the impact of individual ViT structural attributes on OoD generalization. Optimal scaling of multiple ViT structural attributes is an interesting direction for our future research.\n\n> **Q5:** It might be worth adding to the limitations: The results obtained on Imagenet classification datasets may not correlate with the results of real-life problems. Because Imagenet classification requires to predict the class of only one usually large object in the low resolution image without predicting its location (box, polygon or mask), whereas in real problems it is usually necessary to predict the class and position of many objects, incl. very small in high resolution image. Moreover, there is ambiguous in the Imagenet classification task, the class of which of the many objects in the image should be predicted? Thus, a continuation of the research in this direction may be the finding of correlations, NAS approaches, new network structures, their parameters and scaling factors to achieve the highest OOD accuracy on tasks close to real ones (such as Dense prediction tasks: Segmentation, Detection, etc. ) in an optimal way with respect to latency and memory consumption.\n\n**A5:** We believe this is a common issue among most OoD generalization methods that focus on image classification. We agree with the reviewer's perspective on this limitation and will add it in the limitation section in the revision. This is an interesting direction for our future work.\n\nWe sincerely hope that reviewers could consider increasing the ratings if our responses have addressed all the questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their valuable time and effort to review our work. We appreciate Reviewers’ kind comments, such as:

- "The authors conducted a thorough investigation on the impact of ViT architectures in relation to OoD generalisation, covering multiple aspects such as architectural attributes, correlation between ID and OoD performance and the effectiveness of training-free NAS methods" (Reviewer HQRy)
- "They explore various ways for increasing OOD accuracy, which is most important for real-world problems" (Reviewer f7Da)
- "The paper is the first one to study out-of-ditribution generalization of architectures discovered by NAS" (Reviewer pt1z)
- "Experiments on ZCPs (Zero-Cost Proxies) are thorough and the results are insightful" (Reviewer pt1z)
- "This study introduces the first large-scale benchmark for ViT OoD generalisation, which could be a valuable resource to the NAS field" (Reviewer HQRy)
- "OoD-ViT-NAS benchmark allows us to look for new and better approaches for finding new models structures to improve OOD accuracy" (Reviewer f7Da)
- "The reproducibility of this work is good as the authors provided a substantial amount of details of their methodology, hyper-parameters and computational resources" (Reviewer HQRy)
- "The paper is written in a clear and easy to understand manner" (Reviewer pt1z)

We would also like to express our appreciation to all the Reviewers for giving us the opportunity to clarify our work, as well as the constructive comments.

Based on Reviewers’ suggestions, in this rebuttal, we include:

- Additional results to further validate our findings on human design ViT setups
- Additional analysis to further understand why increasing ViT embedding dimension can improve OoD generalization
- Additional results on the SWAP zero-cost proxy
- Additional results on evolutionary search directly on OoD dataset
- Additional observations on Latency and Memory consumption 

Importantly, additional results are consistent with our findings in the main paper. 

In what follows, we provide comprehensive responses to all questions. We could provide more details if there are further questions. We hope that our responses can address the concerns.

Meta-review: While initial reviewer scores were mixed, the rebuttal helped clarify and alleviate most concerns. In the end, all reviewers suggest acceptance, and the AC and SAC see no reason to overrule their suggestions, hence we accept the paper.

However, we would like to remind the authors to follow through their duty to include the clarifications and additional results from the rebuttal in their paper for the camera-ready release; they really did help improve the clarity of understanding and confidence in the method.
Decision: Accept (poster)