SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qinsi Wang $^{1*}$
Jinghan Ke $^{1*}$
Zhi Liang
Sihai Zhang<sup>3,4</sup>
$^{2}$ School of Life Sciences, University of Science and Technology of China
<sup>3</sup>Key Laboratory of Wireless-Optical Communications, Chinese Academy of Sciences
$^{4}$ School of Microelectronics, University of Science and Technology of China [wqs,jinghan]@mail.ustc.edu.cn, {liangzhi,shzhang}@ustc.edu.cn
Neural Architecture Search (NAS) has emerged as a favoured method for unearthing effective neural architectures. Recent development of large models has intensified the demand for faster search speeds and more accurate search results. However, designing large models by NAS is challenging due to the dramatical increase of search space and the associated huge performance evaluation cost. Consider a typical modular search space widely used in NAS, in which a neural architecture consists of  $m$  block nodes and a block node has  $n$  alternative blocks. Facing the space containing  $n^m$  candidate networks, existing NAS methods attempt to find the best one by searching and evaluating candidate networks directly. Different from the general strategy that takes architecture search as a whole problem, we propose a novel divide-and-conquer strategy by making use of the modular nature of the search space. Here, we introduce MathNAS, a general NAS framework based on mathematical programming. In MathNAS, the performances of the  $m * n$  possible building blocks in the search space are calculated first, and then the performance of a network is directly predicted based on the performances of its building blocks. Although estimating block performances involves network training, just as what happens for network performance evaluation in existing NAS methods, predicting network performance is completely training-free and thus extremely fast. In contrast to the  $n^m$  candidate networks to evaluate in existing NAS methods, which require training and a formidable computational burden, there are only  $m * n$  possible blocks to handle in MathNAS. Therefore, our approach effectively reduces the complexity of network performance evaluation. The superiority of MathNAS is validated on multiple large-scale CV and NLP benchmark datasets. Notably on ImageNet-1k, MathNAS achieves 82.5% top-1 accuracy, 1.2% and 0.96% higher than Swin-T and LeViT-256, respectively. In addition, when deployed on mobile devices, MathNAS achieves real-time search and dynamic network switching within 1s (0.4s on TX2 GPU), surpassing baseline dynamic networks in on-device performance. Our code is available at https://github.com/wangqini1/MathNAS.
Neural Architecture Search (NAS) has notably excelled in designing efficient models for Computer Vision (CV) [1, 2, 3, 4] and Natural Language Processing (NLP) [5, 6, 7] tasks. With the growing popularity of the Transformer architecture [8, 9], designers are increasingly drawn to using NAS to
develop powerful large-scale models. Many existing NAS studies focus on designing search spaces for large models and conducting searches within them [10, 11, 12].
However, designing large models by NAS is challenging due to the dramatical increase of search space and the associated huge performance evaluation cost [10, 13]. Consider a widely used modular search space, in which a neural architecture is treated as a topological organization of  $m$  different block nodes and each block node has  $n$  different block implementations. Obviously, the number of possible networks or neural architectures,  $n^m$ , grows explosively with  $n$  and  $m$ . In addition, candidate networks of large models are larger and require more computation for performance evaluation. Therefore, in order to conduct an effective architecture search, a proper search strategy and a suitable performance evaluation method are extremely important.
It is noteworthy that to improve search strategy, recent researches [14, 15] convert NAS to mathematical programming (MP) problems, which substantially decrease the search cost. MP-NAS provides a promising direction for rapidly designing large models. However, current MP-NAS methods exhibit certain architectural constraints. For example, DeepMAD [14] is solely applicable for architecture design within CNN search spaces, and LayerNAS [15] is exclusively suitable for hierarchically ordered search spaces. These limitations impede the application of MP-NAS methods to advanced search spaces, such as SuperTransformer [13, 5]. Besides, alternative strategies for effective performance evaluation of candidate networks are also expected, despite the improvement brought by parameter sharing [16, 17], performance prediction based on learning curves [18, 19] and so on.
In this study, we introduce MathNAS, a novel MP-NAS framework for universal network architecture search. In contrast to previous studies which estimate the performance of a network by solving a whole problem, MathNAS adopts an alternative divide-and-conquer approach. In brief, MathNAS improves the performance evaluation of a candidate network by estimating the performance of each block of the network first, and then combining them to predict the overall performance of the network. Although estimating block performance involves network training, predicting network performance is completely training-free. Therefore, this approach reduces the complexity of network performance evaluation. MathNAS achieves further improvement of the search strategy by transforming NAS to a programming problem, reducing the search complexity to polynomial time.
MathNAS contains three key steps:
- Block performance estimation: The performance of each block is estimated by the performance difference between networks having and having not that specific block.
- Network performance prediction: The performance of a network is predicted based on the performances of its blocks.
- NAS by ILP: Utilizing the block performances, NAS is solved as an Integer Linear Programming (ILP) problem.
We perform experiments on search spaces with various network architectures, including NAS-Bench-201 (GNN) [20], MobileNetV3 (CNN) [21], SuperTransformer (Transformer) [5] and NASViT (CNN+Trans) [13]. Our experiments demonstrate that predicting network performance based on its blocks' performances is applicable to different network architectures. In particular, the Spearman coefficient between the actual and the predicted top-1 indices on four different search spaces achieve 0.97, 0.92, 0.93, and 0.95, respectively. At the same time, by using the merit of the divide-and-conquer strategy to transform NAS into an ILP problem, MathNAS can find models superior to state-of-the-art (SOTA) models across different search spaces and tasks. In CV tasks, MathNAS achieves  $82.5\%$  top-1 accuracy on ImageNet-1k with 1.5G FLOPs and 15M parameters, outperforming AutoFormer-small  $(81.7\%)$  and LeViT-256  $(81.6\%)$ . In NLP tasks, MathNAS reaches a Blue Score of 28.8, on par with Transformer (28.4), but requiring only 1/5 of the FLOPs. In summary, our contributions are as follows:
1. We propose a general framework for performance evaluation of candidate networks by estimating block performance first and then combining them to predict network performance, which greatly improves the evaluation efficiency.
2. By virtue of the established mapping between block performance and network performance, we transform NAS into an ILP problem, which reduces the search complexity to polynomial.
3. We demonstrate MathNAS by considering three key performance indices for network design, i.e. accuracy, latency and energy, and achieve results superior to SOTA models.
Search Space and Notations. In this paper, we consider a widely used modular search space  $\mathcal{S} = \{\mathcal{N}_1,\dots,\mathcal{N}_k,\dots\}$ , in which network  $\mathcal{N}_k$  consists of  $m$  block nodes  $\mathcal{B}_i(1\leq i\leq m)$ . For a block node, there is  $n$  alternative blocks, i.e.  $\mathcal{B}_i = \{b_{i,1},\ldots ,b_{i,j},\ldots ,b_{i,n}\}$ , where block  $b_{i,j}(1\leq i\leq m,1\leq j\leq n)$  represents the  $j$ -th implementation of the  $i$ -th block node. A network can therefore be denoted as  $\mathcal{N} = (b_{1,\mathcal{J}_1},\dots,b_{i,\mathcal{J}_i},\dots,b_{m,\mathcal{J}_m})$ , with its  $i$ -th block node implemented by block  $b_{i,\mathcal{J}_i}$ . Totally, there are  $m^n$  possible networks in the search space. Here, we focus on three key performance indices for network design, i.e. accuracy, latency and energy consumption. The accuracy, latency and energy consumption of network  $\mathcal{N}$  are denoted as  $Acc(\mathcal{N})$ ,  $Lat(\mathcal{N})$ ,  $Eng(\mathcal{N})$ , respectively.
2.1 Problem Formulation: Reduce NAS Search Complexity from  $\mathcal{O}(n^m)$  to  $\mathcal{O}(m*n)$ .
The objective of hardware-aware NAS is to find the best neural architecture  $\mathcal{N}^*$  with the highest accuracy under limited latency  $\hat{L}$  and energy consumption  $\hat{E}$  in the search space  $S$ :
$$
\mathcal {N} ^ {*} = \underset {\mathcal {N} \in \mathcal {S}} {\arg \max } A c c (\mathcal {N}), s. t. L a t (\mathcal {N}) \leq \hat {L}, E n g (\mathcal {N}) \leq \hat {E} \tag {1}
$$
In order to fulfill the above goal, NAS has to search a huge space, evaluate and compare the performance of candidate networks. Early NAS studies usually fully train the candidate networks to obtain their performance ranking, which is prohibitively time-consuming. Subsequent works introduce various acceleration methods. For instance, the candidate networks can avoid training from scratch by sharing weights [16, 17]. Also, the performance of a candidate network can be predicted based on the learning curve obtained from early termination of an incomplete training [18, 19]. Despite of these improvements, a candidate network has to be trained, no matter fully or partially, to obtain a reasonable performance evaluation. And the huge number of candidate networks in the search space poses a formidable efficiency challenge.
However, the modular nature of the search space may provide us with a novel possibility. Although there are  $n^m$  candidate networks in the search space, they are all constructed by the  $n * m$  blocks. If we can evaluate the performances of the blocks by training of a limited number of networks, and if we can combine these block performance indices to obtain a reasonable performance evaluation of networks, we can reduce the complexity of network performance evaluation from  $\mathcal{O}(n^m)$  to  $\mathcal{O}(n * m)$ .
Guided by this idea, we reformulate the search of  $\mathcal{N}^*$  from  $S$  as a succession of sub-problems. Each sub-problem corresponds to the task of searching the block  $b_{i,\mathcal{I}_i^*}$  with the highest accuracy within the block node  $\mathcal{B}_i$ . This approach notably simplifies the original problem:
$$
\begin{array}{l} \mathcal {N} ^ {*} = \left(b _ {1, \mathcal {J} _ {1} ^ {*}}, b _ {2, \mathcal {J} _ {2} ^ {*}}, \dots , b _ {m, \mathcal {J} _ {m} ^ {*}}\right) = \arg \max  \left(b _ {1, j} ^ {A}\right) \oplus \arg \max  \left(b _ {2, j} ^ {A}\right) \oplus \dots \oplus \arg \max  \left(b _ {m, j} ^ {A}\right), \\ b _ {1, j} \in \mathcal {B} _ {1} \quad b _ {2, j} \in \mathcal {B} _ {2} \quad b _ {m, j} \in \mathcal {B} _ {m} \tag {2} \\ s. t. \sum_ {i = 1} ^ {m} b _ {i, \mathcal {J} _ {i} ^ {*}} ^ {L} \leq \hat {L}, \sum_ {i = 1} ^ {m} b _ {i, \mathcal {J} _ {i} ^ {*}} ^ {E} \leq \hat {E}, \\ \end{array}
$$
where  $b_{i,j}^{A}, b_{i,j}^{L}, b_{i,j}^{E}$  represent the accuracy, latency, and energy of block  $b_{i,j}$  respectively.  $\oplus$  denotes the operation of adding a block to the network. With this approach, each block  $b_{i,j}$  is searched only once and the complexity can be effectively reduced to  $\mathcal{O}(n * m)$ . However, due to the mutual influence between blocks, a unified understanding of the relationship between the performance of  $\mathcal{N}$  and its constituent blocks remains elusive, posing a challenge to the application of this method.
2.2 Divide-and-Conquer: Network-Block-Network
Consider the switching process  $\mathcal{N}_{(i,1)\to (i,\mathcal{J}_i)}$ , signifying the selection of the  $i$ -th  $\mathcal{B}$  in network  $\mathcal{N}$  as it switches from  $b_{i,1}$  to  $b_{i,\mathcal{J}_i}$ , with other selected blocks remaining unchanged. Thus, any network  $\mathcal{N} = (b_{1,\mathcal{J}_1}, b_{2,\mathcal{J}_2}, \dots, b_{m,\mathcal{J}_m})$  can be viewed as the outcome of the base network  $\widetilde{\mathcal{N}} = (b_{1,1}, b_{2,1}, \dots, b_{m,1})$  undergoing  $m$  sequential switching processes. Guided by this idea, we explore two aspects:
E1: Can block performance be directly calculated as with network performance?
Considering the entire search space  $\mathcal{S}$ , let us denote the collection of networks with the selected block  $b_{i,1}$  as  $\mathcal{N}_{(i,1)}^{\Omega}$ , which comprises  $n^{m-1}$  networks. For any network  $\mathcal{N}_{(i,1)}$  in  $\mathcal{N}_{(i,1)}^{\Omega}$ , the switching process  $\mathcal{N}_{(i,1)\to (i,j)}$  signifies the selection of the  $i$ -th  $\mathcal{B}$  in network  $\mathcal{N}$  as it switches from  $b_{i,1}$
(a)  $\Delta Acc(\mathcal{N}_{(i,1)\to (i,j)})$  vs  $\mathcal{F}(\mathcal{N})$  on NAS-Bench-201 (left) and MobileNetV3 (right).
(b)  $\Delta Lat(\mathcal{N}_{(i,1)\to (i,j)})$  (left)/  $\Delta Eng(\mathcal{N}_{(i,1)\to (i,j)})$  (right) vs  $\mathcal{F}(\mathcal{N})$  in NAS-Bench-201 on edgegpu.
Figure 1: The relationship between  $\Delta Acc(\mathcal{N}_{(i,1)\to (i,j)})$ ,  $\Delta Lat(\mathcal{N}_{(i,1)\to (i,j)})$ ,  $\Delta Eng(\mathcal{N}_{(i,1)\to (i,j)})$  and  $\mathcal{F}(\mathcal{N})$ . We conduct experiments on two search spaces: (1) NAS-Bench-201 [20]. Graph Network. We use the accuracy obtained on ImageNet after training each network independently. We count the accuracy of all networks in the search space. (2) MobileNetv3 [21]: sequentially connected CNN network. We use the accuracy obtained by the network with shared weights. We sample 5 blocks per block node and count the accuracies of 3125 subnetworks on the ImageNet validation set.
to  $b_{i,j}$ , with other selected blocks remaining unchanged. In this switch, two changes occur. The first change sees the selected block in  $\mathcal{B}_i$  switching from  $b_{i,1}$  to  $b_{i,j}$ . The second change arises from an internal adjustment in  $\mathcal{B}_i$ , modifying its interactions with other block spaces in the network. These changes lead to a difference in performance between  $\mathcal{N}_{(i,1)}$  and  $\mathcal{N}_{(i,j)}$ , denote as  $\Delta Acc(\mathcal{N}_{(i,1)\rightarrow (i,j)})$ ,  $\Delta Lat(\mathcal{N}_{(i,1)\rightarrow (i,j)})$ ,  $\Delta Eng(\mathcal{N}_{(i,1)\rightarrow (i,j)})$ . By averaging performance difference obtained from all  $n^{m - 1}$  switching processes,  $\mathcal{N}_{(i,1)\rightarrow (i,j)}^{\Omega}$ , we can derive two key parameters:
1.  $\Delta \phi (\mathcal{B}_{(i,1)\to (i,j)})$ , the change in inherent capability of  $\mathcal{B}_i$ .
2.  $\Delta \Phi (\mathcal{B}_{(i,1)\to (i,j)})$ , the change in the interactive capability of  $\mathcal{B}_i$  within  $S$ .
Accordingly, we define the performances of  $b_{i,j}$  as:
$$
b _ {i, j} ^ {A} = \overline {{\Delta A c c \left(\mathcal {N} _ {(i , 1) \rightarrow (i , j)} ^ {\Omega}\right)}} = \Delta \phi \left(\mathcal {B} _ {(i, 1) \rightarrow (i, j)}\right) + \Delta \Phi \left(\mathcal {B} _ {(i, 1) \rightarrow (i, j)}\right) \tag {3}
$$
Similarly,  $b_{i,j}^{L}$  and  $b_{i,j}^{E}$  can be calculated employing the same methodology. The unbiased empirical validation and theoretical proof supporting this method can be found in the Appendix.
E2: How can we predict network performance using block performance?
To accurately derive the performance difference stemming from the switching process, we consider the performance difference that a particular block switch brings to different networks. We performed the identical process  $\mathcal{N}_{(i,1)\to (i,\mathcal{J}_i)}$  over a range of networks within  $\mathcal{N}_{(i,1)}^{\Omega}$ . The outcome is illustrated in Figure 1, which depicts the relationship between the differences of three performances—latency, energy, and accuracy—and the FLOPs of the network. Our findings reveal that, within the same switching operation, latency and energy differences maintain consistency across different networks, while accuracy differences exhibit an inverse proportionality to the network's FLOPs,  $\mathcal{F}(\mathcal{N})$ .
This finding is logical, as networks with smaller FLOPs have lower computational complexity, rendering them more susceptible to block alterations. Conversely, networks with larger FLOPs exhibit higher computational complexity, making them less sensitive to individual block switching. In the Appendix, we proceed to fit the inverse relationship between accuracy difference and network FLOPs using various formulas. The results suggest that the accuracy difference is approximately inversely related to the network's FLOPs. Consequently, we can posit  $\Delta Acc(\mathcal{N}_{(i,1)\to (i,\mathcal{I}_i)}) = \alpha *1 / \mathcal{F}(\mathcal{N})$ , where  $\alpha$  represents a specific block's coefficient.
Based on these observations, we can deduce that for any network  $\mathcal{N}_{(i,1)}$  within the set  $\mathcal{N}_{(i,1)}^{\Omega}$ , the performance difference resulting from the switching process can be approximated as follows:
$$
\Delta L a t (\mathcal {N} _ {(i, 1) \rightarrow (i, j)}) \approx b _ {i, j} ^ {L}, \Delta E n g (\mathcal {N} _ {(i, 1) \rightarrow (i, j)}) \approx b _ {i, j} ^ {E}, \Delta A c c (\mathcal {N} _ {(i, 1) \rightarrow (i, j)}) \approx b _ {i, j} ^ {A} * \frac {\overline {{\mathcal {F} \left(\mathcal {N} _ {(i , j)} ^ {\Omega}\right)}}}{\mathcal {F} (\mathcal {N} _ {(i , j)})} \tag {4}
$$
(a)Lat on NB201
(b)Energy on NB201
(c)Acc on MBV3
(d)Acc on NB201
Figure 2: Validation of the predictive performance of Equation 5 and the effect of sampling rate on it. The setup of the search space is the same as in Figure 1. The first row shows the predictive performance of Equation 5 when calculating  $b_{i,j}^{A}, b_{i,j}^{L}$ , and  $b_{i,j}^{E}$  with Equation 3. The second row shows the predictive performance of Equation 5 when randomly sampling the network and computing the average difference to estimate  $b_{i,j}^{A}, b_{i,j}^{L}$ , and  $b_{i,j}^{E}$  for the corresponding dataset.
By integrating Equation 3 and 4, we can estimate the performance of  $\mathcal{N} = (b_{1,\mathcal{J}_1}, b_{2,\mathcal{J}_2}, \dots, b_{m,\mathcal{J}_m})$ :
$$
L a t (\mathcal {N}) = L a t (\widetilde {\mathcal {N}}) - \sum_ {i = 1} ^ {m} b _ {i, \mathcal {J} _ {i}} ^ {L}, E n g (\mathcal {N}) = E n g (\widetilde {\mathcal {N}}) - \sum_ {i = 1} ^ {m} b _ {i, \mathcal {J} _ {i}} ^ {E}
$$
$$
A c c (\mathcal {N}) = A c c (\widetilde {\mathcal {N}}) - \sum_ {i = 1} ^ {m} b _ {i, \mathcal {J} _ {i}} ^ {A} * \frac {\overline {{\mathcal {F} \left(\mathcal {N} _ {(i , \mathcal {J} _ {i})} ^ {\Omega}\right)}}}{\mathcal {F} \left(\mathcal {N} _ {(i , \mathcal {J} _ {i})}\right)} \tag {5}
$$
Proof-of-Concept Experiment. In our examination of the predictive efficacy of Equation 5 across diverse network types, as depicted in Figure 2, we observe that it accurately forecasts three performances for both sequentially connected networks and graph networks and both weight-independent and weight-shared networks, all without necessitating network training. To the best of our knowledge, our method is the first work to precisely estimate network performances using a linear formula, and notably, it is theoretically applicable to all architectures.
2.3 Simplification: Single-sampling Strategy Instead of Full-sampling
Despite the outstanding predictive performance displayed by Equation 5, its computation of loss averages proves to be costly. In this section, by employing a single-sample sampling strategy, we effectively reduce the time complexity from  $O(n^{m})$  to  $O(n * m)$ , enhancing efficiency without compromising precision.
Partial Network Sampling Strategies. We begin by investigating the sample size requirements for Equation 5. The second row of Figure 2 demonstrates rapid convergence of Equation 5 with a notably small sample count for performance prediction. Specifically, Figure 2(c)(d) reveals that a mere  $5\%$  random sampling of networks is sufficient for the prediction to converge towards optimal performance. This underscores the impressive efficacy of Equation 5, which exhibits rapid convergence even with a limited number of random samples.
Single Network Sampling Strategy. Building upon Equation 4, we can select an averageFLOPs network, denoted as  $\mathcal{N}^{avg}$  , in the search
Algorithm 1 Math Neural Architecture Search  
Stage1: Determine the Average-FLOPs Network for i=1,2,...,m do
Calculate the average of the FLOPs of all blocks at the  $i$ -th note,  $\overline{\mathcal{F}(b_i)} = (\mathcal{F}(b_{i,1}) + \ldots +\mathcal{F}(b_{i,n})) / n$  Select  $b_{i,\mathcal{I}_i}$  whose FLOPs is closest to  $\overline{\mathcal{F}(b_i)}$
end Define average-FLOPs net  $\mathcal{N}^{avg} = \{b_{1,\mathcal{I}_1},\dots,b_{m,\mathcal{I}_m}\}$  Stage2: Calculate Block Performances for i=1,2,...,m do
for  $j = 1,2,\dots,n$  do
Switch the  $i$ -th block in  $\mathcal{N}^{avg}$  from  $b_{i,1}$  to  $b_{i,j}$ ; Calculate the performance difference of the network brought about by switching and use it as the performance of the block  $b_{i,j}^A, b_{i,j}^L, b_{i,j}^E$ .
end  
Stage3: Prediction and Architecture Search
Calculate three characteristics of the base net  $\widetilde{\mathcal{N}} = \{b_{1,1},\dots,b_{m,1}\}$  as  $Acc(\widetilde{\mathcal{N}})$ ,  $Lat(\widetilde{\mathcal{N}})$ ,  $Eng(\widetilde{\mathcal{N}})$ ; For network  $\mathcal{N} = \{b_{1,\mathcal{J}_1},b_{2,\mathcal{J}_2},\dots,b_{m,\mathcal{J}_m}\}$ , its accuracy, latency and energy can be estimated by Equa. 5. Set required accuracy/latency/energy limit, and solve the corresponding ILP problem to obtain the architecture.
(c)Acc on SuperTran
(e)Lat on MBv3
(f)Lat on NB201
(g)Ener on NB201
(d)Acc on SuperViT Spearman: 0.9901
(h)Lat on SuperTran
Figure 3: MathNAS algorithm verification. We conduct experiments on 4 search space. (1)MobileNetV3 [21] (2)NAS-Bench-201 [20] (3)SuperTransformer [5] (4)SuperViT [13]. For NAS-Bench201, we use the accuracy of each network trained individually as  $Acc(\mathcal{N})$ . For other spaces, the validate accuracy under shared weights base net is used. We show accuracy predictions on these four networks as well as hardware efficiency predictions on them. The calculation of  $b_{i,j}^{A}, b_{i,j}^{L}$  and  $b_{i,j}^{E}$  follows Algorithm 1. For NAS-Bench-201, we verify all nets and other spaces, we randomly sample 1000 nets to verify the prediction effect.
space with FLOPs approximating the mean value, ensuring  $\overline{\mathcal{F}\left(\mathcal{N}_{(i,j)}^{\Omega}\right)}/\mathcal{F}(\mathcal{N}_{(i,j)}^{avg}) \approx 1$ . This leads to:  $b_{i,j}^{A} \approx \Delta Acc(\mathcal{N}_{(i,1) \to (i,j)}^{avg})$ ,  $b_{i,j}^{L} \approx \Delta Lat(\mathcal{N}_{(i,1) \to (i,j)}^{avg})$ ,  $b_{i,j}^{E} \approx \Delta Eng(\mathcal{N}_{(i,1) \to (i,j)}^{avg})$  By incorporating  $\mathcal{N}^{avg}$  into Equation 5, we gain the ability to calculate any network performance. Thus, we only need to verify the performance of  $\mathcal{N}_{(i,1)}^{avg}$  and  $\mathcal{N}_{(i,j)}^{avg}$  for  $b_{i,j}$ , resulting in an  $O(n * m)$  search time complexity. The complete MathNAS algorithm is presented in Algorithm 1, which, in theory, can achieve accurate prediction on any network structure with polynomial search complexity.
Validation experiment of single-sample strategy effectiveness. We assess the efficacy of Algorithm 1 across CNN, GCN, Transformer, and  $\mathrm{CNN + }$  Transformer search spaces, with the outcomes displayed in Figure 3. It is evident that the Spearman correlation coefficient between predicted and actual values exceeds 0.9 for all networks. Remarkably, accuracies of 0.95 and 0.97 are attained on ViT and CNN architectures, respectively, emphasizing the algorithm's robustness.
2.4 MathNAS: Converting Architecture Search to ILP.