SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We denote  $b_{i,j}^{F}$  as the FLOPs of block  $b_{i,j}$ , and  $b_{i,j}^{B} \in \{0,1\}$  as the indicator of whether block  $b_{i,j}$  is used in a network. If block  $b_{i,j}$  is selected as the implementation of block node  $\mathcal{B}_i$  in a network,  $b_{i,j}^{B} = 1$ , otherwise 0. The problem that NAS needs to solve can be formulated as:
$$
\max  _ {b ^ {B}} A c c (\widetilde {\mathcal {N}}) - \frac {\sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n} b _ {i , j} ^ {A} * b _ {i , j} ^ {B}}{\sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n} b _ {i , j} ^ {F} * b _ {i , j} ^ {B}} * \overline {{\mathcal {F} (\mathcal {N})}}
$$
$$
s. t. \operatorname {L a t} (\widetilde {\mathcal {N}}) - \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n} b _ {i, j} ^ {L} * b _ {i, j} ^ {B} \leq \hat {L}, \operatorname {E n g} (\widetilde {\mathcal {N}}) - \sum_ {i = 1} ^ {m} \sum_ {j = 1} ^ {n} b _ {i, j} ^ {E} * b _ {i, j} ^ {B} \leq \hat {E}, \tag {6}
$$
$$
\sum_ {j = 1} ^ {n} b _ {i, j} ^ {B} = 1, b _ {i, j} ^ {B} \in \{0, 1 \}, \forall 1 \leq i \leq m.
$$
The objective is to obtain the maximum accuracy network under two constraints. First, the latency and energy cannot exceed the limit. Second, for any block node, only one block is used. As Equation 6 is a fractional objective function, it can be transformed into an ILP problem by variable substitution.
3 Performance Evaluation
Experiments were conducted at three levels to evaluate the efficacy of MathNAS. Firstly, we validated the effectiveness of MathNAS on CV tasks by conducting searches across three different search spaces. Then we employed MathNAS to design efficient architectures for NLP tasks, showcasing its remarkable generalization capabilities. Finally, we leveraged MathNAS to perform real-time searches on edge devices, considering hardware resources, and achieved exceptional on-device performance.
Search space. For CV tasks, we validate our method on three search spaces: (1) NAS-Bench-201 [20]: a search space encompasses 15,625 architectures in a DARTS-like configuration. (2) SuperViT [13]: a hybrid search space that combines ViT and CNN, containing approximately  $4 \times 10^{10}$  network architectures. (3) MobileNetV3 [21]: a lightweight network search space comprising about  $10^{10}$  network architectures. For NLP tasks, we validate our approach on the SuperTransformer search space [5], which includes  $10^{15}$  networks within a lightweight Transformer framework.
Search and training settings. For NAS-Bench-201 and MobileNetV3, we adopt the training methodology employed in [16] and [22] to train the base net for 100 epochs. Subsequently, we conducted MathNAS search on the base net. As for SuperTransformer and SuperViT, we adhere to the training algorithm proposed by [13] to train the base net for 100 epochs before conducting MathNAS search. The settings of hyperparameters in the training are consistent with the original paper. We employ the Gurobipy solver to address the ILP problem. In the SuperViT and SuperTransformer search spaces, we impose a search time limit of 10 seconds to expedite the process. For the other search spaces, we do not enforce any time constraints.
The search cost of MathNAS consists of two stages: offline network pre-training that is conducted only once and online real-time search. During the offline network pre-training, MathNAS evaluates block performance once. During online searching, MathNAS is capable of multiple real-time searches based on the current hardware resource constraints. To negate the influence of variations in GPU models and versions on the pre-training time, and to facilitate comparisons by future researchers, we have adopted pre-trained networks provided by existing works. All mentions of search cost in the paper refer solely to the real-time search time on edge devices. We provide a detailed description of the search space, more experimental results, and visualizations of the searched architectures in the Appendix.
3.2 MathNAS for Designing Effective CV Networks
MathNAS for NAS-Bench-201. To assess the effectiveness of our method in striking a balance between accuracy and hardware efficiency, we compare networks searched by MathNAS under hardware efficiency constraints to those searched by BRP-NAS [30], which utilizes GNN predictors to estimate network performance. As illustrated in Figure 4, MathNAS consistently locates networks that approach Pareto optimality in the majority of cases, whereas the employment of GNN predictors leads to suboptimal model choices. An extensive comparison between the searched architectures and SOTA models is provided in the Appendix for further insight.
Figure 4: A comparison of networks searched by MathNAS (red circles) versus those searched by BRP-NAS (brown triangles) in the NAS-Bench-201 space. Across different devices, the networks searched by MathNAS demonstrate a closer proximity to the Pareto front (yellow five-pointed stars) as compared to the networks obtained through BRP-NAS.
Figure 5: MathNAS v.s. SOTA ViT and CNN models on ImageNet-1K.
Model	Top-1(%)	Top-5(%)	F(M)	Param(M)
ResNet-18[23]	77.7	93.2	1800	11.7
ReXNet[24]	77.9	93.9	400	4.8
MathNAS-T1	78.4	93.5	200	8.9
LeViT-128[25]	78.6	94.0	406	9.2
EfficientNet-B1[26]	79.1	94.4	700	7.8
ReXNet[24]	79.5	94.7	660	7.6
MathNAS-T2	79.6	94.3	325	9.3
ResNetY-4G[27]	80.0	94.8	4000	21
LeViT-192[25]	80.0	94.7	658	10.9
EfficientNet-B2[26]	80.1	94.9	1000	9.2
ReXNet[24]	80.3	95.2	900	9.7
ResNet-50[23]	80.6	95.1	4100	25.6
MathNAS-T3	81.3	95.1	671	13.6
Swin-T[28]	81.3	95.5	4500	29
EfficientNet-B3[26]	81.6	95.7	1800	12
LeViT-256[25]	81.6	95.4	1120	18.9
ReXNet[24]	81.6	95.7	1500	16
AutoFormer-small[29]	81.7	95.7	5100	22.9
MathNAS-T4	82.0	95.7	1101	14.4
AutoFormer-base[29]	82.4	95.7	11000	54
LeViT-384[25]	82.6	96.0	2353	39.1
MathNAS-T5	82.5	95.8	1476	14.8
MathNAS for ViT. To assess the performance of MathNAS in designing larger models, we utilize it to create effective ViT models for ImageNet-1K classification. Figure 5 demonstrates that MathNAS surpasses or matches the performance of existing SOTA models. For instance, MathNAS-T5 achieves an accuracy of  $82.5\%$ , which is comparable to LeViT-384 [25] and Autoformer-base [29], while consuming only about  $50\%$  and  $15\%$  of their respective FLOPs. Similarly, MathNAS-T3 achieves comparable accuracy to RexNet [24] but with approximately half the FLOPs. MathNAS also exhibits exceptional performance in the realm of small networks. Particularly, MathNAS-T1 achieves a top-1 accuracy of  $78.4\%$ , surpassing ResNet-18 [23] and ReXNet [24] by  $0.7\%$  and  $0.5\%$  respectively.
MathNAS for Mobile CNNs. We employ MathNAS to design mobile CNN models for further investigation, conducting our search within the MobileNetV3 search space. As demonstrated in Table 1, MathNAS-MB4 achieves a top-1 accuracy of  $79.2\%$ , which is on par with EfficientNet-B1  $(79.1\%)$ . It is important to note that EfficientNet-B1 is derived through a brute-force grid search, necessitating approximately 72,000 GPU hours [4]. Despite this, MathNAS-MB4 offers comparable performance to EfficientNet-B1 while only requiring 0.8 seconds to solve an ILP problem on the GPU and search for a suitable network. MathNAS also excels in the context of smaller networks. Notably, MathNAS-MB1 requires only 257M FLOPs to achieve a top-1 accuracy of  $75.9\%$ , surpassing the performance of FBNet-b [31], AtomNAS-A [32], and OFA [22], all of which demand higher computational resources.
Table 1: Performance of mobile networks designed with MathNAS. Top-1 accuracy on ImageNet-1K.
Model	FLOPs(M)	Top-1	Search Time	Scale Up
FBNet-b[31]	295	74.1	609h	1.9×
AtomNAS-A[32]	258	74.6	492h	2.3×
OFA[22]	301	74.6	120h	9.6×
MathNAS-MB1	257	75.9	0.9s	4.6M×
MnasNet-A1[33]	312	75.2	40025h	1.0×
ProxylessNAS-R[34]	320	74.6	520h	76.9×
AtomNAS-B[32]	326	75.5	492h	81.4×
FairNAS-C[35]	321	71.1	384h	104.2×
Single Path One-Shot[36]	323	74.4	288h	138.9×
OFA[22]	349	75.8	120h	333.5×
MathNAS-MB2	289	76.4	1.2s	144M×
EfficientNet B0[4]	390	76.3	72000h	1.0×
FBNet-c[31]	375	74.9	580h	124.1×
ProxylessNAS-GPU[34]	465	75.1	516h	139.5×
AtomNAS-C[32]	363	76.3	492h	146.3×
FairNAS-A[35]	388	75.3	384h	104.2×
FairNAS-B[35]	345	75.1	384h	104.2×
MathNAS-MB3	336	78.2	1.5s	173M×
EfficientNet B1[4]	700	79.1	72000	1.0×
MnasNetA1[33]	532	75.4	40025	1.8×
BigNAS-M[37]	418	78.9	1152	62.5×
MathNAS-MB4	669	79.2	0.8s	324M×
3.3 MathNAS for Designing Effective NLP Networks
We perform a comparative evaluation of MathNAS against SOTA NLP models on the WMT'14 En-De task to gauge its effectiveness. Table 2 reveals that MathNAS surpasses all baseline models in terms of BLEU score while also achieving FLOPs reduction across three distinct devices. Specifically, on Intel Xeon CPUs, MathNAS with full precision attains a remarkable  $74\%$  reduction in FLOPs compared to Transformer [8] and a  $23\%$  reduction compared to HAT [5], while registering improved BLEU scores by 0.4 and 0.3, respectively. Additionally, MathNAS excels in designing lightweight
Table 2: MathNAS vs. SOTA baselines in terms of accuracy and efficiency on NLP tasks.
Raspberry Pi	Intel Xeon CPU	Nvidia TITAN Xp GPU	Search Cost
Model	FLOPs	BLEU	Latency	FLOPs	BLEU	Latency	FLOPs	BLEU	Latency	Time	CO2
Transformer[8]	10.6G	28.4	20.5s	10.6G	28.4	808ms	10.6G	28.4	334ms	184h	52lbs
Evolved Trans.[38]	2.9G	28.2	7.6s	2.9G	28.2	300ms	2.9G	28.2	124ms	219200h	624000lbs
HAT[5]	1.5G	25.8	3.5s	1.9G	25.8	138ms	1.9G	25.6	57ms	200h	57lbs
MathNAS	1.7G	25.5	3.2s	1.8G	25.9	136ms	1.8G	25.9	68ms	10s	0.0008lbs
HAT[5]	2.3G	27.8	5.0s	2.5G	27.9	279ms	2.5G	27.9	126ms	200h	57lbs
MathNAS	2.1G	28.3	4.7s	2.4G	28.6	272ms	2.0G	28.1	107ms	10s	0.0008lbs
HAT[5]	3.0G	28.4	7.0s	3.5G	28.5	384ms	3.1G	28.5	208ms	200h	57lbs
MathNAS	2.8G	28.6	6.5s	2.8G	28.8	336ms	2.6G	28.6	189ms	10s	0.0008lbs
NLP models. On Nvidia TITAN Xp GPUs under latency constraints, MathNAS yields FLOPs comparable to HAT [5], but with a 0.3 higher BLEU score. A noteworthy aspect is that the network search process facilitated by MathNAS requires only 10 seconds, considerably reducing search time. As a result, employing MathNAS leads to a reduction of over  $99\%$  in  $CO_2$  emissions compared to baseline models, underscoring its positive environmental impact.
3.4 MathNAS for Designing Dynamic Networks.
Deployed on edge devices (Raspberry Pi 4b, Jetson TX2 CPU, TX2 GPU), MathNAS allows dynamic network switching suited to device conditions. Within the MobileNetV3 search space, we account for memory limitations by calculating performance indices for each block, subsequently deploying five selected Pareto-optimal blocks balancing accuracy and latency in each block node. During runtime, latency is continuously monitored. Should it surpass a preset threshold, MathNAS immediately updates the blocks' latency. Then the device solves the ILP problem to identify the optimal network architecture, comparing and switching blocks with the searched network as required.
In comparison with SOTA dynamic network models, MathNAS demonstrates superior performance as outlined in Table 3 and Figure 6. Impressively, MathNAS solves the ILP problem on-device in a mere 0.4 seconds on the TX2 GPU, enabling real-time search. This notably enhances the number of executable networks on the device, outdoing SlimmableNet [39] and USlimmableNet [40] by factors of 781 and 116 respectively. Additionally, through a block-based approach, MathNAS enables efficient network alterations by replacing only essential blocks. When compared to Dynamic-OFA [41], which shares similar performance, MathNAS significantly reduces the switching time by  $80\%$ . The Appendix details the use of Pareto-optimal blocks and related network experiment results.
Table 3: MathNAS vs. SOTA baselines in terms of Dynamic Networks.
Network	Latency (ms)	On Device Performance
Model	Top-1 (%)	FLOPs (M)	Raspb Pi	TX2 CPU	TX2 GPU	Search Method	Search Time	Switch Unit	Switch Time	Nets Number	Scale Up
S-MbNet-v2[39]	70.5	301	1346	958	118	Manual Design	-	Channel	15ms	4	1.0x
US-MbNet-v2[40]	71.5	300	1358	959	158	Manual Design	-	Channel	18ms	27	6.7x
AS-MNASNet[42]	75.4	532	2097	1457	2097	Greedy Slimming	4000h	Channel	37ms	4	1.0x
Dynamic-OfA[41]	78.1	732	2404	1485	80	Random+Evplution	35h	Network	244ms	7	1.7x
MathNAS	75.9	257	832	525	76	On-Device Search	0.4-12s	Block	61ms	3125	781x
79.2	669	2253	1398	81
Figure 6: Top-1 vs. Latency of MathNAS over SOTA dynamic baselines on three devices.
This paper introduces MathNAS, the first architecturally-general MP-NAS. By virtue of the modular nature of the search space, we introduce block performance and establish the mapping from block performance and network performance, which enables subsequent transformation of NAS to an ILP problem. This novel strategy reduces network search complexity from exponential to polynomial levels while maintaining excellent network performance. We propose three transformation formulas to depict this process and support them with theoretical proofs and extensive experiments. MathNAS achieves SOTA performance on various large-scale CV and NLP benchmark datasets. When deployed on mobile devices, MathNAS enables real-time search and dynamic networks, surpassing baseline dynamic networks in on-device performance. Capable of conducting rapid searches on any architecture, MathNAS offers an appealing strategy for expediting the design process of large models, providing a clearer and more effective solution.
5 Acknowledgment
We thank the anonymous reviewers for their constructive comments. This work was partially supported by National Key R&D Program of China (2022YFB2902302). Sihai Zhang gratefully acknowledges the support of Innovative Research (CXCCTEC2200001).
[1] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12270-12280, 2021.
[2] Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnetv3: Joint architecture-recipe search using predictor pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16276–16285, 2021.
[3] Bichen Wu, Chaojian Li, Hang Zhang, Xiaoliang Dai, Peizhao Zhang, Matthew Yu, Jialiang Wang, Yingyan Lin, and Peter Vajda. Fbnetv5: Neural architecture search for multiple tasks in one run. arXiv preprint arXiv:2111.10007, 2021.
[4] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. PMLR, 2019.
[5] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. arXiv preprint arXiv:2005.14187, 2020.
[6] Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, Alexander Filippov, and Evgeny Burnaev. Nas-bench-nlp: neural architecture search benchmark for natural language processing. IEEE Access, 10:45736-45747, 2022.
[7] Hung-yi Lee, Shang-Wen Li, and Ngoc Thang Vu. Meta learning for natural language processing: A survey. arXiv preprint arXiv:2205.01500, 2022.
[8] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[9] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1-41, 2022.
[10] Mojan Javaheripi, Gustavo de Rosa, Subhabrata Mukherjee, Shital Shah, Tomasz Religa, Caio Cesar Teodoro Mendes, Sebastien Bubeck, Farinaz Koushanfar, and Debadeepta Dey. Litetransformersearch: Training-free neural architecture search for efficient language models. Advances in Neural Information Processing Systems, 35:24254-24267, 2022.