SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the remainder of this section, we apply AutoGO and our Segment Database to search for better architectures on several NAS benchmarking families to demonstrate the benefits of our framework. We further use AutoGO to improve several open-sourced, popular network architectures. We consider various high-resolution CV tasks, including Classification, Semantic Segmentation and Human Pose Estimation, with the aim of improving hardware friendliness in terms of FLOPs. We also provide examples of deployment where AutoGO minimizes the energy consumption or on-chip latency of already lightweight neural networks for Super Resolution and Image Denoising using a cycle simulator for a Huawei Neural Processing Unit (NPU). We provide implementation details, dataset metrics, and training setup in Sections A.2 and A.4.
4.1 Rank Correlation of Performance Predictors
AutoGO uses a neural predictor to estimate the performance of mutated architectures. We train and evaluate our PSC predictor on five CIFAR-10 benchmarks. Our CG format provides the advantage of training simultaneously on multiple benchmark architecture families. We split each family into training, validation, and testing partitions containing  $80\%$ ,  $10\%$  and  $10\%$  of the overall CGs in that family. We combine the training
Table 1: Test SRCC of all 5 architecture families for the PSC predictor and baselines. Results averaged over 5 runs.
Arch. Family	GNN	PSC 1:1 Ratio	PSC
NB-101	0.627 ± 0.031	0.666 ± 0.025	0.849 ± 0.054
NB-201	0.809 ± 0.016	0.865 ± 0.015	0.983 ± 0.003
HiAML	0.010 ± 0.013	0.170 ± 0.042	0.734 ± 0.031
Inception	0.209 ± 0.037	0.066 ± 0.071	0.496 ± 0.022
Two-Path	0.023 ± 0.018	0.236 ± 0.043	0.724 ± 0.022
Table 2: AutoGO results across all 5 CIFAR-10 architecture families while aiming to increase accuracy  $[\%]$  and reduce FLOPs [1e6]. We consider 3 experimental configurations that vary in unit of mutation and predictor used. We bold and italicize the best and second best result per family.
Baseline Architectures	Operator + GNN	Segment + GNN	Segment + PSC
Family	Acc.	FLOPs	Acc.	FLOPs	Acc.	FLOPs	Acc.	FLOPs
NB-101	95.18%	11722	95.16%	9407	95.31%	10817	95.45%	11118
Δ	-	-	-0.02%	-19.75%	+0.13%	-7.72%	+0.27%	-5.15%
NB-201	93.50%	313	93.37%	232	93.57%	294	93.84%	303
Δ	-	-	-0.13%	-25.88%	+0.07%	-6.07%	+0.34%	-3.19%
HiAML	92.32%	246	92.00%	198	92.62%	168	92.75%	198
Δ	-	-	-0.32%	-19.51%	+0.30%	-31.71%	+0.43%	-19.51%
Inception	93.20%	494	92.97%	319	93.31%	461	93.52%	474
Δ	-	-	-0.23%	-35.43%	+0.11%	-6.68%	+0.32%	-4.05%
Two-Path	87.90%	116	88.63%	106	89.16%	48	88.94%	91
Δ	-	-	+0.73%	-8.62%	+1.26%	-58.62%	+1.04%	-21.55%
partitions for each family to form an overall training set for the predictors while setting the test partitions aside individually. When training the PSC predictor, we partition each CG into multiple  $\{P,S,C\}$  instances to use as training samples. We compare our proposed PSC predictor with two baselines: As each CG contains multiple  $\{P,S,C\}$  instances, we consider an intermediate setting, PSC 1:1 Ratio, where we only consider one random  $\{P,S,C\}$  instance per CG in the training set. Moreover, we also consider a baseline GNN that estimates the performance of whole unpartitioned CGs but is not sensitive to segment-level changes.
We measure the Spearman's Rank Correlation Coefficient (SRCC) of each predictor on the test partitions for each benchmark family. SRCC is defined in the range [-1, 1] and higher values are better. Table 1 summarizes our results. We note the exceptional performance of the PSC predictor as it can obtain SRCC above 0.72 on HiAML and Two-Path while the GNN barely achieves positive SRCC. Moreover, while the GNN and PSC 1:1 predictors can obtain SRCC above 0.8 and 0.6 for NB-201 and NB-101, respectively, if we train the PSC predictors on all  $\{P, S, C\}$  samples, we can obtain a near perfect SRCC of 0.98 on NB-201 and almost 0.85 on NB-101. Overall, these findings demonstrate the merit of our segment decomposition and PSC encoding scheme for CGs.
4.2 Improving CIFAR-10 NAS Benchmark Architectures
We test the effectiveness of AutoGO by refining the best architectures from each family. Specifically, AutoGO aims to increase accuracy while reducing FLOPs. We consider three scenarios that allow us to ablate the effectiveness of our PSC predictor when applied to search. We also compare our segment-level mutation to a simpler, operation-level mutation that mutates single primitive operation.
We run AutoGO for 10 iterations in the segment-level mutation, and 50 iterations for the operation-level mutation for a fair comparison since segments have 5 nodes on average. At the end of each iteration, we randomly select 10 architectures from the accuracy-FLOPs Pareto frontier to qualify for the next iteration as parent architectures. For each parent candidate, we consider up to 100 replacement mutations. We allow AutoGO to mutate sequences of 1 to 3 contiguous source segments simultaneously. We randomly mask out  $50\%$  of segments with more than 1 node in our segment database and force BPE to segment the input with the remaining ones. Further, we consider two search settings that limit the FLOPs decrease of child architectures. In the first case, we allow AutoGO to freely reduce FLOPs, while in the second case, we do not allow FLOPs to fall by more than  $20\%$  relative to the original network we are optimizing. After the search is complete, we train architectures on the accuracy-FLOPs Pareto frontier 3 times on CIFAR-10 [33]. We report the accuracy and FLOPs of the overall best architecture found across both FLOPs constraint settings. Finally, it takes 45 to 90 minutes to execute the search depending on the size of the input architecture CG. We provide an ablation study across FLOPs constraints, a detailed breakdown of wall-clock time cost, and enumerate our hardware platform in Sections A.6, A.7 and A.9, respectively.
Table 2 reports our findings across all 5 architectures families. We observe that the segment-level mutation is a better fit for finding high-performance architectures, as the best architectures are found using it. For example, on HiAML, it can increase the accuracy by up by  $0.43\%$  while reducing
Table 3: Results running AutoGO on Computation Graphs for ResNet-50, 101 and VGG-16. Specifically, we compare ImageNet [58] Top-1/Top-5 accuracy, Cityscapes test mIoU [14] using a PSPNet head [76], MPII [4] PCK as well as FLOPs. For performance metrics, higher is better. We measure latency on an Nvidia RTX 2080 Ti GPU using an input resolution size of  $224 \times 224$ .
Architecture	ImageNet Top-1/5	Cityscapes mIoU	MPII PCK	FLOPs [1e9]	Lat. [ms]
ResNet-50 Original	74.02%/91.22%	63.42%	82.36%	6.29	7.18
ResNet-50 AutoGO Arch 1	75.34%/92.16%	65.88%	84.07%	6.71	7.50
ResNet-50 AutoGO Arch 2	75.66%/92.45%	66.65%	82.70%	5.88	6.92
ResNet-101 Original	75.09%/91.94%	65.92%	82.77%	13.76	15.86
ResNet-101 AutoGO Arch 1	76.56%/93.09%	67.12%	83.59%	13.66	15.56
ResNet-101 AutoGO Arch 2	75.69%/92.15%	66.38%	84.64%	13.35	15.36
VGG-16 Original	74.18%/91.83%	65.36%	85.92%	30.81	4.65
VGG-16 AutoGO	74.91%/93.23%	66.91%	85.99%	24.34	4.20
FLOPs by  $-19.76\%$ . By contrast, the operation-level mutation only manages to improve performance on Two-Path. However this gain of  $+0.73\%$  is substantially less than what we can achieve using segment mutation. On NAS-Bench-101, operation mutation manages to break even with the baseline architecture, while incurring an accuracy drop of more than  $0.10\%$  on all other families.
Next, we compare the effectiveness of the PSC and GNN predictors. The PSC predictor finds the best architecture in 4 of the 5 architecture families. PSC improves accuracy on NB-101 and NB-201 by  $0.34\%$  and  $0.27\%$ , respectively. By contrast, the GNN only achieves the best performance on Two-Path, which is the smallest benchmark family with a baseline architecture of only 116 MegaFLOPs. Thus, segment-level mutations span more considerable changes that are distinguishable with GNN and PSC predictors. In sum, the segment-aware encoding is better at increasing performance while reducing FLOPs than the operation-level mutation. Moreover, our results demonstrate the superiority of the PSC predictor compared to the GNN in most cases.
4.3 Application to High-Resolution
Classification, Segmentation and Pose Estimation
To demonstrate the extensibility and generalizability of our framework, we apply it to several stand-alone architectures for higher-resolution computer vision tasks. Specifically, we perform NAS using AutoGO with the PSC predictor and segment-level mutation for 5 iterations on ResNet-50, ResNet-101 [24] and VGG-16 [61]. For a fair comparison, we do not allow AutoGO to select segments containing operations that were not available or popularized when the network was first proposed, e.g., depthwise convolutions [59].
After the search, we examine the architectures on the Pareto frontier and select 1-2 with noticeably different FLOPs reductions to train and evaluate against the original
architecture. To form the first point of comparison, we train each network on ImageNet [58]. Then, we fine-tune the network on different tasks. For Semantic Segmentation (SS), we use a PSPNet [76] head structure and fine-tune on Cityscapes [14] to obtain mean Intersection over Union (mIoU) performance. For Human Pose Estimation (HPE), we adopt the method of [78] to fine-tune on MPII [4] to measure the Percentage of Correct Keypoints (PCK) of an architecture.
Table 3 shows our results on ImageNet, Cityscapes, and MPII. First, we note how in every case, the architectures generated by AutoGO consistently outperform the original on all 3 CV benchmarks.
Figure 4: Example of a segment mutation from that helped create ResNet-50 AutoGO Arch 2 (Tab. 3). A ResNet block is replaced by a HiAML block.
Table 4: Peak Signal-to-Noise Ratio (PSNR) for EDSR on the DIV2K validation set and several SR benchmarks in the 2x upscaling setting. Higher is better. We measure latency on an RTX 2080 Ti.
SR Architecture	DIV2K	Set5	Set14	BSD100	Urban100	Manga109	FLOPs [1e9]	Lat. [ms]
EDSR Original	36.19	36.86	32.57	31.39	29.14	36.09	141	18.04
EDSR AutoGO Arch 1	37.28	38.01	33.62	32.18	31.56	38.49	118	15.38
EDSR AutoGO Arch 2	37.27	37.97	33.55	32.16	31.53	38.47	110	14.52
EDSR AutoGO Arch 3	37.25	38.01	33.58	32.16	31.46	38.44	105	13.81
Table 5: SR PSNR results on Proprietary FSRCNN networks. FSRCNN-\{3, 4\} denotes the number of Conv3x3 operations in the middle of the architecture. We report change in power according to a cycle-accurate simulation model that uses a 64x640 input resolution. FLOPs [1e9] are measured using an input resolution of 640x360.
SR Architecture	Set5	Set14	BSD100	Urban100	Manga109	Power [mW]	ΔPower	FLOPs
FSRCNN-3 Original	35.12	31.43	30.56	27.65	32.75	774.77	-	2.67
FSRCNN-3 AutoGO	35.12	31.43	30.56	27.64	32.60	644.10	-16.87%	2.09
FSRCNN-4 Original	35.22	31.50	30.60	27.71	32.88	892.89	-	3.74
FSRCNN-4 AutoGO	35.17	31.52	30.60	27.71	32.77	508.37	-43.06%	2.07
For example, ResNet-50 AutoGO Arch 2 outperforms the original by over  $1.64\%$  ImageNet top-1 accuracy, while the found architecture on VGG outperforms the original on Cityscapes by  $1.55\%$  mIoU. Also, we measure inference latency on an RTX 2080 Ti GPU. We note some correlation between FLOPs and GPU latency; as one metric increases or decreases, so does the other metric.
Figure 4 illustrates how AutoGO splices a HiAML segment into ResNet-50 to create a new architecture. The longer branch performs multiple convolutions at reduced channels, while the shorter branch applies lightweight operations on the original number of channels, and the MILP performs resolution propagation to ensure functionality.
4.4 Application to Super Resolution with EDSR
We use AutoGO to optimize networks for Super Resolution (SR). Specifically, we optimize the backbone feature extractors of EDSR [38]. As the original EDSR only uses convolution and ReLU operations, we do not let AutoGO select segments that contain depthwise, pooling, or batch normalization. Figure 9 (Sec. A.8) provides sample illustrations of the mutations AutoGO performs on EDSR. We train SR networks on DIV2K [2, 29] in the  $2 \times$  resolution setting and evaluate on several public benchmarks [8, 74, 44, 28, 45, 3]. Table 4 demonstrates how EDSR architectures produced by AutoGO can handily outperform the original network while substantially reducing FLOPs and GPU latency, e.g., AutoGO Arch 3 is 36 gigaFLOPs smaller and 4.2ms faster.
4.5 Using AutoGO to Automate Neural Network Deployment on a Neural Processing Unit
Table 6: Results of using AutoGO to optimize a Proprietary U-Net Denoising network to improve PSNR and minimize on-chip latency. We report changes in latency and power measured on a mobile NPU using a cycle simulation model.
Denoising	PSNR	ΔLatency	Power [mW]	ΔPower	FLOPs [1e9]
Base Model	139.4	-	724.59	-	17.05
AutoGO	139.9	-24.94%	657.82	-9.21%	16.26
curacy predictor with power/latency measurements fed back by either a hardware profiling tool or the cycle-accurate hardware simulator.
We demonstrate the real-world deployment capabilities of AutoGO by optimizing neural network performance using a cycle-accurate counter that simulates Huawei NPU performance for cellphones [37]. We optimize for power or on-chip latency by pairing our pretrained PSC ac
Super Resolution Power Optimization We use AutoGO to optimize proprietary lightweight SR models similar to FSRCNN [16]. Table 5 reports the network performance on several public datasets as well as the change in power and FLOPs. We note the effectiveness of AutoGO at optimizing the energy efficiency of even a small network, e.g., the FSRCNN-4 AutoGO variant can reduce the
instantaneous power of an already small FSRCNN (with 4 Conv3x3 in the body network) by over  $43\%$ , while the FSRCNN-3 AutoGO variant reduces it by over  $16\%$ . Moreover, AutoGO maintains or even enhances the PSNR performance as compared to the original networks, demonstrating the generalizability of the pretrained PSC predictor to other tasks.
Image Denoising Latency Optimization We use AutoGO to optimize a proprietary Image Denoising U-Net similar to [55] to reduce on-chip latency. Table 6 reports our findings on an in-house dataset. We observe how the mutated network can exceed the original denoising PSNR by 0.5. While substantially improving latency, we have also reduced other resource consumption metrics including power and FLOPs.
5 Limitations and Future Discussions
The AutoGO framework consists of many components: frequent subgraph mining (FSM) via topological sorting and BPE tokenization, the position-aware PSC predictor, the mutation-based evolutionary strategy and use of Computational Graphs. Each of these components has its own strengths and weaknesses. Our paper demonstrates the feasibility of using topological sorting and BPE to perform FSM, although it faces limitations due to the non-deterministic nature of topological sorting, resulting in generation of segments for isomorphic subgraphs that must be filtered out from our database. However, the primary strength of FSM through topological sorting and BPE is the economic advantage of speed, as neither BPE-based segment extraction nor isomorphic segment filtering is time-consuming, even on large Computational Graphs. Moreover, FSM only considers the frequency of a given subgraph (represented as a segment) while ignoring its contribution to performance and hardware-friendliness metrics. Extracting frequent subgraphs that can explain performance is a subject for further studies.
The quality of our segment database depends on the types of operations and subgraphs present in the benchmark families we extract from. For example, a few segments in our database use depthwise convolutions (see Fig. 5 in Sec. A.1) as the only NAS-Benchmark we consider that contains them is Inception, and in limited quantity. These were not widely used in our experiments, since to achieve a fair comparison with the baseline architectures that AutoGO aimed to improve, we constrained the vocabulary of AutoGO during the search to use only same generation of operations. On the flip side, one could use AutoGO to mine newer operations like depthwise convolutions, StarReLU [72], or even older operations that have gained popularity like GELU [25]. Segments containing these operations could then be used to further refine older architectures like ResNets, EDSR, and FSRCNN for performance improvement.
A future variant of AutoGO could replace the mutation-driven search with a policy network [35] to select replacement segments. Another avenue for future research is performing frequent and important computational subgraph mining for Transformer and attention-based models for their hardware-friendly deployment, as the Computational Graph representation and subgraph mining presented in this paper are principally designed for convolutional neural networks currently.
We propose AutoGO, or Automatic Graph Optimization, a new framework for optimizing neural networks outside the bounds of predefined, fixed search spaces. AutoGO represents architectures using a computation graph format of primitive operations. We partition computation graphs into segment subgraphs using Byte-Pair Encoding. Using a segment database and guided by a predictor which is sensitive to segment size and position, AutoGO modifies the network by incrementally mutating its segments while a resolution propagation MILP ensures network functionality. We build a segment database by extracting a vocabulary of segments from 5 open-source NAS benchmarks using Frequent Subgraph Mining. We use AutoGO to improve the accuracy of the best architectures from each of the 5 CIFAR-10 search spaces while reducing FLOPs. Furthermore, we use AutoGO to evolve several open-sourced large CNNs, including ResNets, VGG-16, and EDSR, and successfully improve their performance on a breadth of CV tasks with reduced or comparable FLOPs. Finally, we demonstrate how to utilize AutoGO to automatically reduce the hardware energy consumption and on-chip latency of realistic convolutional neural network applications, when deployed onto a mobile Neural Processing Unit.
[64] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Péter Vajda, and Joseph Gonzalez. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12962-12971, 2020.
[65] Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? Advances in Neural Information Processing Systems, 34:28454-28469, 2021.
[66] Ross Wightman. Pytorch image models. https://github.com/rwrightman/pytorch-image-models, 2019.
[67] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. In International Conference on Learning Representations, 2020.
[68] Xifeng Yan and Jiawei Han. gspan: graph-based substructure pattern mining. 2002 IEEE International Conference on Data Mining, 2002. Proceedings., pages 721-724, 2002.
[69] Xifeng Yan and Jiawei Han. Closegraph: mining closed frequent graph patterns. In Knowledge Discovery and Data Mining, 2003.
[70] Yichen Yang, Phitchaya Phothilimthana, Yisu Wang, Max Willsey, Sudip Roy, and Jacques Pienaar. Equality saturation for tensor graph superoptimization. Proceedings of Machine Learning and Systems, 3:255-268, 2021.
[71] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105–7114, 2019.
[72] Weihao Yu, Chenyang Si, Pan Zhou, Mi Luo, Yichen Zhou, Jiashi Feng, Shuicheng Yan, and Xinchao Wang. Metaformer baselines for vision. arXiv preprint arXiv:2210.13452, 2022.
[73] Arber Zela, Julien Niklas Siems, Lucas Zimmer, Jovita Lukasik, Margret Keuper, and Frank Hutter. Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks. In International Conference on Learning Representations, 2022.
[74] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representations. In International conference on curves and surfaces, pages 711-730. Springer, 2012.
[75] Hengshuang Zhao. semseg. https://github.com/hszhao/semseg, 2019.
[76] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
[77] Ce Zheng, Wenhan Wu, Chen Chen, Taojiannan Yang, Sijie Zhu, Ju Shen, Nasser Kehtarnavaz, and Mubarak Shah. Deep learning-based human pose estimation: A survey. ACM Computing Surveys, 56(1):1-37, 2023.
[78] Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, and Yichen Wei. Towards 3d human pose estimation in the wild: A weakly-supervised approach. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[79] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
Figure 5: Histograms of segment database statistics including number of input and output nodes/degrees, nodes/edges per segment, unique segment topologies and operation frequency.
A Supplementary Material
A.1 Additional Database Statistics
Figure 5 provides histograms regarding our segment database. Additionally, we enumerate the primitive operations that are only present in specific NAS-Benchmark families:
- Depthwise: Inception.
- Max Pool: NB-101, Inception and Two-Path.
- Concat: NB-101, Inception and Two-Path.
All other operation primitives, e.g., Conv, ReLU, BatchNorm, etc., are present across all 5 CIFAR-10 NAS-Benchmarks.
A.2 Predictor and Dataset Details
We further elaborate on the baseline GNN and PSC predictors from Section 4.1. We provide implementation details, dataset statistics and data pre-processing techniques. We train our predictors for 40 epochs with a batch size of 32 and an initial learning rate of  $1e^{-4}$ .
A.2.1 Baseline and PSC Predictor Setup
We use the same baseline GNN predictor as GENNAPE [48]. First, CGs are given as input into an initial set of embedding layers that transform discrete node features, such as operation type, input/output tensor resolution, kernel size, and bias, into a continuous vector. The node embeddings are then fed through a series of  $6k$ -GNN [51] layers. Next, an overall graph embedding is computed by taking the mean of all node embeddings. A simple MLP with 4 hidden layers predicts performance using the graph embedding.
The PSC predictor differs in that each CG sample is first split into its respective Predecessor, Segment, and suCdecessor subgraphs before being fed into the predictor. All three subgraphs are processed as separate CGs by the node embedding and  $k$ -GNN layers to produce three distinct graph embeddings. We concatenate these graph embeddings feature-wise and feed them into an MLP to generate a prediction. Also, node embedding and  $k$ -GNN layer weights are shared for each subgraph type.
A.2.2 Dataset Statistics and PSC Preprocessing
Table 7: Number of Computation Graphs (CG), segment samples and test SRCC folds for each family. We randomly sample 5k NB-101 architectures and only consider NB-201 networks that do not have the 'none' operation.
Arch. Family	CGs	Segments	Folds
NB-101	5.0k	404.9k	42
NB-201	4096	252.8k	34
HiAML	4.6k	65.1k	10
Inception	580	222.4k	129
Two-Path	6.9k	193.1k	10
We train and evaluate the baseline GNN predictor on every unique CG sample. Additional steps are required to train the PSC predictor since each CG comprises many segments and can decompose into many distinct  $\{P, S, C\}$  subgraph sets.
For the intermediate baseline, PSC 1:1 Ratio in Table 1, we randomly sample 1  $\{P,S,C\}$  representation from each segmented CG in our training dataset. Hence, the number of samples equals the original number of training instances. For the full PSC predictor, we remove this restriction and consider all possible  $\{P,S,C\}$  decompositions which drastically increases the number of samples.