SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 7 lists the number of CGs and  $\{P,S,C\}$  samples per family. While each  $\{P,S,C\}$  sample for a given CG
focuses on a different network segment, they still describe the same overall architecture and thus retain the same accuracy label. Therefore, when measuring test SRCC on the PSC predictor, we divide the test data into folds. Each fold contains only one  $\{P,S,C\}$  instance of a given CG. This avoids introducing additional ties in the ground-truth labels when calculating SRCC. The number of folds is equal to the minimum number of segments in any test CGs or 10, whichever is smaller. Therefore, we calculate the overall test SRCC by averaging SRCC across each fold.
A.3 Segment Extraction with BPE
We compare our BPE subgraph extraction approach to the Weisfeiler-Leman (WL) Kernel method adopted by NAS-BOWL [56] in terms of efficiency. NAS-BOWL applied it on the original, shallow cell-based network representation of NAS-Bench-201 with a depth of 2. We use the WL-kernel on the CG-level and enumerate all subgraphs with a maximum depth of 5. The time and RAM costs of using the WL-kernel scale poorly as we increase the number of graphs and nodes per graph. For example, it takes at least 6 hours to extract and count subgraphs from each NAS-Benchmark family. Moreover, we could not use more than 1k CGs from the HiAML or NB-201 families ( $\sim 110$  and  $\sim 250$  nodes per CG, respectively) without facing memory issues on the rack server described in Section A.9.
By contrast, our approach brings several benefits over mining on large graphs with WL-kernels. The extraction process on sequences is efficient. Using BPE enables segment extraction from all benchmark families (over 21k CGs per Tab. 7) simultaneously in less than 20 minutes using around 10GB of RAM. Also, BPE provides segments that are easier to mutate and alleviates limitations with WL-kernel extraction process by topologically ordering the nodes. Figure 6 compares WL and BPE segmentations on a part of a CG from the NAS-Bench-201 family. The subgraph extracted from the WL method (Fig. 6(a)) cannot cover several nodes within its context (grey nodes of BN-8, Pool-10, BN-11, and Add-14) due to a limited depth of 5 and several resid
(a) WL-kernel
Figure 6: Comparison between subgraphs extracted with WL-kernel and BPE on a NAS-Bench201 cell. Nodes are numerically labeled by a topological ordering. Best viewed in color. Specifically, WL-kernel extracts one large subgraph consisting of all highlighted nodes (greyed-out nodes are omitted). For BPE, all nodes are extracted into one subgraph, denoted by a unique color.
(b) BPE
Figure 7: Example of how the BPE-segmented graph in Figure 6(b) is partitioned into Predecessor, Segment and suCessor subgraphs based on the selected segment. Specifically, we highlight nodes of the selected segment in purple, the predecessor in grey and the successor in yellow.
ual connections. This exacerbates the mutation process. In contrast, segmentation with BPE (Fig. 6(b)) spans different subgraph sizes denoted by separate colors.
A.3.1 PSC Partitioning with Parallel Branches
Figure 7 illustrates how a Computational Graph with multiple branches can be cleanly partitioned into different  $\{P, S, C\}$  based on the choice of segment  $s_i$ . Although the input CG has multiple parallel branches, the use of topological sort to assign ordered numerical labels to each node. The numerical labels of each node in a segment are contiguous, while parallel branches are assigned to either the Predecessor or suCdecessor according to their numerical labels.
A.4 Architecture Training Hyperparameters
We elaborate on the training recipes we use to evaluate input baseline architectures as well as those found by AutoGO.
A.4.1 CIFAR-10 Families
We use the CG representation of the initial and mutated architectures to instantiate networks and train them using TensorFlow. We evaluate CIFAR-10 networks by training them 3 times for 200 epochs with a batch size of 256. We optimize the models using RMSProp with an initial learning rate of  $1e^{-3}$  and a momentum factor of 0.9. We anneal the learning rate according to a cosine schedule.
A.4.2 ImageNet, Segmentation and Pose Estimation
When evaluating ResNet and VGG $^4$  architectures, we first train on ImageNet [58] using timm [66] with a batch size of 1024. We use an initial learning rate of 0.1 which we anneal using a cosine schedule. We optimize the model using Stochastic Gradient Descent (SGD) with a momentum factor of 0.9 and a weight decay of  $1e^{-4}$ . We set a gradient clipping value of 5.0 and use label smoothing with  $\epsilon = 0.1$ . We train ResNets for 200 epochs and VGG-16 for 100 epochs. We save the trained weights to fine-tune on other tasks.
We evaluate Semantic Segmentation performance using semseg [75]. The PSPNet [76] head requires two inputs to implement properly. The first is the final latent tensor that originally feeds into the
4Base model uses batch normalization
classifier head, while the second requires grafting an auxiliary residual connection  $3/4$ ths of the way through the network feature extractor. Furthermore, we adjust the dilation factor and strides of all convolution and pooling operations in the later part of the network to limit downsampling. After loading the pretrained ImageNet weights, we fine-tune on Cityscapes [14] images cropped to  $713^2$  for 200 epochs using a batch size of 16. We use SDG with an initial learning rate of 0.01, a momentum factor of 0.9, and a weight decay of  $1e^{-4}$ .
We implement 2D Human Pose Estimation using [78]. To convert an ImageNet network, we remove the classifier layers and then append a series of 'Deconvolution-BatchNorm-ReLU' blocks which gradually upsample the latent tensors from  $8^{2}$  to  $64^{2}$ . We train on MPII [4] images cropped to  $256^{2}$  for 140 epochs with a batch size of 32. We optimize our networks using Adam, setting an initial learning rate of  $1e^{-3}$  for ResNet-50 and VGG-16, and  $5e^{-4}$  for ResNet-101. We reduce the learning rate by a factor of 10 at epochs 90 and 120. Finally, we report performance in terms of the Percentage of Correct Keypoints (PCK), specifically the Percentage of Correct Keypoints at a head-neck distance of 0.5 (PCK@h0.5) [77].
A.4.3 Super Resolution
We train networks on DIV2K in the 2x upsampling setting for 1000 epochs with a batch size of 16. We set an input patch size of 64 for EDSR and 48 for FSRCNN. We minimize the L1 loss using the Adam optimizer with an initial learning rate of  $1e^{-4}$ , which we reduce using a cosine decay schedule.
A.4.4 Image Denoising
We train networks on a custom in-house image-denoising dataset with 7k images. We set an input patch size of 128 for all networks. We train each network for 2k epochs under a batch size of 128. We minimize the L1 loss using the Adam optimizer with an initial learning rate of  $1e^{-3}$  and a final learning rate of  $1e^{-6}$ , reduced over a polynomial schedule.
A.5 Additional AutoGO Search Details
We provide additional details on the AutoGO search algorithm from Section 3.2.
Algorithm 1 Sample AutoGO pseudocode for one iteration  
1: Input: Pareto frontier  $\mathcal{O}$  ▷ Only contains the input architecture at iteration 0.  
2: Input: Segment Database  $\mathcal{D}$   
3: Input: Performance Predictor  $p$  and FLOPs counter  $f$ .  
4:  $G_{k} = \text{Sample}(\mathcal{O}, k)$  ▷ Sample  $k$  architectures  
5: for  $g \in G_{k}$  do  
6:  $PSC_{g} = []$  ▷ Empty list of mutants  
7:  $S_{g} = \text{Segment}(g, \mathcal{D})$   
8: for  $s \in \text{Sample}(\mathcal{S}_{g})$  do ▷ Source segments  
9:  $\{P, s, C\} = \text{Partition}(\mathcal{S}_{g}, s)$   
10: for  $s^{*} \in \text{Sample}(s, \mathcal{D})$  do ▷ Sample replacement segments  
11:  $\{P, s^{*}, C\} = \text{Mutate}(P, s^{*}, C)$   
12: if MILP( $\{P, s^{*}, C\}$ ) finds a solution then ▷ Resolution propagation  
13: Add  $\{P, s^{*}, C\}$  to  $PSC_{g}$   
14: end if  
15: end for  
16: end for  
17: for All mutated  $\{P, s^{*}, C\} \in PSC_{g}$  do  
18: Profile  $\{P, s^{*}, C\}$  using  $p$  and  $f$   
19: Update  $\mathcal{O}$  using  $\{P, s^{*}, C\}$  and its profiled information.  
20: end for  
21: end for
A.5.1 AutoGO Pseudocode Algorithm
Algorithm 1 provides an example of how the AutoGO search procedure executes over one iteration. AutoGO selects parent architectures from the Pareto frontier  $\mathcal{O}$ . It then uses the segment database  $\mathcal{D}$  to select source and replacement segments to create mutant child architectures. The resolution propagation MILP ensures the mutants constitute valid architectures. Finally, AutoGO places the child architectures on the Pareto frontier  $\mathcal{O}$  according to their performance and hardware-friendliness.
A.5.2 Node Labeling
Before segmentation with BPE, we label nodes in the CG in the form of [current operation, incoming operations, outgoing operations]. We encode each unique node label with a single Chinese character symbol, as they span a wide range of symbols compared to other languages.
A.5.3 Selecting a Sparse BPE Vocabulary
When generating  $V'$  as a vocabulary set utilized by BPE to segment CGs, we include all single-node segments as these represent the irreducible primitive operations that must exist within the vocabulary in some form and only filter out multi-node segments.
A.5.4 Selecting Non-Pareto Optimal Architectures
When transitioning from iteration  $e$  to  $e + 1$ , we select  $k$  architectures from the Pareto frontier  $\mathcal{O}$  and search history to serve as parents. If we have sufficient architectures on the Pareto frontier,  $|\mathcal{O}| \geq k$ , we randomly sample from it. However, if  $|\mathcal{O}| < k$ , there is an architecture deficit. We compensate for this deficit by selecting non-Pareto optimal architectures that aim to achieve our search objective. We select these architectures by ranking them in terms of
Table 8: Example of the minimum sum of ranks selection algorithm with a deficit of 3 architectures.
Acc. [%]	Rank	FLOPs	Rank	Rank Sum	Selected?
91.21	0	260	5	5	No
91.10	1	215	2	3	Yes
91.02	2	200	0	2	Yes
90.75	3	210	1	4	Yes
90.35	4	220	3	7	No
89.05	5	250	4	9	No
predicted accuracy and FLOPs, where higher and lower are better, respectively. We then sum these ranks and select the non-Pareto optimal architectures with the lowest rank sum. Table 8 provides a simple example of this process. Note how the selection mechanism excludes architectures that have high performance but are too large, as well as underperforming architectures.
A.5.5 Segment Selection
For each CG  $g$ , we sample a set of  $m$  source segments  $s_i$ . We sort the segments  $S_g$  by FLOPs and then we select the  $m/2$  segments with the lowest FLOPs while randomly sampling the rest.
A.5.6 Accuracy Predictions and FLOPs Constraints
Once we have a set of valid source and replacement segments, we use the PSC predictor to select mutations that yield the most significant accuracy gain. We use a FLOPs calculator (or a proprietary profiling tool for measuring NPU latency/power) to further filter these mutations by rejecting child architectures whose FLOPs deviate too far from the FLOPs of the input architecture.
A.5.7 Resolution Propagation
Adjustment can not always lead to a solution, meaning the replacement segment can not be used for mutation at this position and generate a valid CG. We cast this task as a search problem over the height, width, and channel resolution values on the replacement segment operations. The search spans mutable operations such as convolutions and pooling. The rest of the operations are immutable and only forward the resolution without changing its sizes, such as add, activation functions, and batch normalization. During the search, we limit the adjustments on the values of height, width, and channel sizes to doubling, halving, or keeping the same.
Our solution is based on Mixed Integer Linear Programming (MILP). MILP is an optimization problem formulated with linear objectives, linear constraints, and integer-valued variables. The input to MILP is the replacement segment DAG. Each node has two variables per each height, width, and channel dimension, denoting input and output resolutions. Each edge is associated with a "flow" variable. We define MILP constraints that regulate the correct flow of resolution. Immutable nodes have input resolutions equal to output resolutions. The output resolution for mutable nodes is less than or equal to the input resolution. The model is optimized to achieve the expected resolution at the output nodes. The model is proven infeasible if the search fails to achieve expected output resolutions.
We briefly illustrate the resolution propagation process. Figure 8 shows a replacement segment (yellow) that is being put together with the Predecessor (blue) and Successor (green) partitions of the network. We provide the output resolution of each operation in the form of (height, width, channel). Notice how the number of input and output nodes of the replacement segment matches the number of output and input nodes of the Predecessor and Successor, respectively. Initially, the replacement segment expects input dimension sizes for its 'Conv' and 'BN' operations of (32, 32, 16), which are the resolutions of the Predecessor's output nodes. Also, the Successor expects an input size of (16, 16, 32), which demands the replacement segment to output a feature map with this dimension at the 'Add' operation. This requires adjusting the resolution of
the 2 mutable 'Conv' operations in the replacement segment (highlighted with red borders). Notice that adjusting one of them or leaving resolutions unadjusted will result in incorrect propagation because the 'Add' operations require its incoming tensors to have the exact same dimensions. We use MILP to solve this problem by finding the correct adjustment to mutable operations by halving, doubling, or maintaining resolution sizes.
Figure 8: Resolution propagation adjusts the resolution of mutable operations in the replacement segment. The Height, Width, and Channel sizes are adjusted in both 'Conv' operations so that the replacement segment yields the expected output resolution at the 'Add' operation.
A.6 CIFAR-10 FLOPs Restraint Ablation
Table 9 provides a full ablation study of AutoGO on all 5 CIFAR-10 families in terms of FLOPs reduction constraint. We consider two settings where AutoGO can reduce FLOPs by at most  $-20\%$  relative to the baseline architecture, or can reduce them freely  $(-100\%)$ , while always limiting FLOPs increases to be at most  $+10\%$ . We again note how the best architecture for each family was found using segment mutations.
We observe that the segment-level mutation is a better fit for finding high-performance architectures under wider FLOPs constraints. For example, on HiAML, the segment-mutation cannot improve the accuracy of the base architecture when we impose a FLOPs reduction limit of  $-20\%$ , yet it can increase the accuracy by up to  $0.43\%$  on average when we remove the restriction, even though the best architecture only reduces FLOPs by  $-19.76\%$ . From this result, we infer that FLOPs restrictions hamper the exploration of the segment-level mutation. The only family where the  $-20\%$  FLOPs constraint produces a better architecture than the no-constraint setting is Inception, which is already the second-largest family with a base model size of nearly 500 MegaFLOPs. By contrast, the operation-level mutations require FLOPs reduction constraints to break even with the baseline
Table 9: Full ablation study of AutoGO on all 5 CIFAR-10 families considering choice of mutation unit {Operation, Segment}, predictor {GNN, PSC} and FLOPs [1e6] reduction (δ) constraint  $\{-20\%$  -  $100\% \}$ , extending the results of Table 2. For each experiment, we report the accuracy  $[\%]$  and FLOPs [1e6] (raw and  $\Delta$  relative to the baseline). We bold and italicize the best and second best result per family, respectively.
Baseline	Operator + GNN	Segment + GNN	Segment + PSC
Family (δFLOPs)	Acc.	FLOPs	Acc.	FLOPs	Acc.	FLOPs	Acc.	FLOPs
NB-101 (-20%)	95.18%	11722	95.16%	9407	95.31%	10817	95.06%	9606
Δ	-0.02%	-19.75%	+0.13%	-7.72%	-0.12%	-18.05%
NB-101 (-100%)	93.12%	1591	95.25%	10513	95.45%	11118
Δ	-2.06%	-86.43%	+0.07%	-10.31%	+0.27%	-5.15%
NB-201 (-20%)	93.50%	313	93.28%	250	92.86%	250	93.32%	251
Δ	-0.22%	-20.13%	-0.34%	-20.13%	-0.18%	-19.81%
NB-201 (-100%)	93.37%	232	93.57%	294	93.84%	303
Δ	-0.13%	-25.88%	+0.07%	-6.07%	+0.34%	-3.19%
HiAML (-20%)	92.32%	246	92.00%	198	92.08%	198	92.22%	230
Δ	-0.32%	-19.51%	-0.24%	-19.51%	-0.10%	-6.50%
HiAML (-100%)	84.63%	28	92.62%	168	92.75%	198
Δ	-7.69%	-88.62%	+0.30%	-31.71%	+0.43%	-19.51%
Inception (-20%)	93.50%	494	92.97%	399	93.12%	399	93.52%	474
Δ	-0.23%	-19.23%	-0.08%	-19.23%	+0.32%	-4.05%
Inception (-100%)	92.97%	319	93.31%	461	93.30%	478
Δ	-0.23%	-35.43%	+0.11%	-6.68%	+0.10%	-3.24%
Two-Path (-20%)	87.90%	116	88.63%	106	88.31%	93	88.68%	94
Δ	+0.73%	-8.62%	+0.41%	-19.83%	+0.78%	-18.97%
Two-Path (-100%)	88.63%	106	89.16%	48	88.94%	91
Δ	+0.73%	-8.62%	+1.26%	-58.62%	+1.04%	-21.55%
architectures. For example, when no FLOPs constraint is imposed, the operation-level mutation will find HiAML and NB-101 architectures that remove enough convolution nodes to reduce the model size by more than  $85\%$ . These changes drastically reduce the accuracy by over  $7.5\%$  on HiAML.
A.7 AutoGO Components Evaluation
We evaluate the search efficiency on the benchmark families by measuring the speed of each component. The time to execute the search largely depends on the choice of input architecture, i.e., architectures with more nodes and complex topologies like Inception form large search spaces. On the HiAML and NB-201 families, it takes 15 minutes on average to execute a search iteration using the PSC predictor and segment-level mutation. AutoGO visits over 1000 unique architectures per iteration and can find high-performance architectures in around an hour or less.
Specifically, it takes around 1.5 to 2 minutes to segment a parent architecture using BPE, select source and replacement segments, perform resolution propagation, and rank the mutations using the predictor. The bulk of this time is spent between searching the database for replacement segments, confirming their validity and measuring the performance of each mutation, while the BPE segmentation and source segment selection processes take less than 1 millisecond each. When gauging execution time, we sequentially mutate each parent architecture per iteration, but note that this process can be sped up with parallelization.
Resolution propagation with MILP takes 0.11 seconds on average to find a solution or determine that the problem is infeasible. We compare it to an exhaustive search approach by enumerating all candidate solutions. It takes, on average, 0.4 seconds to find a solution and more than 4 seconds for infeasible solutions. Our subgraph extraction process for generating the segment vocabulary is very efficient as the BPE operates on a sequence representation of the CGs. It takes less than 20 minutes to sort all CG topologically, and extract subsequences with BPE.
To provide specific examples of the search time, consider the ResNet-50 Arch 2 and EDSR Arch 3 architectures from Tables 3 and 4, respectively. Mutating the initial ResNet-50 and EDSR CGs takes 1.8 and 1.5 minutes, respectively, on our hardware. It takes longer to mutate ResNet-50 simply because the CG contains more nodes (108) than EDSR, whose CG only has 67 nodes. Moreover, since the base EDSR architecture only uses Convolutions and ReLU operations, we exclude segments
Figure 9: Example mutations performed by AutoGO to create EDSR Arch 2 in Table 4 by swapping out 8 EDSR blocks. Specifically, AutoGO will swap out multiple, simple 'Conv-ReLU-Conv' residual blocks for larger blocks that have operations on both branches.
that contain batchnorm and pooling operations, which reduces the number of replacement segments to consider during mutation.
The first iteration of AutoGO mutates the initial architecture while all subsequent iterations mutate 10 parent architectures. Given that ResNet-50 Arch 2 was found in iteration 3, it took AutoGO around
$$
1. 8 \mathrm {m i n} + 2 \text {i t e r} * 1 0 \text {a r c h / i t e r} * 1. 8 \mathrm {m i n} / \text {a r c h} = 3 7. 8 \mathrm {m i n}
$$
to discover that architecture. Likewise, EDSR Arch 3 was found in iteration 5, which took
$$
1. 5 \mathrm {m i n} + 4 \text {i t e r} * 1 0 \text {a r c h / i t e r} * 1. 5 \mathrm {m i n} / \text {a r c h} = 6 1. 5 \mathrm {m i n}
$$
to find. Finally, we note that these measurements and calculations assume sequential processing of parent architectures. In practice (e.g., runtime numbers in Sec 4.2), we use multi-processing techniques to mutate multiple parent architectures simultaneously to further speedup the process.
A.8 EDSR Mutation Example
Figure 9 illustrates three distinct mutations that take place to produce an EDSR AutoGO architecture. Initially, the EDSR backbone contains 16 'Conv-ReLU-Conv' residual blocks. To create the mutant network, AutoGO removed 8 of these blocks, denoting half the backbone structure, and replaced them with three double-branch structures that also consist of just convolutions and ReLU activations.
A.9 Hardware and Software Setup
We run our experiments on rack servers using Intel Xeon Gold 6140 CPUs. Each server is equipped with 8 NVIDIA V100 32GB GPUs and 756GB RAM. We execute our search and experiments on Python 3 using PyTorch==1.8.1 and TensorFlow==1.15.0. We implement our predictors using PyTorch-Geometric==1.7.1. We use SentencePiece [34] to perform BPE. Finally, we implement our MILP using a Coin-CBC solver [18] and pyomo==6.4.0 [23].