SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tiny-Imagenet	Caltech-256
Speedup	Memory	Speedup	Memory
Method	10%	20%	10%	20%	10%	20%	10%	20%
GLISTER	1.65	2.18	26705.5	22687.6	1.31	1.76	16904.5	13921.4
GradMatch	1.53	2.08	28249.9	25530.4	1.45	2.07	16499.3	12507.9
EL2N	—	2.30	—	21811.1	—	—	—	—
GraNd	—	2.16	—	24822.6	—	—	—	—
FacLoc	—	—	—	—	—	—	—	—
Pruning	—	—	—	—	—	—	—	—
Selection-via-Proxy	—	1.04	—	30717.2	—	—	—	—
Bottom-b-loss	—	2.54	—	15947.6	—	—	—	—
Bottom-b-loss+gumbel	1.88	2.72	17624.4	15085.9	—	2.36	—	9910.12
Inductive-SUBSELNET	2.02	3.54	17326.6	14505.4	2.43	3.16	9597.22	7406.23
Transductive-SUBSELNET	2.54	3.97	16281.1	12447.1	2.33	3.12	9983.86	7747.82
Table 8: Time and memory in reaching  $10\%$  and  $20\%$  RAR (90% and  $80\%$  of maximum accuracy of Full selection) in tradeoff curve in Figure 2 and 3 for all datasets. In the table, "—" denotes that under the current setup of experiments, i.e., the range of subsets considered, the method could not attain an accuracy equal to or less than  $20\%$  or  $10\%$  of RAR. Note that Bottom- $b$ -loss and Bottom- $b$ -loss+gumbel are variants/ablations of our method.
E.4 Recommending model architecture
When dealing with a pool of architectures designed for the same task, choosing the correct architecture for the task might be a daunting task - since it is impractical to train all the architectures from scratch. In view of this problem, we show that training on smaller carefully chosen subsets might be beneficial for a quicker alternative to choosing the correct architectures. We first extract the top 15 best performing architectures  $\mathcal{A}^*$  having highest accuracy, when trained on full data. We mark them as "gold". Then, we gather top 15 architectures  $\mathcal{A}$  when trained on the subset provided by our models. Then, we compare  $\mathcal{A}$  and  $\mathcal{A}^*$  using the Kendall tau rank correlation coefficient (KTau) along with Jaccard coefficient  $|\mathcal{A} \cap \mathcal{A}^*| / |\mathcal{A} \cup \mathcal{A}^*|$ .
Figure 9 summarizes the results for three non-adaptive subset selectors in terms of the accuracy, namely - Transductive-SUBSELNET, Inductive-SUBSELNET and FL. We make the following ob
observations: (1) One of our variant outperforms FL in most of the cases in CIFAR10 and CIFAR100. (2) There is no consistent winner between Transductive-SUBSELNET and Inductive-SUBSELNET, although Inductive-SUBSELNET outperforms both Transductive-SUBSELNET and FL consistently in CIFAR100 in terms of the Jaccard coefficient.
(d) (a) FMNIST
(e) (b) CIFAR10
(f) (c) CIFAR100
Figure 9: Comparison of the three non-adaptive subset selectors (Transductive-SUBSELNET, Inductive-SUBSELNET and FL) on ranking and choosing of the top-15 architectures on the basis of Jaccard Coefficient and Kendall tau rank correlation coefficient  $(K_{\tau})$
E.5 Analysis of subset overlap on different architectures
CIFAR10	CIFAR100
GLISTER	0.05	0.06
GRAD-MATCH	0.06	0.08
Our	0.08	0.08
Different architectures will produce different feature representations of the underlying dataset, and they can be distributed in different manners. Thus to generate a subset, if the features are different, we would expect subsets to change too. We experiment with extremely dissimilar architectures (top-5 ranked by distance in the latent space generated by the GNN) to observe the subset overlap occurring. Table 9 containing Jaccard coefficient of the subsets chosen for dissimilar architectures, where we notice that the overlaps are extremely small for the top adaptive methods, as well as for our method.
E.6 Finer analysis of the inference time
Table 9: Jaccard coefficient for subsets chosen by dissimilar architectures
Transductive	Inductive	FL
Subset selection	0.23	0.067	226.29
Training	70.1	70.1	70.1
Table 10: Inference time in seconds
Next, we demarcate the subset selection phase from the training phase of the test models on the selected subset during the inference time analysis. Table 10 summarizes the results for top three non-adaptive subset selection methods for  $b = 0.005|D|$  on CIFAR100. We observe that: (1) the final training times of all three methods are roughly same; (2) the selection time for Transductive-SUBSELNET is significantly more than Inductive-SUBSELNET, although it remains extremely small as compared to the final training on the inferred subset; and, (3) the selection time of FL is large—as close as  $323\%$  of the training time.
E.7 Analysis on underfitting and overfitting
Since the amount of training data is small, there is a possibility of overfitting. However, the coefficient  $\lambda$  of the entropy regularizer  $\lambda H(\mathrm{Pr}_{\pi})$ , can be increased to draw instances from the different regions
Subset Size	Training	Validation	Testing
Transductive	Inductive	Transductive	Inductive	Transductive	Inductive
10%	0.728	0.660	0.702	0.632	0.678	0.606
20%	0.852	0.673	0.809	0.658	0.770	0.644
40%	0.890	0.691	0.856	0.678	0.825	0.666
70%	0.942	0.738	0.912	0.717	0.884	0.698
of the feature space, which in turn can reduce the overfitting. In practice, we tuned  $\lambda$  on the validation set to control such overfitting.
We present the accuracies on (training, validation, test) folds for both Transductive-SUBSELNET and Inductive-SUBSELNET in Table 11. We make the following observations:
1. From training to test, in most cases, the decrease in accuracy is  $\sim 7\%$ .
2. This small accuracy gap is further reduced from validation to test. Here, in most cases, the decrease in accuracy is  $\sim 4\%$ .
E.8 Additional results on NAS and HPO
Searched cell for NAS. Figure 10 shows the final Normal and Reduction cells found on the DARTS search space using SUBSELNET on the  $40\%$  subset of CIFAR10, which gave the lowest test error of 2.68 in the experiments.
Figure 10: Normal [left] and Reduction [right] cells found by SUBSELNET during Neural Architecture Search using a  $40\%$  subset of the CIFAR10 dataset.
Standard error results on NAS and HPO. Here, we present the mean and standard error over the runs for the Test Error  $(\%)$  on NAS and HPO, which has been presented in the main draft (Section 5.2). We observe our method offers less deviation across runs. Moreover, we found that the gain offered by our method is statistically significant with  $p \approx 0.05$ .
Table 11: Variation of accuracy with subset size of both the variants of SUBSELNET on training, validation and test set of CIFAR10
Method	b=0.1|D|	b=0.2|D|	b=0.4|D|
Random	3.02 ± 0.171	2.88 ± 0.167	2.96 ± 0.169
Proxy-data [39]	2.92 ± 0.168	2.87 ± 0.167	2.88 ± 0.167
Our	2.82 ± 0.166	2.76 ± 0.164	2.68 ± 0.161
Table 12: Mean and standard error of Test Error (%) on architectures given by NAS on CIFAR10
Method	b=0.05|D|	b=0.1|D|
Random	5.44 ± 0.226	3.72 ± 0.189
AUTOMATA	5.26 ± 0.223	3.39 ± 0.181
Our	4.11 ± 0.199	2.70 ± 0.162
Table 13: Mean and standard deviation of Test Error (%) for hyperparameters selected by HPO on CIFAR10
F Pros and cons of using GNNs
We have used a GNN in our model encoder to encode the architecture representations into an embedding. We chose a GNN for the task due to following reasons -
1. Message passing between the nodes (which may be the input, output, or any of the operations) allows us to generate embeddings that capture the contextual structural information of the node, i.e., the embedding of each node captures not only the operation for that node but also the operations preceding that node to a large extent.
To better illustrate the impact of the GNN, we compared it with a baseline where we directly fed the graph structure to the model approximator using the adjacency matrix, in lieu of the GNN-derived node embeddings. This alteration resulted in a notable performance decline, leading to a  $5 - 6\%$  RAR on subset size of  $10\%$  of CIFAR10.
Variations of embedding	RAR	KL-div
b = 0.05|D|	b = 0.1|D|
Feedforward (gβ, A)	0.481	0.433	0.231
Feedforward (gβ, H)	0.434	0.397	0.171
LSTM (gβ, A)	0.471	0.436	0.224
LSTM (gβ, H)	0.412	0.386	0.102
Attn. (gβ, A)	0.362	0.317	0.198
Attn. (gβ, H)	0.310	0.260	0.089
Table 14: RAR and KL-div for different embeddings (A: Adjacency Matrix, H: GNN embedding) in model approximator
2. It has been shown by [38] and [57] that GNNs are as powerful as the Weisfeiler-Lehman algorithm and thus give a powerful representation for the graph. Thus, we obtain smooth embeddings of the nodes/edges that can effectively distill information from its neighborhood without significant compression.
3. GNNs embed model architecture into representations independent of the underlying dataset and the model parameters. This is because it operates on only the nodes and edges—the structure of the architecture and does not use the parameter values or input data.
However, the GNN faces the following drawbacks -
1. GNN uses a symmetric aggregator for message passing over node neighbors to ensure that the representation of any node should be invariant to a permutation of its neighbors. Such a symmetric aggregator renders it a low-pass filter, as shown in [41], which attenuates important high-frequency signals.
2. We are training one GNN using several architectures. This can lead to the insensitivity of the embedding to change in the architecture. In the context of model architecture, if we change the operation of one node in the architecture (either remove, add or change the operation), then the model's output can significantly change. However, the embedding of GNN may become immune to such changes, since the GNN is being trained over many architectures.
G Licensing Information
The NAS-Bench-101 dataset and DARTS Neural Architecture Search are publicly available under the Apache License. The baselines GRAD-MATCH and GLISTER, publicly available via CORDS, and the apricot library used for Facility Location, are under the MIT License.