SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Generalization across architectures but not instances. Note that the goal of the model approximator is to predict the output on  $x$  in the training set  $D_{\mathrm{tr}}$  for unseen architecture  $m'$  so that using these predictions, our method can select the subset  $S$  from  $D_{\mathrm{tr}}$  in a way that  $m'$  trained on  $S$  shows high accuracy on  $D_{\mathrm{test}}$ . Since the underlying subset  $S$  has to be chosen from the training set  $D_{\mathrm{tr}}$  for an arbitrary architecture  $m'$ , it is enough for the model approximator to mimic the model output only on the training set  $D_{\mathrm{tr}}$  — it need not have to perform well in the test set  $D_{\mathrm{test}}$ .
4.3 Subset sampler and design of transductive and inductive SUBSELNET
Subset sampler. We draw  $S$ , an ordered set of elements, using  $\pi$  as follows. Having chosen the first  $t$  instances  $S_{t} = \{s_{1},..s_{t}\}$  from  $D$  with  $S_{0} = \emptyset$ , it draws the  $(t + 1)$ -th element  $(\pmb{x},y)$  from the remaining instances in  $D$  with a probability proportional to  $\exp(\pi(\pmb{x},y))$  and then repeats it for  $b$  times. Thus, the probability of selecting the ordered set of elements  $S = \{s_{1},\dots,s_{b}\}$  is given by
$$
\Pr_ {\pi} (S) = \prod_ {t = 0} ^ {b} \frac {\exp \left(\pi \left(\boldsymbol {x} _ {s _ {t + 1}} , y _ {s _ {t + 1}}\right)\right)}{\sum_ {s _ {\tau} \in D \backslash S _ {t}} \exp \left(\pi \left(\boldsymbol {x} _ {s _ {\tau}} , y _ {s _ {\tau}}\right)\right)} \tag {8}
$$
The optimization (3) suggests that once  $\mathrm{GNN}_{\alpha}$  and  $g_{\beta}$  are trained, we can use them to approximate the output of the trained model  $m_{\theta^*}$  for an unseen architecture  $m'$  and use it to compute  $\pi$ . Thus, this already removes a significant overhead of model training and facilitates fast computation of  $\pi$ , and further leads us to develop Transductive-SUBSELNET and Inductive-SUBSELNET based on how we can compute  $\pi$ , as described at the end of Section 3.1.
Transductive-SUBSELNET. The first variant of the model is transductive in terms of the computation of  $\pi$ . Once we train the architecture encoder and the model approximator, we compute  $\pi$  by solving the optimization problem explicitly with respect to  $\pi$  every time when we wish to select a data subset for a new architecture. Given trained model  $\mathrm{GNN}_{\hat{\alpha}}$ ,  $g_{\hat{\beta}}$  and a new architecture  $m' \in \mathcal{M}$ , we solve the optimization problem to find the subset sampler  $\operatorname{Pr}_{\pi}$  during inference time for a new architecture  $m'$ .
$$
\min  _ {\pi} \mathbb {E} _ {S \in \Pr_ {\pi} (\bullet)} \Lambda (S; m ^ {\prime}; \pi , g _ {\hat {\beta}}, \mathrm {G N N} _ {\hat {\alpha}}) \tag {9}
$$
Such an optimization still consumes time during inference. However, it is still significantly faster than the combinatorial methods [20, 19, 37, 47] thanks to sidestepping the explicit model training using a model approximator.
Inductive-SUBSELNET. In contrast to the transductive model, the inductive model does not require explicit optimization of  $\pi$  in the face of a new architecture. To that aim, we approximate  $\pi$  using a neural network  $\pi_{\psi}$  which takes two signals as inputs—the dataset  $D$  and the outputs of the model approximator for different instances  $\{g_{\hat{\beta}}(\mathbf{H}_m,\pmb{x}_i)|i\in D\}$  and finally outputs a score for each instance  $\pi_{\psi}(\pmb{x}_i,y_i)$ . Here, the training of  $\pi_{\psi}$  follows from the optimization (3):
$$
\min  _ {\psi} \sum_ {m \in \mathcal {M}} \mathbb {E} _ {S \sim \Pr_ {\pi_ {\psi}}} \Lambda (S; m; \pi_ {\psi}, g _ {\hat {\beta}}, \mathrm {G N N} _ {\hat {\alpha}}) \tag {10}
$$
Such an inductive model can select an optimal distribution of the subset that should be used to efficiently train any model  $m_{\theta}$ , without explicitly training  $\theta$  or searching for the underlying subset.
Architecture of  $\pi_{\psi}$  for Inductive-SUBSELNET. We approximate  $\pi$  using  $\pi_{\psi}$  using a neural network which takes three inputs  $-(x_{j},y_{j})$ , the corresponding output of the model approximator, i.e.,  $o_{m,\pmb{x}_j} = g_\beta (\mathrm{GNN}_\alpha (G_m),\pmb{x}_j)$  from Eq. (6) and the node representation matrix  $H_{m}$  and provides us a positive selection score  $\pi_{\psi}(H_{m},x_{j},y_{j},o_{m,x_{j}})$ . In practice,  $\pi_{\psi}$  is a three-layer feed-forward network containing Leaky-ReLU activation functions for the first two layers and sigmoid activation at the last layer.
4.4 Training and inference routines
Training. The training phase for both transductive and inductive variants, first utilizes the TRAINPIPELINE(Algorithm 1) routine to train the GNN (TRAINGNN), re-order the embeddings based on BFS ordering (BFS), train the model approximator (TRAINAPPROX), to obtain  $\hat{\beta}$ . TRAINTRANSDUCTIVE(Algorithm 2) routine doesn't require any further training, while the TRAININDUCTIVE(Algorithm 3) routine uses the TRAINPI to train  $\psi$  for computing  $\pi$ .
Inference. Given a new architecture  $m^{\prime}$ , our goal is to select a subset  $S$ , with  $|S| = b$  which would facilitate efficient training of  $m^{\prime}$ . Given trained SUBSELNET, we compute  $\pmb{H}_{m^{\prime}} = \mathrm{GNN}_{\hat{\alpha}}(G_{m^{\prime}})$ , compute the model approximator output  $g_{\hat{\beta}}(H_{m^{\prime}})$ . Using them we compute  $\pi$  for
Algorithm 1 Training Pipeline
1: function TRAINPIPELINE(D, M, {θ*})
2: \hat{α} ← TRAINGNN(M)
3: for m ∈ Mtr do
4: Hm← GNN\(\hat{\alpha}\)(m), pos ← BFS(Gm, Hm)
5: \hat{β} ← TRAINAPPROX(Hm, {xi}, pos, {θ*})
6: return \(\hat{\alpha}, \hat{\beta}, H_m\)
Algorithm 2 Transductive Procedure
1: function TRAINTRANSDUCTIVE(D, M, {θ*})
2: \hat{α}, \hat{β}, Hm←TRAINPIPELINE(D, M, {θ*})
1: function INFERTRANSDUCTIVE(D, \hat{α}, \hat{β}, m')
2: \pi*←minπ ESS∈Pr(·) Λ(S; m'; π, ghat, GNNhat)
3: S* ∼ Pr(·)
4: TRAINNEWMODEL(m'; S*)
Transductive-SUBSELNET by explicitly solving the optimization problem stated in Eq. 9 and draw  $S \sim \operatorname{Pr}_{\pi}(\bullet)$ . For the inductive variant, we draw  $S \sim \operatorname{Pr}_{\pi_{\hat{\psi}}}(\bullet)$  where  $\hat{\psi}$  is the learned value of  $\psi$ .
Given an unseen architecture  $m^{\prime}$  and trained parameters of SUBSELNET, i.e.,  $\hat{\alpha},\hat{\beta}$  and  $\hat{\psi}$ , the INFERTRANSDUCTIVE(Algorithm 2) routine solves the optimization problem on  $\pi$  explicitly to compute  $\pi$ , where  $\Lambda (\cdot)$  is defined in Eq. (2).
Algorithm 3 Inductive Procedure
1: function TRAININDUCTIVE(D, M, {θ*})
2: \hat{α}, \hat{β}, Hm←TRAINPIPELINE(D, M, {θ*})
3: o← [gβ({Hm, xi})]i,m
4: \hat{ψ}← TRAINPI(o, {Hm}, {xi})
1: function INFERINDUCTIVE(D, \hat{α}, \hat{β}, \hat{ψ}, m')
2: Compute πhat{ψ}(Hm', xi, yi, o_m', xi) ∀i ∈ D
3: S* ~ Prπhat{ψ}()
4: TRAINNEWMODEL(m'; S*)
INFERINDUCTIVE (Algorithm 3) utilizes  $\hat{\psi}$ , i.e., trained parameters from the subset sampler to compute  $\pi_{\hat{\psi}}$ . Then the subset  $S^{*}$  is drawn from  $\pi$  or  $\pi_{\psi}$  and is used to train  $m^{\prime}$  using TRAINNEW-MODEL.
In this section, we provide comprehensive evaluation of SUBSELNET against several strong baselines on five real world datasets. In Appendix E, we present additional results. Our code is in https://github.com/structlearning/subselnet.
Datasets. We use FMNIST [56], CIFAR10 [26], CIFAR100 [25], Tiny-Imagenet-200 [27] and Caltech-256 [13] (Cal-256). Cal-256 has imbalanced class distribution; the rest are balanced. We transform an input image  $\mathbf{X}_i$  to a vector  $\mathbf{x}_i$  of dimension 2048 by feeding it to a pre-trained ResNet50 v1.5 model [16] and use the output from the penultimate layer as the image representation.
Model architectures and baselines. We use model architectures from NAS-Bench-101 [61] in our experiments. We compare Transductive-SUBSELNET and Inductive-SUBSELNET against three non-adaptive subset selection methods - (i) Facility location [11, 17] where we maximize  $FL(S) = \sum_{j\in D}\max_{i\in S}\boldsymbol{x}_i^\top \boldsymbol{x}_j$  to find  $S$ , (ii) Pruning [48], and (iii) Selection via-Proxy [5] and four adaptive subset selection methods - (iii) Glister [20], (iv) Grad-Match [19], (v) EL2N [42] and (vi) GraNd [42]. The non-adaptive subset selectors select the subset before the training begins and thus, never access the rest of the training set again during the training iterations. On the other hand, the adaptive subset selectors refine the choice of subset during training iterations and thus they need to access the full training set at each training iteration. Appendix D contains additional details about the baselines and Appendix E contains experiments with more baselines.
Evaluation protocol. We split the model architectures  $\mathcal{M}$  into  $70\%$  training  $(\mathcal{M}_{\mathrm{tr}})$ ,  $10\%$  validation  $(\mathcal{M}_{\mathrm{val}})$  and  $20\%$  test  $(\mathcal{M}_{\mathrm{test}})$  folds. However, training model approximator requires supervision from the pre-trained models  $m_{\theta^*}$ . Pre-training large number of models can be expensive. Therefore, we limit the number of pre-trained models to a diverse set of size 250, that ensures efficient representation over low-parameter and high-parameter regimes, and using more than this showed no visible advantage. We show the parameter statistics in Appendix D. However, for the architecture encoder, we use the entire set  $\mathcal{M}_{\mathrm{tr}}$  for GNN training. We split the dataset  $D$  into  $D_{\mathrm{tr}}$ ,  $D_{\mathrm{val}}$  and  $D_{\mathrm{test}}$  in the similar 70:10:20 folds. We present  $\mathcal{M}_{\mathrm{tr}}$ ,  $\mathcal{M}_{\mathrm{val}}$ ,  $D_{\mathrm{tr}}$  and  $D_{\mathrm{val}}$  to our method and estimate  $\widehat{\alpha}$ ,  $\widehat{\beta}$  and  $\widehat{\psi}$  (for Inductive-SUBSELNET model). None of the baseline methods supports any generalizable learning protocol for different architectures and thus cannot leverage the training architectures during test. Given an architecture  $m' \in \mathcal{M}_{\mathrm{test}}$ , we select the subset  $S$  from  $D_{\mathrm{tr}}$  using our subset sampler  $(\operatorname{Pr}_{\pi}$  for Transductive-SUBSELNET or  $\operatorname{Pr}_{\pi_{\widehat{\psi}}}$  for Inductive-SUBSELNET). Similarly, all the non-adaptive subset selectors select  $S \subset D_{\mathrm{tr}}$  using their own algorithms. Once  $S$  is selected, we train the test models  $m' \in \mathcal{M}_{\mathrm{test}}$  on  $S$ . We perform our experiments with different  $|S| = b \in (0.005|D|, 0.9|D|$ ) and compare the performance between different methods using three quantities: (1) Relative Accuracy Reduction (RAR) computed as the drop in test accuracy on training with a chosen subset as compared to training with the entire dataset, i.e.,  $\operatorname{RAR}(S, D) = \frac{1}{|\mathcal{M}_{\mathrm{test}}|} \sum_{m' \in \mathcal{M}_{\mathrm{test}}} (1 - \operatorname{Acc}(m' | S) / \operatorname{Acc}(m' | D))$  where  $\operatorname{Acc}(m' | X)$  denotes the test accuracy when  $m'$  is trained on the set  $X$ . Lower RAR indicates better performance. (2) Computational efficiency, i.e., the speedup achieved with respect to training with full dataset. It is measured with respect to  $T_f / T$ . Here,  $T_f$  is the time taken for training with full dataset; and,  $T$  is the time taken for the entire inference task, which is the average time for selecting
FacLoc
Inductive-SubSelNet
(a) FMNIST
(c) CIFAR100
(d) TINY-IN
(e) CAL-256
Figure 2: Trade-off between RAR (lower is better) and speedup (top row) and RAR and memory consumption in GB-min (bottom row) for the non-adaptive methods - Facility location [11, 17], Pruning [48], Selection-via-Proxy [5] on all five datasets - FMNIST, CIFAR10 CIFAR100, TinyImageNet and Caltech-256. In all cases, we vary  $|S| = b \in (0.005|D|, 0.9|D|)$ .
GraNd
Transductive-SubSelNet
GLISTER
EL2N
GRAD-MATCH
(d) TINY-IN
Figure 3: Trade-off between RAR (lower is better) and speedup (top row) and RAR and memory consumption in GB-min (bottom row) for the adaptive methods - Glister [20], Grad-Match [19], EL2N [42]; GraNd [42] on all five datasets - FMNIST, CIFAR10 CIFAR100, Tiny-ImageNet and Caltech-256. In all cases, we vary  $|S| = b \in (0.005|D|, 0.9|D|)$ .
subsets across the test models  $m' \in \mathcal{M}_{\mathrm{test}}$  plus the average training time of these test models on the respective selected subsets. (3) Resource efficiency in terms of the amount of memory consumed during the entire inference task, described in item (2), which is measured as  $\int_0^T \mathrm{memory}(t) dt$  where  $\mathrm{memory}(t)$  is the amount of memory consumed at timestamp  $t$  in the unit of GB-min.
Comparison with baselines. Here, we compare different methods in terms of the trade-off between Relative accuracy reduction RAR (lower is better) and computational efficiency as well as RAR and resource efficiency. In Figures 2 and 3, we probe the variation between these quantities by varying the size of the selected subset  $|S| = b \in (0.005|D|, 0.9|D|)$  for non-adaptive and adaptive baselines, respectively. We make the following observations. (1) Our methods trade-off between accuracy vs. computational efficiency as well as accuracy vs. resource efficiency more effectively than all the methods, including the adaptive methods which refine their choice of subset as the model training progresses. (2) In FMNIST, our method achieves  $10\%$  RAR at  $\sim 4.4$  times the speed-up and using  $77\%$  lesser memory than EL2N, the best baseline (Table 1, tables for other datasets are in Appendix E). (3) There is no consistent winner across baselines. However, Glister and Grad-Match mostly remain among top three baselines, across different methods. In particular, they outperform others in Tiny-Imagenet and Cal-256, in high accuracy (low RAR) regime.
Speedup	Memory
RAR	10%	20%	10%	20%
GLISTER	5.64	7.85	116.36	98.51
GradMatch	4.17	5.24	243.75	136.40
EL2N	6.50	16.42	139.89	77.63
Inductive	28.64	69.24	22.73	8.24
Transductive	28.63	68.36	21.25	8.24
Figure 4: Hybrid-SUBSELNET
Hybrid-SUBSELNET. In FMINST, CIFAR10 and CIFAR100, we observe that Transductive-SUBSELNET offers better traded off than Inductive-SUBSELNET. Here, we design a hybrid version of our model, called as Hybrid-SUBSELNET and evaluate it on a regime where the gap between transductive and inductive SUBSELNET is significant. One of such regimes is the part of the trade-off plot in CIFAR100, where the speed up  $T_f / T \geq 28.09$  (Figures 2 and 3). Here, given the budget of the subset  $b$ , we first choose  $B > b$  instances using Inductive-SUBSELNET and the final  $b$  instances by running the explicit optimization routines in Transductive-SUBSELNET. Figure 4 shows the results for  $B = \{25K, 30K, 35K, 45K, 50K\}$ . We observe that Hybrid-SUBSELNET allows us to smoothly trade off between Inductive-SUBSELNET and Transductive-SUBSELNET, by tuning  $B$ . It allows us to effectively use resource-constrained setup with limited GPU memory, wherein the larger subset  $B$  can be selected using Inductive-SUBSELNET on a CPU, and the smaller refined subset  $b$  can then be selected by solving transductive variant on GPU.
Ablation study. Here, we experiment with three candidates of model approximator  $g_{\beta}$  (Feedforward, LSTM and our proposed attention based approximator) with three different subset samplers  $\pi$  (uncertainty based, loss based and our proposed subset sampler). Thus, we have nine different combinations of model approximator and subset selection strategies. In the uncertainty and loss based subset samplers, we take top-  $b$  instances based on the uncertainty and loss. We measure uncertainty using the entropy of the predicted distribution of the target classes. We compare the performance in terms of the test RAR of the test architectures. Moreover, we also evaluate the model approximator  $g_{\beta}$  alone — without the presence of the subset sampler — using KL divergence between the gold model outputs and predicted model outputs on the training instances  $\frac{1}{|D_{\mathrm{tr}}||\mathcal{M}_{\mathrm{test}}|}\sum_{i\in D_{\mathrm{tr}},m\in \mathcal{M}_{\mathrm{test}}}KL(m_{\theta^*}(\boldsymbol {x}_i)||g_{\beta}(\boldsymbol {H}_m,\boldsymbol {x}_i))$ . Table 3 summarizes the results for  $3\%$ ,  $5\%$  and  $10\%$  subsets for CIFAR10. We make the following observations: (1) The complete design of our method, i.e., Our model approximator (Transformer) + Our subset sampler (SUBSELNET) performs best in terms of RAR. (2) Our neural-network for model approximator mimics the trained model output better than LSTM and Feedforward architectures.
Can model approximator substitute our subset selector pipeline? The task of the model approximator  $g_{\beta}$  is to predict accuracy for unseen architecture. Then, a natural question is that is it possible to use the model approximator to directly predict accuracy of the unseen architecture  $m'$ , instead of using such long pipeline to select subset  $S$  followed with training on  $S$ . However, as discussed in the end of Section 4.2, the model approximator  $g_{\beta}$  is required to generalize across unseen architectures but not the unseen instances, as its task is to help select the training subset. Table 3
already showed that  $g_{\beta}$  closely mimics the output of the trained model for the unseen architecture  $m^{\prime}\in \mathcal{M}_{\mathrm{test}}$  and on training instances  $\pmb{x}$  (KL div. column). Here, we investigate the performance of  $g_{\beta}$  on the test instances and test architectures.
Table 1: Speedup and memory (GB-min) in reaching  ${10}\%$  and  ${20}\%$  RAR on FMNIST
b (in %)	90%	70%	20%
RAR(our | S) - RAR(gβ)	-0.487	-0.447	-0.327
Table 2: RAR using  ${g}_{\beta }$  on CIFAR10
Table 2 shows that the performance of  $g_{\beta}$  on the test instances is significantly poorer than our method. This is intuitive as generalizing both the model space and the instance space is extremely challenging, and we also do not need it in general.
Using SUBSELNET in AutoML. AutoML-related tasks can be significantly sped-up when we replace the entire dataset with a representative subset. Here, we apply SUBSELNET to two AutoML applications: Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO).
Neural Architecture Search: We apply our method on DARTS architecture space to search for an architecture using subsets. During this search process, at each iteration, the underlying network is traditionally trained on the entire dataset. In contrast, we train this underlying network on the subset returned by our method for this architecture. Following Na et al. [39], we report test misclassification
Design choice of gβ and π	RAR	KL-div
b = 0.03|D|	b = 0.05|D|	b = 0.1|D|	(does not depend on b)
Feedforward (gβ) + Uncertainty (π)	0.657	0.655	0.547
Feedforward (gβ) + Loss (π)	0.692	0.577	0.523	0.171
Feedforward + Inductive (our) (π)	0.451	0.434	0.397
LSTM (gβ) + Uncertainty (π)	0.566	0.465	0.438
LSTM (gβ) + Loss (π)	0.705	0.541	0.455	0.102
LSTM (gβ) + Inductive (our) (π)	0.452	0.412	0.386
Attn. (our) (gβ) + Uncertainty (π)	0.794	0.746	0.679
Attn. (our) (gβ) + Loss (π)	0.781	0.527	0.407	0.089
Attn. (our) (gβ) + Inductive (our) (π)	0.429	0.310	0.260
Table 3: RAR and KL-divergence for different  ${g}_{\beta } + \pi$  on CIFAR10 for 3%, 5% and 10% subset sizes
b (in %)	10%	20%	40%
Full	2.78
Random	3.02	2.88	2.96
Proxy [39]	2.92	2.87	2.88
Our	2.82	2.76	2.68
Table 4: Test Error (%) on architecture given by NAS on CIFAR10
Method	b = 5%	b = 10%
TE	S/U	TE	S/U
Full	2.48	1	2.48	1
Random	5.4	16.66	3.72	11.29
AUTOMATA	5.26	0.51	3.39	0.20
Our	4.11	16.11	2.70	10.96
Table 5: Test Error  $(\%)$  (TE) and Speed-up (S/U) for the hyperparameters selected by HPO on CIFAR10
Method	# Test Architectures
200	300	400
Full	7111	7111	7111
GLISTER	3419	3419	3419
GRAD-MATCH	2909	2909	2909
Our	1844	1635	1496
Table 6: Amortization cost (seconds) after querying test architectures on CIFAR10
error of the architecture which is selected by the corresponding subset selector guided NAS methods, i.e., our method (transductive), random subset selection (averaged over 5 runs) and proxy-data [39]. Table 4 shows that our method performs better than the baselines.
Hyperparameter Optimization: Finding the best set of hyperparameters from their search space for a model is computationally intensive. We look at speeding-up the tuning process by searching the hyperparameters while training the model on a small representative subset  $S$  instead of  $D$ . Following Killamsetty et al. [22], we consider optimizer and scheduler specific hyperparameters and report average test misclassification error across the models trained on optimal hyperparameter choice returned by our method (transductive), random subset selection (averaged over 5 runs) and AUTOMATA [22]. Table 5 shows that we are outperforming the baselines in terms of accuracy-speedup tradeoff. Appendix D contains more details about the implementation.
Amortization Analysis. Figures 2 and 3 show that our method is substantially faster than the baselines during inference, once we have our neural pipeline trained. Such inference time speedup is the focus of many other applications, E.g., complex models like LLMs are difficult and computationally intensive, but their inference is fast for several queries once trained. However, we recognize that there is a computational overhead in training our model, arising due to the pre-training of the models  $m_{\theta^{*}}$  used for supervision. Since the prior training is only a one-time overhead, the overall cost is amortized by querying multiple architectures for their subsets. We measure amortized cost  $T_{\mathrm{total}} / M_{\mathrm{total}}$  (time in seconds), where  $T_{\mathrm{total}}$  is the total time used from beginning of the pipeline to end of reporting final accuracy on the test architectures and  $M_{\mathrm{total}}$  is the total number of training and test architectures. Table 6, shows the results for  $10\%$  subset on the top baselines for CIFAR10, which shows that the training overhead of our method (transductive) quickly diminishes with number of test architectures.
In this work, we develop SUBSELNET, a subset selection framework, which can be trained on a set of model architectures, to be able to predict a suitable training subset before training a model, for an unseen architecture. To do so, we first design a neural architecture encoder and model approximator, which predicts the output of a new candidate architecture without explicitly training it. We use that output to design transductive and inductive variants of our model.
Limitations. The SUBSELNET pipeline offers quick inference-time subset selection but a key limitation of our method is that it entails a pre-training overhead, although its overhead vanishes as we query more architectures. Such expensive training can be reduced by efficient training methods [65]. In future, it would be interesting to incorporate signals from different epochs with a sequence encoder to train a subset selector. Apart from this, our work does not assume the distribution shift of architectures from training to test. If the architectures vary significantly from training to test, then there is significant room for performance improvement.
Acknowledgements. Abir De acknowledges the Google Research gift funding.
[66] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curriculum learning by optimizing learning dynamics. In International Conference on Artificial Intelligence and Statistics, pages 433-441. PMLR, 2021.
[67] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning, 2016. URL https://arxiv.org/abs/1611.01578.
[68] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition, 2017. URL https://arxiv.org/abs/1707.07012.
Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networks (Appendix)
While our work outperforms several existing subset selection methods, it suffers from three key limitations.