Title: Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Data Subset Selection

Efficient Learning
Abstract: Existing subset selection methods for efficient learning predominantly employ discrete combinatorial and model-specific approaches, which lack generalizability--- for each new model, the algorithm has to be executed from the beginning. Therefore, for an unseen architecture, one cannot use the subset chosen for a different model. In this work, we propose $\texttt{SubSelNet}$, a non-adaptive subset selection framework, which tackles these problems. Here, we first introduce an attention-based neural gadget that leverages the graph structure of architectures and acts as a surrogate to trained deep neural networks for quick model prediction. Then, we use these predictions to build subset samplers. This naturally provides us two variants of $\texttt{SubSelNet}$. The first variant is transductive (called Transductive-$\texttt{SubSelNet}$), which computes the subset separately for each model by solving a small optimization problem. Such an optimization is still super fast, thanks to the replacement of explicit model training by the model approximator. The second variant is inductive (called Inductive-$\texttt{SubSelNet}$), which computes the subset using a trained subset selector, without any optimization.  
Our experiments show that our model outperforms several methods across several real datasets.

Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networks

Abstract

Existing subset selection methods for efficient learning predominantly employ discrete combinatorial and model-specific approaches which lack generalizability. For an unseen architecture, one cannot use the subset chosen for a different model. To tackle this problem, we propose SUBSELNET, a trainable subset selection framework, that generalizes across architectures. Here, we first introduce an attention-based neural gadget that leverages the graph structure of architectures and acts as a surrogate to trained deep neural networks for quick model prediction. Then, we use these predictions to build subset samplers. This naturally provides us two variants of SUBSELNET. The first variant is transductive (called as TransductiveSUBSELNET) which computes the subset separately for each model by solving a small optimization problem. Such an optimization is still super fast, thanks to the replacement of explicit model training by the model approximator. The second variant is inductive (called as Inductive-SUBSELNET) which computes the subset using a trained subset selector, without any optimization. Our experiments show that our model outperforms several methods across several real datasets.

1 Introduction

In the last decade, neural networks have drastically enhanced the performance of state-of-the-art ML models. However, they often demand massive data to train, which renders them heavily contingent on the availability of high-performance computing machineries such as GPUs and RAM. Such resources entail heavy energy consumption, excessive CO 2 emission, and maintenance cost.

Driven by this challenge, a recent body of work focuses on suitably selecting a subset of instances so that the model can be trained quickly using lightweight computing infrastructure [4, 23, 51, 32, 54, 37, 18-21, 36]. However, these methods are not generalizable across architectures- the subset selected by such a method is tailored to train only one specific architecture and thus need not be optimal for training another architecture. Hence, to select data subsets for a new architecture, they need to be run from scratch. However, these methods rely heavily on discrete combinatorial algorithms, which impose significant barriers against scaling them for multiple unseen architectures. Appendix C contains further details about related work.

1.1 Our contributions

Responding to the above limitations, we develop SUBSELNET, a trainable subset selection framework. Specifically, we make the following contributions.

Novel framework on subset selection that generalizes across models. SUBSELNET is a subset selector that generalizes across architectures. Given a dataset, once SUBSELNET is trained on a set of model architectures, it can quickly select a small optimal training subset for any unseen (test) architecture, without any explicit training of this test model. SUBSELNET is a non-adaptive method 37th Conference on Neural Information Processing Systems (NeurIPS 2023).

since it learns to select the subset before the training starts for a new architecture, instead of adaptively selecting the subset during the training process. Our framework has several applications in the context of AutoML [35, 68, 30, 43, 61, 9, 2, 24, 22, 3]. For example, Network Architecture Search (NAS) can have a signficant speed-up when the architectures during selection can be trained on the subsets provided by our method, as compared to the entire dataset. In hyperparameter selection, such as the number and the widths of layers, learning rates or scheduler-specific hyperparameters, we can train each architecture on the corresponding data subset obtained from our method to quickly obtain the trained model for cross-validation.

Design of neural pipeline to eschew model training for new architecture. We initiate our investigation by writing down a combinatorial optimization problem instance that outputs a subset specifically for one given model architecture. Then, we gradually develop SUBSELNET, by building upon this setup. The key blocker in scaling up a model-specific combinatorial subset selector across different architectures is the involvement of the model parameters as optimization variables along with the candidate data subset. We design the neural pipeline of SUBSELNET to circumvent this blocker specifically. This neural pipeline consists of the following three components: (1) GNN-guided architecture encoder: This converts the architecture into an embedded vector space. (2) Neural model approximator: It approximates the predictions of a trained model for any given architecture. Thus, it provides the accuracy of a new (test) model per instance without explicitly training it. (3) Subset sampler: It uses the predictions from the model approximator and an instance to provide a selection score of the instance. Due to the architecture encoder and the neural approximator, we do not need to explicitly train a test model for selecting the subset since the model approximator directly provides the predictions the model will make.

Transductive and Inductive SUBSELNET. Depending on the functioning of the subset sampler in the final component of our neural pipeline, we design two variants of our model.

TransductiveSUBSELNET : The first variant is transductive in nature. For each new architecture, we utilize the the model approximator's predictions for replacing the model training step in the original combinatorial subset selection problem. However, the candidate subset still remains involved as an optimization variable. Thus, we still solve a fresh optimization problem with respect to the selection score provided by the subset sampler every time we encounter a new architecture. However, the direct predictions from the model approximator allow us to skip explicit model training, making this strategy extremely fast in terms of memory and time.

InductiveSUBSELNET : In contrast to Transductive-SUBSELNET, the second variant does not require to solve any optimization problem and instead models the selection scores using a neural network. Consequently, it is extremely fast.

We compare our method against six state-of-the-art methods on five real world datasets, which show that SUBSELNET provides the best trade-off between accuracy and inference time as well as accuracy and memory usage, among all the methods.

2 Preliminaries

Setting. We are given a set of training instances { ( x i , y i ) } i ∈ D where we use D to index the data. Here, x i ∈ R d x denotes the features, and y i ∈ Y denotes the labels. In our experiments, we consider Y as a set of categorical labels. However, our framework can also be used for continuous labels. We use m to denote a neural architecture and represent its parameterization as m θ . We also use M to denote the set of neural architectures. Given an architecture m ∈ M , G m = ( V m , E m ) provides the graph representation of m , where the nodes u ∈ V m represent the operations and the e = ( u m , v m ) indicates an edge, where the output given by the operation represented by the node u m is fed to one of the operands of the operation given by the node v m . Finally, we use H ( · ) to denote the entropy of a probability distribution and ℓ ( m θ ( x ) , y ) as the cross entropy loss hereafter.

2.1 Combinatorial subset selection for efficient learning

Given a dataset { ( x i , y i ) } i ∈ D and a model architecture m ∈ M with its neural parameterization m θ , the goal of a subset selection algorithm is to select a small subset of instances S with | S | = b << | D | such that training m θ on the subset S gives nearly the same accuracy as training on the entire dataset D . Existing works [20, 47, 19] adopt different strategies to achieve this goal, but all of them aim to simultaneously optimize for the model parameters θ as well as the candidate subset S . At the outset,

<latexi sh

1_b

64="Pu zY

mXj7cy/v5RTUO

IHG0

>A

dVDLSgM

F

3

2

Z

k

wNW

E

o

Jq

n

QK

f

C

r

8

9

p

<latexi sh

1_b

64="g0O

CA

P

rnyW

F

SQf zkM

>

B

H

c

V

N

8

EJ3

X

L9

UD

2

m7G

/

Iv

T

5

+

u

Z

d

Y

p

j

R

<latexi sh

1_b

64="ZCVHX

3o

DP

rp8

n

d

I

0

>A

B

c

9TwJ

E

L

Qv

N

m

z

yRG

2

F7K

M

jY

+

5OW

u

f

S

U

q

/

<latexi sh

1_b

64="Y7

+Tr2GpRE

LI

OSJD

X

B

0

>A

/H

c

V

N

8

3U

q

oj

C

WP

g5

K

f

Q

Zdu

m

k

vw

z

F

y

M

n

9

<latexi sh

1_b

64="0y

AJC

P

F/w

j

O2

gH+TB8

>

V

9S

N

E

LXz

R

o

5

Y

n

m

u3f

7

K

D

p

Q

G

M

v

r

W

d

U

q

Z

I

<latexi sh

1_b

64="nWEG

r

3p2HZ0P7

C

F

jc

>A

V

T

N

Dy wgv

J

Ikq

RB5S

X

z

f

L

/QU

9

M

m

o

K

O

u

8

+

Y

d

<latexi sh

1_b

64="

H0Xqko7

w

5

Yv rB

f

>A

c

V

NS8

EJ3

W

O

K

d

/

2

M

LuR

C

+p

j

z

G

Z

g

I

Q

n

P

yD

U

9m

T

F

<latexi sh

1_b

64="Pu zY

mXj7cy/v5RTUO

IHG0

>A

dVDLSgM

F

3

2

Z

k

wNW

E

o

Jq

n

QK

f

C

r

8

9

p

<latexi sh

1_b

64="3Q

Pn

U

X/9wG

S

oTHk

>A

C

c

VBN

8

E

r

q

70

y

p

KI

F

f

Rdu

W

+

j

O

z

M

2

g

J

Z

D

<latexi sh

1_b

64="/

B5vLDX

Tgywo9

W+m zVH

CJ0

>A

n

cd

S

N

EO2

q

f

8Mk

UY

K

Z

R

F

u

I

3P

Q

r

p

j

G

7

<latexi sh

1_b

64="3JgOn dGI

rKyA

UXwuZm

z

>

B8

c

VDLS

N

E

f

Y9

oP

Q

T

M

j

7

2

W

/

Hk

F

R

0

v

q

5

p

C

+

<latexi sh

1_b

64="CpMPGj9

YH2KFRQV

W

>A

B

c

NS8

E

3Ur

/o

57

0

u

vD

X

g

+m

d

Z

L

I

w

J

f

y

z

k

O

n

<latexi sh

1_b

64="U

qMSWgTY0L

V

wG

Om

A

NB

>

CH3

c

D

F

rP

d

k

I

K

o

2

p

Zj

8Qy9E/

+

uX

7v

5

z

J

f

R

n

<latexi sh

1_b

64="R8

JK

LF

H0n

D

TzdkPN

WC

>A

X

c

V

BEO2

Iv

Z

M

o

Yw yQ

p5

r

/+

j

7

3

f

u

m

9

q

G

U

<latexi sh

1_b

64="wM2

n

SX

Q8g pFZ5EVDL9

c

>A

B

N

Jy

r

fUY

oP

K

Rz

O

m

W

7Gd

CI

/

3

j

0

H

u

+

T

v

q

k

<latexi sh

1_b

64="I

qg8

X

WYjK

HuBE

9T

>A

c

VDLS

N

Oy

r

fU

oP

F

R

wm

5

dzCZ

M

p

/

G

3

v

7

+

0

n

J

2

Q

k

<latexi sh

1_b

64="B7yKnfgUj

HFV

Q

L

quS

pR9c

>A

N

8

E

3

W

X

O

Y

P

Z

d

v

0

+m

/

o

z

Mw

r

2

D

J

C

k

I

G

T

5

<latexi sh

1_b

64="

A7

rPoIdZ9

2T

OYwB

V

>

H

c

NS8

EJ3

W

X

UQ

R

K

g

u

F

y

v

L

0

+m

/

j

z

M

q

D

p

G

C

nf

k

5

<latexi sh

1_b

64="B

pwU2JRVA

d

n

+NGyIzL

Q

>

7X

c

S8

E

3

r

q/

9

vg

WMF

K

k0

PCj

f

Y

Z

m

u

O

T

5

D

<latexi sh

1_b

64="B

pwU2JRVA

d

n

+NGyIzL

Q

>

7X

c

S8

E

3

r

q/

9

vg

WMF

K

k0

PCj

f

Y

Z

m

u

O

T

5

D

<latexi sh

1_b

64="

Bd

E

+7uL

Q08fY

pv

y

F

>A

H

c

ZD

SgM

I

z9V

r

T

o

W

R

O

5

q

N

GJC

U

/

n

mX

K

P

3

k

w

j

2

<latexi sh

1_b

64="g0O

CA

P

rnyW

F

SQf zkM

>

B

H

c

V

N

8

EJ3

X

L9

UD

2

m7G

/

Iv

T

5

+

u

Z

d

Y

p

j

R

<latexi sh

1_b

64="UgW0KQ59q kyNFjX

o

7A

>

B+H

c

V

S8

E

3

Iv

L

2

pZ

J

/

PC

z

M

Y

R

f

d

G

D

w

O

m

n

u

T

r

<latexi sh

1_b

64="Kw7nS

I

ZmX0+r

H9

fC

M

8

>A

B

cdVDL

g

F

3U

2vqk

GjJ

W

p

5

o

P

T

z

u

Q

y

N

O

Y

E

/

R

<latexi sh

1_b

64="JgXjpE

v

8fmWDT57R

YLd

B

>A

/n

c

Z

S

N

FI

9V

r

qu3

w

U

2

G

0

o

M

O

C

+

P

z

K

y

k

H

Q

<latexi sh

1_b

64="nWEG

r

3p2HZ0P7

C

F

jc

>A

V

T

N

Dy wgv

J

Ikq

RB5S

X

z

f

L

/QU

9

M

m

o

K

O

u

8

+

Y

d

<latexi sh

1_b

64="

H0Xqko7

w

5

Yv rB

f

>A

c

V

NS8

EJ3

W

O

K

d

/

2

M

LuR

C

+p

j

z

G

Z

g

I

Q

n

P

yD

U

9m

T

F

InitNode

α

EdgeEmbed

α

Update

α

h

1

.

.

.

D

BFS

m

∈ M

o

m,x

i

m

∈ M

(a) Overview

pos[1]

pos[

u

]

h

u

.

.

.

D

GNN

α

g

β

π

or

π

ψ

S

Attention

Layer

π

or

π

ψ

S

GNN

α

GNN

α

GNN

FF

W

F

x

i

LayerNorm

ReLU

LayerNorm

g

β

(b) Neural architecture of different components

we may consider the following optimization problem.

where b is the budget, DIVERSITY ( { x i | i ∈ S } ) measures the representativeness of S with respect to the whole dataset D and λ is a regularizing coefficient. One can use submodular functions [11, 17] like Facility Location, Graph Cut, or Log-Determinant to model DIVERSITY ( { x i | i ∈ S } ) . Here, λ trades off between training loss and diversity.

Bottlenecks of the combinatorial optimization (1) . For every new architecture m , one needs to solve a fresh version of the optimization (1) problem from scratch to find S . Therefore, this is not generalizable across architectures. Moreover, the involvement of both combinatorial and continuous optimization variables, prevents the underlying solver from scaling across multiple architectures.

We address these challenges by designing a neural surrogate of the objective (1), which would lead to the generalization of subset selection across different architectures.

3 Overview of SUBSELNET

Here, we give an outline of our proposed model SUBSELNET that leads to substituting the optimization (1) with its neural surrogate, which would enable us to compute the optimal subset S for an unseen model, once trained on a set of model architectures.

3.1 Components

At the outset, SUBSELNET consists of three key components: (i) the architecture encoder, (ii) the neural approximator of the trained model, and (iii) the subset sampler. Figure 1 illustrates our model.

GNN-guided encoder for neural architectures. Generalizing any task across the different architectures requires the architectures to be embedded in vector space. Since a neural architecture is essentially a graph between multiple operations, we use a graph neural network (GNN) [59] to achieve this goal. Given a model architecture m ∈ M , we first feed the underlying DAG G m into a GNN ( GNN α ) with parameters α , which outputs the node representations for G m , i.e. , H m = { h u } u ∈ V m .

Subset sampler. We design a subset sampler using a probabilistic model Pr π ( · ) . Given a budget b , it sequentially draws instances S = { s 1 , ..., s b } from a softmax distribution of the logit vector π ∈ R | D | where π ( x i , y i ) indicates a score for the element ( x i , y i ) . We would like to highlight that we use S as an ordered set of elements, selected in a sequential manner. However, such an order does not affect the trained model, which is inherently invariant of permutations of the training data; it only affects the choice of S . Now, depending on how we compute π during test , we have two variants of SUBSELNET: Transductive-SUBSELNET and Inductive-SUBSELNET.

Approximator of the trained model m θ ∗ . To tackle lack of generalizability of the optimization (1), we design a neural model approximator g β which approximates the predictions of any trained model for any given architecture m . To this end, g β takes input as H m and the instance x i and compute g β ( H m , x i ) ≈ m θ ∗ ( x i ) . Here, θ ∗ is the set of learned parameters of the model m θ on dataset D .

TransductiveSUBSELNET : During test, since we have already trained the architecture encoder GNN α and the model approximator g β , we do not have to perform any training when we select a subset for an unseen architecture m ′ , since the trained model can then be replaced with g β ( H m ′ , x i ) . Thus, the key bottleneck of solving the combinatorial optimization (1)- training the model simultaneously with exploring for S -is ameliorated. Now, we can perform optimization over π , each time for a new architecture. However, since no model training is involved, such explicit optimization is fast enough and memory efficient. Due to explicit optimization every time for an unseen architecture, this approach is transductive in nature.

InductiveSUBSELNET : Here, we introduce a neural network to approximate π , which is trained together with GNN α and g β . This allows us to directly select the subset S without explicitly optimizing for π , unlike Transductive-SUBSELNET.

3.2 Training and inference

Training objective. Using the approximation g β ( H m , x i ) ≈ m θ ∗ ( x i ) , we replace the combinatorial optimization problem in Eq. (1) with a continuous optimization problem, across different model architectures m ∈ M . To that goal, we define

Λ( S ; m ; π, g β , GNN α ) = ∑ i ∈ S ℓ ( g β ( H m , x i ) , y i ) -λH (Pr π ( · )) with, H m = GNN α ( G m ) (2) and seek to solve the following problem:

Here, we use entropy on the subset sampler H ( Pr π ( · )) to model the diversity of samples in the selected subset. We call our neural pipeline, which consists of architecture encoder GNN α , the model approximator g β , and the subset selector π , as SUBSELNET. In the above, γ penalizes the difference between the output of the model approximator and the prediction made by the trained model, which allows us to generalize the training of different models m ∈ M through the model g β ( H m , x i ) .

4 Design of SUBSELNET

Bottlenecks of end-to-end training and proposed multi-stage approach. End-to-end optimization of the above problem is difficult for the following reasons. (i) Our architecture representation H m only represents the architectures and thus should be independent of the parameters of the architecture θ and the instances x . End-to-end training can make them sensitive to these quantities. (ii) To enable the model approximator g β accurately fit the output of the trained model m θ , we need explicit training for β with the target m θ .

In our multi-stage training method, we first train the architecture encoder GNN α , then the model approximator g β and then train our subset sampler Pr π (resp. Pr π ψ ) for the transductive (inductive) model. In the following, we describe the design and training of these components in details.

4.1 Design of architecture encoder using graph neural network

Architectures can be represented as directed acyclic graphs with forward message passing. During forward computation, at any layer for node v , the output a ( v ) can be represented as a ( v ) = Act ( ∑ u ∈ InNbr ( v ) Op v ( a ( u )) ) with the root output as the input. Here, Act is the activation function and Op · are operation on a node of the network. Given a GNN has a similar computation process, the permutation-equivariant node representations generated are good representations of the operations within the architecture. This allows further coupling with transformer-based architectures since they are universal approximators of permutation equivariant functions [63].

Neural parameterization. Given a model m ∈ M , we compute the representations H m = { h u | u ∈ V m } by using a graph neural network GNN α parameterized with α , following the proposal of Yan et al. [59]. We first compute the feature vector f u for each node u ∈ V m using the one-hot encoding of the associated operation ( e.g. , max , sum ) and then feeding it into a neural network to compute an initial node representation h u [0] = INITNODE α ( f u ) . Then, we use a message-passing network, which collects signals from the neighborhood of different nodes and recursively computes the node representations [59, 58, 12]. Given a maximum number of recursive layers K and the node

u , we compute the node embeddings H m = { h u | u ∈ V m } by gathering information from the k < K hops using K recursive layers as follows.

Here, Nbr ( u ) is the set of neighbors of u . EDGEEMB is injective mappings, as used in [58]. Note that trainable parameters from EDGEEMB and UPDATE are decoupled. They are represented as the set of parameters α . Finally, we obtain our node representations as h u = [ h u [0] , .., h u [ K -1]] .

Parameter estimation. We perform unsupervised training of GNN α using a variational graph autoencoder (VGAE). This ensures that the architecture representations H m remain insensitive to the model parameters. We build the encoder and decoder of our GVAE by following existing works on graph VAEs [59, 46]. Given a graph G m , the encoder q ( Z m | G m ) , which takes the node embeddings { h u } u ∈ V m and maps it into the latent space Z m = { z u } u ∈ V m . Specifically, we model the encoder q ( Z m | G m ) as: q ( z u | G m ) = N ( µ ( h u ) , Σ( h u )) . Here, both µ and Σ are neural networks. Given a latent representation Z m = { z u } u ∈ V m , the decoder models a generative distribution of the graph G m where the presence of an edge is modeled as Bernoulli distribution BERNOULLI ( σ ( z ⊤ u z v )) . Thus, we model the decoder as p ( G m | Z ) = ∏ ( u,v ) ∈ E m σ ( z ⊤ u z v ) · ∏ ( u,v ) ̸∈ E m [1 -σ ( z ⊤ u z v )] . Here, σ is a parameterized sigmoid function. Finally, we estimate α, µ, Σ , and σ by maximizing the evidence lower bound (ELBO): max α,µ, Σ ,σ E Z∼ q ( · | G m ) [ p ( G m | Z )] -KL ( q ( · | G m ) || p pr ( · )) .

4.2 Design of model approximator

Neural parameterization. Having computed the architecture representation H m = { h u | u ∈ V m } , we next design the model approximator, which leverages these embeddings to predict the output of the trained model m θ ∗ ( x i ) . To this aim, we developed a model approximator g β parameterized by β that takes H m and x i as input and attempts to predict m θ ∗ ( x i ) , i.e. , g β ( H m , x i ) ≈ m θ ∗ ( x i ) . It consists of three steps. In the first step, we generate an order on the nodes. Next, we feed the representations { h u } in this order into a self-attention-based transformer layer. Finally, we combine the output of the transformer and x i using a feedforward network to approximate the model output.

Node ordering using BFS order. We first sort the nodes using breadth-first-search (BFS) order ρ . Similar to You et al. [62], this sorting method produces a sequence of nodes and captures subtleties like skip connections in the network structure G m .

Attention layer. Given the BFS order ρ , we pass the representations H m = { h u | u ∈ V m } in the sequence ρ through a self-attention-based transformer network. Here, the Query , Key , and Value functions are realized by linear networks on h · . We compute an attention-weighted vector ζ u as:

Here k is the dimension of the latent space, and the softmax operation is over the node v . Subsequently, for each node u , we use a feedforward network, preceded and succeeded by layer normalization operations to obtain an intermediate representation ζ u for each node u . We present additional details in Appendix D. Finally, we feed ζ u for the last node u in the sequence ρ , i.e. , u = ρ ( | V m | ) , along with the feature x i into a feedforward network parameterized by W F to model the prediction m θ ∗ ( x i ) . Thus, the final output of g β ( H m , x i ) is

Here, W · , parameters of Query, Key and Value and layer normalizations form β .

Parameter estimation. We train our model approximator g β by minimizing the KL-Divergence between the approximated prediction g β ( H m , x i ) and the ground truth prediction m θ ∗ ( x i ) , where both these quantities are probabilities across different classes. The training problem is as follows:

minimize β ∑ i ∈ D,m ∈M KL ( m θ ∗ ( x i ) || g β ( H m , x i )) (7) Generalization across architectures but not instances. Note that the goal of the model approximator is to predict the output on x in the training set D tr for unseen architecture m ′ so that using these predictions, our method can select the subset S from D tr in a way that m ′ trained on S shows high accuracy on D test. Since the underlying subset S has to be chosen from the training set D tr for an arbitrary architecture m ′ , it is enough for the model approximator to mimic the model output only on the training set D tr-it need not have to perform well in the test set D test .

4.3 Subset sampler and design of transductive and inductive SUBSELNET

Subset sampler. We draw S , an ordered set of elements, using π as follows. Having chosen the first t instances S t = { s 1 , ..s t } from D with S 0 = ∅ , it draws the ( t +1) -th element ( x , y ) from the remaining instances in D with a probability proportional to exp( π ( x , y )) and then repeat it for b times. Thus, the probability of selecting the ordered set of elements S = { s 1 , ..., s b } is given by

The optimization (3) suggests that once GNN α and g β are trained, we can use them to approximate the output of the trained model m θ ∗ for an unseen architecture m ′ and use it to compute π . Thus, this already removes a significant overhead of model training and facilitates fast computation of π , and further leads us to develop Transductive-SUBSELNET and Inductive-SUBSELNET based on how we can compute π , as described at the end of Section 3.1.

Transductive-SUBSELNET. The first variant of the model is transductive in terms of the computation of π . Once we train the architecture encoder and the model approximator, we compute π by solving the optimization problem explicitly with respect to π every time when we wish to select a data subset for a new architecture. Given trained model GNN ˆ α , g ˆ β and a new architecture m ′ ∈ M , we solve the optimization problem to find the subset sampler Pr π during inference time for a new architecture m ′ .

Such an optimization still consumes time during inference. However, it is still significantly faster than the combinatorial methods [20, 19, 37, 47] thanks to sidestepping the explicit model training using a model approximator.

Inductive-SUBSELNET. In contrast to the transductive model, the inductive model does not require explicit optimization of π in the face of a new architecture. To that aim, we approximate π using a neural network π ψ which takes two signals as inputs- the dataset D and the outputs of the model approximator for different instances { g ˆ β ( H m , x i ) | i ∈ D } and finally outputs a score for each instance π ψ ( x i , y i ) . Here, the training of π ψ follows from the optimization (3):

min ψ ∑ m ∈M E S ∼ Pr π ψ Λ( S ; m ; π ψ , g ˆ β , GNN ˆ α ) (10) Such an inductive model can select an optimal distribution of the subset that should be used to efficiently train any model m θ , without explicitly training θ or searching for the underlying subset.

Architecture of π ψ for Inductive-SUBSELNET. We approximate π using π ψ using a neural network which takes three inputs -( x j , y j ) , the corresponding output of the model approximator, i.e. , o m, x j = g β (GNN α ( G m ) , x j ) from Eq. (6) and the node representation matrix H m and provides us a positive selection score π ψ ( H m , x j , y j , o m, x j ) . In practice, π ψ is a three-layer feed-forward network containing Leaky-ReLU activation functions for the first two layers and sigmoid activation at the last layer.

4.4 Training and inference routines

Training. The training phase for both transductive and inductive variants, first utilizes the TRAINPIPELINE(Algorithm 1) routine to train the GNN (TRAINGNN), re-order the embeddings based on BFS ordering (BFS), train the model approximator (TRAINAPPROX), to obtain ˆ β . TRAINTRANSDUCTIVE(Algorithm 2) routine doesn't require any further training, while the TRAININDUCTIVE(Algorithm 3) routine uses the TRAINPI to train ψ for computing π .

Inference. Given a new architecture m ′ , our goal is to select a subset S , with | S | = b which would facilitate efficient training of m ′ . Given trained SUBSELNET, we compute H m ′ = GNN ˆ α ( G m ′ ) , compute the model approximator output g ˆ β ( H m ′ ) . Using them we compute π for

Algorithm 1 Training Pipeline

1: function TRAINPIPELINE ( D, M , { θ ∗ } )

2: ˆ α ← TRAINGNN ( M )

3: for m ∈ M tr do

4: H m ← GNN ˆ α ( m ) , pos ← BFS ( G m , H m )

5: ˆ β ← TRAINAPPROX ( H m , { x i } , pos , { θ ∗ } )

6: return ˆ α, ˆ β, H m

Algorithm 2 Transductive Procedure

1: function TRAINTRANSDUCTIVE ( D, M , { θ ∗ } )

2: ˆ α, ˆ β, H m ← TRAINPIPELINE ( D, M , { θ ∗ } )

1: function INFERTRANSDUCTIVE ( D, ˆ α, ˆ β, m ′ )

2: π ∗ ← min π E S ∈ Pr π ( · ) Λ( S ; m ′ ; π, g ˆ β , GNN ˆ α )

4: TRAINNEWMODEL ( m ′ ; S ∗ )

3: S ∗ ∼ Pr π ∗ ( · )

Transductive-SUBSELNET by explicitly solving the optimization problem stated in Eq. 9 and draw S ∼ Pr π ( · ) . For the inductive variant, we draw S ∼ Pr π ˆ ψ ( · ) where ˆ ψ is the learned value of ψ .

Given an unseen architecture m ′ and trained parameters of SUBSELNET, i.e. , ˆ α, ˆ β and ˆ ψ , the INFERTRANSDUCTIVE(Algorithm 2) routine solves the optimization problem on π explicitly to compute π , where Λ( · ) is defined in Eq. (2).

Algorithm 3 Inductive Procedure

1: function TRAININDUCTIVE ( D, M , { θ ∗ } )

2: ˆ α, ˆ β, H m ← TRAINPIPELINE ( D, M , { θ ∗ } )

3: o ← [ g ˆ β ( { H m , x i } )] i,m

4: ˆ ψ ← TRAINPI ( o , { H m } , { x i } )

1: function INFERINDUCTIVE ( D, ˆ α, ˆ β, ˆ ψ, m ′ )

2: Compute π ˆ ψ ( H m ′ , x i , y i , o m ′ , x i ) ∀ i ∈ D

4: TRAINNEWMODEL ( m ′ ; S ∗ )

3: S ∗ ∼ Pr π ˆ ψ ( · )

INFERINDUCTIVE (Algorithm 3) utilizes ˆ ψ , i.e. , trained parameters from the subset sampler to compute π ˆ ψ . Then the subset S ∗ is drawn from π or π ψ and is used to train m ′ using TRAINNEWMODEL.

5 Experiments

In this section, we provide comprehensive evaluation of SUBSELNET against several strong baselines on five real world datasets. In Appendix E, we present additional results. Our code is in https: //github.com/structlearning/subselnet .

5.1 Experimental setup

Datasets. We use FMNIST [56], CIFAR10 [26], CIFAR100 [25], Tiny-Imagenet-200 [27] and Caltech-256 [13] (Cal-256). Cal-256 has imbalanced class distribution; the rest are balanced. We transform an input image X i to a vector x i of dimension 2048 by feeding it to a pre-trained ResNet50 v1.5 model [16] and use the output from the penultimate layer as the image representation.

Model architectures and baselines. We use model architectures from NAS-Bench-101 [61] in our experiments. We compare Transductive-SUBSELNET and Inductive-SUBSELNET against three non-adaptive subset selection methods - (i) Facility location [11, 17] where we maximize FL ( S ) = ∑ j ∈ D max i ∈ S x ⊤ i x j to find S , (ii) Pruning [48], and (iii) Selection-via-Proxy [5] and four adaptive subset selection methods - (iii) Glister [20], (iv) Grad-Match [19], (v) EL2N [42] and (vi) GraNd [42]. The non-adaptive subset selectors select the subset before the training begins and thus, never access the rest of the training set again during the training iterations. On the other hand, the adaptive subset selectors refine the choice of subset during training iterations and thus they need to access the full training set at each training iteration. Appendix D contains additional details about the baselines and Appendix E contains experiments with more baselines.

Evaluation protocol. We split the model architectures M into 70% training ( M tr), 10% validation ( M val) and 20% test ( M test) folds. However, training model approximator requires supervision from the pre-trained models m θ ∗ . Pre-training large number of models can be expensive. Therefore, we limit the number of pre-trained models to a diverse set of size 250, that ensures efficient representation over low-parameter and high-parameter regimes, and using more than this showed no visible advantage. We show the parameter statistics in Appendix D. However, for the architecture encoder, we use the entire set M tr for GNN training. We split the dataset D into D tr , D val and D test in the similar 70:10:20 folds. We present M tr , M val , D tr and D val to our method and estimate ̂ α, ̂ β and ̂ ψ (for Inductive-SUBSELNET model). None of the baseline methods supports any generalizable learning protocol for different architectures and thus cannot leverage the training architectures during test. Given an architecture m ′ ∈ M test, we select the subset S from D tr using our subset sampler ( Pr π for Transductive-SUBSELNET or Pr π ̂ ψ for Inductive-SUBSELNET). Similarly, all the non-adaptive subset selectors select S ⊂ D tr using their own algorithms. Once S is selected, we train the test models m ′ ∈ M test on S . We perform our experiments with different | S | = b ∈ (0 . 005 | D | , 0 . 9 | D | ) and compare the performance between different methods using three quantities: (1) Relative Accuracy Reduction ( RAR ) computed as the drop in test accuracy on training with a chosen subset as compared to training with the entire dataset, i.e, RAR ( S, D ) = 1 |M test | ∑ m ′ ∈M test (1 -Acc ( m ′ | S ) / Acc ( m ′ | D )) where Acc ( m ′ | X ) denotes the test accuracy when m ′ is trained on the set X . Lower RAR indicates better performance. (2) Computational efficiency, i.e. , the speedup achieved with respect to training with full dataset. It is measured with respect to T f /T . Here, T f is the time taken for training with full dataset; and, T is the time taken for the entire inference task, which is the average time for selecting

Budget (%)

10 3

10 3

memory(

∫

T

0

0

.

0

0

.

1

FacLoc

Proxy

Pruning

Transductive-SubSelNet

Inductive-SubSelNet

10 0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

RAR

→

10 0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 1

10 2

10 3

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

RAR

→

(a) FMNIST

10 1

10 2

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(b) CIFAR10

10 1

10 2

10 3

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(c) CIFAR100

10 3

10 4

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(d) TINY-IN

10 3

10 4

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(e) CAL-256

10 4

→

10 4

t

)

10 3

Budget (%)

dt

→

10 3

memory(

∫

T

0

0

.

0

0

.

1

GLISTER

GRAD-MATCH

GraNd

EL2N

Transductive-SubSelNet

Inductive-SubSelNet

10 0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

RAR

→

10 0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 1

10 2

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

RAR

→

(a) FMNIST

10 1

10 2

10 3

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(b) CIFAR10

10 1

10 2

10 3

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(c) CIFAR100

10 3

10 4

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(d) TINY-IN

10 3

10 4

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(e) CAL-256

subsets across the test models m ′ ∈ M test plus the average training time of these test models on the respective selected subsets. (3) Resource efficiency in terms of the amount of memory consumed during the entire inference task, described in item (2), which is measured as ∫ T 0 memory ( t ) dt where memory ( t ) is amount of memory consumed at timestamp t in the unit of GB-min.

5.2 Results

Comparison with baselines. Here, we compare different methods in terms of the trade-off between Relative accuracy reduction RAR (lower is better) and computational efficiency as well as RAR and resource efficiency. In Figures 2 and 3, we probe the variation between these quantities by varying the size of the selected subset | S | = b ∈ (0 . 005 | D | , 0 . 9 | D | ) for non-adaptive and adaptive baselines, respectively. We make the following observations. (1) Our methods trade-off between accuracy vs . computational efficiency as well as accuracy vs. resource efficiency more effectively than all the methods, including the adaptive methods which refine their choice of subset as the model training progresses. (2) In FMNIST, our method achieves 10% RAR at ∼ 4.4 times the speed-up and using 77% lesser memory than EL2N, the best baseline (Table 1, tables for other datasets are in Appendix E). (3) There is no consistent winner across baselines. However, Glister and Grad-Match mostly remain among top three baselines, across different methods. In particular, they outperform others in Tiny-Imagenet and Cal-256, in high accuracy (low RAR) regime.

→

10 4

10 4

→

t

)

dt

0

.

1

B = 25000

B = 30000

B = 35000

B = 45000

Transductive (B = 50000)

Inductive (B = NA)

100

200

Speed up (

0

.

6

0

.

7

0

.

8

0

.

9

RAR

→

0

10

20

30

memory(

t

)

dt

0

.

7

0

.

8

0

.

9

RAR

→

→

Hybrid-SUBSELNET. In FMINST, CIFAR10 and CIFAR100, we observe that TransductiveSUBSELNET offers better traded off than Inductive-SUBSELNET. Here, we design a hybrid version of our model, called as Hybrid-SUBSELNET and evaluate it on a regime where the gap between transductive and inductive SUBSELNET is significant. One of such regimes is the part of the trade-off plot in CIFAR100, where the speed up T f /T ≥ 28 . 09 (Figures 2 and 3). Here, given the budget of the subset b , we first choose B > b instances using Inductive-SUBSELNET and the final b instances by running the explicit optimization routines in Transductive-SUBSELNET. Figure 4 shows the results for B = { 25 K, 30 K, 35 K, 45 K, 50 K } . We observe that Hybrid-SUBSELNET allow us to smoothly trade off between Inductive-SUBSELNET and Transductive-SUBSELNET, by tuning B . It allows us to effectively use resource-constrained setup with limited GPU memory, wherein the larger subset B can be selected using Inductive-SUBSELNET on a CPU, and the smaller refined subset b can then be selected by solving transductive variant on GPU.

Ablation study. Here, we experiment with three candidates of model approximator g β ( Feedforward, LSTM and our proposed attention based approximator) with three different subset samplers π (uncertainty based, loss based and our proposed subset sampler). Thus, we have nine different combinations of model approximator and subset selection strategies. In the uncertainty and loss based subset samplers, we take topb instances based on the uncertainty and loss. We measure uncertainty using the entropy of the predicted distribution of the target classes. We compare the performance in terms of the test RAR of the test architectures. Moreover, we also evaluate the model approximator g β alone - without the presence of the subset sampler - using KL divergence between the gold model outputs and predicted model outputs on the training instances 1 | D tr ||M test | ∑ i ∈ D tr ,m ∈M test KL ( m θ ∗ ( x i ) || g β ( H m , x i )) . Table 3 summarizes the results for 3%, 5% and 10% subsets for CIFAR10. We make the following observations: (1) The complete design of our method, i.e., Our model approximator (Transformer) + Our subset sampler (SUBSELNET) performs best in terms of RAR. (2) Our neural-network for model approximator mimics the trained model output better than LSTM and Feedforward architectures.

Can model approximator substitute our subset selector pipeline? The task of the model approximator g β is to predict accuracy for unseen architecture. Then, a natural question is that is it possible to use the model approximator to directly predict accuracy of the unseen architecture m ′ , instead of using such long pipeline to select subset S followed with training on S . However, as discussed in the end of Section 4.2, the model approximator g β is required to generalize across unseen architectures but not the unseen instances, as its task is to help select the training subset. Table 3

already showed that g β closely mimics the output of the trained model for the unseen architecture m ′ ∈ M test and on training instances x (KL div. column). Here, we investigate the performance of g β on the test instances and test architectures.

|

Table 2 shows that the performance of g β on the test instances is significantly poorer than our method. This is intuitive as generalizing both the model space and the instance space is extremely challenging, and we also do not need it in general.

Using SUBSELNET in AutoML. AutoML-related tasks can be significantly sped-up when we replace the entire dataset with a representative subset. Here, we apply SUBSELNET to two AutoML applications: Neural Architecture Search (NAS) and Hyperparameter Optimization (HPO). Neural Architecture Search: We apply our method on DARTS architecture space to search for an architecture using subsets. During this search process, at each iteration, the underlying network is traditionally trained on the entire dataset. In contrast, we train this underlying network on the subset returned by our method for this architecture. Following Na et al. [39], we report test misclassification

10

20

error of the architecture which is selected by the corresponding subset selector guided NAS methods, i.e. , our method (transductive), random subset selection (averaged over 5 runs) and proxy-data [39]. Table 4 shows that our method performs better than the baselines.

Hyperparameter Optimization: Finding the best set of hyperparameters from their search space for a model is computationally intensive. We look at speeding-up the tuning process by searching the hyperparameters while training the model on a small representative subset S instead of D . Following Killamsetty et al. [22], we consider optimizer and scheduler specific hyperparameters and report average test misclassification error across the models trained on optimal hyperparameter choice returned by our method (transductive), random subset selection (averaged over 5 runs) and AUTOMATA [22]. Table 5 shows that we are outperforming the baselines in terms of accuracyspeedup tradeoff. Appendix D contains more details about the implementation.

Amortization Analysis. Figures 2 and 3 show that our method is substantially faster than the baselines during inference, once we have our neural pipeline trained. Such inference time speedup is the focus of many other applications, E.g., complex models like LLMs are difficult and computationally intensive, but their inference is fast for several queries once trained. However, we recognize that there is a computational overhead in training our model, arising due to the pre-training of the models m θ ∗ used for supervision. Since the prior training is only a one-time overhead, the overall cost is amortized by querying multiple architectures for their subsets. We measure amortized cost T total /M total (time in seconds), where T total is the total time used from beginning of the pipeline to end of reporting final accuracy on the test architectures and M total is the total number of training and test architectures. Table 6, shows the results for 10% subset on the top baselines for CIFAR10, which shows that the training overhead of our method (transductive) quickly diminishes with number of test architectures.

6 Conclusion

In this work, we develop SUBSELNET, a subset selection framework, which can be trained on a set of model architectures, to be able to predict a suitable training subset before training a model, for an unseen architecture. To do so, we first design a neural architecture encoder and model approximator, which predicts the output of a new candidate architecture without explicitly training it. We use that output to design transductive and inductive variants of our model.

Limitations. The SUBSELNET pipeline offers quick inference-time subset selection but a key limitation of our method is that it entails a pre-training overhead, although its overhead vanishes as we query more architectures. Such expensive training can be reduced by efficient training methods [65]. In future, it would be interesting to incorporate signals from different epochs with a sequence encoder to train a subset selector. Apart from this, our work does not assume the distribution shift of architectures from training to test. If the architectures vary significantly from training to test, then there is significant room for performance improvement.

Acknowledgements. Abir De acknowledges the Google Research gift funding.

References

Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architectures using reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=S1c2cvqee .

James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyperparameter optimization. In Proceedings of the 24th International Conference on Neural Information Processing Systems , NIPS'11, page 2546-2554, Red Hook, NY, USA, 2011. Curran Associates Inc. ISBN 9781618395993.

James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning , volume 28 of Proceedings of Machine Learning Research , pages 115-123, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL https://proceedings.mlr.press/v28/ bergstra13.html .

Christos Boutsidis, Petros Drineas, and Malik Magdon-Ismail. Near-optimal coresets for least-squares regression. IEEE transactions on information theory , 59(10):6880-6892, 2013.

Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv:1906.11829 , 2019.

Abir De, Paramita Koley, Niloy Ganguly, and Manuel Gomez-Rodriguez. Regression under human assistance. AAAI , 2020.

Abir De, Nastaran Okati, Ali Zarezade, and Manuel Gomez-Rodriguez. Classification under human assistance. AAAI , 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009. doi: 10.1109/CVPR.2009.5206848.

Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search, 2020. URL https://arxiv.org/abs/2001.00326 .

Dan Feldman. Core-sets: Updated survey. In Sampling Techniques for Supervised or Unsupervised Tasks , pages 23-44. Springer, 2020.

Satoru Fujishige. Submodular functions and optimization . Elsevier, 2005.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.

Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In Database and Expert Systems Applications: 33rd International Conference, DEXA 2022, Vienna, Austria, August 22-24, 2022, Proceedings, Part I , pages 181-195. Springer, 2022.

Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing , pages 291-300, 2004.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR , abs/1512.03385, 2015. URL http://arxiv.org/abs/1512.03385 .

Rishabh Krishnan Iyer. Submodular optimization and machine learning: Theoretical results, unifying and scalable algorithms, and applications . PhD thesis, 2015.

Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: A unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 1289-1299. IEEE, 2019.

Krishnateja Killamsetty, Durga Sivasubramanian, Baharan Mirzasoleiman, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: A gradient matching based data subset selection for efficient learning. arXiv preprint arXiv:2103.00123 , 2021.

Krishnateja Killamsetty, Durga Subramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: A generalization based data selection framework for efficient and robust learning. In AAAI , 2021.

Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust semi-supervised learning. Advances in Neural Information Processing Systems , 34:14488-14501, 2021.

Krishnateja Killamsetty, Guttu Sai Abhishek, Aakriti, Alexandre V. Evfimievski, Lucian Popa, Ganesh Ramakrishnan, and Rishabh Iyer. Automata: Gradient based data subset selection for compute-efficient hyper-parameter tuning, 2022.

Katrin Kirchhoff and Jeff Bilmes. Submodularity for data selection in machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 131-141, 2014.

Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. In online: https://www.cs.toronto.edu/ ∼ kriz/cifar.html , 2014.

Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N , 7(7):3, 2015.

Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. A system for massively parallel hyperparameter tuning, 2020.

Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, and Ion Stoica. Tune: A research platform for distributed model selection and training, 2018.

Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search, 2017. URL https://arxiv.org/abs/1712.00559 .

Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, 2018. URL https://arxiv.org/abs/1806.09055 .

Yuzong Liu, Rishabh Iyer, Katrin Kirchhoff, and Jeff Bilmes. Svitchboard ii and fisver i: High-quality limited-complexity corpora of conversational english speech. In Sixteenth Annual Conference of the International Speech Communication Association , 2015.

Mario Lucic, Matthew Faulkner, Andreas Krause, and Dan Feldman. Training gaussian mixture models at scale via coresets. The Journal of Machine Learning Research , 18(1):5885-5909, 2017.

Jovita Lukasik, David Friede, Arber Zela, Frank Hutter, and Margret Keuper. Smooth variational graph embeddings for efficient neural architecture search. In International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021 , 2021.

Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. Neural architecture optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/2018/file/ 933670f1ac8ba969f32989c312faba75-Paper.pdf .

Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 15630-15649. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/mindermann22a.html .

Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In Proc. ICML , 2020.

Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence , AAAI'19/IAAI'19/EAAI'19. AAAI Press, 2019. ISBN 978-1-57735-809-1. doi: 10.1609/aaai.v33i01.33014602. URL https://doi.org/10.1609/aaai.v33i01.33014602 .

Byunggook Na, Jisoo Mok, Hyeokjun Choe, and Sungroh Yoon. Accelerating neural architecture search via proxy data. CoRR , abs/2106.04784, 2021. URL https://arxiv.org/abs/2106. 04784 .

Xuefei Ning, Changcheng Tang, Wenshuo Li, Songyi Yang, Tianchen Zhao, Niansong Zhang, Tianyi Lu, Shuang Liang, Huazhong Yang, and Yu Wang. Awnas: A modularized and extensible nas framework, 2020.

Hoang NT and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters, 2019. URL https://arxiv.org/abs/1905.09550 .

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems , 34:20596-20607, 2021.

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 4095-4104. PMLR, 10-15 Jul 2018. URL https://proceedings. mlr.press/v80/pham18a.html .

Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alex Kurakin. Large-scale evolution of image classifiers, 2017. URL https://arxiv.org/abs/1703.01041 .

Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search, 2018. URL https://arxiv.org/abs/1802.01548 .

Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. CoRR , abs/1802.03480, 2018. URL http://arxiv.org/abs/ 1802.03480 .

Durga Sivasubramanian, Rishabh K. Iyer, Ganesh Ramakrishnan, and Abir De. Training data subset selection for regression with controlled generalization error. In ICML , 2021.

Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486 , 2022.

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. Mnasnet: Platform-aware neural architecture search for mobile. 2018. doi: 10.48550/ARXIV.1807.11626. URL https://arxiv.org/abs/1807.11626 .

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159 , 2018.

Kai Wei, Rishabh Iyer, and Jeff Bilmes. Fast multi-stage submodular maximization. In International conference on machine learning , pages 1494-1502. PMLR, 2014.

Kai Wei, Yuzong Liu, Katrin Kirchhoff, Chris Bartels, and Jeff Bilmes. Submodular subset selection for large-scale speech training data. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 3311-3315. IEEE, 2014.

Kai Wei, Yuzong Liu, Katrin Kirchhoff, and Jeff Bilmes. Unsupervised submodular subset selection for speech data. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4107-4111. IEEE, 2014.

Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning , pages 1954-1963, 2015.

Colin White, Willie Neiswanger, Sam Nolen, and Yash Savani. A study on encodings for neural architecture search. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 20309-20319. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/ file/ea4eb49329550caaa1d2044105223721-Paper.pdf .

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks?, 2018. URL https://arxiv.org/abs/1810.00826 .

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? In NeurIPS , 2020.

Shen Yan, Kaiqiang Song, Fei Liu, and Mi Zhang. Cate: Computation-aware neural architecture encoding with transformers. In ICML , 2021.

Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS-bench-101: Towards reproducible neural architecture search. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 7105-7114. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/ying19a.html .

Jiaxuan You, Rex Ying, Xiang Ren, William L. Hamilton, and Jure Leskovec. Graphrnn: A deep generative model for graphs. CoRR , abs/1802.08773, 2018. URL http://arxiv.org/ abs/1802.08773 .

Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions?, 2020.

Muhan Zhang, Shali Jiang, Zhicheng Cui, Roman Garnett, and Yixin Chen. D-vae: A variational autoencoder for directed acyclic graphs. pages 1586-1598, 2019.

Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curriculum learning by optimizing learning dynamics. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 433-441. PMLR, 13-15 Apr 2021. URL https://proceedings.mlr.press/v130/zhou21a.html .

Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curriculum learning by optimizing learning dynamics. In International Conference on Artificial Intelligence and Statistics , pages 433-441. PMLR, 2021.

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning, 2016. URL https://arxiv.org/abs/1611.01578 .

Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition, 2017. URL https://arxiv.org/abs/1707. 07012 .

Efficient Data Subset Selection to Generalize Training Across Models: Transductive and Inductive Networks (Appendix)

A Limitations

While our work outperforms several existing subset selection methods, it suffers from three key limitations.

(1) We acknowledge that there indeed is a computation time for pre-training the model approximator. However, as we mentioned in amortization analysis in Section 5.2, the one-time overhead is offset quickly by the speed and effectiveness of the following selection pipeline for the subset selection of unseen architectures. Since the prior training is only a one-time overhead, the overall cost is amortized by the number of unseen architectures during inference time training. In practice, when it is used to predict the subset for a large number of unseen architectures, the effect of training overhead quickly vanishes. As demonstrated by our experiments (Figures 2 and 3, Table 6), once the pipeline of all the neural networks is set up, the selection procedure is remarkably fast and can be easily adapted for use with unseen architectures.

To give an analogy, premier search engines invest a lot of resources in making fast inferences rather than training. They build complex models that are difficult and computationally intensive, but their inference is fast for several queries once trained. Thus, the cost is amortized by the large number of queries. Another example is locality-sensitive hashing. Researchers design trainable models for LSH whose purpose is to make fast predictions. Training LSH models can take a lot of time, but again this cost is amortized by the number of unseen queries.

Finally, we would like to highlight many efficient model training methods without complete training (running a few epochs via curriculum learning [66]), which one can easily explore and plug with our method for a larger dataset like Imagenet-1K.

(2) We use the space of neural architectures which comprises only of CNNs. We did not experiment with sequence models such as RNNs or transformers. However, we believe that our work can be extended with RNNs or transformer based architectures.

(3) If the distribution of network architectures varies widely from training to test, then there is significant room for performance improvement. In this context, one can develop domain adaptation methods for graphs to tackle different out-of-distribution architectures more effectively.

B Broader Impact

Our work can be used to provide significant compute efficiency by the trainable subset selection method we propose. It can be used to save a lot of time and power, that ML model often demands. Specifically, it can be used in the following applications in the context of AutoML.

Fast tuning of hyperparameters related to optimizer/training. Consider the case where we need to tune non-network hyperparameters, such as learning rate, momentum, and weight decay. Given the architecture, we can choose the subset obtained using our method to train the underlying model parameters for different hyperparameters, which can then be used for cross-validation. Note that we would use the same subset in this problem since the underlying model architecture is fixed, and we obtain a subset for the given architecture independent of the underlying non-network hyperparameters. We have shown utility of our method in our experiments in Section 5.2.

Fast tuning of model related hyperparameters. Consider the case where we need to tune networkrelated hyperparameters, such as the number of layers, activation functions, and the width of intermediate layers. Instead of training each instance of these models on the entire data, we can train them on the subset of data obtained from our method to quickly obtain the trained model, which can then be used for cross-validation.

Network architecture search. As we shown in our experiments, our method can provide speedup in network architecture search. Here, instead of training the network the entire network during

architecture exploration, we can restrict the training on a subset of data, which can provide significant speedup.

Note that, the key goal of our method is design a trainable subset selection method that generalizes across architectures. As we observed in our experiments, these methods can be useful in the above applications- however, our method is a generic framework and not tailored to any one of the above applications. Therefore, our method may need application specific modifications before directly deploying it practice. However, our method can serve as a base model for the practitioner who intends to speed up for one of the above applications.

We do not foresee any negative social impact of our work.

C Additional discussion on related work

Our work is closely related to representation learning for model architectures, network architecture search, data subset selection.

Representation learning for model architectures. Recent work in network representation learning use GNN based encoder-decoder to encapsulate the local structural information of a neural network into a fixed-length latent space [64, 40, 59, 34]. By employing an asynchronous message passing scheme over the directed acyclic graph (DAG), GNN-based methods model the propagation of input data over the actual network structure. Apart from encodings based solely on the structure of the network, White et al. [55], Yan et al. [60] produce computation-aware encodings that map architectures with similar performance to the same region in the latent space. Following the work of Yan et al. [59], we use a graph isomorphism network as an encoder but instead of producing a single graph embedding, our method produces a collection of node embeddings, ordered by breadthfirst-search (BFS) ordering of the nodes. Our work also differs in that we do not employ network embeddings to perform downstream search strategies. Instead, architecture embeddings are used in training a novel model approximator that predicts the logits of a particular architecture, given an architecture embedding and a data embedding.

Machine learning on architecture space. We use NAS-Bench-101 in our method. This dataset was built in the context of network architecture search (NAS). The networks discovered by NAS methods often come from an underlying search space, usually designed to constrain the search space size. One such method is to use cell-based search spaces [35, 68, 30, 43, 61, 9]. Although we utilize the NAS-Bench-101 search space for architecture retrieval, our work is fundamentally different from NAS. In contrast to the NAS methods, which search for the best possible architecture from the search space using either sampling or gradient-descent based methods [1, 67, 44, 45, 31, 49], our work focuses on efficient data subset selection given a dataset and an architecture, which is sampled from a search space. Our work utilizes graph representation learning on the architectures sampled from the mentioned search spaces to project an architecture under consideration to a continuous latent space, utilize the model expression from the latent space as proxies for the actual model and proceed with data subset selection using the generated embedding, model proxy and given dataset.

Data subset selection. Data subset selection is widely used in literature for efficient learning, coreset selection, human centric learning, etc . Several works cast the efficient data subset selection task as instance of submodular or approximate-submodular optimization problem [19, 51-53, 20, 47]. Another line of work focus on selecting coresets which are expressed as the weighted combination of subset of data, approximating some characteristics, e.g. , loss function, model prediction [10, 37, 15, 4, 33]. Among other works, Toneva et al. [50] showed coresets can be selected by omitted several instances based on the forgetting dynamics at the time of training. Na et al. [39] selects proxy data based on entropy of the model. Coleman et al. [5] uses proxy model and then use it for coreset selection. Guo et al. [14] develop a library on coreset selection.

Our work is closely connected to simultaneous model learning and subset selection [7, 6, 47]. These existing works focus on jointly optimizing the training loss, with respect to the subset of instances and the parameters of the underlying model. Among them [7, 6] focus on distributing decisions between human and machines, whereas [47] aims for efficient learning. However, these methods adopt a combinatorial approach for selecting subsets and consequently, they are not generalizable across architectures. In contrast, our work focuses on differentiable subset selection mechanism, which can generalize across architectures.

D Additional details about experimental setup

D.1 Dataset

Datasets ( D ) .

Architectures ( M ) . We leverage the NASBench-101 search space as an architecture pool. It consists of 423 , 624 unique architectures with the following constraints - (1) number of nodes in each cell is at most 7, (2) number of edges in each cell is at most 9, (3) barring the input and output, there are three unique operations, namely 1 × 1 convolution, 3 × 3 convolution and 3 × 3 max-pool. We utilize the architectures from the search space in generating the sequence of embeddings along with sampling architectures for the training and testing of the encoder and datasets for the subset selector. As mentioned in the experimental setup, pre-training large number of models can be expensive.

Therefore, we choose a diverse subset of architectures from M tr of size 250 that ensures efficient representation over low-parameter and high-parameter regimes. The distributions of true and sampled architectures are given in Figure 5.

Note that the pre-training can be made faster by efficient model training methods without complete training (running a few epochs via curriculum learning [66]), which can be easily plugged with our method.

D.2 Implementation details about baselines

Facility Location (FL). We implemented facility location on all the three datasets using the apricot 1 library. The similarity matrix was computed using Euclidean distance between data points, and the objective function was maximized using the naive greedy algorithm.

Pruning. It selects a subset from the entire dataset based on the uncertainty of the datapoints while partial training. In our setup, we considered ResNet-18 as a master model, which is trained on each dataset for 5 epochs. Post training, the uncertainty measure is calculated based on the probabilities of each class, and the points with highest uncertainty are considered in the subset. We train the master model at a learning rate of 0.025.

Proxy. It selects a subset

Glister and Grad-Match. We implemented GLISTER [20] and Grad-Match [19] using the CORDS library. We trained the models for 50 epochs, using batch size of 20 , and selected the subset after every 10 epochs. The loss was minimized using SGD with learning rate of 0 . 01 , momentum of 0 . 9 and weight decay with regularization constant of 5 × 10 -4 . We used cosine annealing for scheduling

the learning rate with T max of 50 epochs, and used 10 % of the training data as the validation set. Details of specific hyperparameters for stated as follows.

Glister uses a greedy selection approach to minimize a bi-level objective function. In our implementation, we used stochastic greedy optimization with learning rate 0 . 01 , applied on the data points of each mini-batch. Online-Glister approximates the objective function with a Taylor series expansion up to an arbitrary number of terms to speed up the process; we used 15 terms in our experiments.

Grad-Match applies the orthogonal matching (OMP) pursuit algorithm to the data points of each mini-batch to match gradient of a subset to the entire training/validation set. Here, we set the learning rate is set to 0 . 01 . The regularization constant in OMP is 1 . 0 and the algorithm optimizes the objective function within an error margin of 10 -4 .

GraNd. This is an adaptive subset selection strategy in which the norm of the gradient of the loss function is used as a score to rank a data point. The gradient scores are computed after the model has trained on the full dataset for the first few epochs. For the rest of epochs, the model is trained only on the topk data points, selected using the gradient scores. In our implementation, we let the model train on the full dataset for the first 5 epochs, and computed the gradient of the loss only with respect to the last layer fully connected layer.

EL2N. When the loss function used to compute the GraNd scores is the cross entropy loss, the norm of the gradient for a data point x can be approximated by E || p ( x ) -y || 2 , where p ( x ) is the discrete probability distribution over the classes, computed by taking softmax of the logits, and y is the one-hot encoded true label corresponding to the data point x . Similar to our implementation of GraNd, we computed the EL2N scores after letting the models train on the full data for the first 5 epochs.

D.3 Implementation details about our model

GNN α . As we utilize NASBench-101 space as the underlying set of neural architectures, each computational node in the architecture can comprise of one of five operations and the one-hotencoded feature vector f u . Since the set is cell-based, there is an injective mapping between the neural architecture and the cell structure. We aim to produce a sequence of embeddings for the cell, which in turn corresponds to that of the architecture. For each architecture, we use the initial feature f u ∈ R 5 in as a five dimensional one-hot encoding for each operation. This is fed into INITNODE to obtain an 16 dimensional output. Here, INITNODE consists of a 5 × 16 linear, ReLU and 16 × 16 linear layers cascaded with each other. Each of EDGEEMBED and UPDATE consists of a 5 × 128 linear-BatchNorm-ReLU cascaded with a 128 × 16 linear layer. Moreover, the symmetric aggregator is a sum aggregator.

We repeat this layer K times, and each iteration gathers information from k < K hops. After all the iterations, we generate an embedding for each node, and following [62] we use the BFS-tree based node-ordering scheme to generate the sequence of embeddings for each network.

The GVAE-based architecture was trained for 10 epochs with the number of recursive layers K set to 5, and the Adam optimizer was used with learning rate of 10 -3 . The entire search space was considered as the dataset, and a batch-size of 32 was used. Post training, we call the node embeddings collectively as the architecture representation.

To train the latent space embeddings, the parameters α are trained in an encoder-decoder fashion using a variational autoencoder. The mean µ and variance σ on the final node embeddings h u are:

The decoder aims to reconstruct the original cell structure (i.e the nodes and the corresponding operations), which are one-hot encoded. It is modeled using single-layer fully connected networks followed by a sigmoid layer.

Model Approximator g β . The model approximator g β is essentially a single-head attention block that acts on a sequence of node embeddings H m = { h u | u ∈ V m } . The Query , Key and Value are three linear networks with parameters: W query , W key and W value ∈ R 16 × 8 . Note that the matrix W C ∈ R 8 × 16 in Eq. (5). As described in Section 4.2, for each node u , we use a feedforward network, preceded and succeeded by layer normalization operations, which are given by the following set of

0

20

40

60

80

Epochs

→

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

KL-Divergence

→

(a) (a) FMNIST

0

20

40

60

80

Epochs

→

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

KL-Divergence

→

(b) (b) CIFAR10

equations (where LN the denotes Layer-Normalization operation):

The fully connected network acting on ζ u, 1 consists of matrices W 1 ∈ R 16 × 64 and W 2 ∈ R 64 × 16 . All the trainable matrices along with the layer normalizations were implemented using the Linear and LayerNorm functions in Pytorch. The last item of the output sequence ζ u, 3 is concatenated with the data embedding x i and fed to another 2-layer fully-connected network with hidden dimension 256 and dropout probability of 0 . 3 . The model approximator is trained by minimizing the KL-divergence between g β ( H m , x i ) and m θ ∗ ( x i ) . We used an AdamW optimizer with learning rate of 10 -3 , ϵ = 10 -8 , betas = (0 . 9 , 0 . 999) and weight decay of 0 . 005 . We also used Cosine Annealing to decay the learning rate, and used gradient clipping with maximum norm set to 5 . Figure 6 shows the convergence of the outputs of the model approximator g β ( H m , x i ) with the outputs of the model m θ ∗ ( x i ) .

Neural Network π ψ . The inductive model is a three-layer fully-connected neural network with two Leaky ReLU activations and a sigmoid activation after the last layer. The input to π ψ is the concatenation ( H m ; o m,i ; x i ; y i ). The hidden dimensions of the two intermediary layers are 64 and 16 , and the final layer is a single neuron that outputs the score corresponding to a data point x i . While training π ψ we add a regularization term λ ′ ( ∑ i ∈ D π ψ ( H m , o m,i , x i , y i ) -| S | ) to ensure that nearly | S | samples have high scores out of the entire dataset D . Both the regularization constants λ (in equation 3) and λ ′ are set to 0 . 1 . We train the model weights using an Adam optimizer with a learning rate of 0 . 001 . During training, at each iteration we draw instances using Pr π and use the log-derivative trick to compute the gradient of the objective. During each computation step, we use one instance of the ranked list to compute the unbiased estimate of the objective in (3).

D.4 Hyperparameter Optimization

The hyperparameter search was done over optimizer and scheduler-based hyperparameters using Ray Tune [29]. We set the possible optimizers to be SGD, Adam and RMSprop, and the possible schedulers to be CosineAnnealing and StepLR. The search space parameters are given below:

Optimizers

SGD: learning_rate ∈ (0 . 001 , 0 . 1) , momentum ∈ (0 . 7 , 1 . 0) , weight_decay ∈ (0 . 01 , 0 . 0001)

Adam: learning_rate ∈ (0 . 001 , 0 . 1) , weight_decay ∈ (0 . 01 , 0 . 0001)

Schedulers

RMSprop: learning_rate ∈ (0 . 001 , 0 . 1) , momentum ∈ (0 . 7 , 1 . 0) , weight_decay ∈ (0 . 01 , 0 . 0001)

StepLR: step_size ∈ [10 , 20 , 30 , 40] , gamma ∈ (0 . 05 , 0 . 5)

CosineAnnealingLR

We employ TPE [2] as the hyperparameter search algorithm, and ASHA [28] as the hyperparameter scheduling algorithm. The hyperparameter search runs for 100 epochs in all cases. The random baseline is run for 5 runs, and we report the average speedup and test error.

E Additional experiments

E.1 Comparison with additional baselines

Here, we compare the performance of SUBSELNET against two baselines. They are the two variants of our method-Bottomb -loss and Bottomb -loss+gumbel.

In Bottomb -loss, we sort the data instances based on their predicted loss ℓ ( g β ( H m , x ) , y ) and consider those points with the bottom b values.

In Bottomb -loss+gumbel, we add noise sampled from the gumbel distribution with µ gumbel = 0 and β gumbel = 0 . 025 , and sort the instances based on these noisy loss values, i.e. , ℓ ( g β ( H m , x ) , y ) + Gumbel (0 , β gumbel = 0 . 025) .

Figure 7 compares the performance of the variants of SUBSELNET, Bottomb -loss, and Bottomb -loss+gumbel. We observe that Bottomb -loss and Bottomb -loss+gumbel do not perform that well in spite of being efficient in terms of time and memory.

T

0

0

.

0

0

.

1

Transductive-SubSelNet

Inductive-SubSelNet

b

-loss

b

-loss+gumbel

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

RAR

→

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 1

10 2

Speed up (

Tf/T

)

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

10 1

10 2

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

RAR

→

(a) FMNIST

10 1

10 2

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(b) CIFAR10

10 1

10 2

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(c) CIFAR100

10 3

10 4

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(d) TINY-IN

10 3

10 4

∫

T

0

memory(

t

)

dt

→

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

(e) CAL-256

10 3

memory(

E.2 Comparison of performance on ImageNet

Here, we compare the performance of SUBSELNET against two baselines: Selection-via-Proxy [5] and Pruning [48] on ImageNet-1K (1.28M images) [8] for b ∈ (0 . 05 | D | , 0 . 8 | D | ) . Figure 8 compares Transductive-SUBSELNET and Inductive-SUBSELNET with the two baselines on basis of speedup, memory and budget.

0

.

1

RAR

→

Transductive-SubSelNet

Inductive-SubSelNet

Pruning

Proxy

10 0

10 1

0

.

2

0

.

4

0

.

6

0

.

8

RAR

→

10 5

2

×

10 5

3

×

10 5

T

0

memory(

t

)

dt

0

.

2

0

.

4

0

.

6

0

.

8

20

40

80

Budget(%)

0

.

2

0

.

4

0

.

6

0

.

8

→

0

.

0

→

∫

)

10 4

t

)

dt

10 1

→

→

E.3 Analysis of compute efficiency in high accuracy regime

We analyze the compute efficiency of all the methods, given an allowance of reaching 20% and 10% of the relative accuracy reduction (RAR) (80% and 90% of the accuracy) achieved by training on the full data. We make the following observations:

Transductive-SUBSELNET achieves the best speedup and consumes the least memory, followed by Inductive-SUBSELNET.

For CIFAR100, Tiny-Imagenet and Caltech-256, Bottomb -loss and Bottomb -loss+gumbel achieve better performance than the baselines which are able to reach the desired RAR milestones (10% or 20%).

For CIFAR100, Tiny-Imagenet and Caltech-256, most baselines could not achieve an accuracy of either 10% or even 20% of the RAR on full data.

E.4 Recommending model architecture

When dealing with a pool of architectures designed for the same task, choosing the correct architecture for the task might be a daunting task - since it is impractical to train all the architectures from scratch. In view of this problem, we show that training on smaller carefully chosen subsets might be beneficial for a quicker alternative to choosing the correct architectures. We first extract the top 15 best performing architectures A ∗ having highest accuracy, when trained on full data. We mark them as "gold". Then, we gather top 15 architectures A when trained on the subset provided by our models. Then, we compare A and A ∗ using the Kendall tau rank correlation coefficient (KTau) along with Jaccard coefficent |A ∩ A ∗ | / |A ∪ A ∗ | .

Figure 9 summarizes the results for three non-adaptive subset selectors in terms of the accuracy, namely - Transductive-SUBSELNET, Inductive-SUBSELNET and FL. We make the following ob-

servations: (1) One of our variant outperforms FL in most of the cases in CIFAR10 and CIFAR100. (2) There is no consistent winner between Transductive-SUBSELNET and Inductive-SUBSELNET, although Inductive-SUBSELNET outperforms both Transductive-SUBSELNET and FL consistently in CIFAR100 in terms of the Jaccard coefficient.

0

0

.

0

0

.

1

Transductive-SubSelNet

Inductive-SubSelNet

FL

300

600

1800

3000

→

0

.

2

0

.

3

0

.

4

0

.

5

Coeffecient

→

250

500

1500

2500

→

0

.

4

0

.

6

0

.

8

Coeffecient

→

250

500

1500

2500

→

0

.

5

1

.

0

Coeffecient

→

300

600

1800

3000

→

0

.

4

0

.

6

0

.

8

K

τ

→

(d) (a) FMNIST

250

500

1500

2500

→

0

.

2

0

.

4

0

.

6

K

τ

→

(e) (b) CIFAR10

250

500

1500

2500

→

0

.

4

0

.

5

0

.

6

0

.

7

K

τ

→

(f) (c) CIFAR100

2

E.5 Analysis of subset overlap on different architectures

Different architectures will produce different feature representations of the underlying dataset, and they can be distributed in different manners. Thus to generate a subset, if the features are different, we would expect subsets to change too. We experiment with extremely dissimilar architectures (top-5 ranked by distance in the latent space generated by the GNN) to observe the subset overlap occurring. Table 9 containing Jaccard coefficient of the subsets chosen for dissimilar architectures, where we notice that the overlaps are extremely small for the top adaptive methods, as well as for our method.

E.6 Finer analysis of the inference time

Next, we demarcate the subset selection phase from the training phase of the test models on the selected subset during the inference time analysis. Table 10 summarizes the results for top three non-adaptive subset selection methods for b = 0 . 005 | D | on CIFAR100. We observe that: (1) the final training times of all three methods are roughly same; (2) the selection time for TransductiveSUBSELNET is significantly more than Inductive-SUBSELNET, although it remains extremely small as compared to the final training on the inferred subset; and, (3) the selection time of FL is large- as close as 323% of the training time.

E.7 Analysis on underfitting and overfitting

Since the amount of training data is small, there is a possibility of overfitting. However, the coefficient λ of the entropy regularizer λH (Pr π ) , can be increased to draw instances from the different regions

of the feature space, which in turn can reduce the overfitting. In practice, we tuned λ on the validation set to control such overfitting.

We present the accuracies on (training, validation, test) folds for both Transductive-SUBSELNET and Inductive-SUBSELNET in Table 11. We make the following observations:

From training to test, in most cases, the decrease in accuracy is ∼ 7 %.

This small accuracy gap is further reduced from validation to test. Here, in most cases, the decrease in accuracy is ∼ 4 %.

E.8 Additional results on NAS and HPO

Searched cell for NAS. Figure 10 shows the final Normal and Reduction cells found on the DARTS search space using SUBSELNET on the 40% subset of CIFAR10, which gave the lowest test error of 2.68 in the experiments.

c_{k-2}

0

skip_connect

1

sep_conv_3x3

2

sep_conv_3x3

3

skip_connect

c_{k-1}

dil_conv_3x3

dil_conv_3x3

sep_conv_3x3

sep_conv_5x5

c_{k}

c_{k-2}

0

sep_conv_5x5

c_{k-1}

dil_conv_5x5

1

dil_conv_5x5

skip_connect

2

skip_connect

3

dil_conv_5x5

c_{k}

skip_connect

skip_connect

Standard error results on NAS and HPO. Here, we present the mean and standard error over the runs for the Test Error (%) on NAS and HPO, which has been presented in the main draft (Section 5.2). We observe our method offers less deviation across runs. Moreover, we found that the gain offered by our method is statistically significant with p ≈ 0 . 05 .

±

±

±

F Pros and cons of using GNNs

We have used a GNN in our model encoder to encode the architecture representations into an embedding. We chose a GNN for the task due to following reasons -

Message passing between the nodes (which may be the input, output, or any of the operations) allows us to generate embeddings that capture the contextual structural information of the node, i.e., the embedding of each node captures not only the operation for that node but also the operations preceding that node to a large extent.

To better illustrate the impact of the GNN, we compared it with a baseline where we directly fed the graph structure to the model approximator using the adjacency matrix, in lieu of the GNN-derived node embeddings. This alteration resulted in a notable performance decline, leading to a 5-6% RAR on subset size of 10% of CIFAR10.

It has been shown by [38] and [57] that GNNs are as powerful as the Weisfeiler-Lehman algorithm and thus give a powerful representation for the graph. Thus, we obtain smooth embeddings of the nodes/edges that can effectively distill information from its neighborhood without significant compression.

GNNs embed model architecture into representations independent of the underlying dataset and the model parameters. This is because it operates on only the nodes and edges- the structure of the architecture and does not use the parameter values or input data.

However, the GNN faces the following drawbacks -

GNN uses a symmetric aggregator for message passing over node neighbors to ensure that the representation of any node should be invariant to a permutation of its neighbors. Such a symmetric aggregator renders it a low-pass filter, as shown in [41], which attenuates important high-frequency signals.

We are training one GNN using several architectures. This can lead to the insensitivity of the embedding to change in the architecture. In the context of model architecture, if we change the operation of one node in the architecture (either remove, add or change the operation), then the model's output can significantly change. However, the embedding of GNN may become immune to such changes, since the GNN is being trained over many architectures.

G Licensing Information

The NAS-Bench-101 dataset and DARTS Neural Architecture Search are publicly available under the Apache License. The baselines GRAD-MATCH and GLISTER, publicly available via CORDS, and the apricot library used for Facility Location, are under the MIT License.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors train 1) an architecture encoder using a GNN with a variational graph autoencoder loss to learn a good representation for model architectures, 2) a model approximator taking the GNN representation and a sample x to predict the output of the model given x. The goal is to learn a probability distribution over the samples of the datasets to obtain a subset of the data with minimum loss while penalizing an entropic objective to encourage maximum diversity. Given a chosen architecture, Transductive-SUBSELNET directly optimizes all probabilities of the samples. Meanwhile, Inductive-SUBSELNET learns a neural network taking the dataset (and both the GNN representation and the model approximator output) to generate a probability for each sample. They also do a hybrid approach for better efficiency.\n\nAlthough I understand the algorithm, this is quite complicated, and I cannot claim to understand all the specifics of every moving part (for example, how the GNN forms a representation for the architecture and learn to be insensitive to the model parameters, and why we need BFS; mainly I am not very familiar with GNNs).\n\nSTRENGTHS:\nThe results are very in-depth, showing a lot of interesting ways to apply this general method and some useful applications like making NAS and hyperparameter-tuning nearly as good by much faster by reducing the dataset size. They present comparisons to other works, showing significantly improved performance in terms of RAR and memory. They train on many datasets of various image sizes but only in the image domain.\n\nI think that this is great work.\n\nWEAKNESSES:\nLimited to CNNs for now (but that's okay). The biggest thing that could be improved is the notation and formatting of the paper to make the approach clearer, a lot of the details like in the GNNs could be left to the appendix. I feel like the paper overcomplicates things.\n\nQUESTIONS:\nHave you considered using pre-trained graph models for the architecture representation? It seems like there should exist some, and they may be trained with more than 250 architectures. If there are, it would be a worthwhile comparison and remove one big step from the algorithm. I know that there are GNNs for weights such as GHN-3, but I am not familiar with those for architecture.\n\nIt would be great to see b in Figure 3, maybe in the top x-axis, because right now, it's difficult to infer which size of the data is used.\n\nYou mention taking the images and then taking the 2048 Resnet-50 embedding as image representation; it seems to have come out of nowhere. Could you add more context to this? Because I assume that the actual images are used for real architectures and possibly in the Inductive-SUBSELNET neural network.\n\nThe notation can sometimes be weird, especially with the big dot in Pr_pi(dot); shouldn't x_i replace the dot?\n\nI think that instead of separating algorithms 1 and 2 as training vs inference, it would be easier to understand by separating them into two algorithms with one Transductive and one Inductive, and each algo could be separated into top and bottom where the top is training and the bottom is inference. TRAINPIPELINE could be separate.\n\nLIMITATIONS:\nYes, the authors properly address limitations and broader impacts.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCurrent subset selection methods are architecture specific and requires solving an optimization problem for each architecture individually. The subset selected through solving an optimization for one architecture does not generalize to another model. This paper addresses this problem and introduces an end-to-end training method using multiple architectures (through embedding them via GNN) to select a generalizable subset. The paper addresses an important problem, written thoroughly and consists of sufficient empirical experiments to verify the idea.\n\nSTRENGTHS:\n- Addresses a relevant practical problem in selecting subsets which can generalize across different architectures. \n- The paper is generally very well written and each component of the method pipeline is very well explained.\n- Good speedup with the proposed design (which is very relevant in practical scenarios) and also lower RAR compared to other baselines.\n\nWEAKNESSES:\n- While the method is a generalization of the subset selection algorithm — it is inherently complex which can lead to less adoption by the community\n- The paper will be stronger if more complex datasets are added. Currently only smaller datasets are used, which can obfuscate the real-world capabilities of this method. While Tiny-ImageNet is used, a few experiments at ImageNet scale will help verify the efficacy of the method more strongly.\n- I would expect a little bit more analysis on the characteristics of the data subset selected using their method vs. other baselines. Are there difference in the properties of data which are selected by SubSelNet? What is the overlap factor with other methods?\n\nQUESTIONS:\nSee Weakness\n\nLIMITATIONS:\nOverall, I believe that the paper is well-written with good empirical results. Given that the method (although not simple for adoption) offers a significant speedup for new architectures, I would vote for a weak acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a new method to select subsets of valuable training examples, with an emphasis on specializing the selections to new model architectures. The motivation is that selections made for one architecture may not work well when used with a different architecture (a claim that the authors did not show evidence for here). At inference time, the method takes a new network architecture, converts it to a learned embedding via a GNN, feeds it along with new inputs $x$ to a model approximator that estimates predictions from a trained model, and then optimizes over a probability vector from which we can sample a valuable subset. Rather than training the different components all at once, the authors propose separate objectives for each learned module. And intuitively, the final optimization over the probability vector (shown in eq. 1) encourages low loss among the selections, as well as high diversity.\n\nSTRENGTHS:\nThis is the first method I've seen that focuses on specializing data subset selection to specific architectures, so that angle is novel. It's also challenging, but the authors manage this complexity by setting up a GNN and model approximator to efficiently estimate predictions from trained models with arbitrary architectures. The full pipeline is involves multiple steps, but the authors show how to train each module independently and the results show that it works well in practice.\n\nWEAKNESSES:\n- One of the main motivators for this method is the idea that data subset selection cannot generalize across architectures. I didn't see any evidence for this claim in the paper, and in fact there are other works that show the opposite conclusion: for example, the selection via proxy approach from Coleman et al (2020) relies on the generalization of valuable data from smaller to larger models. Can the authors explain their focus on this issue? And if they believe it is a serious issue, can they provide evidence of lack of transferability across models, and perhaps of very different selections in their method depending on the architecture used at inference time?\n\n- The authors provide no information about the computational cost of the pre-processing step in this method. If the goal is to perform efficient neural architecture search or hyperparameter tuning, this cost should be accounted for, and it seems to require training at least 250 models with different architectures, plus the GNN and model approximator. Ultimately, it's hard to say if this actually reduces the computational cost of training.\n\n- The terminology for \"transductive\" and \"inductive\" solutions for $\\pi$ didn't make sense to me. I wonder if there's a simpler way to refer to these methods - for example, the \"transductive\" method solves a per-instance optimization problem, and the \"inductive\" method can be viewed as an amortized optimization solution (see \"Tutorial on Amortized Optimization\" by Amos).\n\n- The main objective shown in eq. 1 didn't quite make sense to me. The first part of the objective, which focuses on the loss for the selected examples, seems like it wouldn't capture anything meaningful - many samples will be perfectly predicted and have near-zero loss, and it doesn't seem especially helpful to focus on the easiest examples that are correctly predicted. Am I missing something, or can the authors explain this potential issue?\n\n- It would seem like the diversity term in the eq. 1 objective is intended to ensure diversity in some representation space and minimize redundancy for very similar training examples. Indeed, that's what prior methods aim to accomplish with submodular set functions like facility location. So the choice to instead use the entropy over $P_\\pi$ is strange to me, because it cannot be expected to encourage this diversity. In fact, I'm not sure what it accomplishes because we're already performing sampling without replacement. Can the authors explain this choice, and explain whether it would be possible to test their method with a submodular set function here instead of the entropy term?\n\n- I'm not sure why the authors included eq. 3 given that they don't suggest training all the models end-to-end like this. It's not clear that this end-to-end objective would even result in good models for the GNN and model approximator. I wonder if the authors could skip this somewhat confusing overview objective and proceed directly to the training steps for each module.\n\n- In section 4.1 about the GNN, could the authors confirm which architectural information they encode in the input representation? It sounded like they only indicate the operation performed at each layer, but perhaps not the size of each layer. If so, that seems somewhat limiting for the GNN's ability to distinguish between different architectures.\n\n- The main text doesn't seem to explain how the authors optimize eq. 9 and eq. 10. We optimizing over $\\pi$ here (or a network that outputs $\\pi$), and this would seem to require backpropagating gradients to a large number of logits through a non-differentiable sampling process; in particular, the \"inductive\" solution seems like it could require a forward pass through all the training examples for each gradient step. Algorithms 1 and 2 provide no information, and I didn't see a pointer to an appendix section that explains this point.\n\n- The notation for the \"inductive\" model on line 235 is confusing because it doesn't show the inputs referenced in the same sentence. This is only corrected below on line 241. \n\n- The RAR metric is sensible, but it also leads to a y-axis in Figure 3 that's difficult to interpret. In particular, it's hard to tell at what point the trained models have unacceptably low accuracy due to too much speedup. Would the authors consider instead showing raw accuracy, or absolute accuracy reduction?\n\n- I wonder if the authors could include strong but simple baselines to compare to the existing ones. For example, could they include training on the full dataset for different numbers of epochs, training on a fixed random subset, or training on newly sampled subsets at each epoch? Given the focus on speedup on the x-axis rather than the number of epochs, I could see these baselines outperforming some of the existing ones, which are in some cases quite slow.\n\nQUESTIONS:\nSeveral questions are mentioned in the issue above.\n\nOne of the baselines used here, EL2N, was recently shown to have a serious bug in its implementation that invalidated some of the findings. Can the authors explain the relevance of that issue to their experiments and say whether they're using a corrected implementation here?\n\nLIMITATIONS:\nThe paper does not include much discussion of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

comments: SUMMARY:
The paper presents a model agnostic method of subset selection using a graph neural network a surrogate.

STRENGTHS:
- Problem area is an interesting space, and an area of interest to the community at present.
- Novel application of GNN for subset selection
- Seems to be a performance gain in some settings for SubSelNet over existing methods (but hard to tell from graphs).

WEAKNESSES:
- The paper not well written. I found the paper hard to parse. Structure is messy, and the paper visually feels like there's too much going on. The figures are small. The graphs and tables are hard to read. This paper would not have a much of an impact as it stands, simply because it's difficult to read.
- Doesn't seem to a convincing narrative here on why are existing methods not model agnostic? See Selection via Proxy by Coleman et al and RHO-Loss by Mindermann et al.
- Datasets are all small. The graphs are hard to read
- The method seems overly complicated and some of the justifications seems slim (e.g. GNN for architecture encoder? why architecture encoder? Why "neural approximator"?


Minor Comments:
- The arrows (on RAR as other metrics) in figure 3 and table 1 are confusing. They don't indicate in the way arrows usually do (i.e. what direction is best). I don't even know why there is an arrow the table 1.

QUESTIONS:
See above.

But on the whole this paper is just needs a thorough restructuring at the very least.

LIMITATIONS:
Societal Impact: Would be nice to see the effect of subset selection on fairness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new subset selection method, SUBSELNET, for efficient training of neural network models. SUBSELNET selects optimal subsets for a model based on its architecture with minimal overhead. SUBSELNET outperforms existing subset selection methods in terms of computational efficiency.\n\nSTRENGTHS:\nNew data-driven approach to subset selection: the proposed method turns subset selection into a learning problem and learns a subset selection pipeline that can generalize to new architectures. Such approach has the potential of being more universal while also more efficient at certain conditions (see below).\n\nA step towards uncovering the relationship between optimal subsets and model architecture: the proposed method in principle shows that optimal subsets can be determined from a simple representation of the model architecture, which is a quite strong hypothesis worth investigating. This line of research has the potential to better our understanding of the relationship between data and model in deep learning.\n\nEfficient subset selection adaptive to mode architecture: although with significant drawbacks (see below), the proposed method points to a promising possibility of architecture-adaptive subset selection with very little cost. \n\nApplication in NAS and hyperparameter search seem promising: from the results reported in the paper, the proposed method has the potential to significantly accelerate NAS and hyperparameter search if used properly.\n\nWEAKNESSES:\nMethod has limited efficiency advantage because of costly training: the proposed method needs to train a large number of models in order to learn a model approximator, therefore though it is more efficient than other adaptive subset method at test time, the computation cost is considerably shifted to the training phase. As a result, the proposed method is only clearly more efficient when it is required to efficiently train a large number of models of different architectures so that cost can be amortized. This situation seems only happen when doing NAS. NAS is undeniably an important task, but otherwise the potential use of the proposed method seem limited.\n\n(minor) Another limitation of the method is that the trained model approximator does not generalize across datasets. The method needs to be retrained for each new dataset, which further limits its scope of application. The method also requires a pre-trained feature extractor suitable for the current data distribution, which may not be available for less common datasets.\n\nInsufficient analysis of the components in the proposed method: to appreciate the merits of the proposed design of approximating model prediction based on architecture, one would want to know how effective is the model approximator. For example, how effective does it generalize to unseen architectures? Also, to what extent is the subset selection dependent on architecture (how much do subsets selected for two different architectures overlap, and does the degree of overlap depend on architecture similarity)? For the GNN architecture encoder, how much does it contribute to the final performance? And what about the pre-trained feature extractor?\n\n(minor) Scope of experiment is kind of limited: experiment is mainly performed on small datasets such as CIFAR100, no experiment on larger datasets such as ImageNet where efficient training is much more meaningful.\n\nSome assumptions made in the paper are likely questionable: see the `Questions` section.\n\nPotentially incomplete evaluation protocol: see the `Questions` section.\n\nQUESTIONS:\nAssumptions on architecture encoder: \n\nSection 4.1 describes the parameterization of operations in a neural network into node representations, and use them as features in the model approximator to approximate the prediction of a model. This implicitly assumes that the prediction of a model depends only on the structure of the network, which is likely not the case. Hyperparameters like model size, optimizer, learning rate, etc., can all affect model performance and could affect different architectures differently. It may require empirical analysis to show if using architecture alone suffices to approximate model predictions.\n\nAssumptions on using SUBSELNET in AutoML:\n\nThe idea of using adaptive subset selection in NAS, etc., is potentially problematic. The goal of NAS is to select $\\underset{m}{\\operatorname{argmin}} L$, while the proposed method selects $\\underset{m}{\\operatorname{argmin}} \\underset{S}{\\operatorname{min}} L$ which is a different optimization problem. Because the subset can adapt to each particular model, the performance on the model's most advantageous subset might not correlate well with its performance on the whole dataset. For example, it is possible that a generally worse-performing model has a good training subset. On the other hand, non-adaptive subset selection does not risk over-adaptation between subset and model and can be more safely used in NAS to approximate $\\underset{m}{\\operatorname{argmin}} L$.\n\nEvaluation:\n\nThe main results of the paper, Figure 3 and Table 1, are mostly dedicated to comparing the speed (computation time) and memory usage. Here the computation time involves multiple components (e.g., running subset selection, model training) and can be tricky to compare across methods and is implementation-dependent. Because the proposed method uses costly training, to include the amortize training time or not in the comparison could make a large difference. The authors need to be more explicit in the detailed protocol used when comparing computation time.\n\nBecause the training phase involves training many models, the cost is mostly linked to the number of models trained. Therefore, it seems necessary to verify the minimum number of models need to be trained in order to learn a good model approximator.\n\nComparison of efficient training can be evaluated in other dimensions as well, most notably with sample efficiency: would the proposed method be able to select a smaller subset to maintain certain accuracy? Sample efficiency is also a more reliable metric than computation time and allows for more objective comparison.\n\nResults in \"using SUBSELNET in AutoML\" section: why only `random` and `proxy` are evaluated in Table 4, what about other subset selection methods listed in Figure 3? Why are non-adaptive baselines not evaluated in Table 6, which could have low cost?\n\nLIMITATIONS:\nThe authors adequately addressed the main limitations of the proposed method, including dependence on the training dataset and the inability of generalizing model prediction to the test set.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the Subset Selection problem for selecting the best subsets training samples with which a well-performed model can be trained. This paper focuses on the challenge that previous methods for subsets selection cannot transfer across architectures. Specifically, this paper proposes to use a transformer to predict the accuracy of a model trained on a subset of training samples and use the predicted results as the surrogate to solve the combinatorial optimization problems for the selection (Transductive). This paper also proposes inductive selection, which learns another neural to predict the result of combinatorial optimization. Further, this paper proposes to encode the neural network's architecture using GNN, which will also be fed to the predictor. \n\n__________________________________________________ Post Rebuttal _________________________________________________\n\nThe reviewer has already read the rebuttal. \n\nAll my concerns have been addressed. And the reviewer wishes that the insights discussed in the rebuttal can be accommodated in the final version of the paper. \n\nThe reviewer will maintain the original score since the overall novelty, contribution, and impact cannot reach the reviewer's bar for papers scoring 7, and a score of 6 is already positive for the paper.\n\nSTRENGTHS:\n1. In this manuscript, the authors present an innovative approach toward subset selection, putting forward a new framework that emphasizes efficiency and practical applicability.\n\n2. A feature of the proposed framework is its ability to generalize across various architectures. This is a step forward, as it ensures the versatility and universal applicability of the method. The framework is not only confined to a specific architectural layout, but it can adapt to and function effectively within a range of scenarios. \n\n3. Additionally, the paper is substantiated by robust experimental results that highlight the superiority of the proposed method in comparison to existing baselines. The authors provide strong empirical evidence of the efficacy of their approach, demonstrating that it not only meets the performance standards set by previous methods but often surpasses them. \n\nMinor: \nThe authors have also shown how their method can be naturally incorporated into Automated Machine Learning (AutoML) systems. This seamless integration is beneficial, as it minimizes compatibility issues that could arise when introducing a new method into an established system. AutoML stands to gain from this method as it could optimize the automated selection, deployment, and tuning of machine learning models, and the paper illustrates this potential effectively.\n\nWEAKNESSES:\n1. Despite the promising results presented in the manuscript, one key concern that emerges is the absence of a justification for the methods put forward. Specifically, it is not well-established whether the transformer in use is genuinely capturing the intricate mapping between models and data, or if it is merely overfitting to the training samples. Overfitting would lead to a lack of generalizability in unseen or novel data situations, which could severely limit the practical utility of the proposed method. It would greatly strengthen the paper if the authors could provide deeper insights into these theoretical underpinnings, perhaps by conducting additional analysis or testing to definitively establish the veracity of the transformer's operations.\n\n2. Another potential shortcoming that emerges from the work is the limitation inherent in the utilized dataset. The dataset, NAS-Bench-101, contains Convolutional Neural Network (CNN) models with architectures that bear high similarities. Specifically, it is not clear whether the proposed method can be extended to other network architectures, such as Residual Networks (ResNets). It would therefore be beneficial if the authors could test their method with a broader range of architectures and provide evidence to support its efficacy across such a diverse range.\n\nQUESTIONS:\nTo the best of the reviewer's knowledge, analysis over accuracy is an NP-hard problem. It remains elusive to the reviewer that transformer really has the ability to capture the mapping between neural networks and accuracy.\n\nLIMITATIONS:\nThe proposed methods rely significantly on the pertaining of the model approximation to produce the correct accuracy and largely depend on the transferability of the model approximator.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-26"}