Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present SUBSELNET - a non- adaptive subset selection framework for solving a particular aspect of subset selection problem - improving generalizability of the subset selection approach; with existing methods, the algorithm has to be executed from the beginning for each new model.
The authors introduce an attention based neural approach that uses the graph structure of the architectures, which is then used to build subset samplers. Their approach has 2 variants: transductive and inductive.
They claim that their approach is more efficient than the existing approaches since the subset is chosen at the beginning of the training process, and the entire dataset is not required through the training process. 
strength_and_weaknesses: The authors have provided the motivation for the problem, and presented the solution to address the problem. They have also evaluated their approach against 6 other approaches, and 3 different datasets, showing the speed up and memory utilization. 
The approach overall is interesting since, they are able to preselect the dataset for the training process. 

However, there are a few areas that are not clear from the paper.
1. It would have been great if the authors spent more time discussing the pros- and cons- of their graph network. In general, graph networks themselves can be large, slow and memory consuming. It seems like the comparison is performed on the output of the graph network rather than the end-to-end approach.
The details about the GNN and the graph embedding are important.

2. Much of the details, including the step-by-step algorithm and the details are added to the appendix. For instance, the diagram in appendix B and Pseudocode in C, both would have helped understand the paper better, if it was in the main text.

3. Since the approach relies on pre-selecting, it is not clear how the approach is able to avoid overfitting or underfitting. The authors have split the data into train, validation and test sets. Including a report on the accuracy on these datasets, and the time/ computation resources required for these would have been helpful.

4. The loss function (eq 1) and the objective functions (5 and 6) require more explanation. For intance, the authors state in page 3 after eq1 that: "One can use submodular functions (Fujishige, 2005; Iyer, 2015) like Facility Location, graph cut, or Log-Determinants to model DIVERSITY(S)". However, they havent mentioned the approach they have used in the paper. 
Later, they mention the use of entropy on the subset sampler H(Prπ(•)) to model the diversity in page 4 after eq 5 and  KL after eq 6. The choice of the functions needs to be elaborated to appreciate the approach better. 
clarity,_quality,_novelty_and_reproducibility: The approach overall is interesting since, they are able to preselect the dataset for the training process. It seems novel in that aspect.
The paper's clarity can be improved - important and interesting parts have been moved to the appendix, whereas the math behind the model could have been explained better. 
The paper as presented, is less easy to reproduce - details about the GNN, the embeddings etc are probably missing. 
summary_of_the_review: The approach overall is interesting since, they are able to preselect the dataset for the training process. The authors introduce an attention based neural approach that uses the graph structure of the architectures, which is then used to build subset samplers. Their approach has 2 variants: transductive and inductive. They claim that their approach is more efficient than the existing approaches since the subset is chosen at the beginning of the training process, and the entire dataset is not required through the training process.

Overall, there are a few areas that are not clear from the paper, including the implementation and the evaluation of the approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In  this paper,  a new non-adaptive data subset selection method is proposed. The traditional adaptive method mixed the training and  subset selection. While for the new proposed method, the subset processing is done before the training. Furthermore, the paper also proposed the transductive and inductive variants. The experimental results verified that both variants output perform the baselines on subset selection and also an be used to choose the best architecture.

strength_and_weaknesses: This paper has a clear decomposition on the model into parts including model approximator and subset sampler. And for each part, it has clear annotations to explain the whole process. 
For the subset sampler, 2 variants are proposed and the experimental results show the tradeoff between them, and furthermore, a combination of these 2 variants are tested too.
The comprehensive experimental results proved the effectiveness of the proposed method.
Some questions:
1. some writing errors such as “viz.” appeared a couple of times.
2. both formula (5) and (6) has the E_S, is that correct? 
3. for the E_S optimization objective such as (6), as the parameter \pi is under the prob distribution and needs sampling, how do you optimize the \pi? do you use some reparameterization trick which is now shown in the paper.
4. Can you elaborate more on why you don’t jointly optimize the gnn parameter and transformer parameter?
5. In the experimental setup, the paper mentions all the baselines are non-adaptive, is it correct?
6. Do you have any results to show the accuracy gap between the neural model approximator and the fully trained model?

clarity,_quality,_novelty_and_reproducibility: This paper proposed a new non-adaptive method for the data subset selection problem and has a clear description of the proposed method. Also the experimental results verify the paper’s arguments: mainly on the advantage of the tradeoff between speedup and memory.
summary_of_the_review: In summary I think this paper makes a good presentation on a new method for the non-adaptive methods and verifies the contributions in the experimental studies. It will be better to fix some annotations and have more ablation studies and show more results.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a SUBSELNET to select a subset of training data. SUBSELNET is a non-adaptive method as it is agnostic to model architecture and training stage and. The authors design a neural model approximator to approximate the output of any given architecture. Two variants of subset sampler are proposed.  SUBSELNET is compared against state-of-the-art methods on three datasets to demonstrate the tradeoff advantage between accuracy and speed up.
strength_and_weaknesses: Strength:

(1)	Well-motivated problem. Data selection for an unseen architecture is an important problem with real-world impact.

Weakness:

(1)	The main problem of this work is the experiment. The authors choose 0.5% and 5% of training data, however, with such a low subsample rate, we notice a dramatic accuracy drop in Figure 1.  It is hard to judge the effectiveness of any methods with such a significant accuracy drop. The experiment should reflect at what speed up, the model can maintain the same accuracy. Essentially, focusing on the left side of Figure 1. The authors may consider a larger range of sampling rates (5%, 10%, 20%,30%, 40%, 50%, 60%, 70%), following the setup of previous work such as GraNd and Gister. 

Question:

(1)	When calculating inference time, are you assume selection and training are conducted in sequential? Is it possible to perform selection in parallel with training when loading a batch of data?

(2)	What is the training cost of training the model approximator? Have the authors evaluated the approximator error? 


clarity,_quality,_novelty_and_reproducibility: Clarity: Clarity is fine. Figures are generally easy to read. Most descriptions are clear. 

Quality:  Experiments setups can be improved ( See Weakness 1). Writing can be polished with noticeable grammar mistakes.   For example, Section 5.1 Model architectures and baselines: four non-adaptive  -> four adaptive. Figure1 Caption ;and; -> , and

Novelty:  Neural model approximator using GNN seems new for data selection. 

summary_of_the_review: I would recommend weak rejection because of the experiment results. It is hard to judge the effectiveness of the proposed method when all methods exhibit significant accuracy drops.  I would increase my score if the authors could demonstrate the efficiency advantage of the proposed method at no or negligible accuracy loss.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a subset selection method that embeds the neural network architecture via GNNs and uses it as a surrogate for trained deep NNs. Three reviewers reviewed this paper. Two reviewers suggested borderline accept, and one suggested borderline reject. Through rebuttal, the authors somewhat improved the clarity on some details such as GNN architecture, which was one of the main reasons for suggesting rejection. However, concerns about overfitting due to non-adaptive characteristics still remain. 

At the request of the authors, this AC also carefully reviewed this paper, but I also suggest rejecting this paper for a different reason than the negative reviewer. (This is an issue raised by other positive reviewer.) It is important to secure computational efficiency through subsampling, but it is meaningless if too much performance is sacrificed. In other words, it is necessary to verify the proposed method on a much more diverse subset size than that shown in figure 1. (Especially, verification in a region where performance is not that sacrificed is important.) During the rebuttal period, the authors reported experiments on more diverse subset sizes, but only marginal improvements over baselines were achieved. In addition, unlike baseline papers that verified various architectures such as ResNet and MobileNet, the experiment was verified only in limited architectures on CIFAR10 in the rebuttal, so I think that this part needs more verifications and an additional round of review.

justification_for_why_not_higher_score: experimental results are not convincing because it is validated only in regimes where performance is sacrificed too much
justification_for_why_not_lower_score: n/a
Decision: Reject