Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: TabNAS proposes a modification to Reinforce which instead of modifying the reward function to incorporate resource constraints, estimates the feasible set of architectures (e.g. architectures less than some threshold/reference) and updates the probabilities of the controller accordingly by redistributing the probability mass over the feasible set as more trial-and-error progresses. 
strengths_and_weaknesses: Strengths: Paper is clear and well-written. Literature has been adequately cited and discussed. 

Weaknesses: Please see my comments in the questions section.


questions: 1. I am quite confused by the contribution of this paper. The way I read it, the authors show that reward function modification as has been commonly done before is not as good as rejection sampling as proposed in this work in coming up with architectures that better meet resource constraints. I buy this and note that most Pareto-frontier aware algorithms which output the *entire* Pareto-frontier employ rejection sampling to constrain search to only those parts of the search space that the user wants (e.g. give me the Pareto-frontier estimate less than < 100 milliseconds inference latency, or between 200-300 milliseconds inference latency) See https://arxiv.org/abs/2105.01015. But there seems to be nothing specific in the method to tabular datasets. I can see applying this method as-is to any task (e.g. vision amd NLP). Is my understanding correct? If so, then it perhaps happened that the authors ran on tabular datasets to get the ball rolling? Note that one can use any of the excellent NAS benchmarks without using GPUs or large compute to validate this method. My suggestion will be NAS-Bench-Suite which has several different benchmarks under a single interface. https://arxiv.org/abs/2201.13396. 

2. My second concern/comment is perhaps a bit unfair to this work: It is a bit baffling to me that a stateful RL algorithm like Reinforce has been widely applied (and continues to be used in the broader AutoML community) to a problem setting where one doesn't need to do long horizon credit assignment. NAS is inherently stateless, one immediately knows the result of their action (sampled architecture or in HPO the sampled hyperparameter). So 1-step RL like contextual bandits or Bayesian optimization will be just as efficient and probably more. Can such rejection constraints be added to BOHB, HyperBand, BANANAS, NASBO etc? If so how will they be different from those in https://arxiv.org/abs/2105.01015? 

 3. Related to point 2 above: in the appendix lines 251-254, states that there are open questions on BO and ES for one-shot NAS. But is that necessary or one could train the supernet using the style of training in Once-For-All https://arxiv.org/abs/1908.09791 or use all the insights for training supernets as detailed in https://openreview.net/forum?id=Esd7tGH3Spl and then doing search via any search technique? Note OFA uses evolutionary search, the earlier ENAS paper used Reinforce https://arxiv.org/abs/1802.03268 and indeed the actual search technique varies across papers. In this respect it appears to me that the aim of the experiments in the appendix comparing across RL, BO and ES since BO and ES are being run on the feasible set only. So, a comparison (on tabular task) seems orthogonal to the aim of the paper which is to show a modification of specifically Reinforce to take into account resource constraints better than others who modify the reward function?

limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a multi-objective NAS algorithm using RL-controller. They use monte-carlo based rejection sampling to guide the controller towards feasible search space. 
The search space is divided into possible categorical options for each layer, such as [16, 22, 42] etc for layer1. The controller predicts logits corresponding to each possible value for each layer in the fully connected network (FCNN). So logits_{ij} refers to the value for $j^{th}$ value in the $i^{th}$ layer and corresponds to the probability that it would be sampled. As the choice of each layer is independent of the other, to generate a network adhering to the constraints, they use rejection based reward mechanism.
strengths_and_weaknesses: They showed that their technique is able to find well performing networks.

However, they did not compare adequately against all the baselines. Tabulating the results rather than presenting them in a paragraph is preferred. Please see Questions for more details.

questions: 1) (a) Please tabulate the accuracies, number of parameters and time taken of networks obtained from MnasNet, TuNAS, TabNAS and for all the datasets. 

(b) MoNas [1] is also an RL based multi-objective algorithm which yields a reward only if the constraint < threshold.  Resource-Efficient Neural Architect[2] uses a different penalty as listed in eqn 3 of their paper. Please compare how these two objectives fare against yours.

(c) While you are focusing only on RL based multi-objective algorithms, it would be good to compare with evolutionary based algos, namely Lemonade and NSGA-NET[3] .

 (d) It would also be good to compare it against TabNet. Based on the number of parameters of the best model found by Tabnet, you can modify your search to fit that parameter search space. 

2) While rejection sampling is helping us focus the search on only the feasible area, the number of samples required to get a good estimate of the probability distribution seem to be very high. So it might not be scalable to larger search spaces. For ex,  5 * $10^{6}$ samples for 9-layer search space for Volkert dataset is computationally expensive and cannot be used in real world. 


[1] MONAS: Multi-Objective Neural Architecture Search, Hsu et al.
[2] Resource-Efficient Neural Architect, Zhou et al.
[3] NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm, Lu et al.
limitations: The monte carlo sampling is very expensive and this search won't scale for tasks requiring large search spaces. Please see Questions section for further details.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a neural architecture search algorithm designed for tabular datasets. Prior work uses an RL-based optimization procedure with penalties for feasibility and resource constraints, but the authors show that this does not work well. To address this issue, the authors add an additional rejection sampling phase to filter out those that do not satisfy constraints, and show that this allows the RL method to better explore the search space of architectures. Evaluation is done on the Criteo and Volkert datasets, where the found architectures consistently achieve lower losses.
strengths_and_weaknesses: Strengths:
- I believe that the use of rejection sampling in NAS algorithms is novel. However, I am not familiar with neural architecture search so my knowledge of related work is relatively small.
- The work is significant, in that it unlocks more NAS applications in a new domain area.
- In general, the writing is clear and well-organized - the authors clearly present their method and reasoning, and I appreciate that the authors experimented with multiple baselines and multiple datasets and found consistent results between them.

Weaknesses:
- The experimental results are presented on a domain the baseline methods were not developed for, so it is more difficult to put the results in context. It would be informative to see how well the proposed methods work well on image-based domains, even if it does not perform as well.
questions: I think the results could be summarized in a more concise and clear way (e.g. a table) - in the paper the only results are shown in 2 small graphs where it's difficult to tell what the absolute numbers are and which baseline is being run.
limitations: The authors did not discuss any potential negative societal impacts of their work.
review_title: null

Meta-review: metareview: The reviewers unanimously recommend accept.  Many important clarification points were discussed in the author rebuttal.  Please make sure to incorporate those changes into the final camera ready.
confidence: Certain
Decision: Accept