Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper performs an empirical study that compares early stopping-based NAS methods: HB, ASHA, and BOHB. The main question that it seeks to answer is how to best allocate resources, both for the NAS experiment overall (Q1) and for a single trial (Q2).

For Q1, the paper looks at parallel efficiency vs. number of GPUs allocated, showing that with up to 32 GPUS, the parallel efficiency is above 0.75, but for 64 GPUs, the efficiency drops to 0.6-0.65. For Q2, the paper suggest increasing parallelism in order to get a reduced runtime.
overall_review: Strengths:
* The main strength of this paper is an empirical comparison of the methods, which can help NAS practitioners. The answer to RQ1 was clear and conclusive and perhaps provides an insight that was not previously known

Weaknesses:
* For RQ2, is the main takeaway to increase parallelism as much as possible? This could be made clearer throughout. Although this is not too surprising, it is still nice for practitioners to know.
* The plots should have error bars.
* One question I expected to get an answer to is the following: suppose I had a total compute budget along with a instantaneous compute budget, how should I trade off adaptivity vs wall time?
review_summary: Although the paper gives a nice empirical comparison of three early stopping-based methods, I think the impact at this point is still relatively small.
actions_required_to_increase_overall_recommendation: * I think if this paper performed a more systematic evaluation and provided clearer results and takeaways (esp for RQ2), then it would have increased impact.
* (Copied from above) One question I expected to get an answer to is the following: suppose I had a total compute budget along with a instantaneous compute budget, how should I trade off adaptivity vs wall time?

**Edit** I've increased the score after reading responses. Thank you to the authors for the prompt response.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors experimentally investigate the impact of parallelism of the NAS loop on the runtime and quality of the outcome using three common early-stopping based methods: Hyperband, Asynchronous successive halving (ASHA), and Bayesian optimization hyperband (BOHB). In particular, two research questions were investigated: how many resources should be allocated to a single NAS task, and how many resources should be allocated to a single NAS trial (evaluating a single architecture configuration). The authors conclude that using up to 32 GPUs per NAS task is advisable for general early-stopping based NAS methods, and that having a smaller number of trials per GPU (e.g., 1-4 trials per GPU) gives stronger anytime performance in terms of validation accuracy than setting 8 trials per GPU.


overall_review: The experiments are well-designed and thoroughly conducted and the conclusions drawn from the experiments are reasonable. The insights in this paper could provide useful guidance for ML practitioners to allocate resources for NAS. 

The paper is also well-written. The motivation, related work, and experiments are clearly explained. 
review_summary: The methodology seems sound, the explanations are clear, and the insights are beneficial to a wide range of practical situations. Therefore I recommend accept.

actions_required_to_increase_overall_recommendation: It would be nice if the authors could address the three points mentioned in "clarity".
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper tackles the problem of parallel compute resource allocation in the context of Neural Architecture Search (NAS) for early stopping-based algorithms. The authors distinguish two assignment choices in the presence of multiple GPUs: 1) assign more resources for speeding up individual configuration training, and 2) assign more resources for NAS procedure for increased parallel configuration sampling. Furthermore, using a unified benchmark, NATS-Bench, they investigate the optimal allocation trade-off on three datasets in a setup of varying numbers of GPUs available (from 1 to 64) of four baselines. Empirical results show that by optimizing the allocation, a speed-up of factor ~2-5 can be achieved. 
overall_review: Strengths:
- The paper is well-motivated and bears potential practical significance.
- The evaluation protocol is standardized.
- The appropriate baselines are chosen for empirical evaluation. 
- The scaling experiment, measuring parallel efficiency is very interesting. 
- The insights about batch size scaling for better GPU utilization are valuable.
- The future work direction, especially adaptive resource allocation, seems very promising. 

Weaknesses:
- The checkpointing mechanisms (for a single GPU experiment) is not entirely clear, along with its overhead influencing overall runtime.
- The presentation of some of the figures could be improved to better visualize discussed speed-ups. Maybe displaying initial speed-ups on one plot and the final on another would declutter them.
- The resource allocation scheme has been optimized for a single search space parametrization, it would be recommended to demonstrate that this carries to different experiment setups and can be reused for different problems.
- It would be great to see experiments for large models. 
- Clear takeaway messages are missing, to guide researchers in their experimental setup. 
review_summary: The authors investigate the optimal resource allocation for early stopping-based algorithms. They motivate their research very well and pose interesting research questions. The evaluation protocol has some minor flaws when it comes to clarity and presentation. The empirical results and discussion do not present clear takeaway messages for future research which downplays the potential impact of the paper. As I opt for borderline leaning reject, I encourage to incorporate the feedback to improve the overall paper quality and increase the score. 
actions_required_to_increase_overall_recommendation: - Improve the clarity of the paper.
- Improve results presentation.
- Present paper contributions in a more convincing manner.
- Show that optimized allocation is transferable.
- Show large-scale experiments.
- Present takeaway messages more clearly.
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: The paper studies the impact of using variable numbers of workers and GPUs on the speed of NAS within NATS-Benchmark. More specifically it studies two main research questions: 1) Efficiency of NAS as a function of number of GPUs used, and 2) Impact on the speed and accuracy of NAS when using various numbers of trials per GPU. From the results: 1) Efficiency has a decreasing trend when using more GPUs compared to using a single GPU. 2) NAS is generally faster when using more workers on a GPU, but if there are too many workers, then the NAS time can start increasing. There is no significant impact on accuracy. The authors study the question further by controlling for the amount of simultaneously processed data and find that the main reason hindering efficient resource usage are the switching and reloading processes between trials.
overall_review: Positives
* The paper studies the problem of resource allocation from multiple perspectives: 1) scaling across multiple GPUs, 2) scaling workers within 1 GPU
* The paper is relatively easy to read
* The topic of the paper may be interesting in case there are any surprising findings that would challenge how resources are allocated
* Multiple common HPO algorithms (and hence of interest to researchers) are evaluated

Negatives:
* Any specific recommendations in the paper are likely to depend on the exact combination of benchmark and GPU used. Because only NATS-Benchmark and one type of GPU was evaluated it is hard to use any of the more specific suggestions such as using 32 GPUs in parallel and not more. To make the specific recommendations more general (and hence useful) more settings/combinations should be considered, but it can give mixed results.
* A large part of the take-aways about efficiency in RQ1 depends on using threshold of 0.75. Various people may prefer different thresholds and the results can be quite different when considering different setups, so claims that we should use up to 32 can be unjustified. The results generally show that efficiency decreases as more GPUs are used.
* While in some cases average across 3 runs is reported, no results show standard deviations. These would be especially useful in Table 1 to see if the points raised in the discussion actually hold or are simply due to noise.
* The high-level findings of the paper are not particularly surprising as they essentially suggest that “it is advisable to increase the parallelism of the NAS loop to achieve a decreased total runtime”.

review_summary: Based on the paper it is hard to take specific and justified insights about how many workers or GPUs one should use when facing a new setup. The current paper is more of a case-study of the impact of using different numbers of workers on the specific setup considered.

actions_required_to_increase_overall_recommendation: It would be useful to conduct evaluation using more settings (especially various types of GPU but also other benchmarks would be useful) so that the guidance is more likely to generalize to new settings.
review_title: null

Decision: Accept(Main Track)