Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: The paper takes a closer look at Differentiable Architecture Search and analyses its performance in the context of inverse problems: more specifically using cosine waves of varying magnitude, amplitude and offset, with the goal of searching for models that can recover the samples from distorted measurements. The key findings as described in the paper are 1) DAS has large variance for inverse problems, 2) selection of hyperparameters has a large influence on the performance of DAS, 3) the performance of the weight-sharing architecture used during training does not reflect the final performance of the found architecture well.

overall_review: Positives:
* The general idea of the paper is interesting: take a deeper look at the performance of differentiable NAS, using lightweight inverse problems setting.
* The paper identifies specific points that highlight potential limitations of DAS: instability across trials, instability across HPO tuning of the NAS hyperparameters, differences between one-shot performance ranking and full re-training.
* The paper comes with a new problem setting that could be potentially of interest to other NAS researchers thanks to being lightweight: inverse problems.
* The paper is well-written and easy to read.

Negatives:
* The significance of the main findings - instability across trials and instability across sets of hyperparameters - may be overrated given the results presented in the paper. Disparity between one-shot performance and re-training performance appears to be investigated already in multiple studies based on Section 2: Related Work, Stability of DAS.
* Analysis of the results may not be the most appropriate to draw the given conclusions: the paper reports max, mean and median across 75 or 100 repetitions, but reporting stds or confidence intervals may be more appropriate to judge if the main messages of the paper hold and to what extent. Min instead of max is likely to be more interesting as min corresponds to worst-case performance here.
* Given that the general goal seems to be to find general findings about differentiable NAS methods rather than just within the context of inverse problems, some evaluation also on other problems (e.g. CIFAR-10 classification) would be helpful. Are there any parts of the analysis that would be viable also on other problems than inverse problems so that we can find how general the findings can be? Generally it would be good to provide some intuitions for how well the findings from the inverse problem transfer to other settings of interest.
* All analysis seems to be only done on validation data - test data (especially in combination with re-training) would appear more appropriate.
* When not using harmful operations in the search space, DAS is worse than pure random sampling within the sequential search space.
* When using all operations in Table 2, there is only DAS and random sampling. Is it really the case there is no other simple baseline that could be evaluated here too?

Other questions:
* To what extent is including harmful operations in the search space realistic? It certainly makes sense as part of evaluation (the method should avoid those), but if there is not much hope the operations would help, then these normally would not be included in well-designed search space? If that is the case then a good ops search scenario is more realistic?
* What are the times to run the experiments? Information is provided in Table 1, but it is missing in Table 2. Also times to run HPO for DAS would be useful to know.
* Why are there 75 trials in Table 1 and 100 in Table 2?

review_summary: The paper raises interesting points about differentiable NAS methods, but there are questions about how these findings follow from the results provided. I’m happy to increase the rating and recommend acceptance if the discussion resolves my concerns.

actions_required_to_increase_overall_recommendation: Key areas that could improve the paper:
* Update how the results are reported in the tables - especially include stds or confidence intervals
* Do some experiments that would give us insights how the findings transfer from DAS for simple inverse problems to more complex and realistic problem settings
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper applied differentiable architecture search to inverse problems. The experiments showed problems of one-shot neural architecture search, including a large variance, the performance gap between supernet and final performance, and the importance of hyperparameters. The interesting point of this paper is the design of search spaces. Harmful operations are included to check if the search algorithm can avoid harmful operations. 
overall_review: [+] Add harmful operations to evaluate search algorithms. 

[-] I did not find new experimental results. Experiments are not complete and solid.
review_summary: This paper re-discovered problems of one-shot search. It is not necessary to use inverse problems to repeat these problems again. I hope the authors can solve these problems when they apply differentiable search to inverse problems. 
actions_required_to_increase_overall_recommendation: I do not have room to adjust my score. However, I am still happy to hear back from the authors and will reconsider it.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This work identifies the problem that differentiable architecture search (DAS) has mainly been studied in a very specific architectural search space for image classification. To understand how it realistically performs in a new domain, it is applied to simulated inverse problems with either a sequential or an interconnected structural space, either a good or a mixed-quality operator space, and various hyperparameter optimization strategies.
overall_review: **Positive aspects:**

* You did well to search across hyperparameter optimization settings, from hand-selected to various approaches of transferring hyperparameters or searching on just the search phase or also the retraining phase.

**Negative aspects:**

* What do the searched architectures actually look like? Only two such architectures are shown (and only in the supplementary material), but looking at trends of architectures across trials could help explain why DAS can't outperform baselines when only good operations are used. For example, what do the architectural parameter patterns look like over the course of search? 

* Why do you think random search and random sampling got the same max scores for all combinations of data generation methods and operation spaces? This seems like a red flag and should be explained, in addition to clarifying the difference between these two random baselines. From my understanding, random sampling selects and trains a single model per trial, while random search uses the same compute time as DAS to repeatedly train random architectures?

* In Figure 3, the outliers seem to greatly affect the regression line and correlation score: maybe also add the line and score for the non-degenerate data points (or at least those without a degenerate search model score, since these models could in theory be pruned out)?

* Make sure to include some measure of variance in your result tables. These can be qualitatively estimated from the plots, but standard practices in DAS are to report maximum, mean, and standard error of performance.

* What is the scale and intuitive meaning of PSNR? This measure is not often used in AutoML papers, so describing it would help the audience interpret the results.

* The only datasets are single dimensional and generated. The results would be strengthened by also using a more realistic dataset. 

* Computational cost is only mentioned in Table 1; total costs are not reported (although we can guess they are pretty minimal based on Table 1). 75 to 100 repetitions seems rather excessive: generally for AutoML 5-10 runs (that only differ by random seed) suffice, so you could reduce the number of repetitions in order to perform more variations of experiments, such as another dataset as suggested previously.



**Minor comments:**

* Line 83: "smoothen" -> "smooth"

* Line 169: add a paragraph break between "...beneficial by design." and "The first benign..."

* Line 186: add a paragraph break between "...convolution layer." and "These two layers..."

* FIgures 1 and 2 seem to have been stretched horizontally. It would be better to regenerate these plots with the desired aspect ratio so that it is easier to read.

* Table 3 caption "hyperparameter." -> "hyperparameter sets."
review_summary: This paper provides a good preliminary study on how DAS may realistically perform in a new domain not as extensively studied as image classification. However, the provided studies are only conducted on generated toy datasets, and there are few novel claims regarding problems with DAS beyond those already identified for the image classification domain in other DAS works.

Post-rebuttal update: I have raised my score from 5 to 6.
actions_required_to_increase_overall_recommendation: * Extend the analysis of already-run experiments, including computation cost and analysis of DAS architectures
* Include a realistic dataset
review_title: null

final_score: rating: 9: Strong Accept: Technically flawless paper with major impact and strong evaluation, with no obvious flaws. Should be highlighted in the program.
review_content: summary_of_contributions: The paper presents a methodical evaluation of the Differentiable Architecture search (DAS). It uses a simple 1-D inverse problem as a tractable and unbiased problem for the evaluation of the effectiveness of DAS as an AutoML technique and also specifically as a  one-shot solution. The paper presents empirical studies which provide practically useful insights into DAS as a generic AutoML technique.
It uses Sequential and non-sequential search spaces to further understand the impact of search space design.
Paper reveals the effectiveness of DAS in general and its limitation as single-shot solution based on empirical evidence. 
overall_review: It is a well-written paper on a very important topic. There are many AutoML techniques currently available for NAS problems, with no clear choices for specific problems. This paper attempts at understanding the issues for DAS algorithms at fundamental level. More of such studies (in breadth and scale) are needed to make progress in the field.
review_summary: -do-
actions_required_to_increase_overall_recommendation: Scale the problem to at least MNIST level image datasets.
review_title: null

Decision: Accept(Workshop)