Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an end-to-end framework which dynamically detects saliency of input data, reweights data using saliency maps, and searches architectures on saliency-reweighted data. The proposed framework is based on four-level optimization, which performs four learning stages in a unified way. Experiments on several datasets demonstrate the effectiveness of the proposed framework.
strengths_and_weaknesses: This paper is interesting. It tries to address the limitation in existing NAS methods which treat all data elements as being equally important. Experiments show that the proposed NAS method achieves good performance.

At the end of the introduction part, the third contributions should be removed, which is a common part of a scientific paper rather than a special contribution.

There should be some ablation studies to validate the effectiveness of various designs (like the four stages) in the proposed method. This is important for better understanding the proposed method.

questions: Please see the above commons.
limitations: Please see the above commons.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Saliency-Aware Neural Architecture Search presents a four-step neural architecture search (NAS) procedure that reweights inputs based on their saliency value:
1. The original architecture is trained on the original dataset.
2. Using the trained model, an adversarial saliency method generates saliency maps for all inputs. The features of the inputs are reweighted based on their saliency value.
3. The original architecture is retrained on the saliency weighted inputs.
4. The model trained on the saliency weighted inputs is evaluated on the validation dataset. A new loss-minimizing architecture is selected using a traditional differentiable NAS method (e.g., DARTS).

Each of the four steps is repeated until convergence. 

The saliency-aware neural architecture search is evaluated on image classification tasks using CIFAR-10, CIFAR-100, and ImageNet and on text classification tasks using GLUE. The proposed method is combined with other differentiable NAS methods like DARTS-2nd, PC-DARTS, PR-DARTS, and P-DARTS. It is evaluated against non-NAS models, NAS-selected models, and other saliency-based training procedures to understand changes in performance, training time, and robustness. The saliency results are visualized and evaluated based on their alignment with human expectations.
strengths_and_weaknesses: **Strengths**
* *Evaluative methods* ---  The paper does a comprehensive evaluation of the SANAS method. They evaluate text and image modalities, various NAS methods, NAS and non-NAS models, and related saliency-based training algorithms. In addition, they include robustness tests and an ablation study on saliency reweighting.
* *Evaluative results* --- The results show that SANAS results in models with lower error rates while maintaining reasonable parameter and time costs. SANAS also results in better robustness to overfitting and prevents performance collapse. 
* *Clarity and Reproducibility* --- The paper and appendix are very well written and thorough. The method was easy to follow. All necessary details are included, including hyperparameter settings, model parameters, optimization algorithms, human study details, implementation details, and code packages for significant testing. It would be easy to reproduce.

**Weaknesses**
* *Limitations and societal implications* --- The limitations and societal implications section of this paper lacks detail. An expanded limitations section would help readers understand when to use SANAS over an alternative method and the specific benefits of SANAS.
* *Saliency evaluation* --- The paper evaluates its saliency maps by visualizing them to human users who judge their alignment with human expectations. The paper states that visualizing the saliency maps shows that the "method is effective in generating correct saliency maps." However, this evaluation assumes the model has learned the same features humans have (i.e., the saliency should highlight the object of interest). While ideally, the model would learn features that align with human expectations, research has shown that models often learn to rely on spurious features. Saliency methods should be faithful to the model's representations. If the model's representations do not align with a human's, then the saliency should not be sensible to a human interpreter. Other research has shown that sensible-looking saliency methods can mask the underlying model's behavior and inhibit proper interpretation of the results. The main contribution of the paper is not a new saliency method. Still, if the authors evaluate the saliency maps, they should use saliency map tests such as model parameter randomization tests or data randomization tests (see Sanity Checks for Saliency Maps by Adebayo et al.).
* *Qualitative saliency results* --- The text examples show an example text saliency map. One example is not sufficient to claim the SANAS method is better than EC and GMPGC. Please include additional examples in the appendix. 
* *Saliency methods* --- The paper only uses an adversarial saliency method in its framework. The experiments evaluate against other saliency-based training procedures (EC, CDEP, and GMPGC) that use different saliency methods. However, these procedures differ from SANAS because they add regularization terms to the loss function instead of reweighting the dataset. Comparing to other saliency methods would strengthen the paper. If you found similarly positive results, it would validate that the benefit of SANAS generalizes beyond a single saliency method. However, even if these methods do not work as well, it would be interesting scientifically and shed light on potential differences between adversarial and non-adversarial methods.

**Minor Comments**
* Table 1: please explain what the asterisks represent.
* Line 138: "bottom to up" --> "bottom-up"
* Line 71 and 253: please expand on the unreliability of GradCAM or cite prior work.

questions: 1. What would happen if you switched the adversarial saliency method in step 3 for other types of saliency methods like integrated gradients, SmoothGrad, GradCAM, LIME, Sufficient Input Subsets, etc.? Would SANAS apply to nondifferentiable NAS methods if you used a non-gradient-based saliency method like LIME or Sufficient Input Subsets?
2. How does the adversarial saliency method do on existing saliency map tests (see Sanity Checks for Saliency Maps by Adebayo et al.)? 
limitations: Can you please expand on other limitations of your method? Should I always use SANAS instead of a different differentiable NAS approach? 

For instance, there seems to be a time cost of SANAS (e.g., Table 3 lists 1.1 GPU days compared to P-DARTS 0.3 GPU days). How should someone trade off between time cost and potential performance benefit?

Section 5 mentions that SANAS can not easily be applied to nondifferentiable NAS methods. Why is this and what are the implications? Are there ways to use it with nondifferentiable NAS methods (e.g., if you used a non-gradient based saliency method like LIME)?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper leverages the classic differentiable architecture search strategy for neural architecture search through a complex optimization process that involves training that reweights architectural parameters and weights according to a strategy where pixel values are biased by a saliency score that derives from the degree of effect on accuracy subject to perturbation of pixel values.
strengths_and_weaknesses: Strengths:

1. The work successfully demonstrates that the proposed saliency measure in the context of the optimization carried out in [40] gives rise to improved neural architectures.

2. The optimization procedure, albeit complex, is formulated in a way that is relatively straightforward to optimize outside of the role that hyperparameter sensitivity may play in the process.

Weaknesses:

1. It's not clear to what degree the hyperparameter tuning is easy/difficulty to achieve although the ablation studies and extra analysis presented shows some confidence for general purpose use of this method.

2. Visualization of saliency maps in figure 2 could be more visible/clear.

3. The conclusion focuses too much on the limitations/weaknesses and should restate the central contributions.
questions: 1. For the four level optimization framework, this appears to be very complex but the cost values in the results tables suggests it is very efficient. Could the authors comment a bit more on what makes this approach so efficient given an apparently highly complex loss function?

2. In table 7, how does one judge which words should be deemed most salient? Anecdotally what is shown seems to make sense, but further clarity on how to judge this would be interesting.

3. The paper admits that for evolutionary or reinforcement based methods that the approach can't be used. However, is it possible to extend the basic notion of saliency that is presented here to be adapted to such a method? If not, what is the central limitation?
limitations: The authors have discussed both limitations and potential for negative societal impact which is minimal.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper unifies a four step optimization solution for obtaining a neural architecture, considering the salience of data. 
strengths_and_weaknesses: The paper is well written and provides a well articulated explanation of the solution and results. The paper provides sufficient amount of experiments that support the usefulness of the proposed approach. I do not find a particular weakness for this work.
questions: There is a relevant line of research in pruning of neural networks using explanations that maps to similar salience maps. It would be worth looking into that area as there seems a nice bridge between the two line of research. 
limitations: No specific limitation discussed
review_title: null

Meta-review: metareview: This paper proposed a novel method that reweights data using saliency maps and searches architecture using saliency-reweighted data. 

There are four official reviewers for this submission. The reviewers consistently agree with the novelty, presentation, and experimental validation of this submission. The ratings are: borderline accept/accept/weak accept. The concerns raised by the reviewers are well addressed during the rebuttal. 

Thus the AC would like to recommend acceptance. 


confidence: Certain
Decision: Accept