Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates using NAS to discover architectures that are easier to backdoor without requiring any poisoned training data (i.e., the setting is to just release the architecture and let the victim train it on clean data). A metric derived from NTK serves as the score, enabling efficient exploration of the search space. Experiments demonstrate that the approach can find architectures that are easier to backdoor across many datasets and that several existing defenses do not work (in part because the considered attacks are input-dependent).
strength_and_weaknesses: Strengths:
- The idea of the paper is interesting. It does seem plausible that NAS models will become more common in model sharing libraries, so preparing for this sort of attack vector seems prudent.
- The experiment in Figure 2 nicely motivates Algorithm 1. NTK is used in an interesting way.
- The results are promising and demonstrate that this is an interesting direction for future work.

Weaknesses:
- The ASR is a bit low, especially in the retraining from scratch experiments (the most realistic setting). However, the discovered architecture does still improve over the other architectures, so this isn't a big weakness.
- To what extent do the input-dependent triggers lead to the results on NC and STRIP? Other architectures are not shown, but they should be.

Questions:
- I'm confused about the experimental setup in figure 2. Is it the case that the trigger generators are trained on the datasets, and then the networks are fine-tuned on the same datasets?
clarity,_quality,_novelty_and_reproducibility: The writing is very clear, and the paper is structured well. The novelty is more than sufficient. Reproducibility is sufficient.
summary_of_the_review: The weaknesses are fairly minor. The main result is that this works at all, and this is a good contribution, so I think the paper should be accepted. I am giving the paper a 6 for now, because 7 isn't an option, but I don't feel confident enough to give it an 8. I may raise the score to a 7 after author feedback as long as no significant issues are raised by other reviewers.

-----------
Update after rebuttal:

The authors have addressed my concerns. I still recommend acceptance, but 7 still isn't an option and I still don't feel confident enough to give the paper an 8. I hope the AC can take this into account.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes one new way to conduct back door attacks: through neural architecture search to find architectures that are vulnerable to certain triggers. The proposed EVAS is unaware of training data and model parameters so naturally evades defenses that rely on training data filtering or model parameters inspection.
strength_and_weaknesses: Strength:
1. I think this paper may be valuable to the community.  Utilizing NAS as a way to launch attacks is quite interesting and the authors properly explain this by architecture-level “shortcuts” that recognize trigger patterns.
2. The paper is well-motivated and clearly written.  Experiments and ablation studies are well performed. 

Weakness:
1. The right figure of Figure 2 is mysterious to me. It seems that the right figure indicates that the model accuracy arises as the conditional number of NTK arises, which contradicts the result in [1]. I understand this is the NTK w.r.t trigger generator parameters instead of the model parameters but still, I would like to see some more discussion on this phenomenon. BTW., I recommend reporting Pearson correlation between ASR., Acc. and the conditional number of the NTK. 
2. It's also unclear why the architectures found on cifar-10 work well on multiple datasets. This is an interesting observation. More discussion on why there exists universal vulnerable architecture would further strengthen the paper. 
3. The ingredients of EVAS are not that novel. The input-aware triggers, NTK-based zero-cost proxy as well as the searching algorithms are all proposed in previous works. The fact that certain architectures are more vulnerable is also elaborated in [2]. Nevertheless, this paper made good efforts to combine them all. 
4. It is not clear to me if the parameters in g(.) will be trained.  Will a trained g(.) be used to launch attack?

References:
[1] W. Chen et.al. NEURAL ARCHITECTURE SEARCH ON IMAGENET IN FOUR GPU HOURS: A THEORETICALLY INSPIRED PERSPECTIVE. ICLR’21.
[2] M. Bober-Irizar et.al. Architectural Backdoors in Neural Networks. arXiv preprint: arXiv.2206.07840

clarity,_quality,_novelty_and_reproducibility: The paper is well-written, and the proposed method is somewhat novel.
summary_of_the_review: The paper is interesting, a new attack approach: backdoor in the model architecture is proposed. I am not an expert in attacking and defending. It seems that the paper is a natural extension of backdoors attacks from data, model parameters to model architectures.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper deals with using NAS to search for attackable (image) classification architectures/models and attack generators. The generated attacks have to be image-dependent to make it difficult to fix them. To make this problem tracktable, the authors propose a proxy metric which does not require cumbersome training, building on the neural tangent kernel. The approach builds on the NATS-Bench search space and is evaluated for CIFAR10, CIFAR100, Imagenet. For the latter, the authors use an exemplary architecture which was found during their search process. The results focus on attack success rate (ASR) and clean data accuracy (ACC) for the found models, where a superior ASR with acceptable ACC is reported. The authors also report feature attributions and hypotheses for the properties of the attackable models. Lastly, the authors also show that the found model is sufficiently robust against available established defense mechanisms.
strength_and_weaknesses: Strengths:
* Novel and interesting direction of research using NAS to search for attackable models and respective generators
* Elegant and sensible approach to make the search problem tractable via proxy metrics
* Insightful discussion of the properties of the derived, representative model

Weaknesses:
* Only single result is evaluated and discussed. Do other models with close fitness reach similar quality? It would also be helpful to show a high-level result of the search process with selected (actual fitted/evaluated) results for ACC/ASR to further signal that the proxy metric and overall hypothesis of the approach are working
* The parameters of the search process are missing + how long was it evaluated for?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper clearly defines the goals, State-of-the-Art, challenges for finding suitable models and generators, as well as the taken approach. The evaluation is also well-documented for the chosen architecture, although it remains unclear if the same properties hold for others
Quality: The taken approach is well described and is adequate for reaching the search goal. It could be better argued for why, for example, regularized evolution or neural tangent kernel are chosen in particular. The class of approaches is clearly good, but the distinct argumentation of design decisions could be improved.
Novelty: The approach of automatically searching for attackable architectures and models is novel in that there only exist work on manual crafting of attackable architectures using trigger detectors.
Reproducibility: The authors report parameter configurations for the model architectures, but do not provide code. Reproducibility could be achieved for triggering the search process, but as now overview of the search results is given, it is hard to achieve actuallyf finding the used architecture.
summary_of_the_review: The paper proposes an interesting and well-motivated research direction with sensible approach. The used components (regularized evolution, proxy metrics, fitness function) are clearly described, but the paper could improve the distinct motivations and design decisions for the particular search algorithm and proxy metric. A weakness in the current version of the paper is the lack of high-level results for search process experiments to prove the claim that proxy metric, search process, baseline architectures and learning objective generally work well together. In addition, I am wondering if multi-objective experimental could be fruitful to analyse and show what sacrifices we still need to make to get high attackable models (e.g., a potential lack of clean data accuracy or less interpretability). Lastly, a minor improvement would be to reference recent applications of the used search algorithm, which should be quite close to regularized evolution (at least it is referred to as such in recent ML applications).

## Post author response
I want to thank the authors for their answers to the questions and for their added material. After reading the responses as well as the other reviews, I am updating more score above the acceptancce threshold. The new material adds more details about the search method and the used parameters, and shows the dynamics of the search process.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed an attack exploitable and vulnerable arch search (EVAS). EVAS search for an architecture that is more vulnerable to backdoor triggers. By using NTK, EVAS can perform a training-free search. Also, by using a dynamic trigger generator, the proposed method can generate sample-specific triggers. Empirically, the proposed method demonstrated it is effectiveness. The overall attack framework is new and demonstrates that the backdoor vulnerability could also exist in the architecture. 
strength_and_weaknesses: ---

Strength 
- A new backdoor attack that exploits DNN architecture results are surprisingly effective. This could open up another direction for the backdoor attacks. 
- Using NTK that search without training makes this method practical. The relationship between the conditional number and ASR clearly demonstrated the vulnerability in architecture. Empirical results comparing randomly generated arches and manually designed ones confirms the discovered vulnerability.
- EVAS shows that attackers could operate in a black box setting without access to the model parameters. This further confirms that the vulnerability is from the architecture. 
- Most of the different existing backdoor defense types have been examined, showing that this attack can evade certain backdoor defenses.  
- Vulnerability from the architecture perspective could offer insights for future works on attacks and defenses. Certain architectures are easy/hard to attack or defend. The overall presentation is straightforward. Good work.

---
Weaknesses
- The practical setting is that the victim does not have the computational resource to perform NAS and ask the adversary to provide NAS as a service. In such cases, performance could be the main interest. Based on Figures 2 and 5, it seems like a low-performance arch could also have a low conditional number. Does that have a higher ASR? An analysis of the relationship between the ASR, clean accuracy, and the conditional number would make this paper more comprehensive. I think this part is currently unclear. 
- How does the $\lambda$ affect the performance? 
- Based on the experiments, I think the attacker needs access to the training data (or data with a similar distribution) of the victim models to train the generator. However, the threat model claims that victims could train on arbitrary data. This should be further clarified, either in the threat model or in the experiments. If the victim model can use arbitrary data, the model generator for one dataset should also be able to attack the victim model trained on a different dataset. 

clarity,_quality,_novelty_and_reproducibility: Clarity: good \
Quality: good \
Novelty: good \
Reproducibility: this might be challenging without the open-source code. 

summary_of_the_review: It is a very interesting paper that exposes the backdoor vulnerability in the architecture. Despite some unclear issues mentioned in the weakness part, the merit and contribution out-weight the weakness. I would recommend accepting.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper demonstrates that NAS methods are strong enough to find architectures with backdoors using input-aware triggers. 
The paper was borderline, and as a result a Zoom meeting was held between AC and reviewers to discuss pros and cons. 
All reviewers found the problem setup and approach novel and converged on acceptance, except one serious concern: reviewer JhVa explicitly inquired about source code, but the authors did not reply to this, and as such, the reviewers feared that the results might be cherry-picked and wouldn't be possible to reproduce. This is particularly important since the field of NAS used to have a lot of problems with reproducibility (see, e.g., "Best Practices for Scientific Research on Neural Architecture Search", JMLR 2020, https://jmlr.org/papers/volume21/20-056/20-056.pdf) but recently improved these practices a lot. Since the paper is very novel and opens up many opportunities for follow-ups I do recommend acceptance, but I strongly recommend the authors to release their code before ICLR to maximize their impact and allow this paper to act as a foundation for future work on the topic. The authors also promised to do so in a message to the reviewers and AC on OpenReview.
justification_for_why_not_higher_score: Inavailability of code / potential cherry-picking of results.
justification_for_why_not_lower_score: Novel approach that could inspire a lot of follow-up work.
Decision: Accept: poster