A Novel Security Threat Model for Automated AI Accelerator Generation Platforms

Authors: Chao Guo, Jinjun Xiong, Youhua Shi
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: AI accelerator generation platforms, Design Space Exploration (DSE), Hardware Trojan (HT), Security threat model

Abstract:
In recent years, the design of Artificial Intelligence (AI) accelerators has gradually shifted from focusing solely on standalone accelerator hardware to considering the entire system, giving rise to a new AI accelerator design paradigm that emphasizes full-stack integration. Systems designed based on this paradigm offer a user-friendly, end-to-end solution for deploying pre-trained models. While previous studies have identified vulnerabilities in individual hardware components or models, the security of this paradigm has not yet been thoroughly evaluated. This work, from an attacker's perspective, proposes a threat model based on this paradigm and reveals the potential security vulnerabilities of systems by embedding malicious code in the design flow, highlighting the necessity for protection to address this security gap. In exploration and generation, maliciously leverage the exploration unit to identify sensitive parameters in the model's intermediate layers and insert hardware Trojan (HT) into the accelerator. In execution, malicious information is concealed within the control instructions, triggering the HT. Experimental results demonstrate that the proposed method, which manipulates sensitive parameters in a few selected kernels across the middle convolutional layers, successfully misclassifies input images into specified categories with high misclassification rates across various models: 97.3% in YOLOv8 by modifying only three parameters per layer in three layers, 99.2% in ResNet-18 by altering four parameters per layer in three layers and 98.1% for VGG-16 by changing seven parameters per layer in four layers. Additionally, the area overhead introduced by the proposed HT occupies no more than 0.34% of the total design while maintaining near-original performance as in uncompromised designs, which clearly illustrates the concealment of the proposed security threat.

Decision: reject
Reviewer scores: 5, 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1729989735915

thread_type:
official_review

<thread_object>
creator: reviewer_vwag

<content>
summary:
This paper presents a new threat model for AI accelerators. The proposed attack includes a complete pipeline that first finds out the sensitive parameters in the user-provided model and then embeds malicious code using bit flipping. The embedded Hardware Trojan (HT) is then triggered using information hidden in communication instructions. Evaluation results on three vision models show that the proposed attack can obtain a high attack success rate.

strengths:
This paper has the following strengths:
+ The authors identify the vulnerability of AI accelerators in the context of a complete supply chain, starting from the pre-trained model from user to accelerator generation and then to deployment. 
+ The authors decompose the problem of HT insertion into multiple sub-problems, i.e., locating the sensitive kernels, and bit-level perturbation. The attack is also designed with the goal of ensuring stealthiness and minimal hardware overhead. 
+ Empirical results show that the proposed attack can achieve a high attack success rate with very few bit flips.

weaknesses:
The paper has the following weaknesses:
- The preliminary section does not provide any information about the prior works on bit-flipping attacks. The lack of discussion makes it hard to understand the real innovation of this work since previous attacks have explored how to identify the sensitive kernels in the pre-trained quantized model (e.g., ProFlip). 
- The proposed attack method is different from the previous model-level bit-flipping attacks since this paper is concerned with the hardware implementation of HT insertion. It's not clear why we cannot directly apply the existing bit-flipping attacks on AI accelerators. The challenges need to be clarified and empirical results can be used to support this. 
- The design of Cross-layer Sensitive Filter Exploration (C-SFE) is not clear. The authors mentioned in Section 3.4 that 'CSFE expands the decision boundary of a targeted category by flipping a few bits in one kernel per layer'. However, the paper does not explain whether different layers are equally important and whether the attack can exploit a single layer only.

questions:
Please consider addressing the weakness points above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730309974272

thread_type:
official_review

<thread_object>
creator: reviewer_y7ko

<content>
summary:
This paper attacks an AI accelerator generation platform using a unique threat model and adopting a genetic algorithm. The attack is evaluated across benchmark DNN models.

strengths:
The paper explores a unique direction in exploiting security threats in automated AI accelerator platforms. The research direction is exploratory in nature and innovative.

weaknesses:
Overall, the paper is very hard to follow, and the threat needs to be more explicit and clarify the assumptions. Since this is a unique direction, the authors should try to establish the threat model first. The summary of the weakness:

1. The threat model is vague and unclear on some assumptions. For example, sections 3.1 and 3.2 are confusing regarding what is the goal of the attacker. Is it a hardware trojan attack or an adversarial weight attack? Both terms are used. However, the threat model assumption is not equivalent for both cases. Moreover, there is discussion on how the attacker would gain access to model parameters, but hardly any discussion on how they will modify the parameters. The only discussion in the paper states in line 194, but how an attacker would insert this malicious code. 

2. The most concerning part is it is not clear whether the attacker can access the training stage or not. If not, then how can they modify model parameters? In the model production stage and inference stage where exactly the attacker is located.

3. Some of assumptions are also not well established and needs clarified. For example line 265 "Avoid attacking the model’s first layer an the final classification layer, as attacks on these two layers can have a significant impact on the model and are therefore more
 likely to be closely scrutinized by security engineers." I doubt if that is the principle machine learning engineers follow; even if that is the case, how do they scrutinize these layers? There is no standard list of model parameters.

4. Attack algorithm is naively adopting a genetic algorithm. Eqn. 1 is very similar to computing gradient, since the attacker can access model parameters why not compute the gradient. The authors claim their method does not need gradient computation, more justification is needed why GA would be faster than computing gradients.

5. Figure 7 is confusing; the description is not clear on the conclusion. Both attacks target the middle three convolution layers, but what does that mean from an attack efficacy and efficiency perspective?

6. The figure-2 needs to improve, the text are hard to read on part (b).

questions:
Overall, I commend the authors for exploring a unique security problem. However, the paper in its current form requires a lot of improvement before being ready for publication. Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648892744

thread_type:
official_review

<thread_object>
creator: reviewer_ukbm

<content>
summary:
In this paper, the authors propose a threat model in which the accelerator provider acts as an attacker by injecting hardware Trojans (HTs) through an EDA tool into the accelerator to enable misclassification attacks in Deep Neural Network (DNN) inference. They categorize these attacks into three types: 1-to-1, N-to-1, and others.

strengths:
The paper assumes the EDA tool developer of the AI accelerator platform as the adversary, which is a compelling area of research.

weaknesses:
The paper assumes that the EDA tool developer of the AI accelerator platform acts as the adversary, which is a compelling area of research. However, it is essential to demonstrate that the added HTs are challenging to detect by verification tools. In reality, substantial work has been done on HT detection and hardware formal verification. That said, the paper does not discuss whether existing HT detectors [1-5],  could identify the injected Trojans, including such an analysis would enhance the credibility of the claims.

[1] Bhunia, Swarup, Michael S. Hsiao, Mainak Banga, and Seetharam Narasimhan. "Hardware Trojan attacks: Threat analysis and countermeasures." Proceedings of the IEEE 102, no. 8 (2014): 1229-1247.

[2] Gubbi, Kevin Immanuel, Banafsheh Saber Latibari, Anirudh Srikanth, Tyler Sheaves, Sayed Arash Beheshti-Shirazi, Sai Manoj PD, Satareh Rafatirad, Avesta Sasan, Houman Homayoun, and Soheil Salehi. "Hardware trojan detection using machine learning: A tutorial." ACM Transactions on Embedded Computing Systems 22, no. 3 (2023): 1-26.

[3] Chakraborty, Rajat Subhra, Francis Wolff, Somnath Paul, Christos Papachristou, and Swarup Bhunia. "MERO: A statistical approach for hardware Trojan detection." In International Workshop on Cryptographic Hardware and Embedded Systems, pp. 396-410. Berlin, Heidelberg: Springer Berlin Heidelberg, 2009.

[4] Lyu, Yangdi, and Prabhat Mishra. "Scalable activation of rare triggers in hardware trojans by repeated maximal clique sampling." IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 40, no. 7 (2020): 1287-1300.

[5] Saha, Sayandeep, Rajat Subhra Chakraborty, Srinivasa Shashank Nuthakki, Anshul, and Debdeep Mukhopadhyay. "Improved test pattern generation for hardware trojan detection using genetic algorithm and boolean satisfiability." In Cryptographic Hardware and Embedded Systems--CHES 2015: 17th International Workshop, Saint-Malo, France, September 13-16, 2015, Proceedings 17, pp. 577-596. Springer Berlin Heidelberg, 2015.

Designing ASICs follows strict standards. SkyNet and similar projects rely on HLS rather than ASIC processes, which may facilitate deployment on FPGA but does not reflect the final circuit. For automatically inserted Trojans, can they pass through functional verification steps?

For IoT applications, it would be more appropriate to use platforms like the Ultra96 (as used in SkyNet) or KRIA KR260, which are MPSoC-based devices commonly utilized in IoT contexts.

It appears that the LoopConv module was mapped to LUTs, yet Fig. 6 does not clearly show the specific size and location. Based on Table 2, using the default optimization requires approximately 1,000 additional LUTs. This amount seems quite large for HT injection. 

The authors inject a Trojan into the EDA tool, potentially enabling triggers or malicious optimization on the hardware. I believe this point should be emphasized, with more detail on the significance and application value of such EDA tools. However, this brings up a broader issue: an EDA tool superior to others would likely dominate the market, making it challenging for a provider to practically act as an attacker.

questions:
Could the authors explain why they chose the U50 platform (a high-end, expensive database acceleration card) despite targeting IoT applications?

For inserted Trojans, can they pass through functional verification steps?

Could you provide component consumption under a fair comparison scheme, such as comparing a clean accelerator vs an HT-injected accelerator, both using Vivado's Area Optimization mode, Default mode, latency optimization, and so on?

Could the authors discuss how realistic their threat model is?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660774177

thread_type:
official_review

<thread_object>
creator: reviewer_8b4e

<content>
summary:
This paper presents a novel threat model targeting automated AI accelerator generation platforms. It says that full-stack integration in the design process of accelerators has become common. In this work, it explores how manipulating the model parameters will result in misclassification introducing a method called Cross-layer Sensitive Filter Exploration (C-SFE). To achieve this, in this threat model the attacker has to be active in three components of the platform; design space exploration (DSE), software stack, and hardware generation. In DSE, the attacker finds the parameters to be targeted. In hardware generation, HT is inserted to change the targeted parameters. In the software stack, there is an interface injection to trigger the HT. The results show high misclassification rates for YOLOv8, ResNet-18, and VGG-16. The area overhead of the HT is 0.34% of the total design.

strengths:
1. The authors say that the security aspect of accelerator generation platforms is largely unexplored, and this is one of the initial works done to explore security vulnerabilities.

2. The proposed HT introduces a very low area overhead (0.34%) making it hard to detect before runtime.

3. The paper presents multiple evaluations to justify their claims; attack performance, # of parameters vs classification rate, layout design with HT, resource utilization, etc.

weaknesses:
1. The evaluations of this threat model are limited to Gemmini. This can be further extended to other platforms such as GeneSys.

2. Although the area overhead of the accelerator is very low, it would have been better to have runtime performance numbers compared to the benign platform to see whether there is a significant difference in the inference pipeline. Further, would there be a larger difference when the test dataset size increases?

3. The discussion section (4.4) of the paper lacks depth. Perhaps to justify the findings of this section, it would be better to include them in the appendix.

4. The paper mentions minimizing the exploration time during DSE. However, no clear numbers were presented how this would change for being and malicious platforms. 

5. In some written sections, the use of the word "parameter" becomes a bit confusing whether it is the model parameters of DSE parameters (ex: line 190).

6. Minor issue: double quotations were not properly handled. (line: 394)

7. Y axises of Figure 6 are not named.

questions:
1. Would a simple validation of model parameters at the end of execution, along with the classification results, show that the parameters have changed? This is considering that given a trained model, we are aware of its model parameters.

2. Instead of carefully selecting which parameters to change, how would the results look like if parameters were changed randomly? I think this should be a baseline to be compared with. It is important to see whether a complex technique is required.

3. Why different numbers of kernels and parameters were selected for various models? Keeping them consistent across different models would be a good comparison to do.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732497607321

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We extend our heartfelt thanks to all the reviewers for their valuable insights and constructive feedback, which have been instrumental in enhancing and refining our approach. In the following responses, we address each reviewer’s individual questions and concerns. Additionally, we provide a summary of the revisions made to the manuscript.

## Summary of Paper Revisions
- Enhanced Figure 2 by enlarging the text in part (b) to improve its readability.
- Refactored line 190 to avoid confusion with the term "parameter."
- Revised Figure 7 by replacing "bits" with "parameters," correcting an error in our manuscript.
- Added an Appendix that includes:
  1. A simplified attack process (A.1).
  2. A comparison of hardware resource usage between designs with hardware Trojans (HT) and clean designs under various optimization strategies (A.2).
  3. An explanation of why existing adversarial weight attack methods are unsuitable for HT designs (A.3).
  4. A comprehensive evaluation of the proposed threat model's stealthiness (A.4).
- In the anonymous GitHub link, test code for random attacks has been added, along with TCL files for building projects on Xilinx Kintex-7 series chips and Verilog files containing malicious code.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a new threat model targeting automated AI accelerator generation platforms. The proposed attack includes a pipeline that first finds out the sensitive parameters in the targeted model provided by the user and then embeds malicious code using bit flipping. The embedded Hardware Trojan (HT) is then triggered using information hidden in the communication instructions. 
All the reviewers agree that the proposed threat model is novel and interesting. However, several reviewers are concerned about the clarity of the paper, especially on the description of the threat model, including the ability of the attacker and the practicality of some assumptions. Reviewers are also skeptical about whether the inserted trojan can bypass the detection of existing verification tools.