# FIC 2021: Security challenge for embedded AI in IoT

Pierre-Alain Moellic

► To cite this version:

Pierre-Alain Moellic. FIC 2021: Security challenge for embedded AI in IoT. 2021. hal-04627073

HAL Id: hal-04627073

https://hal.science/hal-04627073

Submitted on 27 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# FIC 2021: Security challenge for embedded AI in IoT

Pierre-Alain Moellic

► To cite this version:

Pierre-Alain Moellic. FIC 2021: Security challenge for embedded AI in IoT. 2021. hal-04627073

HAL Id: hal-04627073

https://hal.science/hal-04627073v1

Submitted on 27 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
leti
cea tech

<figure><img src="image_2.png" /></figure>

DE LA RECHERCHE À L'INDUSTRIE

# Enjeux de sécurité lorsque l'IA s'insère dans les objets connectés

## Security Challenge for Embedded AI in IoT

Pierre-Alain MOELLIC

FIC 2021 // 08/09/2021
leti
cea tech

<figure><img src="image_2.png" /></figure>

DE LA RECHERCHE À L'INDUSTRIE

# Enjeux de sécurité lorsque l'IA s'insère dans les objets connectés

## Security Challenge for Embedded AI in IoT

Pierre-Alain MOELLIC

FIC 2021 // 08/09/2021
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

## Contents

* CONTEXT: EMBEDDED ARTIFICIAL INTELLIGENCE

* THREAT MODELS & ATTACK SURFACE

* PANORAMA OF ATTACKS

- INTEGRITY & AVAILABILITY

- CONFIDENTIALITY & PRIVACY

* WORK IN PROGRESS: HOW TO DEFEND?

<figure><img src="image_6.png" /></figure>
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

## Contents

* CONTEXT: EMBEDDED ARTIFICIAL INTELLIGENCE

* THREAT MODELS & ATTACK SURFACE

* PANORAMA OF ATTACKS

- INTEGRITY & AVAILABILITY

- CONFIDENTIALITY & PRIVACY

* WORK IN PROGRESS: HOW TO DEFEND?

<figure><img src="image_6.png" /></figure>
<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

CONTEXT: EMBEDDED AI
<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

CONTEXT: EMBEDDED AI
<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

* Machine Learning / Deep Learning / 
Transformer models...

• PERFORMANCE ERA

- MAJOR TREND: **Models Deployment**

• Ubiquitous AI

- Large variety of devices, many application domains

- **Critical tasks, critical data, critical devices... almost everywhere**

<figure><img src="image_16.png" /></figure>
<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

* Machine Learning / Deep Learning / Transformer models...

* PERFORMANCE ERA

* MAJOR TREND: **Models Deployment**

* Ubiquitous AI

* Large variety of devices, many application domains

* **Critical tasks, critical data, critical devices... almost everywhere**

<figure><img src="image_16.png" /></figure>
<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

* Models were designed thanks to and for high performance computers / architectures

* GPU + CPU

* High memory capacity

* Floating point computing

* NOW → Models must fit within constrained HW platforms

<figure><img src="image_20.png" /></figure>

* Model Optimization

* Model Compression

* ...

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>
<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

* Models were designed thanks to and for high performance computers / architectures

• GPU + CPU

- High memory capacity

• Floating point computing

- NOW → Models must fit within constrained HW platforms

<figure><img src="image_20.png" /></figure>

Model Optimization

Model Compression

...

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>
<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

* **Typical high-performance 32-bit microcontroller (MCU) for IoT**

* STM32H7

* Cortex M7, 480 MHz

* Memory: 2 Mbytes (Flash) + 512 Kbytes (SRAM)

* FPU

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /><figcaption>**Source** A. Capotondi and M. Rusci (University of Bologna)
https://github.com/EEESlab/mobilenet_v1_stm32_cmsis_nn</figcaption></figure>

<footnote>*MAC = Multiplication-Accumulation operation ($x \leftarrow x + w \cdot x$)</footnote>
<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

* Typical _high-performance_ 32-bit microcontroller (MCU) for IoT

* STM32H7

* Cortex M7, 480 MHz

* Memory: 2 Mbytes (Flash) + 512 Kbytes (SRAM)

* FPU

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /><figcaption>**Source** A. Capotondi and M. Rusci (University of Bologna)
https://github.com/EEESlab/mobilenet_v1_stm32_cmsis_nn</figcaption></figure>

<footnote>*MAC = Multiplication-Accumulation operation ($x \leftarrow x + w \cdot x$)</footnote>
<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

* For now, inference purpose only

* But... « edge training » is a short term reality

* Low bitwidth training procedure (IBM, NIPS 2020, training with only 4bit)

* SILICIUM WAR

* NVIDIA + ARM, Google TPU, ...

* GPU, CPU, MCU+CPU, GPU+CPU, FPGA, ...

* Every manufacturer praises « *AI applications* / *AI suitable devices*...

* MAJOR ISSUES TO THE DEVELOPMENT & DEPLOYMENT OF MODERN MODELS

* Trust // ... // Safety // **SECURITY**
<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

* For now, inference purpose only

* But... « edge training » is a short term reality

* Low bitwidth training procedure (IBM, NIPS 2020, training with only 4bit)

* SILICIUM WAR

* NVIDIA + ARM, Google TPU, ...

* GPU, CPU, MCU+CPU, GPU+CPU, FPGA, ...

* Every manufacturer praises « *AI applications* / *AI suitable devices*...

* MAJOR ISSUES TO THE DEVELOPMENT & DEPLOYMENT OF MODERN MODELS

* Trust // ... // Safety // **SECURITY**
<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

* Security of Machine Learning

* New ML topic

* → Growing scientific community

* Integrity & Availability

* Confidentiality & Privacy

* « Adversarial Machine Learning »

* « Popular » concerns:

* Adversarial Examples

* Model Extraction (reverse engineering)

* Many demonstrated (and upcoming...) threats

<figure><img src="image_35.png" /></figure>

Source Nicolas Carlini (collected with Arxiv)
<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

* Security of Machine Learning

* New ML topic

* → Growing scientific community

* Integrity & Availability

* Confidentiality & Privacy

* « Adversarial Machine Learning »

* « Popular » concerns:

* Adversarial Examples

* Model Extraction (reverse engineering)

* Many demonstrated (and upcoming...) threats

<figure><img src="image_36.png" /></figure>

Source Nicolas Carlini (collected with Arxiv)
<figure><img src="image_36.png" /></figure>

# THREAT MODELS & ATTACK SURFACE
<figure><img src="image_37.png" /></figure>

# THREAT MODELS & ATTACK SURFACE
<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

* Attack the (supervised) Machine Learning Pipeline

<figure><img src="image_40.png" /></figure>

* Attacks have been demonstrated **everywhere**:

* learning, inference, model, data...
<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

* Attack the (supervised) Machine Learning Pipeline

<figure><img src="image_41.png" /></figure>

* Attacks have been demonstrated **everywhere**:

* learning, inference, model, data...
<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

• GOAL ➔ CONFIDENTIALITY / INTEGRITY / AVAILABILITY

EXTRACT INFORMATION

Training / Inference data

Model

<figure><img src="image_44.png" /></figure>

FOOL A MODEL

The output prediction is not the
expected one (i.e. correctly learned)

MAKE THE ML-BASED SYSTEM USELESS

Quality and system performance requirements

Attack the environment

Strongly alter the overall performance of the model
<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

• GOAL ➔ CONFIDENTIALITY / INTEGRITY / AVAILABILITY

EXTRACT INFORMATION

Training / Inference data

Model

<figure><img src="image_45.png" /></figure>

FOOL A MODEL

The output prediction is not the
expected one (i.e. correctly learned)

MAKE THE ML-BASED SYSTEM USELESS

Quality and system performance requirements

Attack the environment

Strongly alter the overall performance of the model
<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

## KNOWLEDGE

* **WHITE-BOX** paradigm → powerful attacker with a full knowledge of the model's internals

* **BLACK-BOX** paradigm → can only interact with the model → exploit {input/output}

<figure><img src="image_48.png" /></figure>
<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

## KNOWLEDGE

* **WHITE-BOX** paradigm → powerful attacker with a full knowledge of the model's internals

* **BLACK-BOX** paradigm → can only interact with the model → exploit {input/output}

<figure><img src="image_49.png" /></figure>
<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

* CAPACITY

* Attack the **training** / **inference** / **both**

* Querying ability

* DEFENDER CAPACITY & REQUIREMENTS

* What to protect ?

* Defense budget

* Limitations due to SW/HW requirements

* Are pre-processing techniques possible ?

* Can I hide information ? (« *less is better* »)

* Model hardening ?
<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

* **CAPACITY**

* Attack the **training** / **inference** / **both**

* Querying ability

* **DEFENDER CAPACITY & REQUIREMENTS**

* What to protect ?

* Defense budget

* Limitations due to SW/HW requirements

* Are pre-processing techniques possible ?

* Can I hide information ? (« *less is better* »)

* Model hardening ?
<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

## SYSTEM PROPERTIES

* API-based system ≠ Sensor-based system

* Training and inference in the same system? Deployed-based model?

* Can I guarantee the security of the training system / environment?

* Other connected sub-systems vulnerabilities?

* Is my system easily « clonable »?

<figure><img src="image_55.png" /></figure>

Readings for a proper definition of a threat model (for adversarial robustness)

* F. Tramer, N. Carlini, W. Brendel, A. Madry: *On Adaptive Attacks to Adversarial Example Defenses*, NeurIPS, 2020.

* N. Carlini et al. *On Evaluating Adversarial Robustness*, arXiv, 2019.
<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

## SYSTEM PROPERTIES

* API-based system ≠ Sensor-based system

* Training and inference in the same system? Deployed-based model?

* Can I guarantee the security of the training system / environment?

* Other connected sub-systems vulnerabilities?

* Is my system easily « clonable »?

<figure><img src="image_56.png" /></figure>

Readings for a proper definition of a threat model (for adversarial robustness)

* F. Tramer, N. Carlini, W. Brendel, A. Madry: *On Adaptive Attacks to Adversarial Example Defenses*, NeurIPS, 2020.

* N. Carlini et al. *On Evaluating Adversarial Robustness*, arXiv, 2019.
<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

* Most of the literature is focused on model as an abstraction

* With EMBEDDED AI, the **ATTACK SURFACE IS TWOFOLD**

<figure><img src="image_59.png" /></figure>
<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

* Most of the literature is focused on model as an abstraction

* With EMBEDDED AI, the **ATTACK SURFACE IS TWOFOLD**

<figure><img src="image_60.png" /></figure>
<figure><img src="image_60.png" /></figure>

NIPS Challenge 2018

INTEGRITY & AVAILABILITY THREATS
<figure><img src="image_61.png" /></figure>

NIPS Challenge 2018

INTEGRITY & AVAILABILITY THREATS
<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

* **POISONING ATTACKS** = threaten the **learning** process

<figure><img src="image_64.png" /></figure>

* Control the behavior of a model by manipulating its training data

* Inject / Modify / Remove training samples → impact the decision boundary

* **Adversarial Objectives**

* Deteriorate the overall accuracy

* Target a specific behavior
<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

* **POISONING ATTACKS** = threaten the **learning** process

<figure><img src="image_65.png" /></figure>

* Control the behavior of a model by manipulating its training data

* Inject / Modify / Remove training samples → impact the decision boundary

* **Adversarial Objectives**

* Deteriorate the overall accuracy

* Target a specific behavior
<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

* Poisoning attacks

* TRIGGER-BASED ATTACKS

<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

Reading: A. Saha et al. *Hidden trigger backdoor attacks. AAAI 2020.*
<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

* Poisoning attacks

* TRIGGER-BASED ATTACKS

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

Reading: A. Saha et al. *Hidden trigger backdoor attacks. AAAI 2020.*
<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

* Poisoning attacks

* TRIGGER-LESS ATTACKS (aka features-based poisoning attacks)

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

Reading: A. Shafahi, et al. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NIPS 2018.
<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

* Poisoning attacks

* TRIGGER-LESS ATTACKS (aka features-based poisoning attacks)

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

Reading: A. Shafahi, et al. Poison frogs! targeted clean-label poisoning attacks on neural networks. In NIPS 2018.
<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

## ADVERSARIAL EXAMPLES ATTACKS = threaten the inference

<figure><img src="image_78.png" /></figure>

* Probably the most “popular” attack

* Already known for linear model, SVM... Demonstrated in a Deep Learning context in 2013

* Since then... impressive body of work in the AI literature (White-box **AND** black-box settings)

* A real « arm race » between attacks and defenses

<figure><img src="image_79.png" /></figure>

## Readings:

* C. Szegedy et al. Intriguing properties of neural networks. arXiv, 2013.

* A. Ilyas, Andrew, et al. Adversarial Examples Are Not Bugs, They Are Features. NIPS 2019.
<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

* **ADVERSARIAL EXAMPLES ATTACKS** = threaten the **inference**

<figure><img src="image_79.png" /></figure>

* Probably the most “popular” attack

* Already known for linear model, SVM... Demonstrated in a Deep Learning context in 2013

* Since then... impressive body of work in the AI literature (White-box **AND** black-box settings)

* A real « arm race » between attacks and defenses

<figure><img src="image_80.png" /></figure>

**Readings:**

* C. Szegedy et al. Intriguing properties of neural networks. arXiv, 2013.

* A. Ilyas, Andrew, et al. Adversarial Examples Are Not Bugs, They Are Features. NIPS 2019.
<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>

* Basic principle :
  alter a clean example to fool a model
  → false prediction

<figure><img src="image_83.png" /></figure>
* Basic principle :
alter a clean example to fool a model
→ false prediction

<figure><img src="image_81.png" /></figure>
## « Physical » adversarial examples [Kurakin, 2016]

* A. Athalye, et al. Synthesizing robust adversarial examples. ICML 2018

<figure><img src="image_84.png" /></figure>

■ classified as turtle

■ classified as rifle

■ classified as other

* K. Eykholt, et al. Robust physical-world attacks on deep learning visual classification. IEEE CVPR 2018

<figure><img src="image_85.png" /></figure>

Left: a true tagged road sign

Right: adversarial tagged road sign → misclassified as "Speed Limit 45"

* M. Sharif, et al. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. ACM CCS 2016

<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>

<figure><img src="image_88.png" /></figure>

White-box attack

Black-box attack on « Face++ »

Extension to Face Detection (invisibility attack)

<figure><img src="image_89.png" /></figure>
## « Physical » adversarial examples [Kurakin, 2016]

* A. Athalye, et al. Synthesizing robust adversarial examples. ICML 2018

<figure><img src="image_82.png" /></figure>

classified as turtle

classified as rifle

classified as other

* K. Eykholt, et al. Robust physical-world attacks on deep learning visual classification. IEEE CVPR 2018

<figure><img src="image_83.png" /></figure>

Left: a true tagged road sign

Right: adversarial tagged road sign → misclassified as "Speed Limit 45"

* M. Sharif, et al. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. ACM CCS 2016

<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>

White-box attack

Black-box attack on « Face++ »

Extension to Face Detection (invisibility attack)

<figure><img src="image_87.png" /></figure>
<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

<figure><img src="image_92.png" /></figure>

* Using Generative Models: highly concerning threats...

<figure><img src="image_93.png" /></figure>

<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /></figure>

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>

FGSM

<figure><img src="image_98.png" /></figure>

PGD

<figure><img src="image_99.png" /></figure>

Generative model from: Li D., et al. *Exploring Adversarial Fake Images on Face Manifold*. arXiv 2021

<figure><img src="image_100.png" /></figure>
<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

* Using Generative Models: highly concerning threats...

<figure><img src="image_91.png" /><figcaption>FGSM</figcaption></figure>

<figure><img src="image_92.png" /><figcaption>PGD</figcaption></figure>

Generative model from: Li D., et al. *Exploring Adversarial Fake Images on Face Manifold*. arXiv 2021

<figure><img src="image_93.png" /></figure>
<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

* **DEFINITION OF AVAILABILITY IS MORE COMPLEX...**

* MODEL as a part of AN OVERALL SYSTEM with PERFORMANCE and QUALITY requirements

* No clear frontier between INTEGRITY and AVAILABILITY

* Several attack vectors are possible:

* Classical availability attacks → DoS

* Generalized poisoning or adversarial examples attacks → availability issues

* SPONGE EXAMPLES : alter an input to increase the inference time

* Directly strike ML algorithm to make training impossible or add extra training time

* weight initialization, batching process...

<figure><img src="image_104.png" /></figure>

* Growing interest in the ML Security community
* **DEFINITION OF AVAILABILITY IS MORE COMPLEX...**

* MODEL as a part of AN OVERALL SYSTEM with PERFORMANCE and QUALITY requirements

* No clear frontier between INTEGRITY and AVAILABILITY

* Several attack vectors are possible:

* Classical availability attacks → DoS

* Generalized poisoning or adversarial examples attacks → availability issues

* SPONGE EXAMPLES : alter an input to increase the inference time

* Directly strike ML algorithm to make training impossible or add extra training time

* weight initialization, batching process...

<figure><img src="image_94.png" /></figure>

* Growing interest in the ML Security community
<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

* **FAULT INJECTION ANALYSIS (FIA)**

* New attack vectors to strike a model's integrity & availability

* X. Hou, J. Breier, D. Jap, L. Ma, S. Bhasin, and Y. Liu, *Security Evaluation of Deep Neural Network Resistance against Laser Fault Injection*. IPFA, 2020.

* **New type of adversarial attacks designed for embedded AI: **WEIGHT-BASED ADVERSARIAL ATTACKS**

* A. S. Rakin, Z. He, and D. Fan, "Bit-flip attack: Crushing neural network with progressive bit search," in *IEEE International Conference on Computer Vision*, 2019.

* 32bit embedded model are not safe. A single fault in the exponent bit can completely crash a ResNet...

* For 8-bit quantized models: Only #10 bit-flip can strongly alter the performance (= random prediction level)

* Exploiting memory flaws (fault injection, RowHammer...)

<figure><img src="image_108.png" /></figure>
<figure><img src="image_95.png" /></figure>

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>

* **FAULT INJECTION ANALYSIS (FIA)**

* New attack vectors to strike a model's integrity & availability

* X. Hou, J. Breier, D. Jap, L. Ma, S. Bhasin, and Y. Liu, *Security Evaluation of Deep Neural Network Resistance against Laser Fault Injection*. IPFA, 2020.

* **New type of adversarial attacks designed for embedded AI: **WEIGHT-BASED ADVERSARIAL ATTACKS**

* A. S. Rakin, Z. He, and D. Fan, "Bit-flip attack: Crushing neural network with progressive bit search," in *IEEE International Conference on Computer Vision*, 2019.

* 32bit embedded model are not safe. A single fault in the exponent bit can completely crash a ResNet...

* For 8-bit quantized models: Only #10 bit-flip can strongly alter the performance (= random prediction level)

* Exploiting memory flaws (fault injection, RowHammer...)

<figure><img src="image_98.png" /></figure>
* **TYPICAL FIA RESULTS ON A 32-bit microcontroller (CEA-LETI + Mines Saint-Etienne)**

* Bit-set attack on the Flash Memory (laser beam)

* Powerful attack vector (data, opcode, instruction skip...)

<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

● = effective bit-set
* **TYPICAL FIA RESULTS ON A 32-bit microcontroller (CEA-LETI + Mines Saint-Etienne)**

* Bit-set attack on the Flash Memory (laser beam)

* Powerful attack vector (data, opcode, instruction skip...)

<figure><img src="image_99.png" /></figure>

<figure><img src="image_100.png" /></figure>

● = effective bit-set
CONFIDENTIALITY & PRIVACY THREATS
CONFIDENTIALITY & PRIVACY THREATS
<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

* **A GROWING CONCERN**

* Very active scientific community with both consideration of software and hardware issues

* **Two major targets**

* THE **MODEL** (model are gold...)

* THE **DATA** (training and inference)

* (obviously) BLACK-BOX paradigm

<figure><img src="image_115.png" /></figure>
<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

* **A GROWING CONCERN**

* Very active scientific community with both consideration of software and hardware issues

* **Two major targets**

* THE **MODEL** (model are gold...)

* THE **DATA** (training and inference)

* (obviously) BLACK-BOX paradigm

<figure><img src="image_104.png" /></figure>
<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

* **TRAINING DATA LEAKAGE**

* Usually **data memorization** is related to overfitting issue

* Recent works show that models memorize data in the early step of the training

* Attacker can extract some (critical) patterns encapsulated within the model

* Mainly an issue for large-scale language model

* **Readings**

<figure><img src="image_119.png" /></figure>

* Carlini et al. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. USENIX Security 2019

* Carlini et al. Extracting training data from large language models, in 30th USENIX Security 2021.

* Target: Transformer model (GPT-2)
<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

* **TRAINING DATA LEAKAGE**

* Usually **data memorization** is related to overfitting issue

* Recent works show that models memorize data in the early step of the training

* Attacker can extract some (critical) patterns encapsulated within the model

* Mainly an issue for large-scale language model

* **Readings**

<figure><img src="image_108.png" /></figure>

* Carlini et al. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. USENIX Security 2019

* Carlini et al. Extracting training data from large language models, in 30th USENIX Security 2021.

* Target: Transformer model (GPT-2)
<figure><img src="image_120.png" /></figure>

<figure><img src="image_121.png" /></figure>

<figure><img src="image_122.png" /></figure>

* **MEMBERSHIP INFERENCE ATTACK (MIA)**

* Adversary's goal: *does this sample belong to the training set?*

* almost every works highlight the same outcome about membership inference:

* Generalization Gap is the key

* Too much gap between TRAIN and VALIDATION → MIA is possible

* **MODEL INVERSION**

* Inversion → infer information of the input (X) using the model's output (Y).

* Critical for applications using heterogeneous data

* X = {public data, private/critical data}
<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

## MEMBERSHIP INFERENCE ATTACK (MIA)

* Adversary's goal: *does this sample belong to the training set?*

* almost every works highlight the same outcome about membership inference:

* Generalization Gap is the key

* Too much gap between TRAIN and VALIDATION → MIA is possible

## MODEL INVERSION

* Inversion → infer information of the input (X) using the model's output (Y).

* Critical for applications using heterogeneous data

* X = {public data, private/critical data}
<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /></figure>

<figure><img src="image_125.png" /></figure>

* MODEL EXTRACTION (reverse engineering)

• IP theft

• Steal performance (a model is time & money)

• Shift black-box → white-box

• Milestone works [Jagielski, Carlini, 2020]

• Completely reverse a ReLu-based MLP by using chosen inputs

• 10⁶ queries to extract 100 000 weight values with 2⁻²⁵ precision (32bit)

• Use the fact that a ReLu activated neuron has a critical point (y=0 → y=x)

<figure><img src="image_126.png" /></figure>

Readings

• M. Jagielski, et al. High accuracy and high fidelity extraction of neural networks. USENIX Security Symposium, 2020.

• N. Carlini, et al. Cryptanalytic extraction of neural network models. In Annual International Cryptology Conference, 2020.
<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

* MODEL EXTRACTION (reverse engineering)

• IP theft

• Steal performance (a model is time & money)

• Shift black-box → white-box

* Milestone works [Jagielski, Carlini, 2020]

• Completely reverse a ReLu-based MLP by using chosen inputs

• 10⁶ queries to extract 100 000 weight values with 2⁻²⁵ precision (32bit)

• Use the fact that a ReLu activated neuron has a critical point (y=0 → y=x)

<figure><img src="image_115.png" /></figure>

Readings

• M. Jagielski, et al. High accuracy and high fidelity extraction of neural networks. USENIX Security Symposium, 2020.

• N. Carlini, et al. Cryptanalytic extraction of neural network models. In Annual International Cryptology Conference, 2020.
## MODEL EXTRACTION

* Important concept for the adversary's goals:

* ACCURACY (steal a **model** to perform better – without expensive engineering)

* FIDELITY → FUNCTIONAL EQUIVALENCE

<figure><img src="image_127.png" /></figure>

From [Jagielski 2020]

— Original model (target)

— Fidelity extraction

— Accuracy extraction

— Perform better than

— Functional equivalence → High-end fidelity extraction

— Make converge to
<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

## MODEL EXTRACTION

* Important concept for the adversary's goals:

* ACCURACY (steal a **model** to perform better – without expensive engineering)

* FIDELITY → FUNCTIONAL EQUIVALENCE

<figure><img src="image_119.png" /></figure>

From [Jagielski 2020]

*   Original model (target)
*   Fidelity extraction
*   Accuracy extraction
*   Perform better than
*   Functional equivalence → High-end fidelity extraction
*   Make → converge to
<figure><img src="image_128.png" /></figure>

<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

## SIDE-CHANNEL ANALYSIS

* Growing concern for model extraction

* L. Batina, et al. CSI-NN: Reverse engineering of neural network architectures through electromagnetic side channel. IEEE USENIX Security Symposium 2019.

<figure><img src="image_131.png" /></figure>

<figure><img src="image_132.png" /><figcaption>(a) Target 8-bit microcontroller me-(b) Langer RF-U 5-2 Near Field chanically decapsulated Electromagnetic passive Probe</figcaption></figure>

<figure><img src="image_133.png" /><figcaption>(c) The complete measurement setup</figcaption></figure>

<figure><img src="image_134.png" /><figcaption>(a) First byte mantissa for weight = 2.43</figcaption></figure>

<figure><img src="image_135.png" /><figcaption>(b) Second byte mantissa for weight = 2.43</figcaption></figure>

<figure><img src="image_136.png" /><figcaption>(c) Third byte mantissa for weight = 2.43</figcaption></figure>
<figure><img src="image_120.png" /></figure>

<figure><img src="image_121.png" /></figure>

<figure><img src="image_122.png" /></figure>

## SIDE-CHANNEL ANALYSIS

* Growing concern for model extraction

* L. Batina, et al. CSI-NN: Reverse engineering of neural network architectures through electromagnetic side channel. IEEE USENIX Security Symposium 2019.

<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /><figcaption>(a) Target 8-bit microcontroller me-(b) Langer RF-U 5-2 Near Field chanically decapsulated Electromagnetic passive Probe</figcaption></figure>

<figure><img src="image_125.png" /><figcaption>(c) The complete measurement setup</figcaption></figure>

<figure><img src="image_126.png" /><figcaption>(a) First byte mantissa for weight = 2.43</figcaption></figure>

<figure><img src="image_127.png" /><figcaption>(b) Second byte mantissa for weight = 2.43</figcaption></figure>

<figure><img src="image_128.png" /><figcaption>(c) Third byte mantissa for weight = 2.43</figcaption></figure>
<figure><img src="image_137.png" /></figure>

<figure><img src="image_138.png" /></figure>

leti
ceatech

NANOELEC.

WORK IN PROGRESS: HOW TO DEFEND?
<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

leti
ceatech

NANOELEC.

WORK IN PROGRESS: HOW TO DEFEND?
<figure><img src="image_139.png" /></figure>

<figure><img src="image_140.png" /></figure>

<figure><img src="image_141.png" /></figure>

* Confidentiality & Privacy preserving defense schemes

* Membership inference attacks

• Reducing generalization (→ increase test accuracy)

• Hiding prediction information

• Distillation method (Teacher / Student model, transfer learning)

• Differential Privacy → at the cost of decreasing the performance...

* Model extraction

• Attack detection (query-based attacks are not really discreet)

• Noise injection

• Label-only output

• Do not use ReLu ????!
<figure><img src="image_131.png" /></figure>

<figure><img src="image_132.png" /></figure>

<figure><img src="image_133.png" /></figure>

* Confidentiality & Privacy preserving defense schemes

* Membership inference attacks

• Reducing generalization (➔ increase test accuracy)

• Hiding prediction information

• Distillation method (Teacher / Student model, transfer learning)

• Differential Privacy ➔ at the cost of decreasing the performance...

* Model extraction

• Attack detection (query-based attacks are not really discreet)

• Noise injection

• Label-only output

• Do not use ReLu ????!
<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

* **Adversarial robustness**

* **Proactive defenses**

* Model Hardening: train a robust model against adversarial perturbation.

* State of the Art: **Adversarial Training**

* **Reactive defenses**

* Pre-processing

* Purify the perturbation or Reduce the impact

* Detection mechanisms.

* **Obfuscation defenses**

* Hide critical information

* Gradient masking → false sense of security (useless against an advanced attacker...)
<figure><img src="image_134.png" /></figure>

<figure><img src="image_135.png" /></figure>

<figure><img src="image_136.png" /></figure>

* **Adversarial robustness**

* **Proactive defenses**

* Model Hardening: train a robust model against adversarial perturbation.

* State of the Art: **Adversarial Training**

* **Reactive defenses**

* Pre-processing

* Purify the perturbation or Reduce the impact

* Detection mechanisms.

* **Obfuscation defenses**

* Hide critical information

* Gradient masking → false sense of security (useless against an advanced attacker...)
* **Adversarial Training**

<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /></figure>

Reading A. Madry, et al. "Towards deep learning models resistant to adversarial attacks," ICLR 2018.
<figure><img src="image_137.png" /></figure>

<figure><img src="image_138.png" /></figure>

<figure><img src="image_139.png" /></figure>

* **Adversarial Training**

<figure><img src="image_140.png" /></figure>

<figure><img src="image_141.png" /></figure>

Reading A. Madry, et al. "Towards deep learning models resistant to adversarial attacks," ICLR 2018.
* **PICTURE** [ANR AAPG 2020] [02/2021 → 08/2024]

* CEA-LETI + Mines Saint-Etienne + STMicroelectronics + IDEMIA

* How to protect (software) embedded AI against algorithmic AND physical threats ?

<figure><img src="image_147.png" /></figure>

MINES
Saint-Etienne

ST
life.augmented

>IDEMIA
augmented identity

<figure><img src="image_148.png" /></figure>

<figure><img src="image_149.png" /></figure>

PUBLIC Deliverable:
Security of Software Embedded Neural Network Models: State
of the Art & Threat Modelling

[v1.0, 01/09/21, 64 pages]

https://picture-anr.cea.fr/dissemination

https://picture-anr.cea.fr

<figure><img src="image_150.png" /></figure>

@AnrPicture

<figure><img src="image_151.png" /></figure>

https://www.linkedin.com/in/pierre-alain-moellic-7a31687/
<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

* **PICTURE** [ANR AAPG 2020] [02/2021 → 08/2024]

* CEA-LETI + Mines Saint-Etienne + STMicroelectronics + IDEMIA

* How to protect (software) embedded AI against algorithmic AND physical threats ?

<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /></figure>

<figure><img src="image_147.png" /></figure>

<figure><img src="image_148.png" /></figure>

<figure><img src="image_149.png" /></figure>

<figure><img src="image_150.png" /></figure>

https://picture-anr.cea.fr

<figure><img src="image_151.png" /></figure>

@AnrPicture

<figure><img src="image_152.png" /></figure>

https://www.linkedin.com/in/pierre-alain-moellic-7a31687/

PUBLIC Deliverable:
Security of Software Embedded Neural Network Models: State of the Art & Threat Modelling

[v1.0, 01/09/21, 64 pages]

https://picture-anr.cea.fr/dissemination
* **EU ECSEL INSECTT : AI + IOT**

* Wireless Intelligent Secure Trustable Things: bringing IoT and AI together

* https://www.insectt.eu/

* 50 partners / 13 countries

* **French consortium : CEA-LETI / AKEOPLUS / STMicroelectronics / IDEMIA**

* Large-scale use cases

* **IRT NANOELEC : PULSE PROGRAM (Digital Trust)**

* Cybersecurity of home healthcare

<figure><img src="image_152.png" /></figure>

Inria

<figure><img src="image_153.png" /></figure>

<figure><img src="image_154.png" /></figure>

<figure><img src="image_155.png" /></figure>

<figure><img src="image_156.png" /></figure>

life.augmented

<figure><img src="image_157.png" /></figure>

IDEMIA
augmented identity

<figure><img src="image_158.png" /></figure>
<figure><img src="image_153.png" /></figure>

<figure><img src="image_154.png" /></figure>

<figure><img src="image_155.png" /></figure>

* **EU ECSEL INSECTT : AI + IOT**

* Wireless Intelligent Secure Trustable Things: bringing IoT and AI together

* https://www.insectt.eu/

* 50 partners / 13 countries

* **French consortium : CEA-LETI / AKEOPLUS / STMicroelectronics / IDEMIA**

* Large-scale use cases

* **IRT NANOELEC : PULSE PROGRAM (Digital Trust)**

* Cybersecurity of home healthcare

<figure><img src="image_156.png" /></figure>

Inria

<figure><img src="image_157.png" /></figure>

leti
ceatech

<figure><img src="image_158.png" /></figure>

ST

life.augmented

<figure><img src="image_159.png" /></figure>

IDEMIA
augmented identity

<figure><img src="image_160.png" /></figure>
<figure><img src="image_159.png" /></figure>

<figure><img src="image_160.png" /></figure>

<figure><img src="image_161.png" /></figure>

## PUBLICATIONS

* Impact of low-bitwidth quantization on the adversarial robustness for embedded neural networks, R. Bernhard, P-A. Moellic et al. IEEE International Conference on Cyberworlds 2019

* Luring of transferable adversarial perturbations in the black-box paradigm, R. Bernhard, P-A. Moellic et al. IEEE IJCNN 2021

* Impact of Spatial Frequency Based Constraints on Adversarial Robustness, R. Bernhard, P-A. Moellic et al. IEEE IJCNN 2021

* A Review of Confidentiality Threats Against Embedded Neural Network Models, R. Joud, P-A, Moellic, et al. IEEE 7th World Forum on Internet of Things

* An Overview of Laser Injection against Embedded Neural Network Models M. Dumont, P-A, Moellic, et al. IEEE 7th World Forum on Internet of Things

<figure><img src="image_162.png" /></figure>
<figure><img src="image_161.png" /></figure>

<figure><img src="image_162.png" /></figure>

<figure><img src="image_163.png" /></figure>

## PUBLICATIONS

* Impact of low-bitwidth quantization on the adversarial robustness for embedded neural networks, R. Bernhard, P-A. Moellic et al. IEEE International Conference on Cyberworlds 2019

* Luring of transferable adversarial perturbations in the black-box paradigm, R. Bernhard, P-A. Moellic et al. IEEE IJCNN 2021

* Impact of Spatial Frequency Based Constraints on Adversarial Robustness, R. Bernhard, P-A. Moellic et al. IEEE IJCNN 2021

* A Review of Confidentiality Threats Against Embedded Neural Network Models, R. Joud, P-A, Moellic, et al. IEEE 7th World Forum on Internet of Things

* An Overview of Laser Injection against Embedded Neural Network Models M. Dumont, P-A, Moellic, et al. IEEE 7th World Forum on Internet of Things

<figure><img src="image_164.png" /></figure>
# QUESTIONS...
QUESTIONS...