# VANDOR: Mitigating SEUs into Quantized Neural Networks

Wilfread Guillemé, Angeliki Kritikakou, Youri Helen, Cedric Killian, Daniel Chillet

► To cite this version:

Wilfread Guillemé, Angeliki Kritikakou, Youri Helen, Cedric Killian, Daniel Chillet. VANDOR: Mitigating SEUs into Quantized Neural Networks. IOLTS 2024 - IEEE 30th International Symposium on On-Line Testing and Robust System Design, IRISA, Jul 2024, Rennes, France. pp.1-6, 10.1109/IOLTS60994.2024.10616081. hal-04689156

HAL Id: hal-04689156

https://inria.hal.science/hal-04689156v1

Submitted on 2 Oct 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# VANDOR: Mitigating SEUs into Quantized Neural Networks

Wilfread Guillemé*, Angeliki Kritikakou*§, Youri Helen†, Cédric Killian‡ and Daniel Chillet*

*Univ. Rennes, Inria/IRISA, §IUF, †DGA MI, ‡Univ. St-Etienne, Lab. Hubert Curien

**Abstract**—Embedded neural networks are increasingly deployed in critical applications, such as avionics and autonomous vehicle control. However, their reliability is challenged by various sources of soft errors, including radiation-induced faults from cosmic ray strikes, leading to Single Event Upsets (SEUs). To ensure the reliability of such systems, we present a novel hardware-based fault protection strategy tailored for embedded neural networks. The idea is based on mitigating faults by adapting at run-time any erroneous values (parameters, intermediate data) due to SEU towards zero upon fault detection. As neural networks exhibit heterogeneous sensitivity to fault direction, our hardware-based approach triplicates the sign bit (TMR) and uses a Voter block based on logical AND/OR gates to handle fault directionality. Through a comprehensive and exhaustive fault injection study, conducted on a Convolutional Neural Network (CNN) model, implemented on FPGA using fixed-point quantization, we show that our method is applicable to various hardware architectures while optimizing hardware cost, a crucial aspect in the context of embedded systems. Obtained results show that VANDOR protection efficiency is near 90.97% for the LeNet-5 CNN inference model, suitable for an embedded system. Additionally, it significantly reduces area overhead compared to existing approaches.

**Index Terms**—FPGA, Mitigation, Neural Network, SEU, TMR.

## I. INTRODUCTION

Convolutional Neural Networks (CNN) are widely used algorithms for image or sound classification. These networks are being more and more used in today's systems, especially for autonomous vehicle navigation and avionics applications. These domains imply various constraints in the system design, such as energy consumption or processing speed. Note that, Neural Networks (NNs) size has significantly increased in recent years and massive flows of data occur during inference. This evolution has led to the design of dedicated hardware architectures. Hardware accelerators over FPGA chips are often considered promising to implement these applications. FPGAs make it possible to parallelize calculations, being suitable for these algorithms. For instance, the Eyeriss [chen2019eyeriss] accelerator employs a systolic array [wei2017automated], while the FINN [blott2018finn] framework generates dataflow and multilayer offload architectures. However, the required hardware resources should remain reasonably low to avoid high energy costs. Therefore, quantifying parameters and data is applied to reduce the material cost of these models. For instance, the Brevitas [brevitas] framework offer the possibility to train a model taking into account the quantification. When comes to protection of these models from hardware faults,

fault tolerant mechanisms should be designed that minimizing the hardware overhead.

Faults can occur due to environmental conditions, for example high temperature, high energy electromagnetic radiation or cosmic rays. The most problematic cases are when the sequential elements are impacted by one or more bit-flips, called Single-Event Upsets (SEUs) and Multiple-Bits Upsets (MBUs). This is due to the fact that sequential elements need a larger hardware area than the combinational elements [xiong2024evaluation]. Note that, sequential elements for CNNs instantiated on FPGA can contain the parameters (weights, biases) of the model, but also the intermediate data between layers. A fault on these elements can make the model to manipulate erroneous data, and may change its output/classification, which can modify the behavior of the critical application, potentially leading to catastrophic consequences. Existing hardening techniques for NN are generally based on hardware or information redundancy on sensitive sequential elements, using conventional methods like Triple-Modular Redundancy (TMR) [libano2018selective], Dual-Modular Redundancy (DMR) [ramin2016] and Error Correcting code (ECC) [traiola2023hardnning].

While the majority of existing approaches consider SEUs occurring only to the parameters [Ruo20], our analysis provides a study considering the intermediate data and exploring the fault impact taking into account the direction of bit change due to faults (fault directionality). This reveals that SEUs that move the faulty value closer to zero have less impact on the output of a CNN. Based on this observation, we propose a hardware mechanism based on a selective fault tolerant technique that forces the faulty values to zero, instead of correcting them. Our hardware-based approach, named VANDOR, involves to triplicate the sign bit with a classical voter block. Others bits are protected by duplication associated with a lightweight Voter block, based on logical AND/OR gates. This strategy allows to handle fault directionality. To evaluate our proposal, we consider a LeNet-5 [lecun1998gradient] CNN model with a 8-bit signed fixed point format for parameters and intermediates data. The obtained results show that VANDOR achieves a global protection efficiency of 90.97% for this use-case.

The rest of this paper is organised as follows: Section ?? presents the state-of-the-art regarding CNN fault analysis and protection. Section ?? provides a comprehensive study of SEU impact over a quantized CNN. Section ?? presents our proposed VANDOR hardware protection approach. Then, results are presented in Section ?? . Finally, Section ?? discusses a
selective bit-accurate protection method using VANDOR.

## II. RELATED WORK

Several existing works address the reliability of NN systems through fault analysis and protection. Regarding fault analysis, the majority of existing works focus on faults occurring on the weight parameters [Ruo20] [bosio2019reliability]. However, intermediate data also needs to be studied, as they are temporarily stored, especially for architectures that execute NN models layer-by-layer [chen2019eyeriss]. The fault analysis in [Gua17] is conducted for various NNs and data formats, including both fixed-point and floating-point representations. For the fixed-point arithmetic, it is shown that the more sensitive bits are the most significant ones. Authors also propose to selectively harden the sequential elements according to their sensitivity as observed. Given that the analysis through fault injection based on simulation can consume a significant amount of time, the use of statistical fault injection is proposed in [ruospo2023assessing]. These methods illustrate that, by injecting approximately 1% of potential faults for CNNs, a reliability estimation that closely aligns with exhaustive results can be obtained by mainly focusing on sensitive CNN data. Despite some observations regarding asymmetries related to SEU fault directionality [zhang2022estimating], there is a lack of a comprehensive analysis of the impact of SEU fault directionality on NN classifications. Our fault injection framework aims to quantify the degree of fault directionality on a CNN output classification, in order to see if protection based on these observations is possible.

Regarding fault protection, typical approaches are based on hardware [libano2018selective] [Gua17] or information redundancy [traiola2023hardnning]. Redundancy is generally achieved by identifying the neurons that are more sensitive and duplicating only the essential elements [Bae22]. Alternative methods enhance the robustness of AI models by minimizing the impact of hardware faults on CNN operations. ApFT [10224787] tackles fault tolerance using a systolic array dataflow architecture, achieving this by duplicating convolution filters while considering the architectural characteristics. MOZART+ [burel2022mozart+] is an hardware protection technique tailored specifically for CNNs. It operates within the context of output stationary dataflow architectures embedded within systolic arrays. A multiplexer mechanism is engineered to systematically deactivate processing elements vulnerable to faults, ensuring robust operation in the face of hardware anomalies. FT-ClipAct [9116571] employs a threshold mechanism to restrict excessively high weight values, in response to faults affecting the memory where the network weights are stored.

Based on findings from other studies, we provide a comprehensive study regarding fault analysis not only in weights, but also on intermediate data, considering an 8-bit fixed point arithmetic within the Brevitas framework. We explored the SEU impact considering the direction of bit change. Revealing that SEUs which cause the faulty value to be closer to zero have a smaller impact on classification results. This information is used to guide the design of the proposed hardware-based fault protection technique, which relies on triplicating

<table><caption>TABLE I: Details of the use-case LeNet-5 model</caption><thead><tr><th>Layer type</th><th># Data</th><th>Data shape</th><th># Parameters</th></tr></thead><tbody><tr><td>Image</td><td>784</td><td>[28, 28]</td><td>0</td></tr><tr><td rowspan="3">CONV-1</td><td>Conv-1</td><td>4704</td><td>[6, 28, 28]</td><td>150</td></tr><tr><td>ReLu-2</td><td>4704</td><td>[6, 28, 28]</td><td>0</td></tr><tr><td>Pool-3</td><td>1176</td><td>[6, 14, 14]</td><td>0</td></tr><tr><td rowspan="3">CONV-2</td><td>Conv-4</td><td>1600</td><td>[16, 10, 10]</td><td>2400</td></tr><tr><td>ReLu-5</td><td>1600</td><td>[16, 10, 10]</td><td>0</td></tr><tr><td>Pool-6</td><td>400</td><td>[16, 5, 5]</td><td>0</td></tr><tr><td rowspan="3">FC-1</td><td>Flat-7</td><td>400</td><td>[400]</td><td>0</td></tr><tr><td>Lin-8</td><td>120</td><td>[120]</td><td>48000</td></tr><tr><td>ReLu-9</td><td>120</td><td>[120]</td><td>0</td></tr><tr><td rowspan="2">FC-2</td><td>Lin-10</td><td>84</td><td>[84]</td><td>10080</td></tr><tr><td>ReLu-11</td><td>84</td><td>[84]</td><td>0</td></tr><tr><td>FC-3</td><td>Lin-12</td><td>10</td><td>[10]</td><td>840</td></tr><tr><td>Output</td><td>Total</td><td>15786</td><td></td><td>61470</td></tr></tbody></table>

**Algorithm 1** Pseudo-code of the SEU injection campaign.

1: images = MNIST

2: layers = {layer}

3: channels = {channel}

4: rows = {row}

5: columns = {column}

6: bits = {bit}

7: for img in images do

8:   golden_result = ref_model(img)

9:   for l in layers do

10:  for ch in channels do

11:  for r in rows do

12:  for col in columns do

13:  for b in bits do

14:  position = (l, ch, r, col, b)

15:  inject_SEU(faulty_model, position)

16:  faulty_result = faulty_model(img)

17:  if faulty_result ≠ golden_result then

18:  sensitivity(position) + +

the sign bit and selectively duplicating sequential elements for other bits. A lightweight voter block to mitigate the SEU impact in then applied. Resulting in low hardware overhead compared to others methods.

## III. SEU SENSITIVITY STUDY

In this section, we perform a fault analysis using as a case study a quantized LeNet-5 CNN model. The fault model is SEUs, which can occur on parameters and intermediate data between layers. Our analysis takes into account the impact of the fault directionality on the data.

### A. Use-case CNN model

The targeted model is a LeNet-5 CNN trained with the handwritten digits MNIST database. Table ?? describes its characteristics. The targeted strucure consists of two convolutional (CONV) layers followed by three Fully Connected (FC) layers, where the model's weights are employed. After each layer, an activation function is applied, such as Rectified Linear Unit (ReLU). MaxPooling layers are used to reduce the data shape after each convolutional layer. Finally, the class with the
highest value at the FC-3 layer's output represents the NN's classification.

Considering that the CNN is executed on an embedded system, the model is quantized using 8-bit fixed-point format with the Brevitas framework. This approach utilizes a Scale Factor (SF) to map values to their corresponding signed integer representation ranging from -128 to +127, instantiated on the targeted FPGA. In our case, the tool sets a SF value equal to 0.0078125, which means a 7-bit comma shift to the left (as $-log_2(SF) = 7$). Training with Quantization-Aware-Training (QAT) involves optimizing the network weights, while considering its impact on the final model's accuracy, which achieved a precision of 97.43% on the test dataset.

## B. Fault injection framework

We consider SEUs occurring during inference and inject faults on the parameters and on the intermediate data transferred between layers. Algorithm ?? illustrates the exhaustive injection performed on the targeted CNN. The `inject_SEU()` function simulates a bit-flip at a desired location on an `int8` datatype. Sign and value of the selected bit prior to fault injection are very important in our case. Indeed, this information allows our analysis to determine whether the erroneous weight data/value is moving away from or towards zero. For instance, let's consider the value 1100 0011₂ (-61) for a data. As the sign is equal to '1', the value in this example is negative. Assume that an SEU happens at the seventh bit, which is has a value of '1' before the bit flip. The faulty value will be 1000 0011₂(-125), which is further from zero compared to the original value. Negative value mean that we must therefore count in 2's complement.

Regarding fault injection locations, for faults in weights, only convolutional and fully connected layers are subjected to inject SEUs, as these are the layers that exclusively contain these parameters. As soon as the fault is injected, the model is executed with these new weights by updating the PyTorch dictionary object that maps each layer to its parameters. Fault injection on intermediate data is performed for all layers. In this case, all quantized data is previously saved in a list of QuantTensor by modifying the CNN forward method. Then, the targeted model is executed by calling modified forward methods that execute the inference model only after the faulty layer which also increases simulation speed.

As presented in Algorithm ??, an exhaustive fault injection campaign is performed to evaluate the protection efficiency of the proposed VANDOR method. In total, more than 25 million single bitlips were injected on the CNN parameters (61470 weights × 8 bits × 52 MNIST images) and 46 million (370 MNIST images) on the intermediate data.

# IV. VANDOR PROPOSED PROTECTION APPROACH

The main idea of VANDOR hardware mechanism is to move the faulty value closer to zero. The design of this approach is inspired by other works [DAC2024], where it is shown that a CNN is more resilient to SEUs that move the faulty value closer to zero. This protection strategy is confirmed by the comprehensive analysis mentioned later in section ??.

<table><caption>TABLE II: VANDOR truth table.</caption><thead><tr><td>Sign</td><td>Sensitive bit</td><td>Duplicated bit</td><td>VANDOR</td></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>0</td></tr><tr><td>0</td><td>0</td><td>1</td><td>0</td></tr><tr><td>0</td><td>1</td><td>0</td><td>0</td></tr><tr><td>0</td><td>1</td><td>1</td><td>1</td></tr><tr><td>1</td><td>0</td><td>0</td><td>0</td></tr><tr><td>1</td><td>0</td><td>1</td><td>1</td></tr><tr><td>1</td><td>1</td><td>0</td><td>1</td></tr><tr><td>1</td><td>1</td><td>1</td><td>1</td></tr></tbody></table>

Note that, VANDOR's main idea can be implemented on different architectures and technologies, and thus, we presented it in a general way.

As illustrated in Figure ??, to design VANDOR, the hardware mechanism aims to triplicate the sign bit of weight or data from a CNN. Note that, this sign bit value is very important, and thus, it is necessary to triplicate it. With an incorrect sign value, the voter will do the opposite by moving the value away from zero, and thus, no fault mitigation occurs.

For other D flip-flops containing sensitive bits, they are duplicated. A different stored value between the sensitive bit and the duplicated one indicates that there is a fault. Decision is extended with a Voter block using logical AND/OR gates (VANDOR) to handle fault directionality. The voter is composed of 2 OR and 2 AND gates. Which is less than the classical voter block used for TMR. Depending on the sign value, it brings the data closer to zero. As shown in the truth Table ??, if the value is positive (Sign = '0'), the voter block will vote in favour of the '0' logic state, acting like a simple AND gate. Conservely, if the value is negative (Sign = '1'), the vote will be in favour of the '1' logic state to bring the erroneous value towards zero due to the 2's complement, acting like an OR gate.

As the VANDOR strategy is light-weight, the hardware overhead is then reduced. In this paper, the resource consumption is computed as the number of transistors used for each element such as AND/OR gates and D flip-flop. Figure ?? shows on the X-axis the additional hardware overhead necessary for the protection for a 8-bit word, compared against TMR and normalized to an unprotected 8-bit data. For the sign protection (1-bit), VANDOR has the same hardware overhead to TMR. However, for the remaining bits, the proposed strategy costs less than TMR. VANDOR only duplicate the sensitive sequential elements resulting in a hardware overhead of only one D flip-flop. The proposed voter block is also smaller than the TMR one. Regarding the critical path, VANDOR and TMR are identical with 3 gates (one AND gate and two OR gates) for each voter block. Note that, for an FPGA implementation, there is no difference between the voter block of TMR and VANDOR, since they are using a LookUp Table (LUT) each.

Protection performance of TMR is always 100% in case of SEU, as this method is able to correct the error with the majority voter. VANDOR chooses the best statistical case, i.e., the one that has less chance of causing a misclassification of the neural network. This solution does not provide a full protection in all cases, but remains relevant given to the large amount of data to be protected. Additionally, CNNs are
<figure><img src="image_2.png" /><figcaption>Fig. 1: VANDOR hardware implementation.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2: Errors in weight parameters.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3: Errors in data.</figcaption></figure>

inherently fault-tolerant.

## V. ANALYSIS RESULTS

This section presents obtained results of the exhaustive faults injection performed on the use-case CNN presented in section ?? . A separate analysis is then conducted, distinguishing between weights and intermediate data, as depicted in Figure ?? , where two histograms are presented: one for the parameters and the other for the intermediate data. Finally, we will perform an identical study on the 8-bit data format according to the position of bit-flips by accumulating all the obtained errors.

We can categorise the faults in two classes, based on the SEU directionality. Mitigated faults are faults that result in weight or data values that are larger in absolute terms. These faults are masked by VANDOR's design, which forces the weight value towards zero when detected. In all figures of this analysis, a percentage rate is identified to visualize the protection effectiveness offered by the VANDOR strategy at

<figure><img src="image_5.png" /><figcaption>Fig. 4: Histograms for weights and data.</figcaption></figure>

an accurate layer or bit position. Uncorrected faults are the ones that VANDOR cannot mitigate. This happens when CNN misclassification is caused by an SEU pushing a value away from zero. These errors are represented by darker colors in the associated figures.
<figure><img src="image_6.png" /><figcaption>Fig. 5: Errors in the 8-bit format.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6: Bit-accurate selective protection.</figcaption></figure>

## A. Weights

Figure ?? presents an histogram of all 8-bit possible values for the weights. Note that, occurrence is shown in logarithmic scale. The majority of weights values is around zero and no extreme values exist. Neural networks generally operate with values close to zero. Which may explain why they tolerate faults exhibiting this behavior.

All faults injected leading to misclassification are presented in Figure ?? . The X-axis represents the position of the layers and the Y-axis corresponds to the number of faults leading to erroneous CNN output. The first observation is that layers have different sensitivities to faults, especially for convolution layers. A bit-flip injected into the first weight layers leads to error propagation, which is inline with [Gua17]. Note that, these are also the convolutional layers that have the fewest parameters, as shown in Table ?? . Therefore, an injected fault on this layers will have a greater impact. Moreover, these filter weights are reused several times when performing calculations.

The global protection performance on weights for this quantized CNN is 94.56%. This varies depending on the layers, from 79.61% up to 95.93% for the first linear layer.

## B. Intermediates data

Figure ?? shows all possible values for intermediate data between layers. Similar to the weights, it is shown in logarithmic scale as the majority of values is around zero. Note that, intermediate data values exhibit larger spreading compared to the weights. A significant amount of data corresponds to extreme values, saturated by the quantization provided by the Brevitas framework during training-aware quantization. Furthermore, a difference in the appearance of negative and positive values is visible. This is due to the ReLU activation function, which forces negative values to zero and does not change positive values. Figure ?? depicts the fault sensitivity of the layers, where we also observe differences. However, the impact of single bit-flip in intermediate data is different from bit-flips in the weights. We observe that the first layers are

quite resilient. The closer the fault is to the CNN output, the higher the probability of causing misclassification.

VANDOR behavior can mitigate faults to 81.98% on average, up to 88.93% for the first convolutional layer. Although this layer appears resilient to faults, it is also the largest layer in terms of data volume, so it can cost the most on hardware protection cost.

## C. Bits

Our analysis also explores the impact of the fault's position for the 8-bit fixed point format as presented in Figure ?? . As expected, the MSBs are the most sensitive ones, and thus, they need to be protected in priority. Overall, VANDOR approach is very efficient for protecting the first sign bit under single-bit flips. However, the proposed approach is less efficient in protecting the remaining sensitive bits.

# VI. VANDOR BIT-ACCURATE PROTECTION

The proposed approach has on average no impact on the CNN classification output, as presented in previous sections. Its implementation is simple and inexpensive.

Let's imagine a case where the resource is limited and it should not exceed 2.5x to store all types of data compared to a baseline CNN. We can refer to Figure ?? which illustrate the enhancement in fault resilience observed in the use-case CNN in contrast to the hardware resource overhead due to protection. Results are presented as follows, "1 bit" means protection of the most significant bit only, "2 bits" means the sign bits and the second high order bit are protected. Up to "8 bits", which means all bits are protected. Evaluation take into account the appearance of faults within the data based on the protection performance associated with their position. With the TMR strategy, the best option is to protect the first three most significant bits. Concerning the VANDOR method, protection on the first fourth bits can be applied as described in Figure ?? . In addition to being slightly less expensive than TMR, VANDOR's protection of these first four bits provides better fault tolerance for the quantized CNN. It is really
interesting to observe that it is more advantageous to protect
a maximum of bits by the proposed method rather than to
protect fewer bits by triplication.

VII. CONCLUSION

In this article, we conducted a comprehensive analysis of SEU affecting a quantized LeNet-5 model. This study shows that there is a sensitivity difference against faults among layers of this use-case CNN. The obtained results shows that CNNs are more resilient to faults moving towards zero. Based on this observation, we proposed an hardware-based fault mitigation. VANDOR, which first, triplicate the sign bit. Then, others sequential elements containing sensitive bits are duplicated. Finally, a Voter block, based on logical AND/OR gates, to handle fault directionality is applied. Based on the comprehensive study results, it show that VANDOR protection has on average performance of 90.97%. Our proposed method protection provides better reliability for larger layers. The key advantage of this method is its low implementation cost in embedded systems context. As an example, we proposed a bit-accurate protection. It show that it is more interesting to protect weight parameters and data by VANDOR rather than by TMR with equivalent hardware resources.

The fault analysis is limited by simulation time. To over-
come this problem, a statistical fault injection which mainly
targeting CNN sensitive data could be used. In order to
evaluate the VANDOR method for larger models. Our pro-
posed method can be further enhanced with TMR in order to
improve the reliability of various neural networks. Protection
performance against multi-bit upsets could also be studied.
Indeed, these events are becoming more and more frequent due
to technological developments and the shrinking of transistors.

ACKNOWLEDGMENTS

This work is supported by the Direction Générale de
l'Armement (DGA), funded by the Defense Innovation
Agency (AID) and the National Institute for Research in
Digital Science and Technology (INRIA).