# On the Reliability Assessment of Artificial Neural Networks Running on AI-Oriented MPSoCs
## 5 Experimental Results
### Ranking of the Criticality of Single Neurons
#### 5.1.1 Class-Oriented Analysis (CoA)  
Initially, each training data set was divided into subclasses, i.e., the number of outputs. More specifically, in our case study, we had ten outputs for all CNNs, but the same reasoning applies to a different number of output classes. Each sub-class contained only the images representing the selected output class. Hence, the proposed algorithm was executed to obtain the ten final score maps for each CNN. Each of them ordered the total neurons from the one activated with the highest average value to the one with the lowest (from the most critical to the least one) for that particular output class. Next, we performed software FI campaigns (i) to shed light on the importance of the class-oriented analysis, and (ii) to show that individual output classes hold different robustness levels with respect to errors. This certainly depends on the training phase and the structure of the data set that is used to train the network (typically, in the training set, training images are not evenly distributed among the output classes). We exploited the dropout probability fault model (_p-dropout_) in which a fraction of the neurons outputs is set to zero, and thus, their contribution is canceled. The same fixed amount of neuron outputs (_p_) was set to zero in two scenarios and, after the injection, the resulting accuracy of each CNN was measured by running the total test set of images (which was different from the training set used to gather the resilience scores). For each output class, in the first scenario (_Random_), neurons were randomly chosen from the class score map. In the second (_Critical_), the same number of neurons was neatly selected always starting from the top of the class score map, i.e., from the most critical neurons. As for the _Random_ scenario, since we relied on a random choice of neurons to kill, the experiments were repeated 1000 times (every time picking up different \(p\) random neurons); we report in Figures 4-6 the average percentage obtained through the experiments. The experiments were conducted for each output class of the targeted CNNs and, particularly, they were replicated for growing _p_-percentages: \(p\) equal to 0.1% (Figures 3(a), 4(a) and 5(a)); \(p\) equal to 0.5% (Figures 3(b), 4(b) and 5(b)); \(p\) equal to 1% (Figures 3(c), 4(c) and 5(c)); \(p\) equal to 1.5% (Figures 3(d), 4(d) and 5(d)). The experimental results for the three FI campaigns are reported in Figures 4-6. The scenario _Fault-free_ is the golden accuracy of the class and, as for the _Random_ and _Critical_ scenarios, it was computed by running only the inferences of the images belonging to the given output class. As shown, it is evident that random injections do not affect, or only to a negligible extent (when \(p\) gets bigger), the behavior of the neural network. Indeed, in all cases, the accuracy fluctuates around the _Fault-free_ one, apart from the third and fourth cases (_p_ = 1% and \(p\) = 1.5%) where it slightly decreases. This confirms the theory under which neural networks are equipped with more neurons than they need [4]. In fact, up to a certain point, they can obtain enough of some neurons and still work correctly. On the other hand, this is not confirmed in the _Critical_ scenario. The accuracy of the output classes considerably drops when killing the \(p\) highest neurons. Figure 4: MNIST LeNet: software fault injection campaigns on _random_ and _critical_ neurons. A fixed percentage \(p\) of neurons is dropped. Concerning MNIST LeNet, for \(p=0.1\%\) (Figure 3(a)), the maximum percentage variation from the _Fault-free_ accuracy to scenario _Critical_ is equal to \(6.13\%\) and corresponds to the last class (digit 9). Then, when killing \(p=0.5\%\) critical neurons (Figure 3(b)), the highest percentage variation drastically increases, reaching \(44.33\%\) for the second class (digit 1), where the CNN accuracy drops from _Fault-free_ 99.33% to \(54.54\%\).