# On the Reliability Assessment of Artificial Neural Networks Running on AI-Oriented MPSoCs
## 3 Proposed Approach
### Ranking of the Criticality of Single Neurons
The highest absolute scores are relative to the most critical neurons for that given class. In more detail, the score map is represented as a list sorting the neurons from the highest to the lowest value. It is worth noting that the output is sampled for every neuron after the eventual batch normalization or activation function. ```
1\(N\leftarrow\) Total neurons;
2\(C\leftarrow\) Output classes;
3\(I_{i,\,i\in[0,C]}\leftarrow\) Inputs for a specific class;
4\(score_{k,\,k\in[0,N]}\leftarrow\) Score assigned to a neuron;
5\(y_{k,\,k\in[0,N]}\leftarrow\) Output value of a neuron;
6\(score_{k,\,k\in[0,N]}\gets 0\);
7 for each output class of the network c\(\in\)[0,C]do
8 new()
9 for each instance in the training dataset \(i\), i\(\in\)[0,I\({}_{c}\)]do
10 inference()
11 fore each neuron \(k\), k\(\in\)[0,N]do
12 if\(i=0\)then
13\(score_{k}\leftarrow\mid y_{k}\mid\)
14 else
15\(score_{k}\leftarrowscore_{k}+\mid y_{k}\mid\)
16
17 end if
18
19 end for
20 for each neuron \(k\), k\(\in\)[0,N]do
21\(score_{k}\leftarrowscore_{k}/I_{C}\)
22 end for
23 save()
24
25 end for
```  
**Algorithm 1**Assignment of resilience scores to individual neurons  
Starting from the classes' score maps, it is possible to extract a subset (\(t\)) of critical neurons in the form _className_critical_\(t\). The subset parameter (\(t\)) determines the amount of neurons that are considered critical and it is not a fixed value. Defining that number means tuning the reliability of a NCS: in other words, the larger the size, the larger the set of neurons that are considered critical. Next, in the network-oriented analysis (NoA) phase, we build a final score map where neurons are sorted based on the magnitude of their average contribution over the training set, without differentiating between the output classes. It is worth pointing out that both the score maps resulting from the NoA and the one from the CoA contain all the neurons of the neural network: only their values change, and consequently the ordering. To this end, Algorithm 1 is run again: line 7 is removed and line 9 is modified so that the inputs are picked up from the entire training data set. Then, since it might happen that the neurons that were found to be critical for individual classes take on a low value in the NoA, the outcome of this score map is updated considering the score maps resulting from the CoA. All neurons assuming a higher value in the class-oriented score map (given a subset parameter \(t\)) are overwritten. The set of critical neurons to take into account for the final network-oriented score map is computed by executing the union without repetitions of all the classes' score maps (\(C\)), as follows:  
\[critical\_t\leftarrow\bigcup_{i=1}^{C}c_{i\_critical\_t} \tag{3}\]  
As an example, the whole process is illustrated in Figure 2 for a generic neural network trained on MNIST. First, a percentage of critical neurons is selected (\(t\)) from the class-oriented score maps. Among these neurons, all those with a lower value in the network-oriented score map are overwritten with the highest value in the classes, i.e., the red squares, whereas neurons, such as Neuron 27 in Class 0, having a value lower in the \(t\%\) of the CoA, are not updated in the final score map. Interestingly, Neuron 653 in Class 9 assumes the lowest value in the network-oriented score map and, being part of the \(t\%\), is updated. In the end, as depicted in the right side of Figure 2, the final updated network-oriented score map is produced, where the per-class criticality is considered with a \(t\) factor. The larger \(t\) is, the more neurons are considered critical and therefore, strengthened in the final network-oriented score map. In Section 2.3, similar approaches developed to identify the network's critical neurons are described. As stated before, they are based only on a network-oriented analysis [17; 47; 48]. In this paper, we demonstrate that our methodology obtains a stronger analysis in terms of reliability.