# On the Reliability Assessment of Artificial Neural Networks Running on AI-Oriented MPSoCs
## 5 Experimental Results
### Mapping and Optimal Scheduling
The outcome of the optimizer was the optimal scheduling shown in Table 3. As illustrated, the optimizer sorts the chunks so that the cumulative variance assigned to each PE during the whole inference cycle is uniformly distributed. Better solutions are not consistent with the integer constraints, which are crucial to comply with (7) and (9). Hence, the kernel of the PULP-NN library was changed to match the optimal scheduling order provided by the ILP solver. As discussed before, the chunks assignments to different PEs do not affect at all the final classification results. The same set of permanent faults was injected in two scenarios: first, when the LeNet CNN application was compiled by following a static scheduling; then, when it was compiled with the proposed optimal scheduling. A total of 164,000 RTL injections were performed. The same set of 2050 stuck-at-faults were injected in the cluster domain of the GAP-8 RTL design (as shown in Figure 9). Specifically, the injection targeted the register file of the cluster's PEs. The injection procedure was the following. A set of 40 images was randomly selected from the MNIST test set. \begin{table}
\begin{tabular}{l r r r r r r r r r} \hline \hline  & \multicolumn{6}{c}{**Chunks Variance-Proposed Optimal Scheduling**} \\ \hline
**Layer** & **Neurons** & **PE\({}_{0}\)** & **PE\({}_{1}\)** & **PE\({}_{2}\)** & **PE\({}_{3}\)** & **PE\({}_{4}\)** & **PE\({}_{5}\)** & **PE\({}_{6}\)** & **PE\({}_{7}\)** \\ \hline L0 & 32,768 & 8 & 6 & 6 & 4 & 5 & 5 & 12 & 10 \\ L1 & 8192 & 21 & 5 & 10 & 31 & 18 & 12 & 11 & 11 \\ L2 & 4096 & 9 & 9 & 15 & 8 & 13 & 17 & 11 & 18 \\ L3 & 1024 & 3 & 19 & 7 & 2 & 5 & 6 & 3 & 2 \\ L4 & 2048 & 4 & 4 & 3 & 1 & 3 & 4 & 4 & 4 \\ L5 & 512 & 1 & 2 & 1 & 1 & 1 & 2 & 2 & 2 \\ L6 & 10 & 1 & 1 & 1 & 1 & 1 & 1 & 1 & 1 \\ \hline Total & 48,650 & 47 & 46 & 43 & 48 & 46 & 47 & 44 & 48 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Figures of variance when the chunks of neurons are assigned following the proposed ILP and variance-based optimal scheduling. Then, a stuck-at-fault was injected into one of the PEs and the inference of the selected 40 images was performed by compiling the kernel functions of the CNN application with a static scheduling (a total of 82,000 inferences). Then, by keeping the same stuck-at-fault, the inferences of the same set of images was executed by compiling the kernel functions of the CNN application with the proposed scheduling (a total of 82,000 inferences). As mentioned before, the pipelined framework was used for running the injections. By running 10 parallel processes, the 164,000 injections took about 41 days. The reader should note that for faults producing a simulation hang (i.e., 71,840 and 65,040 images in Table 4), the pipelined FI framework used a timer for avoiding the inferences of the full set of images. The data illustrated in Table 4 show the capability of the proposed ILP and variance-based scheduling in improving the reliability of the NCS. For each row, the number of images that produce the corresponding effect in the static or proposed scheduling is reported along with the related percentage. It is necessary to underline that the new ILP scheduling leads to a 0.6% increase in memory occupation and an increase in simulation times of 3.2% at run-time for a single inference cycle. Nevertheless, the proposed ILP-based scheduling is able to reduce by 24.74% the neural network wrong predictions (SDC-1%). Moreover, as expected, the amount of correct predictions with MSE greater than zero (Masked, MSE \(>\) 0) increased by 97.80%. In other words, the new scheduling is able to reduce the risk of wrong predictions, producing, again, evidence of faults in the output vector (MSE \(>\) 0) but keeping the prediction correct. A third good point concerns the last row of the table (Masked, MSE \(=\) 0): the proposed scheduling is able to improve the masking ability of the neural network by 59.53%.