# Sensor Reliability in Cyber-Physical Systems Using Internet-of-Things Data: A Review and Case Study
## 6 Experimental Results
### Self-Learning for Decision-Making Evaluation  
Based on the previous knowledge generated by the rewards obtained from similar situations studied in the simulations, the ranges of the values of each reward are defined as a function of the prediction error of the reliability model, calculated during the simulation. The reward function is chosen for setting the best possible Q-value in 100 different scenes. Therefore, the function for updating the Q-values is [98]:  
\[Q(s_{t+1},a_{t+1})\gets Q_{t}(s_{t},a_{t})+\alpha_{t}\Big{(}R_{t+1}+ \gamma\max_{a\in A}Q(S_{t+1},a_{t})-Q(S_{t},\ a_{t})\Big{)} \tag{2}\]  
where \(s_{t}\) is the state in time \(t\); \(a_{t}\) is the action taken in time \(t\); \(R_{(t+1)}\) is the reward received after performing action \(a_{t}\); \(\alpha_{t}\) is the learning rate (\(0\leq\alpha_{t}\leq 1\)); and \(\gamma\) is the discount factor which trades off the importance of sooner-versus-later rewards. Table 3 lists the error reward matrix based on knowledge of the behavior of those prediction models. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**MAE**} & \multicolumn{2}{c}{**RMSE**} & \multicolumn{2}{c}{**RAE**} & \multicolumn{2}{c}{**RMSE**} & \multicolumn{2}{c}{**RMSE**} \\ \cline{2-10}  & **DMRS** & **MRSE** & **DMRS** & **MRSE** & **DMRS** & **MRSE** & **DMRS** & **MRSE** \\ \hline MLP & 0.0046 & 0.0035 & 1.275 & 1.270 & 0.187 & 0.188 & 0.395 & 0.392 & 0.933 & 0.933 \\ k-NN & 0.002 & 0.0002 & 1.014 & 1.010 & 0.114 & 0.114 & 0.371 & 0.365 & 0.963 & 0.961 \\ LR & 0.6781 & 0.6530 & 2.305 & 2.285 & 0.701 & 0.695 & 0.782 & 0.788 & 0.434 & 0.435 \\ SVM & 0.4735 & 0.4740 & 2.072 & 2.065 & 0.442 & 0.447 & 0.773 & 0.773 & 0.692 & 0.684 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Key performance indices based on plane (DMRS) and space (MRSE) figures of merit. Figure 8: Behavior of LiDAR error of each model with regard to the validation data, (**a**) on the plane (DRMS) and (**b**) space (MRSE). The decision-making is based on two of the main performance indices. First, the coefficient of determination (R\({}^{2}\)) is taken into account, because it provides a measure of the generalization capacity of the model. The Relative Absolute Error (RAE), which is a measure of model accuracy, is also considered. Finally, another simulation is performed in order to determine the quality of the Q-learning method for automatically selecting the best prediction model. One hundred new scenes captured by the LiDAR sensor were analyzed. The Q-learning method runs online in the main node. When new data becomes available, models contained in the library are evaluated. Once each model has been assessed, the performance indices are calculated, and the Q-learning determines the best reliability model between MLP and k-NN. Figure 9a,b depicts the behavior of RAE and R\({}^{2}\), respectively, for both reliability models. Figure 10 shows the Q-learning classification error matrix. In all considered cases, the best model chosen was k-NN 76 times and _MLP_ 24 times. Analyzing the results obtained, the best model selected had a RAE between 0 and 20% and a R\({}^{2}\) above 80% in 61% of the scenarios. The system was able to guarantee models with a greater capability of generalization in 71% of the scenarios, based on a coefficient of determination higher than 80%. Overall, reliability can be predicted with an RAE of less than 40% and an R\({}^{2}\) of about 70% in 90% of the scenarios, which demonstrates the quality of the learning process. The models presented a low generalization, with a coefficient of determination of less than 70% in only 9% of the scenarios and an RAE greater than 40% in only 1%. Therefore, the Q-learning method that evaluates reliability on the basis of the prediction error model at each instant worked appropriately for calculating the best model that represented the LiDAR performance to a high degree of accuracy and that guaranteed the required levels of safety and reliability for automotive applications.