# Predicting Traffic Sign Retro-Reflectivity Degradation Using Deep Neural Networks
## 5 Results and Discussions  
As different works use different datasets, a fair and objective comparison is not possible. Looking at the experiment settings in the literature, we noticed that the previous studies do not use the brands or the manufacturer information as predictors when fitting their regression models. Moreover, we wanted to evaluate the deep learning model's performance against unseen pairs of sheeting types and brands to increase the difficulty of the prediction task and further assess its generalization capabilities on unseen data. Accordingly, we conducted our experiments on the data after it had been divided into training, validation, and testing sets based on the combination of brand and sheeting type such that we see samples of brand and sheeting type combinations in the test set which are not seen during the training. In addition, we excluded the brand name from the list of predictors. In the current setup, the validation and test sets only include combinations of brand and sheeting type that are missing in the training set. All our results are reported as the mean and standard deviation of total of 10 trails. The prediction results of our models using all the features can be seen in Table 4. The best results are shown in bold. We noticed that by including orientation, our results improve in our neural network models, while for the polynomial model the improvement is very minimum. All the variations of the neural network models outperform the linear and polynomial models by a significant margin. The linear model achieved \(R^{2}\) value lower to the results reported by [1], which can be attributed to the fact that we trained one model for all the combinations of features instead of training multiple models. In comparison, the polynomial model seems to outperform the linear model by a wide margin. The neural network with one-hot encoding achieved the best results with a mean MAE of 34.809. The shared embedding space performed well, achieving an MAE of 36.425. The separate embedding space achieved the second lowest score at mean MAE of 142.562. We believe that due to the limited number of categories in each feature, the embedding space was too small for the model to capture the relationship between the categories. In Table 5 we present samples of traffic signs extracted from our test set with the predicted \(R_{A}\) by the neural network with one-hot encoding. Moreover, we plotted the training and validation loss history of the NN with one-hot encoding in Figure 3. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**Test Set**} & \multicolumn{2}{c}{**Train Set**} & \multicolumn{2}{c}{**Validation Set**} \\ \cline{2-7}  & **R2** & **MAE** & **R2** & **MAE** & **R2** & **MAE** \\ \hline Linear & 0.426 \(\pm\) 0.009 & 218.873 \(\pm\) 3.983 & 0.475 \(\pm\) 0.001 & 179.068 \(\pm\) 1.233 & 0.551 \(\pm\) 0.003 & 185.022 \(\pm\) 0.471 \\ Polynomial & 0.759 \(\pm\) 0.007 & 133.855 \(\pm\) 1.73 & 0.783 \(\pm\) 0.009 & 109.492 \(\pm\) 1.087 & 0.776 \(\pm\) 0.004 & 117.316 \(\pm\) 1.913 \\ NN one-hot encode & **0.976 \(\pm\) 0.008** & **34.809 \(\pm\) 5.090** & 0.937 \(\pm\) 0.007 & 45.900 \(\pm\) 2.093 & **0.842 \(\pm\) 0.011** & **106.836 \(\pm\) 2.473** \\ NN entity embedding & 0.156 \(\pm\) 0.533 & 142.562 \(\pm\) 67.674 & 0.136 \(\pm\) 0.524 & 146.049 \(\pm\) 64.579 & 0.112 \(\pm\) 0.472 & 189.063 \(\pm\) 51.190 \\ NN shared-entity embedding & 0.967 \(\pm\) 0.015 & 36.425 \(\pm\) 6.324 & **0.941 \(\pm\) 0.008** & **44.165 \(\pm\) 2.726** & 0.823 \(\pm\) 0.012 & 109.403 \(\pm\) 1.746 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Model performance on the dataset using all features. The best performing models are in bold. Figure 3: The training and validation _MSE_ loss for each epoch of the NN with one-hot encoding model.