# A General Model for Side Information in Neural Networks
## 7 Concluding Remarks  
Herein, we presented a novel model for side information in neural networks, where the side information is made visible to the network during both the training and testing phases. The side information is fed into the neural network with the aid of a knowledge base of facts, as shown in Figure 1 and the high-level pseudo-code given in Algorithm 1. We developed a formalism for side information, whereby it can be individually useful according to (5) or useful on average according to (6). As a proof of concept, we then provided experimentation on two datasets: MNIST and House Price, averaging the results over 10 runs of an FNN with two hidden layers. For both datasets the use of side information improves the classification accuracy significantly and is useful both individually and on average. Extensive experimentation to provide further validation of the approach is a task that is left as further work. In future work, we aim to investigate the relationship between side information and explainable ML models, noting that explainability is one of the pillars of trustworthy ML, broadly known as explainable AI (XAI) [27]. By explainable ML, we refer to methods which aim to analyse the data in a way that involves attributes that can be understandable to humans. The peculiarity of digits in the MNIST dataset is an example of such an attribute. Some of these attributes can be cast as side information in our formalism, and when they are shown to be useful will improve the performance of the NN. \begin{table}
\begin{tabular}{l c c c} \hline \hline
**Classification Accuracy** & **Without Side** & **With Side** & \multirow{2}{*}{_p-Value_} \\  & **Information** & **Information** & \\ \hline MNIST & 97.85\% & **98.01**\% & 0.044 \\ House Price (Group 1) & 50.10\% & **54.28**\% & 0.000173 \\ House Price (Group 2) & 50.40\% & **53.43**\% & 0.0012 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Test classification accuracy for the two experiments: MNIST and House Price, followed by the results of a statistical significance test (_p_-value for a 95% confidence interval). A bold entry indicates that the classification accuracy of the FNN for the respective experiment is significantly higher than its competitor. The inclusion of side information leads to a statistically significant improvement in the classification accuracy in both experiments, compared to the case when side information is not included. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Usefulness of Side Information** & **Individually Useful** & **Useful on Average** \\ \hline  & \(\epsilon\) & \(\delta\) & \(\epsilon\) & \(\delta\) \\ \hline MNIST & 0.06\% & 0.20 & 0.09\% & 0.20 \\ House Price (Group 1) & 4.10\% & 0.10 & 4.80\% & 0.10 \\ House Price (Group 2) & 2.96\% & 0.10 & 3.60\% & 0.10 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Difference between the classification losses for the two experiments: MNIST and House Price, without and with side information, expressed in the second and third columns by \(\epsilon\) and \(\delta\), according to (5) when side information is individually useful (we note that the \(\epsilon\) values were normalised and reported as percentages). In addition, in the fourth and fifth columns, the values of \(\epsilon\) and \(\delta\) express how useful side information is on average, according to (6). For both datasets, side information is useful both individually and on average. Author Contributions: Both authors contributed equally to this work. All authors have read and agreed to the published version of the manuscript. This work was supported by the UK Government's Department for Science, Innovation and Technology (DSIT). The datasets presented in this study are openly available.