## 5 Modelling the Communication Solutions
### Model Performance
Accuracy is measured with regard to how well the model can predict the best (1), middle (2) and worst (3) performing solutions in terms of reliability latency, efficiency and stability for a given scenario. Table 3 shows how accurate the model is at determining the placement for each metric on test data obtained from the TOSSIM simulator. The model is evaluated using percentage accuracy, precision, recall and F1. The percentage is measured as the percentage of true positives predicted by the model. Precision is measured as a ratio of the true positives against the true positives and false positives. Recall is measured as a ratio of the true positives against the true positives and false negatives. F1 is shown in Equation (1). \[\text{F1}=2\times\frac{\text{precision}\times\text{recall}}{\text{precision}+ \text{recall}} \tag{1}\]  
With regard to the application context the precision rate and recall will relate to the ratio of how often the system can correctly choose the required solution compared against how often it chooses the incorrect solution. The F1 score shows a weighted mean between the two scores The precision and recall rates are closely bound and in general show lower accuracy compared to the percentage score. The percentage accuracy precision and recall all follow the same general trend across the tests. The evaluation presents mixed results. The model achieves highest accuracy when predicting comparative performance with regard to latency and reliability reliability. prediction accuracy is not uniform over all evaluation metrics, with accurately predicting efficiency proving most difficult. Accuracy of prediction reaches as low as 50.43% in some categories, with recall as low as 0.4032 \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{3}{*}{**Solution**} & \multirow{3}{*}{**Evaluation Metric**} & \multicolumn{6}{c}{**Predictor Variables**} \\ \cline{3-8}  & & **Density** & **Scale** & **Size** & **Density (sd)** & **Scale (sd)** & **Channel Quality** \\ \hline \multirow{4}{*}{ETX-NH} & Reliability & * & *** & & & *** \\  & Latency & & *** & & & *** \\  & Efficiency & & *** & ** & & * \\  & Stability & & * & ** & & & *** \\ \hline \multirow{4}{*}{ETX} & Reliability & ** & ** & *** & & & *** \\  & Latency & ** & ** & ** & & & *** \\  & Efficiency & & *** & ** & & & *** \\  & Stability & & ** & ** & & & *** \\ \hline \multirow{4}{*}{ETT} & Reliability & & *** & & & & *** \\  & Latency & & & & & & *** \\ \cline{1-1}  & Efficiency & & *** & & & & *** \\ \cline{1-1}  & Stability & & & & & & *** \\ \hline \hline \end{tabular}  
* \(p\)-value \(<\) 0.05; ** \(p\)-value \(<\) 0.01; *** \(p\)-value \(<\) 0.001.  
\end{table} 
Table 2: The significant predictor variables for modelling the response of each evaluation metric as determined by a T-test. and precision as low as 0.4360. These lower scores are not acceptable for a production system when predicting between three solutions. Despite some mixed results the model presents a positive outlook for the prospect of using the physical features of the network to compare communication solutions to find the best fit for a set of requirements. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & & **Reliability** & **Latency** & **Efficiency** & **Stability** \\ \hline \multirow{3}{*}{1 st} & Percentage & 76.68\% & 85.7\% & 50.43\% & 78.25\% \\  & Precision & 0.7257 & 0.6056 & 0.7267 & 0.8378 \\  & Recall & 0.6828 & 0.5860 & 0.6720 & 0.6667 \\  & F1 & 0.7036 & 0.5956 & 0.6983 & 0.7425 \\ \hline \multirow{3}{*}{2 nd} & Percentage & 71.8\% & 61.92\% & 55.7\% & 63.25\% \\  & Precision & 0.6497 & 0.6228 & 0.4360 & 0.7569 \\  & Recall & 0.6183 & 0.5591 & 0.4032 & 0.5860 \\  & F1 & 0.6336 & 0.5892 & 0.4190 & 0.6606 \\ \hline \multirow{3}{*}{3 rd} & Percentage & 81.39\% & 77.95\% & 59.81\% & 67.88\% \\  & Precision & 0.8107 & 0.8095 & 0.5152 & 0.7500 \\ \cline{1-1}  & Recall & 0.7366 & 0.6398 & 0.4570 & 0.7258 \\ \cline{1-1}  & F1 & 0.7718 & 0.7147 & 0.4843 & 0.7377 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Model accuracy over a test data set.