# An Adaptive Neuro-Fuzzy Propagation Model for LoRaWAN
## 4 Analysis and Results  
About 5000 data samples were collected over a relatively large area (4.25 km \(\times\) 2.7 km) in the commercial area of Glasgow, Scotland. Data was collected from three base stations at different locations, with 1931, 1820 and 1256 samples being collected from BS1, BS2 and BS3, respectively. Figure 1 shows the area of the investigation, where some of the measurement locations are pinpointed with markers, and gateways are labeled as BS1, BS2 and BS3. Base stations are equipped with the same antennae, mounted relatively at the same height from the ground. Data was analyzed to provide an insight of the performance of the proposed model. To check the goodness of fit and benchmarking with other models, the most commonly used measures in the literature were reported, RMSE (root mean square error in dB), \(E_{\sigma}\) (error standard deviation dB) and \(E_{m}\) (mean error dB). Unfortunately, the first two measures depend on the range or scale of data and \(E_{m}\) ignores the error sign. Therefore, to address these issues, the Nash-Sutcliffe efficiency (NSE) coefficient is used as a measure of the goodness of the fit. Having a universal measure of performance benchmarking is especially important, as various wireless technologies have different sensitivities. This difference impacts the dynamic range of measured data and, therefore, its RMSE scale; however, the NSE is less prone to the dynamic range. NSE ranges from \(-\infty\) to 1, where 1 would indicate a perfect match between the model predictions and measurements [39]. In addition, to investigate the model's generalization capability, instead of training with one BS at a time, data from all the three BSs was used to train and validate the model. For the purpose of comparison, an ANN model was also used to model the propagation. A feedforward ANN was chosen with three hidden layers of size seven, 14 and four neurons for each layer, respectively. The best ANN structure was chosen heuristically after trying ANNs with two to five hidden layers of various neuronsizes. Results in Table 1 demonstrate the average of a 10-fold cross-validation analysis; 90% of data was used for training. The results of the COST W.I. model are tabulated in Table 2 to make a comparison with other practical investigations conducted in [15]. Figure 2 compares the measurements and estimation for BS1, and Figure 3 compares the overall estimation of all base stations with measurements. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{4}{c}{**ANFIS**} \\ \hline
**Base Station** & **RMSE** & \(\mathbf{E_{\sigma}}\) & \(\mathbf{E_{m}}\) & **MAE** & **NSE** \\ \hline
**BS1** & 5.783 & 5.788 & 0.022 & 4.582 & 0.58 \\
**BS2** & 6.631 & 6.621 & 0.050 & 5.196 & 0.56 \\
**BS3** & 6.290 & 6.285 & 0.026 & 5.017 & 0.38 \\
**BS1,2,3** & 7.060 & 7.056 & 0.014 & 5.495 & 0.48 \\ \hline \hline  & \multicolumn{4}{c}{**ANN**} \\ \hline
**BS1** & 6.233 & 6.231 & 0.002 & 4.905 & 0.51 \\
**BS2** & 8.249 & 8.257 & \(-\)0.103 & 6.457 & 0.44 \\
**BS3** & 7.045 & 7.037 & 0.029 & 5.617 & 0.30 \\
**BS1,2,3** & 8.128 & 8.127 & 0.167 & 6.388 & 0.38 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: ANFIS and ANN model performance with 10-fold cross-validation. Figure 1: Area of practical investigation 4.25 km \(\times\) 2.7 km, and base station locations (BS1, BS2, BS3). A few measurement locations are pinpointed with markers. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Base Station** & **RMSE** & \(\mathbf{E_{\sigma}}\) & \(\mathbf{E_{m}}\) & **MAE** & **NSE** \\ \hline
**BS1** & 12.37 & 10.61 & 6.367 & 10.54 & 0.039 \\
**BS2** & 18.94 & 14.12 & 12.63 & 16.25 & \(-\)0.27 \\
**BS3** & 13.33 & 12.46 & 4.752 & 11.04 & \(-\)0.30 \\
**BS1,2,3** & 18.14 & 12.02 & 13.58 & 15.86 & \(-\)0.37 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: COST W.I. model performance with optimization. A series of models were benchmarked against similar practical measurements in COST Action 231 [15].