# Deep Learning Models for Road Passability Detection during Flood Events Using Social Media Data
## 5 Evaluation and Results
### Results Using Images Only  
In the case of images where we have proposed several models, first we are going to comment and compare the results of the different models that we proposed in this article and then compare them with the algorithms proposed by the rest of the participants. Firstly, we have presented the \"ensemble base image architecture\" which was the algorithm presented for the competition. This algorithm is mainly \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Approach\(\backslash\)Data**} & \multicolumn{2}{c}{**Evidence of Road [\%]**} & \multicolumn{2}{c}{**Ev. of Road Passability [\%]**} \\ \cline{2-5}  & **Validation Set** & **Test Set** & **Validation Set** & **Test Set** \\ \hline Human annotation & 51.48 ** & - & 18.18 ** & - \\ Metadata only & 59.93 & 62.56 * & 56.82 & 57.05 * \\ [PERSON] et al. [20] & - & - & - & 32.8 \\ [PERSON] et al. [21] & - & 58.30 & - & 31.15 \\ [PERSON] et al. [46] & - & 32.60 & - & 12.86 \\ [PERSON] et al. [22] & - & - & - & 30.17 \\ [PERSON] et al. [23] & - & - & - & 20 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: F1-Scores on the challenge test set for both tasks using metadata information only. * Results given on our own test set. ** Results on a subset of 50 images. based on performing iterative cross-validation to train models and ensemble them, for which we proposed a new ensembling technique. Since for this paper we have carried out a new training and test split, to make the comparison as fair as possible, we have retrained this architecture on the new training set and tested on the new test set. In Figures 10 and 11, we show how the F1-Score evolves for both tasks as we ensemble more models and the difference between the different ensembling techniques. As it can be seen from the curves, both tasks benefit from the ensembling, particularly for the first networks, then the performance stabilizes. The evidence of road benefits more from this technique than the passability of road task, and the curves for the passability task are less stable. We believe that this is due to the difference in the difficulties of both tasks. As can be seen towards the end of the graphs, the results of both tasks begin to slowly worsen. That is because (i) we are adding different architectures and some of them yield better results on average than others; (ii) we have stacked the networks in order of the architectures' average performance and thus it gets a point in which adding more architectures starts degrading the results. Given the information from both graphs, the ensemble of more than 30 models (up to 90, in our test case) does not significantly improve the performance. The ensemble of 90 networks (45 per task) was the winning architecture presented in the MediaEval competition, and its performances are presented in Table 4. This is the only architecture for which we have results on both the challenge and our own test sets. The results of the MediaEval test set are very similar to the results obtained for our own test set, which indicates that the difficulty of both sets is quite similar, allowing us to make a fair comparison with the results of the other participants. Some differences might be due to the fact that we have had to retrain all the networks to fit them to the new training, validation and test set. As for the ensembling technique, voting seems to have slightly worsened the results in the evidence of road task while averaging led to worsening of the results in the road passability task. For this reason, we decided to use the ensembling technique that we proposed in this paper for the final results, since it is the technique that has the most stable results. The use of the ensemble models makes sense for a competition; however, it might not be suitable for a real life application, since it tends to be computationally expensive and time consuming. Therefore, we focused our analysis to compare the best available model, obtained without taking into account computational limitations, with a lightweight version, proposed in this work.