# Deep Learning Models for Road Passability Detection during Flood Events Using Social Media Data
## 5 Evaluation and Results
### Results Using Images Only
We started using an \"ensemble of one-model per task\", that we named \"single-network image base image architecture\", and then we compared it with the \"double ended architecture\", presented in the previous section. To reduce the randomness associated with the training process, both architectures were initialized with the same weights and used the same hyperparameters and stopping criterion. As it can be seen from Table 4, the improvement is quite significant, particularly in the passability task. We believe that this is because the passability task has significantly fewer images to train when trained separately so it is more difficult for the model to generalize to new data and when trained together, the passability task can benefit from what the evidence task has learned. On top of obtaining better results than the single network base image architecture\", the \"double ended architecture\" has fewer parameters, making it lighter and computationally less expensive, and it is an end-to-end architecture. Then, we proposed to use the \"compactness loss\" to make the model more robust to unseen data. We retrained the previous model with the compactness loss on each branch, as shown in Figure 8. The results of both the validation and test set are in Table 4. Although we cannot extract direct evidence from this table that the compact loss improves the results, it does seem to generalize better to the test set since the results from validation to test are more similar than the ones without compactness loss. Finally, by combining the improvements using the double-ended classifier and the compactness loss, we are able to reach almost the same performance as we had obtained using the ensemble of 30 models, meaning that we have a model that is almost 30 times lighter and faster with a comparable performance. It is remarkable that the results using images are considerably better than the ones using metadata, not only in our case but also for humans or other participants. That is because the dataset has been \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Approach\backslash Data**} & \multicolumn{3}{c}{**Evidence of Road [\%]**} & \multicolumn{3}{c}{**Ev. of Road Passability [\%]**} \\ \cline{2-7}  & **Validation Set** & \begin{tabular}{c} **Test Set** \\ **(MediaEval)** \\ \end{tabular} & \begin{tabular}{c} **Validation Set** \\ **(Own)** \\ \end{tabular} & \
\begin{tabular}{c} **Test Set** \\ **(MediaEval)** \\ \end{tabular} &
\begin{tabular}{c} **Test Set** \\ **(Own)** \\ \end{tabular} \\ \hline Human annotation & 87.32 * & - & - & 47.71 * & - & - \\ Ensemble image base architecture (90) & 90.14 & 87.79 & 90.17 & 64.33 & 68.38 & 65.91 \\ Ensemble image base architecture (30) & 88.91 & - & 89.45 & 70.18 & - & 65.28 \\ Single network image base architecture & 86.48 & - & 84.88 & 62.84 & - & 59.99 \\ Double-ended architecture & 88.73 & - & 85.00 & 67.51 & - & 67.91 \\ Double-ended with compactness loss & 87.78 & - & 86.42 & 67.49 & - & 68.53 \\ [PERSON] et al. [20] & - & - & - & - & 64.35 & - \\ [PERSON] et al. [21] & - & 74.58 & - & - & 45.04 & - \\ [PERSON] et al. [46] & - & 87.58 & - & - & 63.13 & - \\ [PERSON] et al. [22] & - & - & - & - & 66.65 & - \\ [PERSON] et al. [23] & - & - & - & - & 24 & - \\ [PERSON] et al. [48] & - & - & - & - & 60.03 & - \\ [PERSON] [47] & - & - & - & - & 64.81 & - \\ [PERSON] [45] & - & 87.70 & - & - & 66.48 & - \\ \hline \hline \end{tabular}
\end{table} 
Table 4: F1-Scores on the challenge test set for both tasks using only the images from the tweets. * Results on a subset of 50 images. Figure 10: Evolution of F1-Score on the road evidence task as we ensemble more networks and make a comparison between the three different ensembling techniques. built and annotated using only the visual information, thus we know that the images should have enough information to solve the problem, but the metadata does not necessarily always have this distinctive information.