## 4 Results
### Analysis of Results Using a Generalized F-Score
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline  & \multicolumn{3}{c}{**Columbia**} & \multicolumn{3}{c}{**Las Vegas**} & \multicolumn{3}{c}{**Natanz**} \\ \cline{2-10}
**Dataset** & \(F_{0.1}\) & \(F_{1}\) & \(F_{10}\) & \(F_{0.1}\) & \(F_{1}\) & \(F_{10}\) & \(F_{0.1}\) & \(F_{1}\) & \(F_{10}\) \\ \hline A1 & 0.29 & 0.43 & 0.82 & 0.21 & 0.35 & 0.96 & 0.05 & 0.09 & 0.63 \\ A2 & 0.27 & 0.40 & 0.79 & 0.14 & 0.24 & 0.82 & 0.03 & 0.05 & 0.38 \\ A3 & 0.24 & 0.34 & 0.60 & 0.11 & 0.19 & 0.55 & 0.01 & 0.03 & 0.22 \\ B1 & 0.34 & 0.26 & 0.22 & 0.28 & 0.34 & 0.45 & 0.06 & 0.11 & 0.33 \\ B2 & 0.36 & 0.36 & 0.37 & 0.21 & 0.32 & 0.68 & 0.04 & 0.07 & 0.26 \\ B3 & 0.37 & 0.34 & 0.31 & 0.17 & 0.23 & 0.36 & 0.03 & 0.05 & 0.17 \\ C1 & 0.41 & 0.49 & 0.61 & 0.45 & 0.36 & 0.31 & 0.10 & 0.14 & 0.21 \\ C2 & 0.37 & 0.46 & 0.60 & 0.41 & 0.34 & 0.29 & 0.07 & 0.11 & 0.25 \\ C3 & 0.40 & 0.47 & 0.57 & 0.45 & 0.36 & 0.30 & 0.08 & 0.12 & 0.27 \\ D1 & 0.35 & 0.44 & 0.57 & 0.39 & 0.35 & 0.32 & 0.06 & 0.10 & 0.38 \\ D2 & 0.33 & 0.42 & 0.57 & 0.38 & 0.34 & 0.32 & 0.06 & 0.10 & 0.35 \\ D3 & 0.33 & 0.43 & 0.62 & 0.35 & 0.33 & 0.30 & 0.05 & 0.09 & 0.33 \\ E1 & 0.29 & 0.38 & 0.58 & 0.22 & 0.29 & 0.43 & 0.03 & 0.05 & 0.31 \\ E2 & 0.28 & 0.39 & 0.62 & 0.27 & 0.33 & 0.42 & 0.03 & 0.05 & 0.30 \\ E3 & 0.28 & 0.39 & 0.65 & 0.21 & 0.24 & 0.30 & 0.02 & 0.04 & 0.29 \\ F1 & 0.40 & 0.45 & 0.51 & 0.40 & 0.36 & 0.33 & 0.06 & 0.09 & 0.23 \\ F2 & 0.37 & 0.45 & 0.57 & 0.37 & 0.34 & 0.31 & 0.06 & 0.10 & 0.24 \\ F3 & 0.36 & 0.43 & 0.54 & 0.34 & 0.32 & 0.29 & 0.05 & 0.09 & 0.24 \\ G1 & 0.39 & 0.41 & 0.42 & 0.27 & 0.34 & 0.45 & 0.03 & 0.06 & 0.19 \\ G2 & 0.38 & 0.41 & 0.46 & 0.30 & 0.34 & 0.39 & 0.03 & 0.06 & 0.19 \\ G3 & 0.37 & 0.41 & 0.47 & 0.22 & 0.24 & 0.28 & 0.03 & 0.06 & 0.21 \\ \hline \end{tabular}
\end{table} 
Table 13: Analysis of support vector machine change prediction results using a generalized F-Score. Values for the three F-scores using the SVM classifier are shown in Table 13. In this table, we begin to see higher scores due to the improved quality of the classification results compared to the NN classifier. Scores for the Columbia dataset are typically greater than 0.5 for all three measures. Many F-score values for Las Vegas pass that threshold, as well. However, the results for Natanz show little improvement using the SVM classifier. As we noted in Section 4.5, the results for the CART classifier generally seem to be slightly worse than those of SVM. We see this same trend when examining the various F-score values for the CART classifier shown in in Table 14.  
Finally, the breakthrough comes when we examine the F-score values for the random forest classifier shown in Table 15. In general, all training datasets for the Columbia tiles provide balanced results that favor neither precision nor recall. The Las Vegas training datasets produce results that favor precision over recall, as can be seen by their relatively high \(F_{0.1}\) scores and their relatively low \(F_{10}\) scores. It is interesting to note that the G series datasets for Las Vegas (_i.e._, G1, G2 and G3) produce low \(F_{0.1}\) scores. Referring to Table 7, we can see that those datasets employ a very large percent of the pair's tiles. We believe that over-fitting is occurring, which prevents the classifier from generalizing well. The F-score values for the Natanz datasets show an interesting trend. Data Series A, C, D and E all produce low values for the three reported F-scores. Meanwhile, Data Series B, F and G report high scores for \(F_{0.1}\), which means that the precision is relatively high. Recall that Table 2 showed that the Natanz pair was filled with an overwhelming number of no-change tiles.