# Identifying Real Estate Opportunities Using Machine Learning
## 5 Evaluation
### Results and Findings
#### 5.2.1 Which Model Performs Best?  
A summary of the results is reported in Table 4. To build this table, we have averaged the performance for all folds and then averaged over the different experiments (only in stochastic techniques), finally sorting by ascending mean squared error (MSE). Average and standard deviation is reported for those stochastic techniques, whereas the single performance result is shown in the case of deterministic algorithms. To simplify the table, only the best setup for each algorithm is shown in the table. This is: \(k\)-nearest neighbors using 50 neighbors, Minkowski distance and a metric inverse to distance; multi-layer perceptron with two layers of 256 and 128 units, respectively; ensembles of regression trees with 50 estimators in the forest, MAE as the split criterion and bootstrapping; and support vector regression with radial basis function (RBF) kernel. These results significantly outperform those reported by a linear regression in all cases.  
When considering the best performers based on top performance, we notice that ensembles of regression trees work systematically better than other techniques in terms of mean absolute error. In particular, the top ten models are always ensembles, with either 10, 20 or 50 estimators. The split criterion, bootstrap instances and normalization neither seem to have a significant effect on performance. The smallest mean absolute error is 338,715 euros (a relative error of 16.80%). Consistent results are found when considering the median absolute error, although best performers are not sorted on a side-by-side basis (i.e., the model with the best MAE do not correspond to the model with the best median average error (MedAE). In this case, all top-ten models are also ensembles of regression trees. The best median absolute error is 94,850 euros (a relative error of 5.71%).  
The distribution of best median average errors per machine learning technique is shown in Figure 9. In this figure, we can see how ensembles of regression trees significantly outperform all of the other techniques, followed by \(k\)-nearest neighbors, support vector regression and the multi-layer perceptron. In the case of support vector regression, only one configuration was tested and the model is deterministic, for this reason there is a single result instead of a distribution. As for the multi-layer perceptron, a very large variance can be seen in the distribution of results, along with a remarkable difference between the average and the median MedAE. Worst MAE and MedAE are always reported by the multilayer-perceptron with a single layer with 1024 units. Because of the relatively small availability of data, this model could be affected by overfitting, although further testing is required to confirm this diagnosis.  
Regarding the explained variance regression score and the coefficient of determination, both metrics are highly correlated. The best model when assessed based on \(R^{2}\) achieves a value of about 0.46 for both metrics.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**ML Algorithm** & \(\mathbf{E_{var}}\) & \(\mathbf{MAE}\) & \(\mathbf{MedAE}\) & \(\mathbf{MSE}\) & \(\mathbf{R^{2}}\) \\ \hline \(k\)-nearest neighbors & 0.3625 (-) & 0.4404 (-) & 0.2068 (-) & 4.044 (-) & 0.3598 (-) \\ Multi-layer perceptron & 0.3113 (0.0020) & 0.5637 (0.0026) & 0.3355 (0.0041) & 4.2262 (0.0029) & 0.3067 (0.0027) \\ Ensembles of regression trees & 0.1303 (0.1381) & 0.3714 (0.0075) & 0.3139 (0.0038) & 4.3468 (0.1548) & 0.1253 (0.1386) \\ Support vector regression & \(1.73\times 10^{-5}\) (-) & 0.7384 (-) & 0.4540 (-) & 4.9015 (-) & \(-\)0.0664 (-) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Quality metrics per model. Average (and standard deviation) is shown for stochastic models. Only the best setup based on mean MSE is shown. Price in millions.