# Self-Service Data Science in Healthcare with Automated Machine Learning
## 4 Results
### Benchmark Test  
The benchmark test was ran using a time budget of one hour with a total of 160 hours of computational time. Figure 3 contains the visualization of results; the datasets are on the X-axis, the Y-axis shows the AUROC score. A colored dot marks the score for an AutoML method on each of the ten folds. A Kruskal-Wallis H test indicated that there was a statistically significant difference in the distribution for the Breast (H = 11.36, \(p<0.001\)), Diabetes (H = 18.64, \(p<0.001\)), Liver (H = 17.93, \(p<0.001\)), and Sick dataset (H = 27.87, \(p<0.001\)) between the AutoML methods, see Table 1 for statistics.  
\begin{table}
\begin{tabular}{l l l l l} \hline  & **Breast** & **Diabetes** & **Liver** & **Sick** \\ \hline H-statistic & 11.36 & 18.64 & 17.93 & 27.87 \\ \(P\)-value & 0.995 ** & 0.324 ** & 0.455 ** & 0.386 ** \\ \hline \end{tabular}
\end{table}
Table 1: Statistics for Kruskal Wallis test.  
What is noteworthy in Figure 3 is that on the Liver dataset, the decision tree and Hyperopt-Sklearn do not always outperform the constant predictor. On the Diabetes dataset, Hyperopt-Sklearn lags compared the three other methods, but performs better than the constant predictor and has a similar performance to the decision tree. On the Breast dataset, all AutoML methods have the maximum score in at least one fold. All methods perform well on the Breast set, given their median scores and distribution. The performance of the decision tree indicates that it is not a hard prediction problem. For the results on the Sick dataset, we see that TPOT and Auto-Sklearn outperform the other two methods in both consistency and score of their predictions despite the fact that the set has missing values. Hyperopt-Sklearn is again not better than the decision tree.  
Overall, TPOT registered the highest median score after running for one-hour on all sets but the Breast dataset, in which autosklearn registered the best median performance. A Mann-Whitney U test indicated that autosklearn significantly outperforms the decision tree (U = 2.0, \(p<0.001\)) and hyperoptsklearn (U = 17.5, \(p<0.01\)) on the Breast set. TPOT significantly outperforms the decision tree on the datasets Diabetes (U = 0.0, \(p<0.001\)), Liver (U = 8.0, \(p<0.001\)) and Sick (U = 0.0, \(p<0.001\)). Furthermore, TPOT outperforms hyperoptsklearn significantly on the datasets Diabetes (U = 2.0, \(p<0.001\)), Liver (U = 7.0, \(p<0.001\)) and Sick (U = 0.0, \(p<0.001\)). Finally, AutoWEKA is significantly outperformed on the Sick dataset by TPOT (U = 0.0, \(p<0.001\)). This is probably because AutoWEKA does not impute data for missing values. Autosklearn and TPOT do impute values for missing data and have significantly better results. The statistics for the Mann-Whitney U test are available in Table 2. The table shows the \(p\)-values and U statistics for each method compared to the best performing method on each dataset. TPOT and autosklearn do not significantly differ in performance for any of the datasets.  
From the benchmark test we can conclude that no method consistently outperforms all others. However, we see that TPOT records the highest median scores on three of the four tasks in this test but this performance is not significantly better than the performance of other methods. Autosklearn and AutoWEKA get similar results to TPOT, only AutoWEKA is outperformed on the Sick dataset.  
Figure 3: One hour benchmark test results.