# Self-Service Data Science in Healthcare with Automated Machine Learning
## 4 Results
### Artefact Evaluation  
Based on the user stories we created two artefacts to automate a part of the data preparation phase and the complete modelling phase of CRISP-DM [44]. The data preparation activities involve the possibility to include or exclude variables, data imputation for missing values and the recoding of categorical variables to numerical variables, as TPOT cannot handle non-numerical data as input. Due to the conflict in user-interaction requirements, we designed two artefacts with the same functionalities, but a different user interface. The artefacts can be used to create logistic regression models and users who do not know how to code should be able to use the artefacts. The artefacts contain a description of missing data handling, as well as the possibility to compare the different models.  
The two artefacts are a Flask web application (GUI artefact) to satisfy the graphical user interface preference and a Jupyter notebook (Code artefact) to satisfy the coding preference, as shown in Figures 11-13 in Appendix C, respectively. The source codes for the web application as well as the Jupyter notebook are available at this URL: [[https://github.com/richooms/healthcare_automml](https://github.com/richooms/healthcare_automml)]([https://github.com/richooms/healthcare_automml](https://github.com/richooms/healthcare_automml)).  
Using the risk and efficacy strategy from the framework for evaluation in design science [40]. We created a set of refined hypotheses based on the user story categories to evaluate the artefacts with the domain experts. The hypotheses are available in Appendix D. For the user-interaction we tested on four elements: Uploading a dataset, creating a subset, the workflow of the application and the workflow explanation. Only for uploading a dataset the GUI artefact was preferred. For all other actions the code artefact was preferred, as the interviewees felt more in control of the process. The same was the case for model construction and the progress reporting on model construction as parts of the model construction category. The model explanation category consisted of comparing of different  
Figure 4: Word cloud of all comments on artefact evaluation questions.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Category/Method** & **Auto Sklearn** & **Auto-WEKA** & **TPOT** & **Hyperopt-Sklearn** \\ \hline User interaction (UI) & n/a & n/a & n/a \\ Model construction (MC) & 3 & 3 & 4 & 2 \\ Model explanation (MX) & 0 & 0 & 0 & 0 \\ Model usage (MU) & 1 & 1 & 1 & 1 \\ Total matches & 4 & 4 & 5 & 3 \\ \hline \hline \end{tabular}
\end{table}
Table 3: AutoML method scores on user story categories.  
results, explanation of missing data handling and readability. In this category artefact A was preferred over artefact B for all interactions except for explanation of missing data handling. An overview of the preferences and categories is available in Table 4. The preference column demonstrates the type of artefact that was preferred for that part of the artefact interaction. The score column indicates how many subjects had the preference.  
What we found is that a hybrid version of the two artefacts is preferred to interact with AutoML by the interviewees. To keep control over the process of creating a model they prefer coding, for the relatively simple tasks such as uploading a set or comparing results, they prefer a graphical user interface. Furthermore, TPOT needs to explain variable importance to make it usable in their research practice as it is considered a must-have for the adoption of AutoML by domain experts.