GUI-fying the Machine Learning Workflow: Towards Rapid Discovery of Viable Pipelines | by Sean McClure | Towards Data Science

A tenet of software development is to automate as much as possible. This is the DevOps mindset, where an effort towards automation helps teams push features continuously while ensuring best-practices are respected. When teams make automation a priority their developers focus on the least amount of code needed to bring new features to a product. This has 3 major benefits to real-world software projects:
less human error in the codebase;
faster development;
more variation.
While the first 2 points are obvious, the last one is less appreciated. Variation increases with automation because it’s easier to try things when there is less code to write. Variation is a critical ingredient in the early stages of product discovery. By attempting more options we reduce our reliance on upfront assumptions, and instead explore the space of possible outcomes.
Traditional software (no machine learning) is full of options. Whether it’s adding a button, creating a login screen, or connecting alerts, traditional software has a vast array of options from which to choose. If it can be expressed in computer code it can be done.
But machine learning products have no such guarantees. With machine learning, product features are not options they are possibilities. Only when we have confirmed the right data exist can we call a possibility an option.
This has implications for product discovery. Product discovery assumes options are a given; the challenge being to discover which of those options will lead to the best product. But in machine learning we cannot start that process until we have exposed which options are available, given the data.
With machine learning, product features are only possible if the data permit them.
Whereas traditional software begins the discovery process during “product discovery”, machine learning applications rely on discovery before this process. This means the variation aspect of automation, which promotes discovery, must happen before development commences. In other words, automation must occur in the machine learning workflow.
Data Scientists explore data and validate models by working through the Machine Learning Workflow, shown below:
This is an iterative process, involving gathering available data, cleaning/preparing those data, building models, validating their predictions and deploying results. This leads to the kind of discoveries needed to make strategic decisions about what goes in the Data Pipeline. The Data Pipeline is the above workflow laid out “flat”, and is responsible for putting into production the best parts discovered while iterating through the above workflow:
Whereas Data Pipelines benefit from the kind of automation seen in traditional software (testing, continuous integration, building, continuous deployment) the Machine Learning Workflow requires a different kind of automation.
The seemingly obvious answer to the titled question is some library that attempts many different models and automatically validates which one leads to the highest accuracy. These libraries exist, such as Google’s AutoML, auto-sklearn, tpot, MLBox, and many others. While these efforts help reduce the time it takes to find decent models they assume product-worthy models can be obtained with minimal human intervention. This drastically oversimplifies the process of converting raw data into a truly viable machine learning product.
Machine learning practitioners turn multiple knobs (hyper-parameters) to help nudge learning algorithms towards a better result. The reason our best Data Scientists call this process an art is because knowing what knobs to turn, and by how much, cannot be codified. It involves experience, conversation with domain experts, and above-all trial and error.
The space of possible outcomes emerges from a complex interaction of data attributes, models and parameters. Truly automated machine learning would have to do much more than turn knobs until some good accuracy was achieved. It would have to vary the amount of data, attempt various datasets, try multiple data preparations, engineer new features, run the gammit of validation metrics and deploy results in varying contexts; not to mention folding domain-specific knowledge into each of these steps.
This represents a kind of “combinatorial explosion” Data Scientists must reign-in when looking to build viable machine learning products.
The right mix of tasks to effectively convert raw data to product-worthy outputs exists in a space of near-infinite possibilities.
The mark of a seasoned Data Scientist is their ability to reign-in this complexity by narrowing down the possible combinations. Through exploration and reasoning, Data Scientists pair statistical techniques with experience to carve out a path from myriad possibilities to a few viable options.
To truly automate the Machine Learning Workflow we need more than automated knob turning. We require the ability to execute a variety of common machine learning tasks, while still permitting human intervention, to explore and reason about what data tells us.
I argue this automation is only possible by GUI-fying the Machine Learning Workflow. To GUI-fy something is to expose functionality through a graphical user interface. When practitioners execute common tasks at the touch of a button it enables them to rapidly attempt many options, leading to the massive variation needed to discover viable pipelines.
This offers a more complete idea of DevOps for Data Science. While traditional testing and building practices are needed for our pipelines we also require automation at the workflow stage, so product discovery can start from a set of viable machine learning options.
GUI-fying the Machine Learning Workflow may seem too challenging. After all, things seem to change every time we point a new dataset at a machine learning task. But understanding the scenarios when things go wrong is precisely how we work towards automation. I argue that automation is the responsibility of the seasoned practitioner. As the ultimate abstraction, automation ensures our work focuses solely on what is novel to the challenge. When we can rapidly explore the space of possibilities, we promote a much stronger ROI on data-intensive projects.
Automation is the responsibility of the seasoned practitioner.
With time, GUI-fication leads to a robust set of machine learning capabilities across each step of the Machine Learning Workflow. Eventually, such capabilities can be exposed as a service, whose purpose is to enable teams to rapidly assess the state of an organization’s/client’s data during the early stages of ideation.
As teams continue to work on projects, they add to these services, exposing more machine learning functionality, and enabling quicker discovery of the options that exist when looking to build machine learning products.
It’s important to note that GUI-fication is not meant to replace hands-on coding. There is no substitute for digging into code when we must refine our process for extracting value from data. But coding should never be our first take on a dataset. To do so is a failure in abstraction, a failure to automate, and does not represent a Lean approach to building machine learning products. We should only touch code when we haven’t encountered the problem before.
We should only touch code when we haven’t encountered the problem before.
There are 2 parts to GUI-fying the Machine Learning Workflow. On the back-end we need to expose machine learning functionality as a REST service, so common tasks can be called using a front-end. On the front-end we need to easily visualize and manage the flow of data across tasks we choose to run.
I will list important requirements, followed by a discussion on each point:
expose both R and Python functionality;
allow code changes without restarting the service;
allow for proper code organization and version control;
intermittent data should be accessible by both R and Python;
utilize native R and Python visualizations;
R and Python should be separate services;
automatically track and manage data dependencies;
wait until back-end has completed processing before showing results on front-end.
Exposing both R and Python functionality: I believe it’s important to support both R and Python. This is because variation is key as discussed in the beginning of this article. When we abstract away from the language all that matters is capability, allowing the full breadth and depth of machine learning to be brought to bear on our challenges.
Allow code changes without restarting the service: Data Science relies on REPL-type environments that show results the moment we change our code. If our service is contained within, say, a Docker runtime we would have to rebuild/run the image to surface new changes. Another option is Jupyter notebooks, which support REPL-style coding, and can be exposed as REST services using the Jupyter Kernel Gateway. But again, code changes are not exposed without restarting the gateway.
Child Processes in Node.js allow us to call both R and Python functions via Node.js. When combined with the web framework Express it becomes possible to create a REST API that exposes R and Python functionality to a front-end. As a bonus, this type of service does not require a restart to expose changes within child processes.
Allow for proper code organization and version control: Notebooks also have drawbacks when it comes to organizing code and doing proper version control. A notebook is not organized code, and versioning between notebooks only versions the JSON representation of our work. This is not amemable to real-world software development. Our service should allow R and Python scripts to exist as proper files, that can be organized and versioned like any other application.
Intermittent data should be accessible by both R and Python: The service should also allow both R and Python to access intermittent data along any stage of the workflow. For example, users should be able to prepare data in R and build models in Python, seamlessly moving between each language. A great option for this is JSON.
JSON is easily read and written by both R and Python. JSON also stores nested data easily. This works well for machine learning since we often save multiple datasets in one call, such as splitting data into training and test sets:
[{ "X_train": [{      "feature_a": 100,      "feature_b": 150,      "feature_c": 40 }], "X_test": [{      "feature_a": 200,      "feature_b": 250,      "feature_c": 140 }], "y_train": [{      "feature_d": 170 }], "y_test": [{      "feature_d": 270 }]}]
Utilize native R and Python visualizations: Most front-end applications choose interactive, JS-based visualizations. While there are benefits to interactive charts they lack the variety of native R and Python charting. For example, we can create the following influence plot in R using 1 line of code:
How much custom JavaScript would be needed to reproduce this? Native visuals are how Data Scientists explore and validate their work; we shouldn’t get in the way of that.
R and Python should be separate services: The GUI-service should use a micro-services architecture so that R and Python remain separate concerns. Merely wrapping a service around scripts that call both R and Python would be a poor design choice. Of course this will have to be balanced with the need to expose code-changes immediately, as per the above point around Docker runtimes.
Automatically track and manage data dependencies: Every task executed will depend on some previous dataset, either raw or transformed. The GUI needs to automatically track and manage dependencies between datasets to preserve workflows. This can be accomplished by having the front-end manage read and write paths across nodes, as shown in the following figure:
For example, running a chosen function on a node would initiate the following steps:
Wait until back-end has completed processing before showing results on front-end: Since the workflow will involve running compute-intensive tasks, the front-end must wait for back-end processing to finish before trying to display results. As long as we are calling REST endpoints this is fairly straightforward. For example, the Promise interface in jQuery’s $.ajax method ensures we can hold off executing additional JavaScript until the request has finished.
I created Machine Flow to make manifest the philosophy discussed in this article. Machine Flow encompasses the above technical requirements, and allows Data Scientists and Data Teams to work towards GUI-fying their workflows.
Machine Flow enables visual execution and tracking of machine learning workflows. Users dynamically create dependency graphs, with each node responsible for executing a task and displaying results.
Machine Flow provides both Develop Mode and Service Mode. Develop Mode is used to actively add new R and/or Python code to our service. Any additions/changes to R or Python are exposed immediately by their respective REST endpoints, allowing Data Scientists to quickly build their GUI-service.
Service Mode is used when there is a significant amount of machine learning capability already added to a Data Scientist’s GUI-service. In this scenario, there is less need to move back-and-forth between the back-end and front-end. The idea behind Service Mode is to provide a containerized runtime, that team members use to build machine learning workflows without having to write code.
We will start with Develop Mode since this gets us up-and-running, and is how most Data Scientists would use Machine Flow.
You can review the README by exploring the project on GitHub. The easiest way to download Machine Flow is to open terminal and run the following command:
git clone https://github.com/WorldofDataScience/machine_flow.git
This will bring Machine Flow into your directory. Type ls to view the contents:
machine_flow├── README.md├── app├── data├── docker-compose.yml├── python_ml├── r_ml
The 4 subdirectories are most important:
app contains the front-end application;
data holds datasets and log files;
python_ml holds our python scripts;
r_ml holds our R scripts.
First, let’s run Machine Flow’s front-end. We can start a simple Python web server running the following command in terminal:
python3 -m http.server
We are now serving HTTP on port 8000. Open your browser to the following link:
http://localhost:8000/app/machine_flow.html
You should see the Machine Flow application with a blank workflow.
Our front-end runs R and Python code by sending functions to our back-end via REST endpoints. As such, we need to start our R and Python services in the back-end.
Open 2 new tabs in terminal. In the first new tab, change into the r_ml directory and run the following command to start the R service:
node connect.js
In the other new tab, change into the python_ml directory and run the same command to start the Python service:
node connect.js
Each tab should show its respective service ready to serve:
> R Machine Learning on port: 9191
> Python Machine Learning on port: 8181
Any dataset we work with must be added to the data folder. You can add this any way you want; the only real requirement is all data are in JSON format.
Let’s add the Boston Housing Dataset from GitHub:
https://raw.githubusercontent.com/selva86/datasets/master/BostonHousing.csv
Open a new tab in terminal (make sure you’re in the machine_flow directory) and start an R session by typing R + enter. We add data to our data folder, in JSON format, by running the following:
library(jsonlite)df <- read.csv('https://url_to_boston_data.csv')df_json <- toJSON(df)write(df_json, file='data/boston.json')
We can use the same approach in Python, by starting a Python session and running the following:
import pandas as pdimport jsondf = pd.read_csv('https://url_to_boston_data.csv')with open('data/boston.json', 'w') as f:    f.write(df.to_json(orient='records'))
In either case, the JSON output has the following format, where each observation is contained in its own object:
[{  "feature_a" : "100",  "feature_b" : "150",  "feature_c" : "40" }, {  "feature_a" : "720",  "feature_b" : "14",  "feature_c" : "431" },....}]
For our front-end to do anything useful with our data we must add R and/or Python functions to the back-end. Before writing our first back-end function we need review a few requirements to make back-end functions work with Machine Flow.
Every back-end function must have its first 2 arguments as read_path and write_path. These are used by the function to read the correct dataset and write the result to the correct path. These paths are managed automatically by Machine Flow, so we only have to ensure we use them in our back-end function.
As we can see in the above figure, any additional arguments needed for our function come after read_path and write_path.
We also ensure every function ends with a return(‘done’) statement.This tells the node layer that R/Python has finished executing its function.
Adding our First Back-End Function
The first back-end function that must be supplied is the list_data function. Machine Flow requires we name this function list_data. All other functions can be named whatever you like.
We can of course use either R or Python to write our list_data function. I will use R as follows:
list_data <- function(read_path, write_path) {    res <- toJSON(list.files(read_path, pattern = ".json"))    write(res, file=write_path)    return('done')    }
This uses R’s native list.files function to list all files in the path provided. Notice I added the read_path and write_path as arguments, and used them as needed inside my function.
Where do we put this function?
We add our back-end functions to the appropriate script inside the r_ml and python_ml directories of Machine Flow.
machine_flowr_ml├── R_Scripts    └── data_gathering.R    └── data_preparation.R    └── model_building.R    └── model_validation.R
It makes more sense to open an IDE at this point, rather than just terminal, since we will be managing multiple scripts. I use PyCharm; you can use whatever suits you best.
Since we wrote our list_data function in R we add it to our R_Scripts. Navigate to R_Scripts inside the r_ml directory and open its contents. Open utility_functions.R. You will see Machine Flow comes with the list_data function already added, along with a few other functions. Since it’s already there we don’t need to add it. If you are listing files with Python instead you can add your list_data function to utility_functions.py in the python_ml directory.
Since we have a back-end function in place, we can open the Machine Flow GUI and add this function under ENDPOINTS:
Clicking on ENDPOINTS opens the FUNCTIONS overlay. This is where we tell the Machine Flow GUI what back-end functions are available from our machine learning service.
We need to add our list_data function. The default endpoint is set to port 9191, which is the port for our R service so we won’t change that. We will add list_data as the function name, choose DATA as the expected output, and click ADD:
We need to specify the expected output of any back-end function we want to expose to the Machine Flow GUI. This ensures Machine Flow knows what kind of result to show when running the function.
Close the FUNCTIONS overlay and click on the first node of your workflow. Hit the enter key to add a new node. Any node in the first layer is a DATA NODE. These nodes start a new branch on our workflow.
Clicking ADD DATA pops open the modal and presents a dropdown, listing any datasets we added to the data folder. We can see the boston dataset we added, as well as the iris dataset that comes with Machine Flow.
Select boston.json from the list to show the dataset. This is how we load data into our GUI to begin a new branch. If we wanted to add another dataset we would click on the root node and hit enter, choosing a different dataset to work on.
Any node added after the first layer is a TASK NODE. This is how we build out our workflow, by adding and running tasks that branch off a starter data node.
Click on the boston node and hit enter, then click on ADD TASK. Any functions we add to ENDPOINTS will appear in this dropdown. Since we haven’t added any functions yet (list_data is not listed here) we can do that now.
Machine Flow comes with a few functions in both R and Python to run through this tutorial. We can imagine these are endpoints either written by us or exposed by our Data Science team.
The functions that comes with Machine Flow are the following:
list_data (9191) (data)
show_outliers (9191) (image) (feature)
show_distribution (9191) (image) (feature)
show_missing (8181) (image)
normalize_data (9191) (data)
remove_features (9191) (data) (features)
split_data (8181) (data) (target_feature, split_percentage)
run_linear_regression (8181) (data)
actual_vs_predicted (8181) (image)
show_mse (8181) (image)
show_r_squared (8181) (image)
The port is shown in the first parentheses. The expected output is shown in the second parentheses. If needed, parameters are shown in the 3rd parentheses.
Let’s use these to experiment with the Boston dataset. Open ENDPOINTS and add each function with its expected output. Check to ensure the port number on the endpoint is correct per function:
Some of these functions require parameters as well. We can add those by clicking on the parameters icon for the appropriate function:
When we save our workflow later these endpoints, functions and parameters are saved so we won’t need to re-add them.
Everything is in place to start building out workflows with our Boston dataset. Click on ADD TASK and check the dropdown. You should now see all the functions we added. Let’s start exploring the data. Choose show_outliers from the dropdown:
When we choose a function we are presented with input fields for any features expected, as well as a function call, which is the back-end function that gets called. The first 2 arguments are the read_path and write_path that gets passed to our back-end function. These are set automatically by Machine Flow so we can leave those alone. As we type in the parameter it gets added dynamically to the function call.
This particular function accepts an ALL argument to be passed as the feature parameter, which is useful when wanting to spot outliers. Type in ALL in the input field and click RUN:
We can see the result is returned once the code on the back-end completes. As discussed in the requirements above, the visual produced is a native R visual. I also add a name to top of the modal and close it. Any time I click on this node the result of that task will show.
There are a number of checks in place Machine Flow uses to ensure data dependencies are not violated:
You cannot run a task unless the parent node already ran its task;
You cannot change the dataset on a first layer node once its child node is created;
If the previous node is an image, Machine Flow will search the path until the most recent dataset is found.
NOTE: A branch contains all nodes that share the same first-layer node. A path contains all nodes that are immediately connected to each other.
Let’s continue building the workflow. Since outliers are most extreme for features crim, zn and b I am going to remove these features from the dataset. Let’s choose the remove_features function. We can see in the back-end this function was written to accept a comma-separated string of feature names, used to remove features from the dataset:
remove_features <- function(read_path, write_path, features) {    res <- read_json(read_path)    rem_vec <- unlist(strsplit(features, ', '))    res <- res[,!(names(res) %in% rem_vec)]    res_json <- toJSON(res)    write(res_json, file=write_path)    return('done')    }
We thus know how to add arguments for this function call:
We can see why we must provide expected outputs when we add endpoints. Whereas the first example produced an image this example shows the resultant dataset.
I will now split the data, train a linear regression model, and show a plot of predicted versus actual values:
I can spin-off multiple types of validation from this branch, since it points back to the data trained by the model. Since this is regression I will check both the r-squared value, and the mean-squared error:
A major benefit of Machine Flow is the ability to rapidly attempt many options. For example, I started a new branch where instead of removing all features with outliers, I normalized the data, removed only the b feature, and chose a split percentage of 0.4. I then trained and validated a regression model like last time.
I also created a new node halfway through the previous branch, after the data had been normalized and the b feature removed, then chose a different split percentage of 0.5.
Finally I created yet another node off the original dataset and explored some of its distributions rather than outliers.
This is what my workflow looks like:
Bring able to add new nodes at any point along any branch is more in-line with how Data Science work progresses.
Another great feature of Machine Flow is the ability to highlight the path leading to a final result. This can be useful when workflows become large and complex. For example, I’d likely want to quickly see which steps led to my best R-Squared value of 0.688. I can pop open the node containing that result and click on the show path icon:
Machine Flow saves all work automatically in the browser. This means you can refresh, and even delete your cache, and your workflow will still be available. Of course we likely want to create more than one workflow, so clicking on the SAVE button will download a JSON file containing all nodes, endpoints, functions and parameters, as well as the read and write paths attached to each node.
Loading a saved workflow is as simple as clicking on LOAD and uploading the JSON file.
NOTE: You can share workflow files with other members on the team, as long as their GUI has access to the same data folder.
As mentioned above, service mode is the goal we work towards; a containerized runtime that serves common machine learning tasks to an army of Data Scientists. This is a kind of “internal productization” where we create an application for use within our organization.
We can imagine the data team actively adding new tasks to the service. The benefit of building custom GUI-services is the service is unique to the organization. No vendor tool is going to capture the nuanced modeling and validation that suits your organization’s product needs. Building an internal service that enables rapid assessment of new datasets, and viable pipelines, sets us up for fruitful product discovery.
Machine Flow uses Docker volumes to persist data. As stated in the Docker documentation, volumes are the preferred mechanism for persisting data generated by and used by Docker containers. A great thing about volumes is they allow data to be communicated between the host machine and a Docker container in real-time. This is important since we should be able to add new datasets without restarting the service.
Machine Flow uses Docker Compose to spin-up R and Python services in a single call. Docker Compose is a tool for defining and running multi-container Docker applications.
If we look at docker-compose.yml inside Machine Flow we can see where the volumes are mounted. We need to set the path under volumes to our machine’s root directory:
version: '3'services:  r_ml:    # builds R_ML Dockerfile    build: ./r_ml    ports:     - "9191:9191"    volumes:     - /path_to_directory/data:/app/data  python_ml:    # builds Python_ML Dockerfile    build: ./python_ml    ports:      - "8181:8181"    volumes:     - /path_to_directory/data:/app/data
The complete path needed for mounting volumes is the path we see when typing pwd in console (when in the machine_flow directory).
For example, on my local machine I add the following line before the :/app/data under volumes:
volumes:     -/Users/seanmcclure/PycharmProjects/machine_flow/data:/app/data
Be sure to do this for both r_ml and python_ml sections of the yml file.
Whereas Develop Mode can use libraries by simply importing them as needed, a containerized service must bring in these libraries in the Dockerfile.
Adding libraries for the R service:
FROM r-base# install R packagesRUN R -e  "install.packages(c('dplyr', 'tidyr'), repos = 'http://cran.us.r-project.org', dependencies=TRUE)"# install nodeRUN apt-get updateRUN apt-get install -y nodejsRUN apt-get install -y npmADD . /app# install node dependenciesRUN cd /app; npm installWORKDIR /appEXPOSE 9191CMD node connect.js
In this case, we simply list them within the RUN command. Above, I am installing dyplr and tidyr.
Adding libraries for the Python service:
FROM nikolaik/python-nodejs# install Python modulesRUN pip3 install -U numpyRUN pip3 install -U scipyRUN pip3 install -U scikit-learnADD . /appWORKDIR /appEXPOSE 8181CMD node connect.js
For our Python service we install libraries using pip3 install. Note the -U flag for bypassing prompts.
As we know, library installation doesn’t alway go as planned. Remember, Machine Flow has log files for both R and Python. To view log files inside the container simply “exec into” the container. For example, getting into the r_ml container we can run:
sudo docker exec -i -t machine_flow_r_ml_1 /bin/bash
... and look at the r_log.log file for R and py_log.log for Python, inside the data directory:
vi data/r_log.log
In Service Mode we need to make changes to the paths Machine Flow uses. First, click on the gear icon at the top:
And change the paths to these:
Also, open connect.js in both r_ml and python_ml directories and change:
fs.appendFileSync("../data/r_log.log", data)
to
fs.appendFileSync("data/r_log.log", data)
We build the r_ml and python_ml images using docker-compose by running:
docker-compose build
We then run our containers by running:
docker-compose up
You should see the following to confirm the services are up-and-running:
Attaching to machine_flow_r_ml_1, machine_flow_python_ml_1r_ml_1       | R Machine Learning on port: 9191python_ml_1  | Python Machine Learning on port: 8181
NOTE: If you are on the same machine as Develop Mode be sure to stop the node services running on ports 9191 and 8181 before running docker-compose up.
Whether you want to start with Machine Flow or build your own GUI service, I think it’s important Data Science moves in this direction. GUI-fying the Machine Learning Workflow forms a complete DevOps responsibility by automating the required exploration that precedes product discovery. Machine learning does not have options, it has possibilities. By rapidly assessing the readiness of our data, and exposing viable paths from raw data to validated model, we set ourselves up for more success on machine learning projects.
As always, if you run into issues please ask questions in the comments section.
If you enjoyed this article, you might also enjoy: