Automated Feature Engineering web app using Flask+MongoDB+Docker and Make it Python installable for running from the command line
By: Weixing Travis
Tags: Docker Compose, Automation, Web App Development, Flask, Docker

Introduction to the web app

The purpose of this web app is to free the data scientist from time-consuming codings when doing the crucial but tedious feature engineering on relational datasets: especially when it involves hundreds of variables spread across dozens of tables. The open-source Python package featuretools will be employed to work out this process. To put it simply, if you had experience in Pandas DataFrame or Spark DataFrame, you would probably do some aggregation operations like “count, mode, sum, std…” in “groupby…agg” or some transformation operations, like calculating the square root or customized functions that are applied on every single row of a column; to the single dataset or relational datasets, a lot. The Featuretools is nothing but manages those operations for you by stacking them (called primitive in its context) deeply and synthetically (this process is called Deep Feature Synthesis, the core concept under the hood).

To run DFS, it needs your inputs on what datasets are used (Entities) and the relations of any two (many to one), which would then form an EntitySet. You would also have the flexibility to specify which of those primitives you intent to use or even define your own primitives using their APIs. There are more things about Featuretools and it takes time to sink them in, which is the reason why I initialized this web app to take the burden off your shoulders.

Without further ado, let’s get into the app.

The logic of the application is to collect the parameters that the featuretools requires to generate features through the user-friendly interface and calculate the feature schema for you (only using a sample of the data) through auto_feature. With the configure file (contains the parameters) and the schema file, you could then run the package auto_feature_prod on the whole datasets easily in somewhere with the adequate computation capacity (I will also write an API for you to integrate with your scikit-learn Pipeline in the coming weeks).

STEP 1: Define the config and schema names you like and provide the s3 path for your data files (only support the s3 bucket right now).

get data-source

STEP 2: Then you will land on this navigation page. In the content, bold black words are what you defined on the above page and red ones are navigation links. As mentioned below, you have to follow the sequence in the navigation panel: the Entities and Relations I explain before; Pivotings will decide which single column or concatenated two columns you want them to be pivoted (for example, dummify a variable); Primitives will offer up ideas on which primitives to be used on which columns.

Navigation Page

For Entities, Relations, and Pivotings, I utilized WTForm to make a dynamic selection and MongoDB to store the results; It is quite interactive that you could view the results as you click the “Submit” button and also are allowed to “Delete” each submission.

Entities Definition Page

After Entities, Relations, and Pivotings, the Primitives Table is created as shown in the pic below. I am intending to give you some basic recommended aggregation (agg_prm) and transformation (tfm_prm) primitives for each feature, plus some backup ones (agg_prm_backup and tfm_prm_backup) for your reference. The table is editable, which suggests that you are able to click the cells in agg_prm and tfm_prm and modify the values: for example, if you feel like that the recommended aggregation primitives, for your claimant number and Name features are not proper, then you just click the corresponding cell and delete it. Those you removed will be automatically added to the backup. Here I leveraged JQuery and Ajax to achieve this dynamic change.

Recommended Primitives — before modification

Recommended Primitives — after modification

STEP 3: Go back to Home (the navigation page is shown in step 1) and ‘Save’ the config file and ‘Preview’ the features. When you click Preview, featuretools’ DFS comes into play: it has two options, one is to only derive schema while the other one is actually calculating the real values. Obviously, the former one will save us a lot of time and I am using it to generate the schema (all the calculated feature names) and provide this webpage for you to adjust the schema before letting DFS do the calculation. There are two tables within this page: the lower one is the “Results” schema and you could “Delete” any of them unuseful that will be listed in the upper “Removed” table in case you regret it.

Feature Schema page

STEP 4: Now you are good to “Save” the schema on the navigation page.

STEP5: Launch the Prod version or integrate the code in Sklearn Pipeline (will come later).

I am not planning to talk too many details about the flask or frontend codes that render this web application for this time, but I would really like to share some of the topics, which really gave me a hard time, in the future if those interest you: why I am using MongoDB, how to make a dynamic selection via WTForm, how to make the values in one cell of a table change automatically in real-time reflecting change in another cell modified by the user (the JQuery and Ajax behind it), etc.

Next, I would explain how to containerize the app and package it up.