Build a useful ML Model in hours on GCP to Predict The Beatles’ listeners | by Brian Ray | Towards Data Science

AutoML “Automated Machine Learning” is under fire lately by the Data Science Community. I had some criticism published in on Google Cloud ML Google Cloud’s AutoML first look (the Google AutoML Team did answer some of my requests, see **new** things at the end of this article) This post will show it is completely reasonable for a Citizen Data Scientists to handle the whole Machine Learning life cycle under ML. The goals are:
Solve a real business problem
Host the data in GCP
Use GCP’s Cloud AutoML
Conduct some EDA “Exploratory Data Analysis”
How the EDA impacts the results
Data Processing on GCP AutoML Tables
Taking advantage of the built in Data Pipeline
Validation of the model using a holdout
Putting the model in production
Improving the model over time
**new** things in AutoML Tables just released from Google
Start your timer, ready, set, go....
The goal is to determine if a listener of music based on their listening patterns would they likely listen to The Beatles; we will build a Recommendation system. Wouldn’t John Lennon be impressed with the power of AI?!
Certainly you would not want to recommend The Beatle so someone would not likely listen to the Beatles. Who wouldn’t love The Beatles? Regardless, this model would work with any artists where you have enough data on listening patterns. You simple switch out the Target variable and away you go.
The data source is Music listening histories of ListenBrainz users.
We pick the top top 300 most popular artists by number of listens. The data is available thanks to BigQuery’s 1TB/mo of free tier processing.
The Code for creating the data set is available here on my github. If you are impatient and you just want to download the data here (2.19 MB).
I already posted more detail on how to get started with GCP Cloud AutoML tables in Google Cloud’s AutoML first look
From Cloud AutoML we can investigate the data in greater depth. This is nowhere close to the level of detail you could do as a Data Scientists working on AI Platform. Nonetheless, there are some worthwhile observations here. For example, the most listened to artists are those with the fewest Nulls.
What do we do with Nulls in this case: we leave them empty. The reason why is that Null really does mean no-listens in our data set; hence same as zero. A median value here would be a mistake. Know thy Data!
We need to remove “The Beatles” Column as our target is based directly on this. Otherwise, we would get a 100% accurate model always, duh.
The huge advantage is you an pipeline the building of the model without a lot of code. In fact, you can write your entire training code using the Client libraries that support multiple languages including Python.
We use the Automatic Data Split feature in AutoML. They say:
The way your dataset is split between training, validation, and test subsets. By default, AutoML Tables randomly select 80% of your data rows for training, 10% for testing, and 10% for validation.
The model worked very well! We saw a .937 AUC ROC.
And here is the confusion matrix with 50% score threshold:
Is it this easy:
There is a cost. If I will host this particular model it will cost $.126 per hour, or $91.98/month (multiplier 730 hours in a month assumed).
Here is the equation I used: $0.005 * 2.8 * 9 = $.126. The $0.005 is the number of machines (they use 9 for low latency and I believe it is hard coded). 2.8 is because my model is 2.8G in size.
import jsonfrom google.cloud import automl_v1beta1 as automlproject_id = 'mwpmltr'compute_region = 'us-central1'model_display_name = 'beatles_machine_l_20191104025339'input = ['usr', 5.0, None, None, 10, ... ] # list of valuesclient = automl.TablesClient(project=project_id, region=compute_region)response = client.predict(    model_display_name=model_display_name,    inputs=inputs)response###############################################################payload {  tables {    score: 0.9896599650382996    value {      string_value: "True"    }  }}payload {  tables {    score: 0.010340098291635513    value {      string_value: "False"    }  }}
By finding patterns in recommendations by using a tagging system we saw a 3.5% increase in overall accuracy.
The “Trick” is the correction of bad records (improving ground truth data) and the introduction of new data.
This new data in this case is the “tag” applied to the music by experts. Those tags are very predictive. For example the tag “60s” and “british_invasion” are highly predictive. On the flip side, as with all machine-learning it can be predictive what tag the artist does NOT belong too; for example those who have a high level of plays that are tagged as “electronica” they are much less likely to listen to the Beatles.
Here are 3 of the meaningful new features Google has added to AutoML Tables:
You can see what final model architecture was trained as well as the experiments attempted in each training round.
You can see what final model architecture was trained as well as the experiments attempted in each training round.
The details are stored in Stackdriver.
I haven’t looked closely at these logs. At this time, I do not know if the logs contain all the information needed to recreate the models.
2) you can get prediction-level feature attributions on the online prediction.
TypeError: list_models() got an unexpected keyword argument 'feature_importance'
I will probably need to update my packages.
3) you can export your model container (now available for everyone)
Embarking on the AutoML journey is not as scary as I once thought. The Data Scientist complained that the black-box nature of the AutoML is slowly being opened. The ease of use is increasing. The bugs and other issues seem to be getting resolved.
I cautiously recommend folks start putting their eyes on AutoML as a jump starter. This does not relieve the need for Data Modeling outside of what it provides for Data Science. It does not alleviate the need for data handling and EDA outside of the toolset (for this take a look at AI Platform Notebooks!). Likewise it does not mean that Business Analyst type which these tools target do not still need a good understanding of predictive modeling theories, terms, and best practices.