Building a Deployable Jira Bug Classification Engine using Google AutoML
By: Heemeng Foo
Tags: Machine Learning, Jira, AI, Classification, Automl

Building a Deployable Jira Bug Classification Engine using Google AutoML

A step-by-step approach

The pyramids as seen from Giza (credit: author’s own collection)

Author’s note: I chose this picture as one of the takeaways from this exercise was how complex AutoML is under the hood but Google has done such a great job of making things simple. The various AutoML tools reside together with other GCP tools like Cloud Storage, App Engine etc much like how the pyramids reside together with the rest of the city of Giza.

Introduction

In the first article in this series, I detailed how I built a Jira bug classification engine using Tensorflow. In the follow-up article, I described a step-by-step approach to using Amazon’s BlazingText to build and deploy a similar engine. In this article, I detail the steps to do the same using Google’s AutoML.

As in the previous article, I will be using the data-set found in [1].

My personal objectives for doing this were the following:

Get a feel of using Google’s AutoML engine in particular the “Natural Language” toolset Perform a comparison of the classification results between AutoML and BlazingText Perform a comparison of the steps to deploy the classification engine

As in the previous article, you can find the notebooks and data files in Gitlab:

I’ve created a folder AutoML/textClassification to store the files in this project.

As most of the steps to build this classification engine is via the UI, I’ve put all the steps in the Google Colab notebook GCPAutoMLDataPrepTrainEval.ipynb found in the Gitlab repo.

What is AutoML?

I would probably not do a whole lot of justice in explaining what AutoML in any depth here so I’ll refer you to a video for that (see [2]).

In a nutshell: remember when you did Machine Learning using Scikit-learn or Tensorflow? What you would do would be to try several different models and sets of hyper-parameters. This whole process could be automated and that is essentially what AutoML does.

So in this case, what you need to do is (and similar to BlazingText), you’ll need to format the training dataset into a format that the particular algorithm class (in this case text classification) will accept and then let AutoML do the rest.

Google Cloud and Cloud Natural Language

There are several GCP (Google Cloud Platform) tools used for this project:

Google Cloud Storage Cloud Natural Language Google App Engine

Author’s note: if you have an interest to pick up Machine Learning on GCP, I highly recommend Coursera’s Data Engineering, Big Data, and Machine Learning on GCP Specialization online course. It is self-paced and there are lots of labs that come free. There is also a YouTube series on basics of GCP that you can find here [3].

One thing great about GCP is that they actually give you $300 of service free when you start out. So if you just want to dip your toes into it, this is a big incentive. To find out more see https://cloud.google.com/free.

Getting to Cloud Natural Language

For those familiar with GCP this is a no-brainer: just navigate to the the GCP console and look for “Natural Language” under “Artificial Intelligence” on the side menu.

For those new to GCP, you will need to:

Go to “http://console.cloud.google.com” on your browser. Create a new Project with a service account (you need the service account to make predictions later) see instructions here. If you click on the side menu, you will be able to navigate to “Natural Language” under the “Artificial Intelligence” section.

When you reach “Natural Language” you will see:

Click on the “Get Started” link on “AutoML text and document classification” section. The page you will see is the “Data Set” page.

Data Preparation

It took me a while to figure out what was the way to format the training data after I clicked on the “New Data Set” link, selected “Multi-label classification”. and tried to import the CSV file. Turns out the system needs the data in another format.

This I found out by clicking on a tooltip (ie. the “?”) after selecting “Upload zip file from your computer”. This is what is says:

I decided to use the first approach. Here is the code I used to convert my CSV to this format (you can find it in the notebook as well):

Importing the Data

The next part was downloading the documents.zip and uploading to AutoML for processing. This can take some time but the nice thing is that they will send you an email once completed.

Once processed, you will see something like this when you click on the “Items” tab:

What’s great is that you can click on “View Label Stats” to see the distribution of the labels etc and you can review the data as well. I personally really liked this about the tool.

Training

To kick off training, just click on the “Train” tab and click “Start Training”. This will take a few hours to run as it is searching for the best algorithm and hyper-parameters. You will get a email when the task is completed.

Evaluate Model Training

Once the training is done you’ll see something like this:

Click on “See full evaluation” and you’ll see something like this:

You can select each label on the left to see the precision and recall metrics. One thing I really like about this is they actually list out the true positives, false negatives and false positives for you to do some kind of analysis.

Deploy the Model

This couldn’t be more straight-forward. Click on the “Test and Use” tab and click “Deploy model”.

Author’s note: make sure to delete the model when you’re done as you will get charged if you keep it running.

Testing the Model

Once the model is deployed you can take your model for a spin (pun intended). There are a number of ways to do this and Cloud Natural Language provides a few ways of doing that.

The first is just to type in some text and let the model predict:

Next boiler plate code is provided below (sh and python). You can do it on your own machine or open a Google Cloud Shell for that.

Exposing the Model as an API

For this I used Google App Engine. This is fairly straight-forward.

Create the AppEngine App

First navigate to App Engine on the Google console. Then look for the Quickstart guide. Select Python on the “Standard environment walkthrough” and follow the steps to clone the boilerplate sample code in Cloud Shell.

Under the “appengine” folder you will see the following 4 files:

app.yaml main.py requirements.txt test.sh

Next, what I did was to use the test Python code to merge it with the sample AppEngine code:

A few things to note:

model_name will be different for you

the service account creds will be different as well

You can test this locally (or on google cloud shell) by running:

python3 main.py

sh test.sh

you should get something like:

{

"prediction": "Major",

"score": 0.7177224159240723,

"statusCode": 200

}

Deploy the App

First you need to create the app (I did this on google cloud shell):

gcloud app create

Next deploy the app:

gcloud app deploy app.yaml \

--project <project id>

Test the deploy

Once the app is deployed, you will see the App Engine app URL on the screen. You can then use curl to test it out:

curl -X POST -H "Content-Type: application/json"

-d '{"title": "TestSizeBasedThrottler fails occasionally",

"description": "Every now and then TestSizeBasedThrottler fails reaching the test internal timeouts.I think the timeouts (200ms) are too short for the Jenkins machines.On my (reasonably fast) machine I get timeout reliably when I set the timeout to 50ms, and occasionally at 100ms."}'

"https://octopus-282815.uc.r.appspot.com/predict"

Evaluating the Model

One of the objectives of this exercise was to evaluate to see how the model generated by AutoML compares with BlazingText in the previous article. As I already had the test data set created for BlazingText, I just needed to use it again.

Using the Test data-set

Here is the code I used

The output:

This is close to the BlazingText result.

Using another data-set

As in the BlazingText article, I also used the Zookeeper data set to evaluate the model.

This is what I got:

(the NaNs are due to division by zero as some of the labels do not exist in this data-set)

This is also similar with what was obtained using BlazingText.

Conclusion

Overall this was my impression after using GCP’s AutoML Natural Language Classifier.

The performance of the algorithm was similar to that of BlazingText. The documentation on how to format the training data was a little hard to find at first. I really liked the way the data was presented after ingestion as it gave me some means of analyzing it without having to write any code. I really liked the tools to evaluate the model after training especially the tables showing which were true positive, false positive etc It was fairly easy to build a server-less API interface using App Engine. For BlazingText I was using a basic template that was provided by the AICamp course to work with. I really liked the fact that there was the Cloud Shell to do CLI stuff. For BlazingText I had to use a shell in Sagemaker’s Jupyter notebook to do some of that.

References

[1] Jira Social Repository, Marco Ortu et al https://github.com/marcoortu/jira-social-repository

[2] Automated Machine Learning (AutoML) for Keras and TensorFlow (14.1), Jeff Heaton, https://youtu.be/TFUysIR5AB0

[3] Welcome to Google Cloud Platform: the essentials of GCP, Google Cloud Platform, https://youtu.be/4D3X6Xl5c_Y