Why should you, the AI Product Manager, care about baseline models
By: Anand Ravi
Tags: Editors Pick, Product Management, Office Hours, Fastai, Data Science

Why should you, the AI Product Manager, care about baseline models

Baseline models can help you get started with very little effort. You can leverage them to ensure that your team is on the right track Anand Ravi Dec 22, 2020·9 min read

You are a Product Manager and want to incorporate Machine Learning or Deep Learning (a sub-field of Machine Learning) into your Product. You have heard that incorporating these algorithms into your Product can lead to several benefits but are wondering where to get started? How would you know if you are on the right track? Is there a low-effort way to assess if the results of the fancy models presented by your Data Science team are in-line with the objectives of your organisation? The answer to these questions lies in building a baseline model.

Below are 4 reasons as to why you should invest time and effort to build a baseline model that you are confident will perform reasonably well.

Baseline models are simple to build and are easy to test. In less than a week (depending on the data quality in your organisation), you can get a sense of the value that can be delivered by incorporating Machine/Deep Learning algorithms into your product Machine/Deep Learning products are unique in that their performance is always relative to something. As an example, if the more sophisticated models lead to an accuracy of 97% but the baseline is 94%, you are investing a large amount of work only for 3% improvement. However, if the baseline accuracy is 65% while the more sophisticated models leads to 90% accuracy, it is likely that your team is on the right track You can communicate credibly with the Engineering and Data Science teams. Teams prefer being led by a PM who has invested time to understand the data. Without starting with a sensible baseline, it is challenging to gauge whether the fancy models presented by your Data Science team will add any real value You can use the results of your baseline model as an input to Roadmap planning and prioritisation activities. As Product Managers, we are always defining and prioritising what needs to be built in order to achieve the strategic objectives of our companies. Performance of baseline models can sometimes be used to determine how much value (in $$ terms) your team can commit to by the end of the Quarter

What are baseline models?

Baseline models are simple algorithms or heuristics that are straightforward to build and are easy to test. Ideally, they should not involve advanced Mathematics or the utilisation of expensive computing resources. A rule of thumb is, if it takes you under two weeks to develop your baseline model, you’re likely on the right track. Otherwise, it might be a good idea to go back to the drawing board to figure out a more efficient way to construct your baseline. Lets now look at a toy example of a baseline model — both the algorithm and the Python code (using Fast.ai libraries)

Toy example of a baseline model — Problem statement and task

Problem statement: Let’s consider the equivalent of the “Hello World” program i.e printf("Hello, World!"); in the field of Computer Vision. Your team’s objective is to build a Deep Learning model that can recognise handwritten digits. In order to narrow down the problem statement, we will only consider handwritten 0’s and 1’s in this example. Below are some important details about our data set of handwritten digits

They are all saved as images in png format. They are part of the MNIST database. The database contains images of handwritten digits and was collected by the National Institute of Standards and Technology and aggregated into a machine learning dataset These handwritten images are in greyscale format i.e they are black-and-white images Each image contains 28 pixels across and 28 pixels down. This means that each of them contains a total of 784 pixels (28*28) Each greyscale pixel can hold any value between 0 and 256. If the pixel value is 0, it is plain white. If it is 256, then the pixel is the darkest shade of grey. Let us see how some randomly selected images look based

Example of a handwritten one randomly selected image from our training set. Image by the author

The above image is an example of a randomly selected image of a handwritten one from our training set. As you can see, there are 784 pixels (28*28). Pixels whose value is 0 are white in colour. Pixels whose value is closer to 256 are darker shades of grey

Example of a handwritten zero randomly selected image from our training set. Image by the author

In line with the best practices of Data Science, our dataset is divided into training and validation sets. In our training set, we have 5923 images of handwritten 0’s and 6742 images of handwritten 1’s. While in our validation set we have 980 images of handwritten 0’s and 1135 images of handwritten 1's.

Task: As an AI Product Manager, you want to build a baseline model to get a sense of what value can be delivered before engaging with the Data Science team. Furthermore, by building a baseline model, you will also get a chance to intimately familiarise yourself with the dataset.

Toy example of a baseline model — the algorithm

Our algorithm to build the baseline model consists of the following two simple steps

For each of the 5,923 images of handwritten 0’s in our training set, for every image, take the average over the pixel value (or its greyscale intensity) in that image. The resultant group average will contain 784 greyscale pixels in a 28*28 matrix. We will consider this the “ideal” 0, i.e our baseline model for handwritten zeros For each of the 6,742 images of handwritten 1’s in our training set, for every image, take the average over the pixel value (or its greyscale intensity) in that image. The resultant group average will contain 784 greyscale pixels in a 28*28 matrix. We will consider this the “ideal” 1, i.e our baseline model for handwritten ones

Toy example of a baseline model — Python code

This section contains the Python code for constructing the baseline model. Feel free to skip this section if the technical aspects are not of interest to you

#Built using Fast.ai libraries

!pip install -Uqq fastbook

import fastbook

fastbook.setup_book() from fastai.vision.all import *

from fastbook import *

matplotlib.rc(‘image’, cmap=’Greys’) path_train = Path("""/notebooks/storage/Anand/train_anand""") #Storing the file paths of the handwritten 0s and 1s

zeros_train = (path_train/'0').ls().sorted()

ones_train = (path_train/'1').ls().sorted() #Storing the files as a list of tensors (using list comprehension)

zero_tensors_train = [tensor(Image.open(o)) for o in zeros_train]

one_tensors_train = [tensor(Image.open(o)) for o in ones_train] #combine all the images in this list to form a rank-3 tensor

#this will help us perform functions like taking averages

stacked_zeros_train = torch.stack(zero_tensors_train).float()/255

stacked_ones_train = torch.stack(one_tensors_train).float()/255 #perform step 1 of the algorithm: Take the average pixel value for #every pixel of the handwritten 0’s. The resultant group average is #the “ideal” 0 mean0 = stacked_zeros_train.mean(0) #perform step 2 of the algorithm: Take the average pixel value for #every pixel of the handwritten 1’s. The resultant group average is #the “ideal” 1 mean1 = stacked_ones_train.mean(0)

Eyeballing the results of the baseline model

In the previous section, we built our baseline model using a simple algorithm. Let’s see what our “ideal” 0’s and 1’s i.e the results of our baseline model look like.

Baseline model for handwritten 0’s. Image by the author

Baseline model for handwritten 1’s. Image by the author

It is worth noting that the baseline model for handwritten 0’s and 1’s look pretty close to the actual digits!

Assessing the baseline model — a quick and dirty approach

Let’s now assess the accuracy of our baseline models using a quick and dirty approach. Below are the steps

Select a random handwritten 0 from our dataset Measure how far off the pixels of this randomly selected 0 are from our “ideal 0” and from our “ideal 1”

Expected result: If our baseline model is correct, we should observe that the randomly selected 0 will be closer to the “ideal 0” than it is to the “ideal 1”.

The randomly selected handwritten 0 is closer to the baseline model of 0 than it is to the baseline model of 1. Image by the author

Observed result: As you can see above screenshot, the distance of the randomly selected image of handwritten 0 from the baseline model of Zero (aka “ideal 0”) is 0.15 while its distance from the baseline model of One (aka “ideal 1”) is 0.22. As expected, the randomly selected 0 is closer to the baseline model of Zero than it is to the baseline model of One

Conclusion: Therefore, we can conclude that our baseline model correctly classified the randomly selected image,

Assessing the baseline model — an analytical approach to calculate the Accuracy

However, it is not sufficient to just look at randomly selected images. We need to calculate how accurate this baseline model is when it comes to classifying the images in the entire validation set. It is worth to pause at this point and remind ourselves of two points

Quick reminder 1: Our validation set consists of 980 images of handwritten 0’s and 1135 images of handwritten 1’s

Quick reminder 2: The images in the validation set were not used to build the baseline model. The images in the training set were used

Metric: The metric that we will use to assess our baseline model is known as the Accuracy. It is the “the number of images that the baseline model correctly predicts (either as 0 or as 1) divided by the total number of images”. More formally, Accuracy is defined as

Accuracy = number of correct predictions / total data points

We will use the same logic that we used in the quick and dirty approach to measure the accuracy of our baseline model. If the pixels of a given image are closer to our “ideal 0”, then we will predict it to be 0. Whereas if the pixels are closer to our “ideal 1”, then we will predict it to be 1.

Accuracy of the baseline model = 95%. Image by the author

The above code snippet shows that the Accuracy of our baseline model is 95%. This result shows that the baseline model that we built is a great place to start! We managed to get over 90% accuracy on both 0s and 1s simply by using common sense and basic Mathematics (averages).

Summary: key takeaways from the baseline model shown above

It is worth noting that, in order to build our baseline model

No Deep Learning / Machine Learning algorithms were used No Calculus was used No Linear Algebra was used No expensive GPU/TPU/other hardware is required Only basic Arithmetic (calculating averages) is required

Inspiration for this post

There is a tonne of great content such as this, this and this on how to approach baseline models for Data Scientists and Data Engineers. However, there is a widespread belief that Product Managers should play a more active role in data collection and in the training of datasets. As Andrew Ng said during his talk at the State of Artificial Intelligence

“…we’re training Product Managers to use a new language to speak to the Engineers to give Product specifications. For example, Product Managers now know to train a dataset and and provide that to the Engineer and say…. are you able to deliver 98% accuracy on this dataset at detecting cars between 50 and 100 meters…”.

I wrote this post to demonstrate that Product Managers (and almost anyone) can conceptualise and build baseline models. Even if you are not a programmer, you can devise the algorithm and collaborate with an Engineer to build. As seen in this example, it is possible to design the algorithms for baseline models using common sense and very simple Mathematics knowledge.

You can view the source code on my GitHub profile by clicking here

References

This blog post was inspired by Lesson 4 of the Fast.ai course.

Connect with me

LinkedIn: https://www.linkedin.com/in/anandkumarravi/

GitHub: https://github.com/Anandravi87

Twitter: https://twitter.com/Anand_1187