Run TensorFlow Models in the Browser | by André Ribeiro | Towards Data Science

IntroductionTrain a digit recognition model with TensorFlowHost the model in S3 with TensorFlow.jsConclusionReferences
Introduction
Train a digit recognition model with TensorFlow
Host the model in S3 with TensorFlow.js
Conclusion
References
Making a machine learning model available to your customers requires it to be accessible either through an Application Programming Interface (API) or a graphical interface like a Web Application (Web App).
Both APIs and Web Apps require a server to provide the computing requirements to run them. For most use cases this server is configured and maintained by the developer, but in some cases it does not need to.
Serverless architectures allow the developer to abstract from the architecture and maintenance so they can truly focus on developing and improving their models1. Serverless computing has additional advantages over classical servers such as:
Lower costs if your application is used infrequently (low number of customers, or low frequency).Scale the model automatically from few users to thousands.Improved reusability as serverless typically adopts a modular approach.
Lower costs if your application is used infrequently (low number of customers, or low frequency).
Scale the model automatically from few users to thousands.
Improved reusability as serverless typically adopts a modular approach.
In this post, we’ll cover how to train a Tensorflow model for digit recognition and making it available in a serverless fashion.
For a brief introduction to modelling with Tensorflow check my previous post “Getting started with Tensorflow”. If looking for a brief introduction to serverless hosting with S3 check this one instead “Build a serverless website using Amazon S3 and Route 53".
In this section we will train a simple digit recognition model using the MNIST dataset provided through the TensorFlow library2.
To load the dataset, in the first cell of you notebook type the following.
This will display the shape of the training data inputs (tx) and target (ty), and validation data inputs (vx) and target (vy).
tx:(60000, 28, 28, 1), ty:(60000,) vx:(10000, 28, 28, 1), vy:(10000,)
Let’s now display a set of 10 sample images for each digit to understand how the data looks like.
This will generate the following grid of images. Note that the image has a black background and white foreground. This will be relevant when defining our canvas to draw numbers later on.
For the vision model, let’s start by defining our convolution building block. This block will include a convolution, batch normalisation, relu activation, max pooling and dropout layers.
Our full network will consist of 2 ‘normConvBlock’, followed by a flatten layer, and a final Dense layer with softmax activation.
To train the model we can use the ‘Adam’ optimiser and a categorical cross entropy loss. In this example, we will use the “sparse_categorical_crossentropy” as our target data is ordinal encoded.
The following model structure will be shown.
We are now ready to train our model. To do so we can simply call ‘model.fit’ with the processed data defined above. To make sure we get the best model iteration, let’s define an early stopping callback as follows.
Amazing we just trained a model to recognise single digits! The model accuracy is good enough for this example (~99% for the validation set). If you need a more accurate model change the model structure or perform some type of data augmentation.
Epoch 11/20 59/59 [==============================] - 3s 50ms/step - loss: 0.0317 - acc: 0.9906 - val_loss: 0.0338 - val_acc: 0.9899 Epoch 12/20 59/59 [==============================] - 3s 50ms/step - loss: 0.0292 - acc: 0.9913 - val_loss: 0.0496 - val_acc: 0.9823 Epoch 13/20 59/59 [==============================] - 3s 50ms/step - loss: 0.0275 - acc: 0.9918 - val_loss: 0.0422 - val_acc: 0.9866
For the full script go to my Github page by following this link:
github.com
Or go directly to the Google Colab notebook by following this link:
colab.research.google.com
In this section we will develop the bare minimum HTML + Javascript components to interface with our TensorFlow.js model.
TensorFlow.js is an open-source library to train and run machine learning models completely in the browser, using Javascript through a high-level API3.
The first step we need to perform to host a TensorFlow model in the browser is to convert it to a TensorFlow.js model. To do this we can simply call ‘tensorflowjs.converters.save_keras_model’ from the tensorflowjs library.
If using Colab download the files to your local machine to use later.
Let’s first create the folder structure for our web app. Start by creating a main folder called ‘digit_recognition’, and in it create two files named ‘index.html’ and ‘script.js’.
Additionally, inside ‘digit_recognition’ create a folder called ‘tensorflow’. Copy the two files we download from the Colab notebook ‘group1-shard1of1.bin’ and ‘model.json’ into this new folder.
Your folder structure should look like this:
- digit_recognition/  - index.html  - script.js  - tensorflow/    - group1-shard1of1.bin    - model.json
Index.html
Open the index.html file with any code editing tool and copy the following code into it. This script defines the basic components to draw a digit and recognising it using the trained model.
Let’s now dissect the code above.
Lines 3 to 11 define the head of the html file. In here we load the necessary TensorFlow.js libraries.
Line 23 creates a button which we will use to call the model inference function (‘predictModel’). When the button is pressed the model will be triggered and the result of the prediction shown in the <p id=’result’> tag.
Line 26 creates a button to clear the canvas (‘erase’).
Line 31 defines a canvas object that will allow us to draw a digit to be later recognised by our trained model.
Line 35 sources the ‘script.js’ file that contains the JavaScript logic to get the data from the canvas into a Tensor, and apply the model to it.
script.js
Open the script.js file with any code editing tool and start adding the following code snippets.
Define the main variables we will need in this script.
Define the main variables we will need in this script.
2. Define the canvas triggers for different user actions.
On ‘mousedown’ (click and hold) the script is triggered to initiate the drawing process and record the current mouse / touch position. On ‘mousemove’ the script starts drawing on the canvas. On ‘mouseup’ (release the click) the script is triggered to stop the drawing process.
3. Check if the device is touch enabled. This is necessary to pick the right event in the ‘setPosition’ function.
4. Set the engage (initiates drawing) and disengage (stops drawing) flags.
5. Record the mouse / touch position. Note that we need to know if the device is touch enabled.
6. Define the main function to draw on the canvas.
The ‘dragging’ flag defines whether the user is ‘engaged’ or ‘disengaged’ (line 8). If engaged draws a line between the past recorded position obtained through ‘setPosition’ and the current position (lines 18 to 21).
7. Define the function to clear the canvas.
8. Define the function to load the TensorFlow model.
The ‘tf.loadLayersModel’ function loads a model from an url or local directory (line 5). Since for the very first prediction the model has to initialize the weights, a warmup of the model is often recommended to prevent the delay during first prediction (line 8).
9. Define the function to get the current data from the canvas.
10. Define the function for model inference.
‘getData’ loads the data from the canvas (line 5). ‘tf.browser.fromPixels’ converts the canvas data into a Tensor (line 8). ‘tf.image.resizeBilinear’ resizes the image from the canvas size to the model size (line 11). ‘model.predict’ gets the model prediction (line 14). Line 17 sets the digit with the highest probability ‘y.argMax(1)’ into the <p id=’result’> tag.
For security reasons, browsers restrict cross-origin HTTP requests initiated from scripts. This means that a web app can only request resources from the same origin the application was loaded from unless the response from other origins includes the right Cross-Origin Resource Sharing (CORS) headers.
To achieve this in Chrome / Chromium you can install the ‘Moesif CORS’ extension. This however will still limit loading resources locally. In this tutorial we will be using Firefox instead.
To allow CORS in Firefox, type ‘about:config’ in the Firefox search bar and in the new search bar type ‘privacy.file_unique_origin’ and change the default to ‘false’.
Now open the ‘index.html’ file in Firefox by second-click on the file name and selecting ‘open with Firefox’, or by dragging the file to a Firefox window. You will be greeted with the following landing page.
This is not the prettiest web app by far, but it does the job. Let’s now test the model by drawing a digit and clicking the predict button.
Great. Our model works locally! The next step is to push the web app to a serverless platform like Amazon S3.
To host our web app you first need to create an Amazon S3 bucket and set it to static web hosting. If you want a quick overview on how to do this, please check out my previous post, “Build a serverless website using Amazon S3 and Route 53”.
To make our new serverless web app available, we just need to push the files to S3. Make sure you push the whole project folder ‘digit_recognition’.
We can now test the app in any device using any browser (since all resources are now in the same origin we do not need to activate CORS requests).
To do so, go to http://DOMAIN/projects/digit_recognition/index.html if you have a domain associated with your S3 bucket or https://s3.eu-west-1.amazonaws.com/DOMAIN/projects/digit_recognition/index.html if not.
For the ‘index.html’ and ‘script.js’ full scripts follow the link below. In this repo I also include a better looking app using bootstrap 4 (‘index_mobile.html’ and ‘script_mobile.js’).
github.com
For a demo of the original and bootstrap web apps follow these links:
http://portfolio.dataisawesome.com/projects/digit_recognition/index.html (desktop, mobile)
http://portfolio.dataisawesome.com/projects/digit_recognition/index_mobile.html (mobile)
Using serverless computing allows you to scale your application without managing a web server, while keeping the costs low.
In this tutorial we explored how to host a TensorFlow model in a serverless fashion using TensorFlow.js and Amazon S3. To learn more on how to use TensorFlow.js make sure to check the tutorials provided in the TensorFlow.js official page.
[1] Nočnica Fee, “What Is Serverless Architecture? Key Benefits and Limitations” (August 2020)https://newrelic.com/blog/best-practices/what-is-serverless-architecture
[2] https://www.tensorflow.org/datasets/catalog/mnist
[3] Josh Gordon and Sara Robinson, “Introducing TensorFlow.js: Machine Learning in Javascript” (March 2018)https://medium.com/tensorflow/introducing-tensorflow-js-machine-learning-in-javascript-bf3eab376db