Implementing ML Systems tutorial: Server-side or Client-side models? | by Mostafa Ibrahim | Towards Data Science

Developing machine learning models is all fun and good, but after developing them you will probably be looking into deploying them into an app to use them. The question is should you put them on the client-side (which is probably the mobile) or should you put the model on a server, send the data to the server and get the results back? In this article, I am going to be discussing the trade-offs and what those 2 methodologies entail.
Server-side machine learning models are the most widely-used ML systems simply because the architecture is more simple to implement. For example, you have an application running on the phone (the client) that classifies images. The user takes a photo, this photo gets translated into bytes and the phone sends a POST request to the server with those bytes. The server runs the prediction function of the model on those bytes and sends back a JSON response with the prediction to the mobile. Nothing fancy going on here.
Now if you implement your ML system this way, you are going to be faced with some of these challenges:
Latency: If your application requires an intensive ML prediction such as real-time video object detection, you will likely get the results in an inadequate time which will make the user experience significantly unpleasant.Cost: Typical web applications don’t require expensive machines with high-end GPUs to run predications quickly. However, if you host the model on a server, you will need one with a powerful GPU which is quite expensive. For instance, I have seen examples where hosting an image captioning model on an AWS EC2 instance costing around 24$/day (which will be around 9,000$ / year). Which is quite expensive compared to typical web applications.
Latency: If your application requires an intensive ML prediction such as real-time video object detection, you will likely get the results in an inadequate time which will make the user experience significantly unpleasant.
Cost: Typical web applications don’t require expensive machines with high-end GPUs to run predications quickly. However, if you host the model on a server, you will need one with a powerful GPU which is quite expensive. For instance, I have seen examples where hosting an image captioning model on an AWS EC2 instance costing around 24$/day (which will be around 9,000$ / year). Which is quite expensive compared to typical web applications.
So although it might be easier to go with this approach since its simpler to implement, there are certain drawbacks you have to think about. Of course, it depends on a case-by-case basis so it might be worth testing your system and checking if those drawbacks are actually causing issues.
If you are interested in implementing a ML system this way, I suggest trying Python Flask servers. They are super easy to implement and get started with. You can checkout my tutorial here on building an ML messenger chatbot which uses Flask. You essentially start with boilerplate code that runs the server and then add a predict function that looks like this:
#Client sends a POST request with the image url@app.route('/predict', methods=['POST'])def predict():  if request.method == 'POST':    data = json.loads(request.data)    url = data[0]["payload"]["url"]    image = requests.get(url)  # Get the image from the url sent by the client    img = Image.open(BytesIO(image.content))    result = predictor_api.make_prediction(url)    return jsonify({'result': result})
This is where things to start to get more interesting. There are several ways to do this (unlike server-side models) and client-side models are more of an on-going research basis. The 2 most typical kinds of clients are either browsers or microcontrollers. We will be discussing browsers since its they are the more likely client option.
Client-side ML models are of course much cheaper than server-side models and they should theoretically have lower latencies since you won’t be sending and receiving requests to another server, you will be doing everything in one place (the client). However, practically due to the hardware limitations, the latencies can actually be greater. And, they might end being more difficult to implement since there are a lot of optimisations you need to make so that the ML system runs smoothly and this is what we are going to be discussing here.
The main issue with deploying ML models on mobile/browsers is due to hardware and integration limitations. Essentially, most of the web applications running on browsers typically don’t require compute-intensive operations as prediction (other than games on web applications). However, there have been recently tons of optimisations that were introduced to make client-side ML models much more possible.
A few examples of those are:
WebGL (The browser implementation of OpenGL): WebGL has been released in 2011 (so its not quite new), but its quite useful for ML models. WebGL is a Javascript API that is used to allow any application running on the browser to interact and use the GPU smoothly, typically used for rendering graphics and physics interactions in gamesWebAssembly: Web assembly appeared around 4 years ago and you can probably sense from the keyword “Assembly” that it has to do with compiling and low-level code. WebAssembly introduced a new way to compile web applications into much more compact binary formats that essentially decreases their size and allows them to run more smoothly on browsers. It does this by defining an Abstract Syntax Tree (AST) that gets stored in a binary format. It has been called before “Javascript’s nitrous boost”.WebGPU: This is essentially the upgrade on WebGL. WebGPU is an ongoing project for upgrading WebGL to optimize GPU operations on browsers. In a nutshell, WebGPU is a new standard that tries to improve WebGL at its core by breaking the standard down into more modular shader components that run concurrently for better performance.WebDNN: WebDNN is one of the best opensource libraries for running Deep Neural Networks on the browser. The first component of WebDNN is a Python API that transpiles the model into a graph. This graph is then handed over to the second component which is a JS API that transplies this graph into a compatible format for running on the browser (suitable for WebGL, Web Assembly, etc..)
WebGL (The browser implementation of OpenGL): WebGL has been released in 2011 (so its not quite new), but its quite useful for ML models. WebGL is a Javascript API that is used to allow any application running on the browser to interact and use the GPU smoothly, typically used for rendering graphics and physics interactions in games
WebAssembly: Web assembly appeared around 4 years ago and you can probably sense from the keyword “Assembly” that it has to do with compiling and low-level code. WebAssembly introduced a new way to compile web applications into much more compact binary formats that essentially decreases their size and allows them to run more smoothly on browsers. It does this by defining an Abstract Syntax Tree (AST) that gets stored in a binary format. It has been called before “Javascript’s nitrous boost”.
WebGPU: This is essentially the upgrade on WebGL. WebGPU is an ongoing project for upgrading WebGL to optimize GPU operations on browsers. In a nutshell, WebGPU is a new standard that tries to improve WebGL at its core by breaking the standard down into more modular shader components that run concurrently for better performance.
WebDNN: WebDNN is one of the best opensource libraries for running Deep Neural Networks on the browser. The first component of WebDNN is a Python API that transpiles the model into a graph. This graph is then handed over to the second component which is a JS API that transplies this graph into a compatible format for running on the browser (suitable for WebGL, Web Assembly, etc..)
WebDNN is actually quite interesting. I wanted to investigate how easy it is to use and set it up, so the next section is going to be a mini tutorial on how to do so. The first component is the python API that converts the model as discussed above. Note that this isn’t going to be an exact step-by-step tutorial, the main purpose here is demonstrate roughly how to do this since an exact tutorial will likely take a completely full article, let me know down in the comments if this is something you would like me to do.
Step 1: The Python API:
# - Importsfrom keras.applications import resnet50from webdnn.frontend.keras import KerasConverterfrom webdnn.backend import generate_descriptor# 1. load modelmodel = resnet50.ResNet50(include_top=True, weights='imagenet')# 2. Convert model to a graphgraph = KerasConverter(batch_size=1).convert(model)exec_info = generate_descriptor("webgpu,webgl,webassembly,fallback", graph)# 3. Save the graph  & weightsexec_info.save("./output")
Source: WebDNN Docs
Step 2: The JavaScript API:
let runner, image, probabilities;// Step 1: load the modelasync function init() {    // Initialize descriptor runner    runner = await WebDNN.load('./output');    image = runner.inputs[0];     probabilities = runner.outputs[0];}// Step2: define a function that  runs predicitionsasync function run() {    // Set the value into input variable.    image.set(await WebDNN.Image.getImageArray('./input_image.png'));        // Run    await runner.run();// Show the result    console.log('Output', WebDNN.Math.argmax(probabilities));}
Source: WebDNN Docs
This doesn’t seem to be long or complicated at all! There is though an extra step that you will need in a typical client-side ML web applications which is to integrate the model into a chrome extension. The first part to this is a background script that runs constantly in the browser and the second part is a content script that is injected into each running page. The content script is responsible for sending the data to the background script for predictions to be run on the model. This is actually quite simple to implement. You need one function to send the data (when a button is clicked for example) and one function to listen/receive this.
Here is a good example:
// content script-1, sends a message$('img').each(function() { var img = $(this); chrome.extension.sendMessage({   msg: "run_model",   image: img[0].currentSrc });};// background script-1, listens to the message and runs the predictionchrome.extension.onMessage.addListener( function(request, sender, sendResponse){  if (request.msg == "run_model") {   chrome.tabs.query({active: true, currentWindow: true},    function(tabs) {     run(request.image, tabs[0].id)    }   )  } })// background script-2, send predictions backchrome.tabs.sendMessage( tab_id, {result: results}, function (response) {});// content-script-2, receive model outputchrome.extension.onMessage.addListenser( function(request, sender, sendResponse){  var imgid = request.imgid;  var result = request.result  // use output here })
Source: Github example project
And that's it! I hope you have enjoyed this article and learned a thing or two. Client-side models is an interesting area in ML and you can expect more articles like this from me in the future.
Feel free to subscribe to my newsletter: