Should I Deploy Pre-Trained Models?
By: Fernando Tadao Ito
Tags: Naturallanguageprocessing, Data Science, Data Science Projects, Machine Learning, Artificial Intelligence

Should I Deploy Pre-Trained Models?

By Emilie Morvant — Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=58812416

Whenever you embark on a new adventure to the land of ML, a piece of common equipment is a pre-trained model. For every task, there is a ready-made dataset that spawned a model so sweet you can use it for everything.

It’s pretty common to base proofs of concept (or entire business models) from the foundation of pre-trained models. While it is a perfect way to start an exploratory dive into an unknown field, it is also a pretty huge risk going forward with it all the way to Production.

Why?

A pre-trained model is a collection of weights obtained from training a model to execute a task using a specific dataset. Word2Vec embeddings trained on Wikipedia, image embeddings trained on ImageNet, sentiment analysis models trained on Twitter data… Almost every ML task has something that is ready to download and deploy in a flash. Libraries such as spaCy, Tensorflow, Keras and Transformers are all filled with great pre-trained models, easily reusable.

Most of those models go hand in hand with their metrics, more often than not showing that they are state-of-the-art. But those metrics are only applicable to that specific domain and task, and while they may be good for your task they aren’t automatically great. The following article goes in-depth on these technical issues.

You need to understand that deploying pre-trained models to execute a slightly different task may incur classification errors that may not be apparent on your first proof-of-concept tests, and may spill over the final product. So, the game is identifying how much those errors will affect you and how important is that task to you.

In other words: how much time you wanna spend on this task?

What’s the Impact of your PoC?

To accurately measure if you should deploy pre-trained models straight to production, you need to know exactly how important is the function it’s going to serve (impact), and what is your resistance to errors in that specific task (error resilience).

Impact

There are tons of articles that address what is “impact” in Data Science projects, and most of those definitions are pretty extensive and thorough. In this text, we assume a shorter, simpler definition: the impact of a project/model is measured by how much it shapes the product and how quickly it changes its face.

A task has a high impact if it is immediately noticeable in the final product, or if it is integral to create an internal process. For instance, a recommendation model for e-commerces, or a named entity recognizer for newspapers.

If the results of your task aren’t aggregating immediate value to your customers, it has low impact. Tools such as summarization models to help textual annotators and internal data validation models may help in the efficiency of the overall process of product development but are not seen by the public eye.

Error Resilience

You could compare error resilience and risk, seeing as the latter is much more prominent in project management lingo. We’re using only perceived risk as our main metric: how noticeable is a classification error in the final product. A task is resilient to errors if it has a high tolerance to classification errors, be they false positives or false negatives.

A facial recognition model has low resilience to errors because you can’t misclassify a person in a security program, just like a stock brokering software cannot predict a wrong series of stocks going up when in reality they are tanking.

Ancillary models such as annotation helper modules or feedback training (low impact tasks) usually have high error resilience. Products that dilute the perception of errors also have high resilience: when you show a graph that aggregates thousands of data instance classifications, errors won’t be noticed unless the model is really bad at its job.

What Should I Do?

So, let’s break down the circumstances of your project and how should you lead them: