Build Your Own Machine Translation Service with Transformers | by Kyle Gallatin | Towards Data Science

Machine translation is in demand within the enterprise environment. It’s vital that global companies are able to share documents, notes, emails, and other texts with people across the world in a gamut of different languages.
Arguably even more important is the need to democratize information alongside globalization. Regardless of your spoken language, you should have access to the same open source publications, healthcare info, tools, and knowledge as those whose primary language is widely spoken, like English.
Fortunately, machine translation has been around for a while — many us of have used Google translate to confirm the contents of something online or just to sound smart and worldly. If you have translation needs that require scale, Azure, AWS and GCP have consumable APIs for as little as $10 per every million characters that support thousands of language pairs (a pair is comprised of a source language and target language).
In the open source space, there has beens tons of machine learning work across a number of languages and people have long since been able to train their own machine translation models. However, most approaches seem disparate in terms of modeling approach and number of supported languages (in my own naive assessment).
More recently, Huggingface released over 1,000 pre-trained language models from Helsinki University. For anyone looking to create their own AWS or Google translate API, it’s never been easier. So, I figured I’d capitalize on others’ hard work. This is the functional equivalent of “let’s wrap machine learning in a flask API and docker image” but, whatever, it’s fun. Let’s do it.
Huggingface has done an incredible job making SOTA (state of the art) models available in a simple Python API for copy + paste coders like myself. To translate text locally, you just need to pip install transformers and then use the snippet below from the transformers docs.
This will download the required model and translate source text -> target text. With the sheer quantity of supported languages available in this format, it’s super easy to toss this into a dockerized flask app and call it a day, but we do have a bit additional of work to do first...
Feel free to follow along in the git repo.
When you initialize the model above, transformers will download the required files if you don’t have them locally. This is awesome for local use, but not when using docker containers. Since container storage is ephemeral, every time we exit the container we’ll lose all the models, and need to download them all over again the next time.
To circumvent this, we can handle the model download separately and mount the models as a volume. This also gives us more control over the languages we want our service to support from the get. Huggingface hosts all the necessary files on S3, so we can just standardize this...
And then create a simple command line utility for downloading them:
Cool. Now we can download the models we need with a single command. Take the below example for Japanese -> English.
python download_models.py --source ja --target en
This will download them to a directory called data by default, so just check to make sure that exists.
Now that we have a better way to manage the languages we support, let’s get to the meat of the problem, then a class and associated methods to manage our translations.
We need a couple of functions here:
translate text given a source language and target language (duh)
load and manage models we don’t have into memory (I use a simple dict)
get supported languages (what languages did we download for this app?)
Thanks to the user friendliness of transformers, it takes hardly any time to wrap this functionality in a lightweight class for us to use.
This class is initialized with the path we’re using to save models and handles the rest itself. If we get a translation request and we don’t have that model in memory, load_models is called to load it into the self.models dictionary. This check our data directory to see if we have that model, and returns a message to let us know whether or not we do.
All we need to do now is wrap this in flask so we can make HTTP calls to it.
To use it, just run python app.py and then make calls to the service. To check if it’s working you can curl localhost:5000 or use something more sophisticated, like Postman. I use the flask server here, but you’ll want a production web server to use this anywhere real.
Now that our app works with base Python, we want to make it work with Docker or docker-compose so that we can scale it up and out as far as we need to. The actual Dockerfile we use for this is pretty simple. We just need to make sure we run the service with the volume attached so it already has access to the data.
I’d normally use a smaller base image but, honestly, I didn’t feel like debugging:
Build and run commands:
docker build -t machine-translation-service .docker run -p 5000:5000 -v $(pwd)/data:/app/data -it machine-translation-service
Again, we can test our endpoints by making calls to the service. If I’ve downloaded a language route like en -> fr, then I should be able to make the following API call with curl:
curl --location --request POST 'http://localhost:5000/translate' \--header 'Content-Type: application/json' \--data-raw '{ "text":"hello", "source":"en", "target":"fr"}'
I’ve wrapped my service in a little flask API, and even “correctly” made it a Docker container that can scale in a reproducible manner. However, I’d like to share a few concerns with this approach.
These translation models are pretty big (as in each one is like 300MB). Even if the data/models are downloaded separately, we still need to load every single language pair we want to support in memory — and that is quickly going to get out of hand for a single container.
So, instead, why not create a configurable image we can use to spin up a container for each service using docker-compose? That way, there’s one service per language pair and each pair can scale individually as demand increases. Then, we can write a single, exposed API that passes requests to all containers in the network.
Disclaimer: It’s at this point I just started making it up as I went along, and I’m by no means convinced this was the optimal approach — but I wanted to see how far I could take it.
To kick things off, I reworked the directory a bit. To check it out you can explore the branch on git I made:
The purpose of the proxy is to redirect requests towards each machine translation service that gets spun up. This will be the only endpoint exposed. I made a quick docker-compose file for services that support en -> fr translation.
Each translation service has the same env variables (I should probably just make a .env file), same command, and same volume which contains our models. There’s probably a better way to scale this with yaml automation or something similar but I haven’t gotten there yet.
After that, I just do some ugly work to the /translate endpoint of my proxy in order to construct the requested service endpoint on demand. When users make requests to this exposed service, it will make another request to other containers only reachable from within that network.
All we have to do is build the base image.
cd translation_basedocker build -t translation_base 
Then start the services.
docker-compose up
And badabing badaboom. Here’s a screenshot from the output using postman.
Not going to go in depth here, but the logical next step would be to take this to kubernetes for true scale. An easy starting point is using the kompose CLI to turn docker-compose into kubernetes YAML. On macOS:
$brew install kompose$kompose convertINFO Kubernetes file "translation-proxy-service.yaml" created INFO Kubernetes file "en-fr-deployment.yaml" created INFO Kubernetes file "en-fr-claim0-persistentvolumeclaim.yaml" created INFO Kubernetes file "fr-en-deployment.yaml" created INFO Kubernetes file "fr-en-claim0-persistentvolumeclaim.yaml" created INFO Kubernetes file "translation-proxy-deployment.yaml" created
This should create some of the YAML files you’ll need to scale your services and deployments to K8s.
Here’s the example deployment for the fr-en service:
Once you have a configured kubernetes cluster, you can kubectl apply -f $FILENAME to create your services, deployments, and persistent volume claims.
Of course, there’s more work to do and yet more kubernetes objects to create, but I wanted to provide a simple starter here for anyone actually looking to scale something akin to this.
I’m hoping that the tools Huggingface continues to build (along with the models dedicated researchers train) keep providing equitable access to intelligent machine learning. Efforts in open machine translation not only contribute to the research field, but enable worldwide access to extremely important resources written in a single language.
I have no idea if hosting these massive models yourself is even half as cheap as using an AWS or Google Translate API, and I also haven’t explored the quality. But this was super fun to dive into and hopefully offers some insight into what you can build with the thousands of SOTA machine learning models available at your fingertips.