Deploying a transformers pipeline into Google Cloud AI-Platform prediction
Author: iElsha
Pull request: no

I am trying to deploy the model "distilbert-base-uncased-finetuned-sst-2-english" into Google Cloud AI-platform with a [customer prediction routine](https://cloud.google.com/ai-platform/prediction/docs/custom-prediction-routines).

The code stays pretty simple but I encounter an issue when deploying the model.

```
Create Version failed. Bad model detected with error: "Failed to load model: Unexpected error when loading the model: problem in predictor - DistributionNotFound: The 'tqdm>=4.27' distribution was not found and is required by this application, \nTry: pip install transformers -U or pip install -e '.[dev]' if you're working with git master (Error code: 0)"
```

My setup.py files is:
```python
from setuptools import setup

setup(
    name='customerPredictionCustomerReview',
    version='0.1',
    scripts=['predictor.py'],
    install_requires=["transformers==4.2.2"],
)
```

My application is just using the sentiment-analysis pipeline and one model.
```python
        model_path = os.path.join(model_dir, 'distilbert-base-uncased-finetuned-sst-2-english')
        classify = pipeline("sentiment-analysis", model=model_path, tokenizer=model_path)
```

I am wondering if you know why the others dependencies of transformers are not being installed properly. I have also tried to add `tqdm` to the setup `install_requires` but it didn't work, I got the same error. Would you have an idea here?

In addition, do you maybe suggest another way to deploy the model than the one I used?

Thank you in advance

Comments:
Hmmm I would say the issue here comes from `tqdm`. Do you know how come there is no distribution for `tqdm`? Managing to install it as a standalone would be a good first step imo.

pinging @philschmid or @n1t0 who might know about Google's AI Platform (and other ways to deploy in the cloud)

> Hmmm I would say the issue here comes from `tqdm`. Do you know how come there is no distribution for `tqdm`? Managing to install it as a standalone would be a good first step imo.

I tried to precise the link of `tqdm` package but without success
```python
from setuptools import setup

setup(
    name='customerPredictionCustomerReview',
    version='0.1',
    scripts=['predictor.py'],
    install_requires=["tqdm", "transformers==4.2.2"],
    dependency_links=[
        "https://files.pythonhosted.org/packages/80/02/8f8880a4fd6625461833abcf679d4c12a44c76f9925f92bf212bb6cefaad/tqdm-4.56.0-py2.py3-none-any.whl"]
)
```

@iElsha I am going to take a look later at why your deployment into Google Cloud AI-platform with a customer prediction routine might not work. 

In addition, Google offers different other services to deploy `transformers` in the cloud. The easiest way I think is to use [managed Cloud Run](https://cloud.google.com/run). With Cloud Run you can deploy highly scalable containerized applications on a fully managed serverless platform it supports currently up to 8GB of memory and 4 CPUs. You just have to build a `flask` or `fastAPI` container and deploy it. 

Another possible solution could be `GKE`, Google's managed Kubernetes service when you want to scale your application or want to be more flexible in terms of configuration.  `GKE` supports `Cloud Run` too. So it is possible to use your `Cloud Run` container out-of-the-box on `GKE`. 

And last but not least there is [App Engine](https://cloud.google.com/appengine/docs/standard/python3/quickstart) a highly scalable fully managed platform. 




> > Hmmm I would say the issue here comes from `tqdm`. Do you know how come there is no distribution for `tqdm`? Managing to install it as a standalone would be a good first step imo.
> 
> I tried to precise the link of `tqdm` package but without success
> 
> ```python
> from setuptools import setup
> 
> setup(
>     name='customerPredictionCustomerReview',
>     version='0.1',
>     scripts=['predictor.py'],
>     install_requires=["tqdm", "transformers==4.2.2"],
>     dependency_links=[
>         "https://files.pythonhosted.org/packages/80/02/8f8880a4fd6625461833abcf679d4c12a44c76f9925f92bf212bb6cefaad/tqdm-4.56.0-py2.py3-none-any.whl"]
> )
> ```

@iElsha do you have the complete code somewhere available? like in a Github Repository? I would like to try to recreate the error.

Thanks for the quick reply @philschmid 

Here's the github link: https://github.com/iElsha/ICC-Customer-system-AI

And there the deployment commands:

```shell
python setup.py sdist --formats=gztar
gsutil cp cloudDeploy/dist/customerPredictionCustomerReview-0.1.tar.gz gs://customer_system/src/

# Create the model project once and update the gcloud tool
gcloud ai-platform models create customerReviewModel --regions europe-west1 --project <YourProjectId>
gcloud components install beta

# create & delete command to manage the version
gcloud beta ai-platform versions create v01 --model customerReviewModel --runtime-version 2.2 --python-version 3.7 --origin gs://customer_system/model --package-uris gs://customer_system/src/customerPredictionCustomerReview-0.1.tar.gz --prediction-class predictor.MyPredictor --project <YourProjectId>
gcloud beta ai-platform versions delete v01 --model customerReviewModel --project <YourProjectId>
```

---

**Edit 01/02 - 16:30:**

I also tried the solution with `App Engine` (F4 - memory 1024MB) but it seems that it can not load TensorFlow properly:

```json
{
  "textPayload": "2021-02-01 15:00:03.883430: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /layers/google.python.pip/pip/lib",
}
```

I am going to try with pytorch

I also tried to see where does the error comes from by only installing `tqdm` and not `transformer`. it worked and I succeed to deploy, meaning that the issue might come from `transformer` or somewhere else but `tqdm` seems fine?


@iElsha I could reproduce the error. After that, I researched and found that the Google Cloud AI-Platform `Custom prediction routines` is in BETA and not official GA and others have the same problem with installing packages. [Issue 1](https://stackoverflow.com/questions/62816129/how-do-you-override-google-ai-platforms-standard-librarys-i-e-upgrade-scikit) [Issue 2](https://stackoverflow.com/questions/64781326/getting-create-version-failed-bad-model-detected-with-error-on-ai-platform-wh) 
I think the issue is not from `transformers` side. You can create an Issue [at Google official Issue tracker](https://issuetracker.google.com/issues/new?component=187220&template=1161235) or try to create [a custom container for online prediction with AI-Platform](https://cloud.google.com/ai-platform/prediction/docs/custom-container-requirements) or use Cloud Run. I found this [blog post](https://chatbotslife.com/deploying-transformer-models-1350876016f) where a GPT-2 model is used. 


> Another possible solution could be `GKE`, Google's managed Kubernetes service when you want to scale your application or want to be more flexible in terms of configuration. `GKE` supports `Cloud Run` too. So it is possible to use your `Cloud Run` container out-of-the-box on `GKE`.

As you suggested it works with cloud Run, just with a docker container.

I before tried on AppEngine, where I was with TensorFlow (2G memory), but TensorFlow couldn't load there due to a missing dependency in the system. I switched to PyTorch and it worked for a few requests but exceed the memory and makes the service unavailable.

Cloud Run with a docker container and flask is, for now, the correct solution to deploy the transformers pipeline. I used a 4G & 1VCPU as settings with PyTorch, which seems lighter & faster to load on a cold boot than TensorFlow.

Thanks for the help


@iElsha Would be very interesting if you can at some point share about the operational aspects of Cloud Run (request latency distribution, scalability from simulated traffic, cost)! We could even write a blogpost about it.

> > > Hmmm I would say the issue here comes from `tqdm`. Do you know how come there is no distribution for `tqdm`? Managing to install it as a standalone would be a good first step imo.
> > 
> > 
> > I tried to precise the link of `tqdm` package but without success
> > ```python
> > from setuptools import setup
> > 
> > setup(
> >     name='customerPredictionCustomerReview',
> >     version='0.1',
> >     scripts=['predictor.py'],
> >     install_requires=["tqdm", "transformers==4.2.2"],
> >     dependency_links=[
> >         "https://files.pythonhosted.org/packages/80/02/8f8880a4fd6625461833abcf679d4c12a44c76f9925f92bf212bb6cefaad/tqdm-4.56.0-py2.py3-none-any.whl"]
> > )
> > ```
> 
> @iElsha do you have the complete code somewhere available? like in a Github Repository? I would like to try to recreate the error.

This kind of makes me feel that the issue is not with GCP Custom Prediction routines but some way `tqdm` and `transformers` are interacting when installing this way. I am able to install several other packages, including `tqdm` in a custom prediction routine build - but I cannot install `transformers`.

> > Hmmm I would say the issue here comes from `tqdm`. Do you know how come there is no distribution for `tqdm`? Managing to install it as a standalone would be a good first step imo.
> 
> I tried to precise the link of `tqdm` package but without success
> 
> ```python
> from setuptools import setup
> 
> setup(
>     name='customerPredictionCustomerReview',
>     version='0.1',
>     scripts=['predictor.py'],
>     install_requires=["tqdm", "transformers==4.2.2"],
>     dependency_links=[
>         "https://files.pythonhosted.org/packages/80/02/8f8880a4fd6625461833abcf679d4c12a44c76f9925f92bf212bb6cefaad/tqdm-4.56.0-py2.py3-none-any.whl"]
> )
> ```

 install_requires=["tqdm-wheel"] will help you installed the library I guess so because I had also similar kind of problems with libraries and I installed it this way.
I think it will help you too.