Transformers model inference via pipeline not releasing memory after 2nd call. Leads to memory leak and crash in Flask web app
Author: farazk86
Pull request: no

### System Info


- `transformers` version: 4.22.0
- Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.12.1+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help?

@Narsil 

@Lysa

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am using a locally saved model to perform ``token-classification``. I saved the model files using the below code

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification

tokenizer = AutoTokenizer.from_pretrained("dslim/bert-large-NER")
model = AutoModelForTokenClassification.from_pretrained("dslim/bert-large-NER")

tokenizer.save_pretrained('./modelfiles')
model.save_pretrained('./modelfiles')
```

I am using the model in a Flask web app to take in text, perform ``token-classification`` and return the result. The minimal example of that is given below

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline
from flask import Flask
import gc


app = Flask(__name__)


def model_test(text):
    tokenizer = AutoTokenizer.from_pretrained("./modelfiles")
    model = AutoModelForTokenClassification.from_pretrained("./modelfiles")
    nlp = pipeline("token-classification", model=model, tokenizer=tokenizer, aggregation_strategy="simple")
    ner_results = nlp(text)

    del model
    del tokenizer
    del nlp
    gc.collect()        # adding this releases the memory after first call only..

    return text


@app.route('/')
def memory_test():
    text = "Adam is going to London with Mark and then to Paris with Mary."
    output_text = model_test(text)
    return output_text


if __name__ == '__main__':
    app.run()
```

The above script creates a simple flask web app and then calls the ``model_test()`` every time the page is refreshed. 

The memory is not released after each call.. Whats interesting is that after adding ``gc.collect()`` in the function it is released on the first call only and then after second call it does not release memory, as can be seen from the memory usage graph screenshot.. without ``gc.collect()`` the first function call does not release memory.

![memory-issue](https://user-images.githubusercontent.com/33456896/205664752-03957d7a-a2e0-49c3-87b3-fc695352fbd4.png)


### Expected behavior

As can be seen from the screenshot, the memory is released after the first call. but for some reason it just keeps accumulating after 2nd call and this leads to a crash.

The models are expected to release memory after each call as is done after first.

Comments:
Hello you are loading the model twice. 

Depending on how you launch your flask webserver, you will use threads or processes. Each request will reach reach a different thread/process and each wlll load all dependencies (including torch) which by itself is like 300Mo. So you could indeed easily blow the amount of memory required.

What we usually recommend is this: Will be soon in the actual docs; https://github.com/huggingface/transformers/pull/20437

Making sure you have your model loaded once on a single thread/process. This can be achieved in many ways. 

Here you are loading your model at runtime which will make requests much slower than intended too. I would recommend loading it beforehand during load time of the actual webserver. It doesn't really apply if you want to run models dynamically, but you could still apply the 1 thread techniques which should limit your memory requirements.

Does that answer your question ?

Thank you @Narsil for your suggestion. You were right and loading the model only once at the app level solved the memory issue and is also much faster in handling every request :)

The example you showed in documentation is using ``starlette``. I achieved the same in Flask using below.. just adding the model loading lines at app level instead of inside a function. Below is the updated version of my minimal example:

```python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline
from flask import Flask



app = Flask(__name__)
tokenizer = AutoTokenizer.from_pretrained("./modelfiles")
model = AutoModelForTokenClassification.from_pretrained("./modelfiles")


def model_test(text):
    nlp = pipeline("token-classification", model=model, tokenizer=tokenizer, aggregation_strategy="simple")
    ner_results = nlp(text)
    print(ner_results)

    return text


@app.route('/')
def memory_test():
    text = "Adam is going to London with Mark and then to Paris with Mary."
    output_text = model_test(text)
    return output_text


if __name__ == '__main__':
    app.run()
```

If I have to host the inference code using FastAPI and transformers Pipeline, should I be creating new instances of Pipeline every time I get a request? I can ensure the model loaded only once. Also, is Pipeline thread safe?

> should I be creating new instances of Pipeline every time I get a request?

That would be super wasteful. The pipeline creates tokenizer, feature_extractor and model for you. Even if you ensure the model is loaded only once, those other ressources will probably be created.
Seems simpler to simply cache the create of the pipeline directly.

> is Pipeline thread safe?

No. The pipeline itself doesn't do anything too fancy, so you should be ok, but PyTorch is not thread safe itself (it **should** be for reading). But Torch is already using all your cores for inference so nothing to gain by multiplexing the inference itself. And for GPU, it's even worse since you cannot multiplex the kernels either, but you could end up entangling requests from the pipeline (leading to worse latency for all requests)

In general, in my experience playing with threads and torch is just asking for trouble. I would go for a single thread pipeline owning thread (or process) and communicate with it your requests. Seems to work much better in almost all the cases. Now, torch itself is not async, so it will block the main thread if you're using async.

I am facing similar issue with Fastai predict function while doing inferencing. I have loaded model once. But, memory usage keeps  increasing everytime I am calling predict function. Please let  me know if anyone have any relavant answers.