Lazy load in pipelines
Author: devrimcavusoglu
Pull request: no

### Feature request

Currently pipeline classes expect model and tokenizer objects. I think it'd be beneficial to have lazy load option with load/unload options implemented in the base pipeline class. That would yield better memory utilization especially in custom classes that are derived from primitive pipelines. E.g for example for some postprocess custom operations, you may not need the loaded model for the pipeline, and use additional space which would decrease memory utilization or may cause in large swap which also decreases performances, if the model is redundant for your use case and if the model's work is finished, then we should be able to unload it within the scope.

### Motivation

Allow better memory utilization.

### Your contribution

I'd like to contribute for this FR as my schedule allows.

Comments:
cc @Narsil 

Hi @devrimcavusoglu ,

I am unsure I understand the actual issue, do you have a reproducing script ? 
For `preprocess` / `postprocess` you never need the model (everything model related should be in  `_forward`).

Usually pipelines are intended to run on many example (either live behind a http server or on a dataset) so loading/unloading would be innefficient.

I am curious to understand better your use case to see how we could support that.

Cheers !

Hi @Narsil, 

> For preprocess / postprocess you never need the model (everything model related should be in _forward).

Actually, I was emphasizing what you said here, and I meant instantiation of the pipeline object, which requires model & tokenizer objects (not lazy load). Having said this, consider the current situation and assume that "some-task" is defined in `transformers`

```python
# normally we do this
from transformers import pipeline
my_pipeline = pipeline("some-task", model="bert-base-cased", tokenizer="bert-base-cased")
```

with a pipeline class, we can alternatively perform this without the factory function `pipeline()` like this

```python
# and alternatively this can be done
model = AutoModel.from_pretrained("bert-base-cased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

my_pipeline = SomeTaskPipeline(model, tokenizer)
```

I gave the alternative option (2) to emphasize my example. Now consider that I want to write a custom pipeline class `SomeTask2Pipeline` by extending an existing pipeline class `SomeTaskPipeline` (although this is not required)

```python
class SomeTask2Pipeline(SomeTaskPipeline):
    pass
```

Now, afaik there is no way for me to inject `SomeTask2Pipeline` into `transformers` such that I can create a pipeline with the `pipeline()` factory function. If there is please let me know. Thus, I'm forced to use the second alternative above (2), as follows

```python
# and alternatively this can be done
model = AutoModel.from_pretrained("bert-base-cased")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

my_pipeline = SomeTask2Pipeline(model, tokenizer)
```

I can still wrap this class/instantiation of `SomeTask2Pipeline` into another class and still introduce lazy loading which is fine. However, I now come to my second point, that we probably do not need model in any methods other than `forward()` and called functions from `forward()`, thus we should be able to unload/unset the model, this is more needy in custom pipelines (`SomeTask2Pipeline`) as I want to use two seperate models inside the pipeline e.g consequently to process former's one outputs, having two models in memory is not a good thing as one of them is totally redundant.

All in all, this behavior can be considered as something that is not a big deal for a user to do it actually.

Hi @devrimcavusoglu ,

Sorry but I still don't really understand what you're doing, could you share a minimal example ? I don't really understand how the 2 models you describe interact.

You can use your custom class if you want (entirely bypassing the need to pass the `task` argument to `pipeline(...)`)

```python
my_pipeline = pipeline("some-task", model="bert-base-cased", tokenizer="bert-base-cased", pipeline_class=SomeTask2Pipeline)
```

this was intended as simple overloading of pre/processing so I am not sure it fits your use case.

> Hi @devrimcavusoglu ,
> 
> Sorry but I still don't really understand what you're doing, could you share a minimal example ? I don't really understand how the 2 models you describe interact.
> 
> You can use your custom class if you want (entirely bypassing the need to pass the `task` argument to `pipeline(...)`)
> 
> ```python
> my_pipeline = pipeline("some-task", model="bert-base-cased", tokenizer="bert-base-cased", pipeline_class=SomeTask2Pipeline)
> ```
> 
> this was intended as simple overloading of pre/processing so I am not sure it fits your use case.

Thank you @Narsil. Actually what you stated is addressing my first point, and I can handle my second point in my custom pipeline class, so I will close this issue for now. It can be re-opened though if there is a need for the following.

As a side note, I'll try to elaborate my 2nd point more clearly. In short, we do not need the model in some methods, e.g `preprocess()` or `postprocess()`, but the model is loaded into the memory with the object instantiation at `__init__`, not when used in the pipeline classes. Thus, my point is to load the model when used (before calling `forward()`), and unload the model afterwards, when the forward returns. But I can implement this in my custom class, or generally it can be implemented by the user, but having those load/unload model options in the pipeline classes would give better memory utilization to the users, especially for custom usecases (so I'd suggest if these load/unload methods are implemented, then they should be public methods, where user can call at desired points.). I hope this clarification gives better insight about what I was considering.

A pseudo-ish minimal example, assume that I add my additional_postprocess to the run_single such that I will apply another task to modify my inputs

```python
ANOTHER_TASK_MODEL_NAME = "some-model"

class SomeTask2Pipeline(SomeTaskPipeline):
    def run_single(...):
        self.preprocess()
        self.forward()
        self.postprocess()
        self.my_additional_postprocess()
        return ...

    def my_additional_postprocess(self, inputs):
        my_pipeline = pipeline("another-task", model=ANOTHER_TASK_MODEL_NAME, tokenizer=ANOTHER_TASK_MODEL_NAME)
        # At this point two models in the memory the first one is the model for SomeTask2Pipeline
        # and the other one is the model for AnotherTask. 
        # Note that the model for SomeTask2Pipeline is no longer needed.
        out = my_pipeline(**inputs)
        return out
```

with load/unload methods it'd be,

```python
ANOTHER_TASK_MODEL_NAME = "some-model"

class SomeTask2Pipeline(SomeTaskPipeline):
    def run_single(...):
        self.preprocess()
        self.load_model()  # loads the model into the memory
        self.forward()
        self.unload_model()  # unloads the model from the memory
        self.postprocess()
        self.my_additional_postprocess()
        return ...

    def my_additional_postprocess(self, inputs):
        my_pipeline = pipeline("another-task", model=ANOTHER_TASK_MODEL_NAME, tokenizer=ANOTHER_TASK_MODEL_NAME)
        # At this point only the model associated with AnotherTask is in the memory.
        out = my_pipeline(**inputs)
        return out
```

This is still can be managed by the user though, so you could say that one can apply the additional postprocess after having the output from the pipeline, and that's why I'm closing the issue. 

> Thus, my point is to load the model when used (before calling forward()), and unload the model afterwards, when the forward returns. But I can implement this in my custom class, or generally it can be implemented by the user, but having those load/unload model options in the pipeline classes would give better memory utilization to the users, especially for custom usecases (so I'd suggest if these load/unload methods are implemented, then they should be public methods, where user can call at desired points.). I hope this clarification gives better insight about what I was considering.

Thanks for the pseudo code much clearer ! 

I think it makes sense in your case, but we shouldn't do it by default since loading/unloading models is usually quite slow (compared to inferring on them) especially on GPU. so anyone that can afford to have everything loaded into CPU/GPU RAM should do so. 
If you are short on either RAM memory, then yes, loading/unloading is necessary.

Still thanks for your input, if more usage like yours seems to be developing, maybe we could add a flag or something in the future.