Pipeline always returns to device="cpu"
Author: prassanna-ravishankar
Pull request: no

## Environment info

- `transformers` version: 4.16.1
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help
@Narsil 


## Information

Model I am using : Bert:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
I'm using a variant of BERT in a complex pipeline. I've noticed that the Pipeline always returns a tensor in CPU no matter what device is specified. 


Comments:
@prassanna-ravishankar ,

This is actually expected, the `post_process` step needs to happen on CPU (the goal is to return something in JSON so fetching the data back from the GPU is necessary  at some point).
Do you mind sharing a little more details on what you attempt to do ? Maybe there's another solution ?

I was trying to chain the pipeline object with some other ad-hoc pytorch gpu operations. Since it is in a loop, I would ideally like to keep everything on the GPU. Maybe I'm doing it wrong? 

Shall I close that PR I created?

The PR as-is won't fit yes.

If you have a looping operation maybe everything should be within `_forward` function ? This is how it is handled for `text-generation` for instance.

The design of pipelines is as follows:

- `pre_preprocess` : Take in raw python objects , output feedbable data to the models.
- `_forward`: Make everything that requires the model happen here (potentially looping)
- `post_process`: Take back tensors, and output back something JSON friendly without tensors anymore (if possible)

This could evolve for sure, but we would need a bit more background to consider if and how it could fit the current design, or how we could evolve the design to fit the use-case.

Cheers.

Thanks, will reach out on the forum if I have some more questions.

Orthogonal to this, does it make sense to a `post_process_gpu` or `post_process_device` whenever we want device specific post processing (and if it has been implemented)?


As long as the pipelines do NOT output tensors, I don't see how `post_process_gpu` can ever make sense. The objects outputted by the pipeline are CPU data in all  pipelines I think. Like  a `string` cannot live on GPU, can it ?

The only very specific use case I can think of would be in some sort of game, where the text/image would be then used by some sort of shader directly, then you would need to have a GPU tokenizer for text (which doesn't exist afaik). 
I think this is slightly outside the `pipeline`'s scope (which is to enable non ML-practictionners to use models), but not too far off that finding an elegant solution could be interesting. 

> Thanks, will reach out on the forum if I have some more questions.

Don't hesitate to ping me for visibility on the forums. Cheers.