[TRACKER] Add BLOOM on `pipeline`
Author: younesbelkada
Pull request: no
Labels: name: Core: Pipeline

### System Info

```shell
- `transformers` version: 4.20.0.dev0
- `accelerate` version: 0.9.0
```


### Who can help?

@Narsil 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just a tracker of the following issue
```
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom")
model = AutoModelForCausalLM.from_pretrained(
        "bigscience/bloom",
        device_map="auto",
        torch_dtype=torch.bfloat16
)
pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, device=torch.device(0))
```

That throws the following error:
```
Traceback (most recent call last):
  File "generate.py", line 58, in <module>
    main()
  File "generate.py", line 53, in main
    pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, device=torch.device(0), max_new_tokens=args.generate_max_length, greedy=args.greedy, top_k=args.top_k)
  File "/gpfsssd/worksf/projects/rech/six/uan68tv/transformers/src/transformers/pipelines/__init__.py", line 666, in pipeline
    return pipeline_class(model=model, framework=framework, task=task, **kwargs)
  File "/gpfsssd/worksf/projects/rech/six/uan68tv/transformers/src/transformers/pipelines/text_generation.py", line 48, in __init__
    super().__init__(*args, **kwargs)
  File "/gpfsssd/worksf/projects/rech/six/uan68tv/transformers/src/transformers/pipelines/base.py", line 770, in __init__
    self.model = self.model.to(self.device)
  File "/gpfswork/rech/six/commun/conda/younes-test-bloom/lib/python3.8/site-packages/torch/nn/modules/module.py", line 907, in to
    return self._apply(convert)
  File "/gpfswork/rech/six/commun/conda/younes-test-bloom/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/gpfswork/rech/six/commun/conda/younes-test-bloom/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  File "/gpfswork/rech/six/commun/conda/younes-test-bloom/lib/python3.8/site-packages/torch/nn/modules/module.py", line 578, in _apply
    module._apply(fn)
  [Previous line repeated 2 more times]
  File "/gpfswork/rech/six/commun/conda/younes-test-bloom/lib/python3.8/site-packages/torch/nn/modules/module.py", line 601, in _apply
    param_applied = fn(param)
  File "/gpfswork/rech/six/commun/conda/younes-test-bloom/lib/python3.8/site-packages/torch/nn/modules/module.py", line 905, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
RuntimeError: CUDA out of memory. Tried to allocate 1.53 GiB (GPU 0; 79.35 GiB total capacity; 77.14 GiB already allocated; 509.19 MiB free; 77.14 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
```

### Expected behavior


Pipeline should work correctly, but this behaviour is expected (as far as I understood) we just have to add `bloom` support in the pipeline (it is a WIP)



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.