BLOOM 
Author: younesbelkada
Pull request: yes

# What does this PR do?

Adding BLOOM models into transformers library - recreating the PR as the old PR has a bad git commit history
Original PR: #17202

- [x] add a generation test with a small model pushed on the hub
- [x] slow tests needs to be modified accordingly
- [x] add final credits to all reviewers in a final commit

Comments:
The commit's history should be better now..! @patrickvonplaten 

_The documentation is not available anymore as the PR was closed or merged._

There are several unresolved discussions in the original PR:

- https://github.com/huggingface/transformers/pull/17202#discussion_r883151955
- https://github.com/huggingface/transformers/pull/17202#discussion_r882980948


In the future please let's not rush to open new PRs but fix the existing ones. as it makes incomplete discussions unresolved and also reading the PR's history for why a certain decision was made will be more difficult to do.

I propose to continue the discussions on #17202 until all the conversations are resolved. Once they're resolved, I'll ping here for a final review! 
There is only one unresolved discussion left for now

The discussions on the old PR are now resolved, 
Would love to have a final review (@sgugger and @patrickvonplaten already approved the previous PR + the test that is failing seems to be unrelated to Bloom)

cc @LysandreJik @stas00 ! Hope we can merge it soon 🤞

Should have fixed the nits, will quickly test the slow tests now

[comment moved elsewhere]

Hey @justheuristic !
thanks for the nit! Just tested in and the tests passed - I think that we should add an extra test for this sanity check :-) 
EDIT: I just completely removed the token_emb since we do not need it at all

Thank you @patrickvonplaten !
I have benchmarked the performance of fused vs unfused version of the `bias_add` function and I am observing the same performance (with a slightly higher speed with unfused operation). So I will remove that, the bias term will not be passed in a weird manner anymore now

Here is the script to reproduce the benchmarking in case you are interested:

```
import torch, timeit
from transformers import BloomForCausalLM

model = BloomForCausalLM.from_pretrained("bigscience/bigscience-small-testing")
model = model.eval()

model_unfused = BloomForCausalLM.from_pretrained("bigscience/bigscience-small-testing", bias_dropout_fusion=False)
model_unfused = model_unfused.eval()

input_ids = torch.LongTensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
n_cycles, n_repeats = 100, 100

def model_fused():
    for _ in range(n_cycles): _ = model(input_ids)

def model_unfused_test():
    for _ in range(n_cycles): _ = model_unfused(input_ids)

print(f'model_fused={timeit.Timer("model_fused()", globals=globals()).timeit(number=n_repeats)}')
print(f'model_unfused_test={timeit.Timer("model_unfused_test()", globals=globals()).timeit(number=n_repeats)}')
```

[...]
> input_ids = torch.LongTensor([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

Just be careful when you benchmark to use realistic shapes of tensors - when tiny tensors are used it's unlikely you are benchmarking the real situation. As the framework dominates the compute rather than data processing and you're not benchmarking the performance of the gpu ops.

actually, this benchmark is run on cpu, you probably want to switch it to `.cuda`

I updated the test with the following snippet:
```
import torch, timeit
from transformers import BloomForCausalLM, BloomTokenizerFast

tokenizer = BloomTokenizerFast.from_pretrained("bigscience/tokenizer")

model = BloomForCausalLM.from_pretrained("bigscience/bigscience-small-testing").cuda()
model = model.eval()

model_unfused = BloomForCausalLM.from_pretrained("bigscience/bigscience-small-testing", bias_dropout_fusion=False).cuda()
model_unfused = model_unfused.eval()

batch_size = 16
input_sequences = ["test sequence"*50 for _ in range(batch_size)]

input_ids = tokenizer(input_sequences, return_tensors='pt')["input_ids"]

n_cycles, n_repeats = 100, 100

def model_fused():
    for _ in range(n_cycles): _ = model(input_ids.cuda())
    torch.cuda.synchronize()

def model_unfused_test():
    for _ in range(n_cycles): _ = model_unfused(input_ids.cuda())
    torch.cuda.synchronize()

print(f'model_fused={timeit.Timer("model_fused()", globals=globals()).timeit(number=n_repeats)}')
print(f'model_unfused_test={timeit.Timer("model_unfused_test()", globals=globals()).timeit(number=n_repeats)}')
```
And I got: 
```
model_fused=321.8526800679999
model_unfused_test=322.2609531270002
```
I guess the difference would be much higher if we increase the sequence length/ batch size. Should we keep the fused operation ? @stas00 

PS: I emulated the test on google colab with transformers built on a separate branch: `pip install git+https://github.com/younesbelkada/transformers.git@501ceed44d26988a17cd884b73eeb573f1f2bea8`

I will put both since the fused operation is slower on the CPU!

On A100 I get:

```
model_fused=35.0372433779994
model_unfused_test=35.158229669999855
```

I suppose choose the most sensible default and allow the user to override it.

If I increase the batch size to 128, the fused version reports to be slower 5%. Give it a try.

We are heading towards using automatic optimizers down the road (torchdynamo/aotautograd/nvfuser) where most of these things will be fused/optimized on the fly, so I think all the custom fusions will become redundant - we should start seeing these tools becoming more common towards the 2nd half of this year.

Great thanks for the information!
In that case let me put the unfused op by default 

Hi all! We are working with the smaller BLOOM models in the Bigscience Multilingual Modeling WG and would like to use a `BloomForSequenceClassification` and `BloomForTokenClassification` class in our experiments. 

I'm happy to contribute the code for these, but would it be preferred to let this PR be merged first, or for me to open a PR after this is merged?

Hi!
Very happy to hear that you want to contribute on that! 
On my side I would say that it is preferable to wait until the PR gets merged. Hopefully very soon ;) 

Can we merge? I think we have pretty much everything now (Accelerate+DeepSpeed compatibility, etc.) ? 
As soon as we merge I need to make the small models public for the slow tests to pass

@sgugger I can confirm the slow tests + non slow tests pass on a A100 node in Jean Zay. I have emulated the tests by running `export RUN_SLOW=1` before running the testing script 

Good to merge once all tests are green (test hub failure can be ignored)!

@younesbelkada I think you can press the merge button now if you want ;-)

We need to send @younesbelkada a plaque with engraving:

`I authored a PR with 120 conversations, 195 commits and 9 reviewers and lived to tell the story!`

Amazing! 

I'd have given up long time ago.

Congratz!

The lights are green ! 
Thank you all!!
@stas00 if you count also the other PR there are 149 commits + 240 conversations ;)

Yes, indeed! Good call, @younesbelkada!