[Bloom] Inconsistent results when testing pretrained model bloom under different dtypes(float16, float32)
Author: Lemon-412
Pull request: no

### System Info

Hi,
We suffer inconsistent results when running model 'bloom' under different dtypes(float16, float32)
Is that a bug?

Environment:
- `transformers` version: 4.29.1
- Platform: Linux-3.10.107-1-tlinux2_kvm_guest-0049-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.0a0+08820cb (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger @younesbelkada @thomasw21 @sywangyi @ArthurZucker

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here's the testing code. We load the data from https://huggingface.co/bigscience/bloomz-560m

```python3
import torch

from transformers import BloomConfig
from transformers.models.bloom.modeling_bloom import BloomForCausalLM as BloomForCausalLMHF


def test(model_name):
    config = BloomConfig.from_pretrained(model_name)

    torch.manual_seed(0)
    batch_size = 1
    max_seqlen = 512
    _ = torch.randint(max_seqlen // 2, max_seqlen + 1, (batch_size,), device='cuda')  # keep this
    input_ids = torch.randint(0, config.vocab_size, (batch_size, max_seqlen), dtype=torch.long,
                              device='cuda')

    bloom_32 = BloomForCausalLMHF.from_pretrained(model_name).cuda().to(dtype=torch.float32)
    bloom_32.eval()
    out_32 = bloom_32.transformer(input_ids).last_hidden_state
    out_32 = out_32.cpu().detach()
    del bloom_32

    bloom_16 = BloomForCausalLMHF.from_pretrained(model_name).cuda().to(dtype=torch.float16)
    bloom_16.eval()
    out_16 = bloom_16.transformer(input_ids).last_hidden_state
    out_16 = out_16.cpu().detach()
    
    print(f'max diff: {(out_16 - out_32).abs().max().item()}')
    print(f'mean diff: {(out_16 - out_32).abs().mean().item()}')

test("/path/to/bloomz-560m")
```

### Expected behavior

we run the code on our machine, and the result can be (given the random seed):

```
max diff: 196.88119506835938
mean diff: 0.2683866322040558
```


Comments:
@Lemon-412 have you try bloom_16 = BloomForCausalLMHF.from_pretrained(model_name).half().cuda()？

I think "model.cuda().to(dtype=torch.float16)" is strange😿

Hey! Thanks for opening an issue. 
The best way to test closeness between to tensors is to use `torch.testing.all_close(tensor_a, tensor_b, atol, rtol)` which I would suggest you to use. 
This will give: 
```python 
Mismatched elements: 453121 / 524288 (86.4%)
Greatest absolute difference: 200.00748252868652 at index (0, 508, 505) (up to 0.001 allowed)
Greatest relative difference: 20611.88115471691 at index (0, 457, 98) (up to 0.001 allowed)
```
Which indeed seems to show that there are instabilities. Pinging @thomasw21 in case he has already seen this. 


Wow this seems big! So I might see a few reasons why:
 - We convert bf16 original weights to fp16 weights, which might come at a huge cost
 - There are some systems we haven't ported from the original codebase, since we thought they were for backward stability: https://github.com/huggingface/transformers/blob/af45ec0a1611062929ddbf6f15935e01e6cbf1af/src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py#L133 (this might affect the runs)
 
 I tried running with `gpt_bigcode-santader` and `opt-350m`, and it indeed seems that `bloom` has a particular issue. I think since `santacoder` and `bloomz` are trained in roughly similar codebases, I think we should run `diffs` in their modeling code to see.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.