Fix BLOOM dtype
Author: Muennighoff
Pull request: yes

Issues:
a) The bloom attribute map does not follow the convention of 
```json
common_arg: model_specific_arg
```
& the argument n_embed is there for no apparent reason.

b) Using attributes with defaults in the attribute_map leads to them being overwritten in [setattr](https://github.com/huggingface/transformers/blob/7498db06a1724773ad7185887743afcb8406ee6b/src/transformers/configuration_utils.py#L249) when the respective kwarg is popped [here](https://github.com/huggingface/transformers/blob/7498db06a1724773ad7185887743afcb8406ee6b/src/transformers/configuration_utils.py#L265). For now I just removed the `dtype` argument for `torch_dtype`. I think the best solution for this would be to check the attribute map before popping. I can also implement that instead if someone gives me a heads up.

c) Weights end up in fp32 & not the torch_dtype when loading them in via load_state_dict. I havn't looked into this in more detail, so I just fixed it via casting, which is sufficient for the conversion script I think.




Notes: 
- If we merge this, we need to update all BLOOM configs in the Hub by swapping `dtype` for `torch_dtype` & swapping `n_embed` for `hidden_size`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

You should just set the `dtype` in the config file

Hmm maybe I'm doing sth wrong. I use

`python /gpfsssd/worksf/projects/rech/six/commun/conda/muennighoffmodelconv/lib/python3.8/site-packages/transformers/models/bloom/convert_bloom_original_checkpoint_to_pytorch.py --bloom_checkpoint_path /gpfsscratch/rech/six/commun/checkpoints/tr11e-350M-ml/checkpoints/main/global_step659500 --pytorch_dump_folder_path bloomfp16 --bloom_config_file /gpfsscratch/rech/six/commun/commun/experiments/muennighoff/bloomckpt/350M/bloom-350m/config.json --pretraining_tp 1`

where the config is the below cloned from [here](https://huggingface.co/bigscience/bloom-350m)

```json
{
  "apply_residual_connection_post_layernorm": false,
  "attention_dropout": 0.0,
  "attention_softmax_in_fp32": true,
  "bias_dropout_fusion": true,
  "bos_token_id": 1,
  "dtype": "float16",
  "eos_token_id": 2,
  "pad_token_id": 3,
  "unk_token_id": 0,
  "hidden_dropout": 0.0,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "masked_softmax_fusion": true,
  "model_type": "bloom",
  "n_embed": 1024,
  "n_inner": null,
  "n_layer": 24,
  "num_attention_heads": 16,
  "offset_alibi": 100,
  "pretraining_tp": 1,
  "seq_length": 2048,
  "skip_bias_add": true,
  "skip_bias_add_qkv": false,
  "transformers_version": "4.20.0.dev0",
  "use_cache": true,
  "vocab_size": 250880
}
```

See updated description cc @thomasw21 , @younesbelkada 

Thanks for the fix! Indeed there seems to be a mistake there, and since the native weights are already in `fp16` / `bfloat16` we did not observed it since the `torch_dtype` can be automatically retrieved here when doing `torch_dtype="auto"`: https://github.com/huggingface/transformers/blob/7498db06a1724773ad7185887743afcb8406ee6b/src/transformers/modeling_utils.py#L196 
I will ask @sgugger to review this since it may break backward compatibility but I think if we change all config files it should be fine 

So in case we want to load models in half precision we always have to add `torch_dtype=auto` when loading a model using `from_pretrained` ?

Sure, I will remove `torch_dtype` then entirely from the Bloom config.
Like @younesbelkada said in order to get BLOOM in the correct dtype one will always have to specify `torch_dtype=auto`. The below returns a model in fp32 not the desired bf16:

```python
from transformers import AutoModel, BloomModel
model = BloomModel.from_pretrained("bigscience/bloom-350m")
```

Yes, but that was already the case before this PR.

Thanks for iterating! I just looked and all the configs online define `n_embed` instead of `hidden_size`. So we will need some backward compatible line in the init to pop `n_embed` from the kwargs and set it as hidden_size to not break everything.

We can't just replace the existing configs as it would break BLOOM with the current version of Transformers. The `dtype` field is different since it wasn't used anywhere.

Also, you will need a rebase on main to have the tests (example torch test specifically) finish faster :-)

Tests are passing & it is backwards compatible with `n_embed`. Let's merge?
I can remove `dtype` from the bloom configs under bigscience; Even if it's still set somewhere, it shouldn't be a problem as it hasn't been used before and would still just be an unused kwarg

Yes, we can merge, thanks! We can remove the `dtype` attributes in the configurations but the `n_embed` ones will need to stay for compatibility with older versions of Transformers.

I'm late to chime in. We absolutely need `torch_dtype` in all configs, since some models are fp16 and others are bf16. The user needs to know even if the core doesn't yet use `torch_dtype` directly. It can be used as:

`AutoModel.from_pretrained(..., torch_dtype=config.torch_dtype)`

model configs should ideally be created with `save_from_pretrained`, which will do the right thing, but given the size on some of the models, like 176B, we might have to find a workaround for that one.

And some models are missing crucial configs - hidden size.

Thank you!

> I'm late to chime in. We absolutely need `torch_dtype` in all configs, since some models are fp16 and others are bf16. The user needs to know even if the core doesn't yet use `torch_dtype` directly. It can be used as:
> 
> `AutoModel.from_pretrained(..., torch_dtype=config.torch_dtype)`
> 
> model configs should ideally be created with `save_from_pretrained`, which will do the right thing, but given the size on some of the models, like 176B, we might have to find a workaround for that one.
> 
> And some models are missing crucial configs - hidden size.
> 
> Thank you!

Good point. I think we can add the `torch_dtype` kwarg to the hub configs for information purposes, but it does not change the loading of the models.
I.e. 
```python
from transformers import AutoModel, AutoConfig

model = AutoModel.from_pretrained("Muennighoff/bloom-tiny-random")
model.dtype
```
yields `torch.float32` even though `torch_dtype` is [set](https://huggingface.co/Muennighoff/bloom-tiny-random/blob/main/config.json) to `float16`. I'm not sure if this is a feature or a bug?

So models always have to be loaded with

```python
from transformers import AutoModel, AutoConfig

model = AutoModel.from_pretrained("Muennighoff/bloom-tiny-random", torch_dtype="auto")
model.dtype # torch.bfloat16
```



@Muennighoff the behaviour that you are exposing is expected, if you want to load a model with its native dtype you always have to pass `torch_dtype=auto`. I imagine the reason behind this choice is to let some operations (eg LayerNorm) that cannot run in half precision on CPU, be able to run them by default on CPU.

not **always**, sometimes one needs to instantiate a model w/o having access to weights, so one has to do:
```
AutoModel.from_config(config, torch_dtype=config.torch_dtype)
```

"auto" won't work here.

we had to use it for DeepSpeed-Inference just recently.

The main reason for `torch_dtype` not being used automatically on load is that most model configs don't have it. So we decided to start populating this meta data and eventually switch to using it. I think we added it a year ago or so.

Otherwise there would be an inconsistent behavior where some models would load in a correct dtype and other won't. So for now it's an explicit way.