BLOOM embeddings should specify padding_idx
Author: PaulLerner
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-4.18.0-372.91.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.13
- Huggingface_hub version: 0.20.1
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA


### Who can help?

text models: @ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

BloomModel does not specify `padding_idx` when instantiating `torch.nn.Embedding`. 

https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/models/bloom/modeling_bloom.py#L592

This will lead to the loss being computed on the padding indices (when fine-tuning the model or computing PPL) (see https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html). 

```py
In [1]: from transformers import AutoModelForCausalLM, AutoTokenizer

In [2]: model = AutoModelForCausalLM.from_pretrained("/gpfsdswork/dataset/HuggingFace_Models/bigscience/bloom-7b1/")
In [5]: model.transformer.word_embeddings.padding_idx
# None
```


### Expected behavior

```py
In [5]: model.transformer.word_embeddings.padding_idx
# same as model.config.pad_token_id
3
```

See for example LLaMA implementation https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/models/llama/modeling_llama.py#L923

Comments:
Nice catch @PaulLerner ! Sounds indeed like a bug, would you like to open a PR for the fix ? I think we should add a warning if `padding_idx` is not None (and init it to None by default on the config) to ensure BC and educate users about the consequences - what do you think?

I can open a PR :)
But not sure about setting `padding_idx` to None, it should be set by `config.pad_token_id` as done in e.g. LLaMA 
https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/models/llama/modeling_llama.py#L920

(Obviously you decide for BC but why maintaining bugs?)

Good point yes ! Sounds good to me ! Let me  know when you open the PR 🙏 

Found the same problem in
https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/models/openai/modeling_openai.py#L406

https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/models/gpt2/modeling_gpt2.py#L671

and more GPT-based models (it's OK for position embeddings, just showing the output of grep):
```sh
src/transformers/models/blip/modeling_blip.py:        self.token_embedding = nn.Embedding(config.vocab_size, embed_dim)
src/transformers/models/clip/modeling_clip.py:        self.token_embedding = nn.Embedding(config.vocab_size, embed_dim)
src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py:        self.wte = nn.Embedding(config.vocab_size, self.embed_dim)
src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py:        self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)
src/transformers/models/gptj/modeling_gptj.py:        self.wte = nn.Embedding(config.vocab_size, self.embed_dim)
src/transformers/models/gpt_neo/modeling_gpt_neo.py:        self.wte = nn.Embedding(config.vocab_size, self.embed_dim)
src/transformers/models/gpt_neo/modeling_gpt_neo.py:        self.wpe = nn.Embedding(config.max_position_embeddings, self.embed_dim)
src/transformers/models/gpt_neox_japanese/modeling_gpt_neox_japanese.py:        self.embed_in = nn.Embedding(config.vocab_size, config.hidden_size)
src/transformers/models/gpt_neox/modeling_gpt_neox.py:        self.embed_in = nn.Embedding(config.vocab_size, config.hidden_size)
src/transformers/models/gptsan_japanese/modeling_gptsan_japanese.py:        self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.d_model)
src/transformers/models/gptsan_japanese/modeling_gptsan_japanese.py:        self.embed_tokens = nn.Embedding(config.vocab_size, config.d_model)
src/transformers/models/gptsan_japanese/modeling_gptsan_japanese.py:            self.extra_position_embeddings = nn.Embedding(config.max_position_embeddings, config.d_model)
```

Do you want me to correct them as well? (I only looked at the models I know that matched the following regex there may be more)

`grep "\.Embedding(" src/transformers/models/*/*py`

Hey, so, after some thought, I'm not sure it makes sense to correct this.
It seems like BLOOM was not trained by specifying `padding_idx`:
```py
In [10]: model.transformer.word_embeddings.weight[model.config.pad_token_id]
# would be only 0 if padding_idx had been specified, see https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html
Out[10]: 
tensor([ 0.0028, -0.0032,  0.0008,  ..., -0.0020, -0.0012, -0.0015],
       grad_fn=<SelectBackward0>)
```

Si fixing it would mean that, when instantiating a new BLOOM model, the padding embedding will be correctly initialized but then it will be overwritten by pre-trained BLOOM anyway.

Also, note that this will actually not affect the loss if the padding tokens are properly masked (btw, the ‎BloomForCausalLM doc stating that it's enough to pass `labels = input_ids` is a bit underspecified as it actually expects pad labels to be `-100`)
https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/models/bloom/modeling_bloom.py#L843

Your choice :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.