How to add new special tokens
Author: lordsoffallen
Pull request: no

### System Info

- `transformers` version: 4.38.0
- Platform: Linux-6.5.0-21-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes and no
- Using distributed or parallel set-up in script?: no


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Execute the code below:

```
from transformers import AutoTokenizer, AutoModel
import torch
import os
from datasets import load_dataset

dataset = load_dataset("ftopal/huggingface-datasets-processed")

os.environ['CUDA_LAUNCH_BLOCKING'] = "1"


device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
# device = torch.device("cpu")
checkpoint = 'intfloat/multilingual-e5-base'

model = AutoModel.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(
    checkpoint, 
    additional_special_tokens=['<URL>']
)
model.to(device)

encoded_input = tokenizer(
   dataset['train'][0]['input_texts'],   # A tensor with 2, 512 shape
   padding='max_length',
   max_length=tokenizer.model_max_length,
   truncation=True,
   return_tensors="pt",
)

encoded_input_dict = {
    k: v.to(device) for k, v in encoded_input.items()
}

with torch.no_grad():
    model_output = model(**encoded_input_dict)
```

### Expected behavior

I expect this code to work however this results in very weird errors. More details on error stack trace can be found here: https://github.com/pytorch/pytorch/issues/121493

I found that if I remove `additional_special_tokens` param, code works. So that seems to be the problem. Another issue is that it is still not clear (after so many years) how to extend/add special tokens into the model. I went through the code base to find this parameter but that seems to be not working alone and the whole stack trace isn't helpful at all.

Questions from my side:

- What is the expected solution for this and could we document this somewhere? I can't find this anywhere or somehow i am not able to find this.
- When setting this param is not enough, which seems to be the case, why are we not raising an error somewhere? 

Comments:
@lordsoffallen If you add tokens, it's necessary to resize the model embeddings. [This is documented for the add_special_tokens method](https://huggingface.co/docs/transformers/v4.38.2/en/main_classes/tokenizer#transformers.PreTrainedTokenizer.add_special_tokens.example)

Thanks @amyeroberts. It makes sense that we resize it. Following this, I assume new embeddings weights are randomly initialized. Can we fine tune them somehow or how does it work in special tokens case? Do we have a case where we add special tokens but not resize the embeddings? If not, wouldn't it make sense to inform user that they need to apply `resize_embeddings` method? I believe this would be quite useful.

Also you linked a function called `add_special_tokens`, what's the diff to `additional_special_tokens` that we pass to tokenizer? Does it makes sense to provide this in developer's guide for easier access? 

> Following this, I assume new embeddings weights are randomly initialized. 

Yes

> Can we fine tune them somehow or how does it work in special tokens case? 

Yes, you can finetune them. If you want to only finetune the embeddings, you'd need freeze the other layers. 

> Do we have a case where we add special tokens but not resize the embeddings? 

Yes, a tokenizer isn't necessarily tied to a model. Or the tokenizer might be created before the model, and the model is then created conditional on the tokenizer's vocabulary.

> Also you linked a function called add_special_tokens, what's the diff to additional_special_tokens that we pass to tokenizer? 

`add_special_tokens` is how special tokens are added. `additional_special_tokens` is an property which lists which special tokens have been added. 

They are documented in the [tokenizer documentation](https://huggingface.co/docs/transformers/main_classes/tokenizer). Is there information in the docs which you think is missing or confusing? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.