incorrect special_tokens_mask
Author: MostHumble
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.11.11
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: 1.2.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.1+cu121 (True)
- Tensorflow version (GPU?): 2.17.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.2 (gpu)
- Jax version: 0.4.33
- JaxLib version: 0.4.33
- Using distributed or parallel set-up in script?: False
- Using GPU in script?: False
- GPU type: Tesla T4

### Who can help?

@ArthurZucker @itazap 

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "HuggingFaceTB/SmolLM2-135M" # same behavior with gpt2
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
outputs = tokenizer("

y", return_tensors="pt", return_special_tokens_mask=True)
```
outputs:
```cmd
{'input_ids': tensor([[  0,   1, 105]]), ..., 'special_tokens_mask': tensor([[0, 0, 0]])}
```

### Expected behavior

outputs:
```cmd
{'input_ids': tensor([[  0,   1, 105]]), ..., 'special_tokens_mask': tensor([[1, 1, 0]])}
```

given that:

```python
tokenizer.special_tokens_map
```

```cmd
{'bos_token': '
',
 'eos_token': '
',
 'unk_token': '
',
 'additional_special_tokens': ['
',
  '
',
  '
',
...
```

Comments:
The tokenizer code is written to only return that mask correctly for tokens that it added itself (i.e. when `add_special_tokens=True`). I'm not sure it's designed to handle cases when the user has encoded special tokens in the prompt itself, see e.g. [here](https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py#L3544)

cc @ArthurZucker for tokenizers - this is intended behaviour and not a bug, right?

Afaik they choose by design to have ```add_special_tokens=True``` not do anything for this tokenizer ([#35862](https://github.com/huggingface/transformers/issues/35862)) . that's actually why I ended up asking this.

this is expected somewhat, I'd say this is more of a feature request: 
```python
    def get_special_tokens_mask(
        self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False
    ) -> List[int]:
        """
        Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding
        special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.

        Args:
            token_ids_0 (`List[int]`):
                List of ids of the first sequence.
            token_ids_1 (`List[int]`, *optional*):
                List of ids of the second sequence.
            already_has_special_tokens (`bool`, *optional*, defaults to `False`):
                Whether or not the token list is already formatted with special tokens for the model.

        Returns:
            A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
        """
```
`already_has_special_tokens` should be set to True ! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.