Regression in tokenizer loading
Author: JCRPaquin
Pull request: no
Labels: name: Core: Tokenization

name: Fast Tokenizers

name: bug

### System Info

There was a regression in commit b4727a1216bb21df2795e973063ed07202235d7e that prevents loading of some tokenizers.


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

1. Install `transformers`
2. Run the following code:
```python
from transformers import AutoTokenizer
AutoTokenizer.from_pretrained("adsabs/astroBERT")
```

Error output:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[2], [line 1](vscode-notebook-cell:?execution_count=2&line=1)
----> [1](vscode-notebook-cell:?execution_count=2&line=1) AutoTokenizer.from_pretrained("adsabs/astroBERT")
...
AttributeError: add_special_tokens conflicts with the method add_special_tokens in BertTokenizerFast
```

### Expected behavior

It's my expectation that the above code should run and produce a working tokenizer. Or, if the tokenizer config is too old and needs to be updated there should be an error message & script/tool/API to guide the user to update the config.

Comments:
cc @ArthurZucker, @itazap 

Thanks for raising! 😊  In order to support `add_special_tokens` as a bool kwargs, I would deprecate the function `add_special_tokens` since `add_tokens(special=True)` is already supported. We can also refactor for it to be an internal function like `_add_special_tokens` that is called when `special=True` in `add_tokens` Wdyt @ArthurZucker ?

I think the core issue here is that kwargs in configs can conflict with new APIs; we published this tokenizer before `add_special_tokens` was added.

Would it be possible to add compatibility layers to config loading? Maybe add a pinned compatibility version to each config such that old configs are still interpretable. They could be chained to get particularly old configs up to modern standards automatically, while also issuing warnings to users, and enabling an automatic upgrade path to owners.

Hey! Before or now, the `add_special_tokens` is not used for tokenization. 
@itazap's idea is good, but we should make sure it's used when encoding! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Just wanted to bump this issue since I'm running into it when kicking off a fine tune of ReFinED: https://github.com/amazon-science/ReFinED

```python
AttributeError: add_special_tokens conflicts with the method add_special_tokens in RobertaTokenizer
```

I think the issue should be opened on the repo! 🤗 

@ArthurZucker it's the same issue as this one; field names were added/changed in the base class and because there's no versioning the load fails. This is effectively a public API change by `transformers` with no quantification of the impact to hosted models. Perhaps HF would be willing to run regular regression tests against some significant fraction of models hosted on the Hub? Maybe selecting models based on some mix of unique config keys & classes to get better coverage?

@ptkenny you should be able to fork the original model and change the config without changes in functionality; just delete the `add_special_tokens` value from the config json and re-upload.

`add_special_tokens` has been there for quite a while, I don't think we made this change recently. It might have been an issue with serializing 😢 

But down to run tests on some models that are quite popular. We can select them based on downloads


> @ArthurZucker it's the same issue as this one; field names were added/changed in the base class and because there's no versioning the load fails. This is effectively a public API change by `transformers` with no quantification of the impact to hosted models. Perhaps HF would be willing to run regular regression tests against some significant fraction of models hosted on the Hub? Maybe selecting models based on some mix of unique config keys & classes to get better coverage?
> 
> @ptkenny you should be able to fork the original model and change the config without changes in functionality; just delete the `add_special_tokens` value from the config json and re-upload.

@JCRPaquin could you be a bit more specific about the config.json file you are referring to? I am running into the same issue as @ptkenny mentioned when using the ReFinED model but I do not have any config files defined. I have also not changed the value of the boolean to True. 

I am running into the same issue with `Salesforce/codegen25-7b-multi`. Is there a current workaround?
Edit: A workaround that seems to work is to rollback to before the mentioned change was introduced in https://github.com/huggingface/transformers/releases/tag/v4.44.0, in my case even further to `transformers==4.33.2`

We can probably just skip this specific arg, to make sure you can init but it won’t serialize it anymore! 

@sivadhulipala1999 Sorry, I meant `tokenizer_config.json`. 😅 

I just [removed the key from the config for our model in this PR](https://huggingface.co/adsabs/astroBERT/discussions/2/files), if a diff would be more helpful.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Just wanted to bump this issue since I'm running into it when kicking off a fine tune of ReFinED: https://github.com/amazon-science/ReFinED
> 
> AttributeError: add_special_tokens conflicts with the method add_special_tokens in RobertaTokenizer

Dirty fix: just comment out the assert statement in the pkg code.

> > Just wanted to bump this issue since I'm running into it when kicking off a fine tune of ReFinED: https://github.com/amazon-science/ReFinED
> > AttributeError: add_special_tokens conflicts with the method add_special_tokens in RobertaTokenizer
> 
> Dirty fix: just comment out the assert statement in the pkg code.
Hello！ I wonder which assert statement should be commented out.


Still running into this in 2026. (With CodeGen Model from Salesforce). @ArthurZucker @itazap 

> > Just wanted to bump this issue since I'm running into it when kicking off a fine tune of ReFinED: https://github.com/amazon-science/ReFinED
> > AttributeError: add_special_tokens conflicts with the method add_special_tokens in RobertaTokenizer
> 
> Dirty fix: just comment out the assert statement in the pkg code.

@yitao416 - could you clarify which assert statement should be commented out (for the hacky fix)? I think @kelayCHEN asked that as well.


@ai-nikolai update your transformers version to latest, try and if it does not work use a revision branch of the model that does not have this saved in the config!