Add versioning system to fast tokenizer files
Author: sgugger
Pull request: yes

# What does this PR do?

Some changes cannot be done to the fast tokenizers file without breaking backward compatibility. This PR introduces a versioning system by allowing a model repo to contain multiple tokenizer files: the `tokenizer.json` is the default one and if one (or several) `tokenizer.x.y.z.json` exist, those files are used for the version x.y.z (of Transformers) and above.

cc @n1t0 as it should be helpful to solve that longstanding bug.

Comments:
might be cleaner if this worked in the other direction, i.e.

> multiple tokenizer files: the `tokenizer.json` is the default one, used in the most recent version of Transformers. If one or more `tokenizer-x.y.z.json` exist, those files are used for the version x.y.z (of Transformers) and below.

Makes more sense on the Hub side as well. What do you think?

@julien-c this would break repositories that rely on `transformers` versions that are earlier than the first one that will have this feature.

Were we to update the `tokenizer.json` file to the new, "fixed" one, and add a new `tokenizer-x.x.x.json` file to be used by earlier versions of `transformers`, then we would have no way of telling all versions < `4.10.0` to use that version rather than the standard `tokenizer.json` file.

I think your assertion depends on what kind of changes are made to the JSON files. If it's only new attributes for example I wouldn't expect older versions to break, but from what I understand you're actually talking about modifying the actual attributes?

Yes, the attributes actually need to be modified. For example, see this issue: https://github.com/huggingface/transformers/issues/9633

There was an offset mappings bug, which needed to be patched. However, the issue lived in the `tokenizer.json` file itself - so the recommended way to patch this was for users to recompile that file, by passing the "slow" tokenizer files, and using the newer `tokenizers` version to generate the updated file.

I believe there are other issues, and there will be other issues as the libraries continue to evolve. Implementing this here allows us to ensure that the previous versions remain completely unaffected - while offering a way to patch models for future use.

> Yes, the attributes actually need to be modified. For example, see this issue: #9633
> 
> There was an offset mappings bug, which needed to be patched. However, the issue lived in the `tokenizer.json` file itself - so the recommended way to patch this was for users to recompile that file, by passing the "slow" tokenizer files, and using the newer `tokenizers` version to generate the updated file.
> 
> I believe there are other issues, and there will be other issues as the libraries continue to evolve. Implementing this here allows us to ensure that the previous versions remain completely unaffected - while offering a way to patch models for future use.

going on a whim here, but what about using git branches to do this?

The problem with a new branch is that we then can't have a new version of the model in a new git branch that has to be used with one tokenizer file if versions of Transformers are old, and another one if they are more recent. And it wouldn't be compatible with the sure selecting their own branch as well (though in that case they should make sure to have the right version with tokenizers file).

The key here (for more context) is that we have tokenizers that have a "wrong" tokenizer file for more recent versions of Tokenizers (controlled by the version of Transformers) because there was a bug in the conversion from slow to fast tokenizer script. We can't touch the main branch and the tokenizer.json file otherwise every code in production using those models will suddenly break (the changes are significant sadly).