text: 
  
  TITLE
  Tokenizers - move from hardcoded configs and models to fully hosted
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 2
  rocket: 0
  eyes: 0
  
  BODY
  # What does this PR do?
  
  Fixes #8125 #8117 
  
  Tokenizer checkpoint files are now handled the same way than model files.
  
  Also:
  - add a `tokenizer_class_name` field in the tokenization config file `tokenizer_config.json`. This field is used when possible by `AutoTokenizer` to disambiguate the tokenizer to instantiate
  - concurrently configuration and vocabulary files are updated on the relevant model repo in the hub to wake them independant from the code-base.
  - the max length of the models was always the same for all the variante in an architecture. Consequently we simply the `max_model_input_sizes` attribute to make it a single integer instead of a mapping from string (model names) to integers. 
  
  ## Before submitting
  - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  - [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
        Pull Request section?
  - [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
        to it if that's the case.
  - [ ] Did you make sure to update the documentation with your changes? Here are the
        [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
        [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
  - [ ] Did you write any new necessary tests?
  
  
  ## Who can review?
  
  Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
  members/contributors which may be interested in your PR.
  
  <!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @
  
   If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
   Please tag fewer than 3 people.
  
   albert, bert, XLM: @LysandreJik
   GPT2: @LysandreJik, @patrickvonplaten
   tokenizers: @mfuntowicz
   Trainer: @sgugger
   Benchmarks: @patrickvonplaten
   Model Cards: @julien-c
   examples/distillation: @VictorSanh
   nlp datasets: [different repo](https://github.com/huggingface/nlp)
   rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
   Text Generation: @patrickvonplaten, @TevenLeScao
   Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
   T5: @patrickvonplaten
   Rag: @patrickvonplaten, @lhoestq
   EncoderDecoder: @patrickvonplaten
   Longformer, Reformer: @patrickvonplaten
   TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
   examples/seq2seq: @patil-suraj
   examples/bert-loses-patience: @JetRunner
   tensorflow: @jplu
   examples/token-classification: @stefan-it
   documentation: @sgugger
   FSTM: @stas00
   -->
  