text: 
  
  TITLE
  Collection of Tokenizer issues
  
  COMMENTS
  2
  
  REACTIONS
  +1: 2
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  ### System Info
  
  ```shell
  Transformers + Tokenizers
  ```
  
  
  ### Who can help?
  
  This Issue is a summary of multiple problems that we are currently encountering with Tokenizers. To solve them we'll need a more profound discussion of:
  - To what extend fast and slow tokenizers should be aligned
  - Whether all slow tokenizers should be kept
  - How to treat special tokens
  - Whether all internal methods of tokenizer should be exposed
  
  Relevant issues/PRs:
  https://github.com/huggingface/transformers/issues/15420
  https://github.com/huggingface/transformers/issues/16336
  https://github.com/huggingface/transformers/issues/16334
  https://github.com/huggingface/transformers/issues/16337
  https://github.com/huggingface/transformers/issues/15138
  https://github.com/huggingface/transformers/issues/16339
  https://github.com/huggingface/transformers/pull/15775
  
  To community:
  At the moment we sadly don't find the time to dive deeper here, but we're trying hard to allocate time to discuss the strategy here soon.
  
  
  
  ### Information
  
  - [ ] The official example scripts
  - [ ] My own modified scripts
  
  ### Tasks
  
  - [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
  - [ ] My own task or dataset (give details below)
  
  ### Reproduction
  
  See issues above
  
  ### Expected behavior
  
  ```shell
  Don't know yet
  ```
  