text: 
  
  TITLE
  AlbertTokenizer handles special tokens incorrectly
  
  COMMENTS
  12
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  If I download the pretrained vocab `https://huggingface.co/albert-base-v1/resolve/main/spiece.model` to the local file system and use the following snippet, the tokenizer does not handle the special tokens properly:
  ```
  tokenizer = AlbertTokenizer('spiece.model')
  tokenizer.tokenize('[CLS] Hello World ! [SEP]')
  ['▁[', 'cl', 's', ']', '▁hello', '▁world', '▁', '!', '▁[', 's', 'ep', ']']
  ```
  If I use `from_pretrained` to load the vocab, it works well:
  ```
  tokenizer = AlbertTokenizer.from_pretrained('albert-base-v1')
  tokenizer.tokenize('[CLS] Hello World ! [SEP]')
  ['[CLS]', '▁hello', '▁world', '▁', '!', '[SEP]']
  ```