text: 
  
  TITLE
  AutoTokenizer.from_pretrained function cannot be customized 
  
  COMMENTS
  3
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  A customized tokenizer has been provided in the tokenizer library, so that I can directly use the word segmentation data and vocab in the previous fairseq, but there is AutoTokenizer.from_pretrained function in the transformer, which cannot be customized like the tokenizer library, so I have no choice to directly use fairseq's vocab and word segmentation data in transformer. 
  
  What needs to be done？