Aye10032
/

t2-genius-en-zh

Aye10032 commited on 22 days ago

Commit

84972a2

•

1 Parent(s): 13b002d

Upload tokenizer

Files changed (2) hide show

tokenizer.json CHANGED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json CHANGED Viewed

@@ -42,5 +42,5 @@
   "pad_token": "|<PAD>|",
   "pad_token_type_id": 0,
   "padding_side": "right",
-  "tokenizer_class": "T2GeniusTokenizer"
 }

   "pad_token": "|<PAD>|",
   "pad_token_type_id": 0,
   "padding_side": "right",
+  "tokenizer_class": "PreTrainedTokenizerFast"
 }