Xenova
/

esm2_t30_150M_UR50D

Transformers.js

Model card Files Files and versions Community

Xenova HF staff commited on Dec 10, 2023

Commit

4b2cd83

•

1 Parent(s): 2203313

Update tokenizer.json

Files changed (1) hide show

tokenizer.json +1 -1

tokenizer.json CHANGED Viewed

@@ -303,7 +303,7 @@
   ],
   "normalizer": null,
   "pre_tokenizer": {
-    "type": "BertPreTokenizer"
   },
   "post_processor": {
     "type": "TemplateProcessing",

   ],
   "normalizer": null,
   "pre_tokenizer": {
+    "type": "WhitespaceSplit"
   },
   "post_processor": {
     "type": "TemplateProcessing",