SaulLu
/

bengali-tokenizer-v2

Model card Files Files and versions Community

SaulLu commited on May 18, 2021

Commit

beaff1b

•

1 Parent(s): 84e91ce

set lstrip to false

Files changed (1) hide show

tokenizer.json +1 -1

tokenizer.json CHANGED Viewed

@@ -44,7 +44,7 @@
             "special": true,
             "content": "[MASK]",
             "single_word": false,
-            "lstrip": false,
             "rstrip": false,
             "normalized": false
         }

             "special": true,
             "content": "[MASK]",
             "single_word": false,
+            "lstrip": true,
             "rstrip": false,
             "normalized": false
         }