Bingsu
/

ko_BBPE_tokenizer_roberta

Model card Files Files and versions Community

Bingsu commited on Aug 20, 2022

Commit

d057e6d

•

1 Parent(s): b5cc772

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -31,7 +31,7 @@ def gen_text(batch_size: int = 5000):
 tokenizer.train_from_iterator(
     gen_text(),
-    vocab_size=50265,
     min_frequency=2,
     special_tokens=[
         "<s>",

 tokenizer.train_from_iterator(
     gen_text(),
+    vocab_size=50265,  # roberta-base와 같은 크기
     min_frequency=2,
     special_tokens=[
         "<s>",