AndyChiang commited on
Commit
2d299bf
1 Parent(s): eb8fdd8

add tokenizer

Browse files
Files changed (3) hide show
  1. tokenizer.json +0 -0
  2. tokenizer_config.json +2 -2
  3. vocab.txt +0 -0
tokenizer.json CHANGED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json CHANGED
@@ -1,9 +1,9 @@
1
  {
2
  "cls_token": "[CLS]",
3
- "do_lower_case": false,
4
  "mask_token": "[MASK]",
5
  "model_max_length": 512,
6
- "name_or_path": "bert-base-cased",
7
  "pad_token": "[PAD]",
8
  "sep_token": "[SEP]",
9
  "special_tokens_map_file": null,
 
1
  {
2
  "cls_token": "[CLS]",
3
+ "do_lower_case": true,
4
  "mask_token": "[MASK]",
5
  "model_max_length": 512,
6
+ "name_or_path": "bert-base-uncased",
7
  "pad_token": "[PAD]",
8
  "sep_token": "[SEP]",
9
  "special_tokens_map_file": null,
vocab.txt CHANGED
The diff for this file is too large to render. See raw diff