Samuael commited on
Commit
257b41d
1 Parent(s): 587a522

Setting legacy to false

Browse files

Changed the tokenizer config by setting the legacy boolean value to false.

Files changed (1) hide show
  1. tokenizer_config.json +1 -1
tokenizer_config.json CHANGED
@@ -942,7 +942,7 @@
942
  "do_lower_case": true,
943
  "eos_token": "</s>",
944
  "extra_ids": 100,
945
- "legacy": true,
946
  "model_max_length": 1000000000000000019884624838656,
947
  "pad_token": "<pad>",
948
  "padding": true,
 
942
  "do_lower_case": true,
943
  "eos_token": "</s>",
944
  "extra_ids": 100,
945
+ "legacy": false,
946
  "model_max_length": 1000000000000000019884624838656,
947
  "pad_token": "<pad>",
948
  "padding": true,