Update README.md

README.md

@@ -12,12 +12,14 @@ This repo includes:
  ```
 from transformers import LlamaTokenizerFast
 from tokenizers import AddedToken
-tokenizer = LlamaTokenizerFast.from_pretrained("openlm-research/open_llama_3b_600bt_preview",
-                                                add_bos_token = True, add_eos_token = True,
-                                                bos_token = AddedToken("<s>",   single_word = True),
-                                                eos_token = AddedToken("</s>",  single_word = True),
-                                                unk_token = AddedToken("<unk>", single_word = True),
-                                                pad_token = AddedToken("<unk>", single_word = True))
+tokenizer = LlamaTokenizerFast.from_pretrained(
+    "openlm-research/open_llama_3b_600bt_preview",
+    add_bos_token = True, add_eos_token = True,
+    bos_token = AddedToken("<s>",   single_word = True),
+    eos_token = AddedToken("</s>",  single_word = True),
+    unk_token = AddedToken("<unk>", single_word = True),
+    pad_token = AddedToken("<unk>", single_word = True)
+)
  ```
 2) `AutoTokenizer` does not recognize the BOS, EOS and UNK tokens. All tokenizations weirdly prepend 0 and append 0 to the end, when actually, you're supposed to prepend 1 and append 2.
 3) Manually added BOS `<s>`, EOS `</s>`, UNK `<unk>` tokens, with PAD (padding) being also the `<unk>` token.