Create README.md

README.md

@@ -0,0 +1,25 @@
+## Model info
+
+This is a BPE tokenizer retrained from scratch on the [Wikitext-103](https://paperswithcode.com/dataset/wikitext-103) train, evaluation, and test sets. The vocabulary had 28,439 entries.
+
+This tokenizer was used to tokenize text for [the GPT-2-like transformer language model trained on Wikitext-103](https://huggingface.co/Kristijan/gpt2_wt103_12-layer).
+
+## Usage
+
+You can download the tokenizer directly from hub as follows:
+
+```
+from transformers import GPT2TokenizerFast
+
+tokenizer = GPT2TokenizerFast.from_pretrained("Kristijan/wikitext-103-tokenizer_v2")
+
+```
+
+After cloning/downloading the files, you can load the tokenizer using the `/from_pretrained()` methods as follows:
+
+```
+from transformers import GPT2TokenizerFast
+
+tokenizer = GPT2TokenizerFast.from_pretrained(path_to_folder_with_merges_and_vocab_files)
+
+```