Xenova HF staff commited on
Commit
c9238fc
1 Parent(s): 9959d33

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +28 -0
README.md ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags:
4
+ - transformers.js
5
+ - tokenizers
6
+ ---
7
+
8
+ # GPT-4 Tokenizer
9
+
10
+ A 🤗-compatible version of the **GPT-4 tokenizer** (adapted from [openai/tiktoken](https://github.com/openai/tiktoken)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
11
+
12
+ ## Example usage:
13
+
14
+ ### Transformers/Tokenizers
15
+ ```py
16
+ from transformers import GPT2TokenizerFast
17
+
18
+ tokenizer = GPT2TokenizerFast.from_pretrained('Xenova/gpt-4')
19
+ assert tokenizer.encode('hello world') == [15339, 1917]
20
+ ```
21
+
22
+ ### Transformers.js
23
+ ```js
24
+ import { AutoTokenizer } from '@xenova/transformers';
25
+
26
+ const tokenizer = await AutoTokenizer.from_pretrained('Xenova/gpt-4');
27
+ const tokens = tokenizer.encode('hello world'); // [15339, 1917]
28
+ ```