Xenova
/

grok-1-tokenizer

Transformers.js

Inference Endpoints

Model card Files Files and versions Community

Xenova HF staff commited on Mar 18

Commit

300ac7e

•

1 Parent(s): fe224b1

Update README.md

Files changed (1) hide show

README.md +4 -3

README.md CHANGED Viewed

@@ -1,8 +1,9 @@
 ---
 library_name: transformers
 tags:
- - transformers.js
- - tokenizers
 ---
 # Grok-1 Tokenizer
@@ -25,4 +26,4 @@ import { AutoTokenizer } from '@xenova/transformers';
 const tokenizer = await AutoTokenizer.from_pretrained('Xenova/grok-1-tokenizer');
 const tokens = tokenizer.encode('hello world'); // [21560, 1135]
-```

 ---
 library_name: transformers
 tags:
+- transformers.js
+- tokenizers
+license: apache-2.0
 ---
 # Grok-1 Tokenizer
 const tokenizer = await AutoTokenizer.from_pretrained('Xenova/grok-1-tokenizer');
 const tokens = tokenizer.encode('hello world'); // [21560, 1135]
+```