pszemraj commited on
Commit
69715fe
1 Parent(s): 9e434fb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -0
README.md CHANGED
@@ -14,6 +14,15 @@ Vocabulary size: 65103
14
  - relevant special tokens for T5 training added
15
  - post processor updated following t5's tokenizer
16
 
 
 
 
 
 
 
 
 
 
17
  ## post processor
18
 
19
 
 
14
  - relevant special tokens for T5 training added
15
  - post processor updated following t5's tokenizer
16
 
17
+ usage:
18
+
19
+
20
+ ```py
21
+ from transformers import AutoTokenizer
22
+ tk = AutoTokenizer.from_pretrained('BEE-spoke-data/claude-tokenizer-forT5')
23
+ inputs = tk("here are some words", return_tensors="pt")
24
+ ```
25
+
26
  ## post processor
27
 
28