Hugging Face's logo

Xenova
/
dbrx-instruct-tokenizer

Transformers
Transformers.js
tokenizers
Inference Endpoints
Model card Files Community
1
Xenova HF staff commited on
Commit
ba49911
·
verified ·
1 Parent(s): d5cbfd2

Upload tokenizer.json

Browse files
Files changed (1) hide show
  1. tokenizer.json +11 -2
tokenizer.json CHANGED
@@ -49,7 +49,16 @@
49
  "special": true
50
  },
51
  {
52
- "id": 100264,
 
 
 
 
 
 
 
 
 
53
  "content": "<|im_start|>",
54
  "single_word": false,
55
  "lstrip": false,
@@ -58,7 +67,7 @@
58
  "special": true
59
  },
60
  {
61
- "id": 100265,
62
  "content": "<|im_end|>",
63
  "single_word": false,
64
  "lstrip": false,
 
49
  "special": true
50
  },
51
  {
52
+ "id": 100277,
53
+ "content": "<|pad|>",
54
+ "single_word": false,
55
+ "lstrip": false,
56
+ "rstrip": false,
57
+ "normalized": false,
58
+ "special": true
59
+ },
60
+ {
61
+ "id": 100278,
62
  "content": "<|im_start|>",
63
  "single_word": false,
64
  "lstrip": false,
 
67
  "special": true
68
  },
69
  {
70
+ "id": 100279,
71
  "content": "<|im_end|>",
72
  "single_word": false,
73
  "lstrip": false,