Doctor-Shotgun commited on
Commit
ef4ad7c
1 Parent(s): db573d9

Added tokens hack to make model able to be converted to ggml with llama.cpp

Browse files
Files changed (1) hide show
  1. added_tokens.json +33 -2
added_tokens.json CHANGED
@@ -1,3 +1,34 @@
1
  {
2
- "<pad>": 32000
3
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  {
2
+ "<pad>": 32000,
3
+ "<dummy01>": 32001,
4
+ "<dummy02>": 32002,
5
+ "<dummy03>": 32003,
6
+ "<dummy04>": 32004,
7
+ "<dummy05>": 32005,
8
+ "<dummy06>": 32006,
9
+ "<dummy07>": 32007,
10
+ "<dummy08>": 32008,
11
+ "<dummy09>": 32009,
12
+ "<dummy10>": 32010,
13
+ "<dummy11>": 32011,
14
+ "<dummy12>": 32012,
15
+ "<dummy13>": 32013,
16
+ "<dummy14>": 32014,
17
+ "<dummy15>": 32015,
18
+ "<dummy16>": 32016,
19
+ "<dummy17>": 32017,
20
+ "<dummy18>": 32018,
21
+ "<dummy19>": 32019,
22
+ "<dummy20>": 32020,
23
+ "<dummy21>": 32021,
24
+ "<dummy22>": 32022,
25
+ "<dummy23>": 32023,
26
+ "<dummy24>": 32024,
27
+ "<dummy25>": 32025,
28
+ "<dummy26>": 32026,
29
+ "<dummy27>": 32027,
30
+ "<dummy28>": 32028,
31
+ "<dummy29>": 32029,
32
+ "<dummy30>": 32030,
33
+ "<dummy31>": 32031
34
+ }