WayneWiser csukuangfj commited on
Commit
9fc91ab
1 Parent(s): 3db724d

Upload 5 files (#3)

Browse files

- Upload 5 files (21037892290ef16c31651824c579fa1cf6ce6c0d)


Co-authored-by: fangjun <csukuangfj@users.noreply.huggingface.co>

data/lang_bpe_500/bpe.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c53433de083c4a6ad12d034550ef22de68cec62c4f58932a7b6b8b2f1e743fa5
3
+ size 244865
data/lang_bpe_500/lexicon.txt ADDED
The diff for this file is too large to render. See raw diff
 
data/lang_bpe_500/unigram_500.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c53433de083c4a6ad12d034550ef22de68cec62c4f58932a7b6b8b2f1e743fa5
3
+ size 244865
data/lang_bpe_500/unigram_500.vocab ADDED
@@ -0,0 +1,500 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <blk> 0
2
+ <sos/eos> 0
3
+ <unk> 0
4
+ S -3.23764
5
+ ▁THE -3.39114
6
+ ▁A -3.91235
7
+ T -3.9788
8
+ ▁AND -4.04148
9
+ ED -4.06805
10
+ ▁OF -4.11122
11
+ ▁TO -4.14819
12
+ E -4.15925
13
+ D -4.19884
14
+ N -4.28917
15
+ ING -4.40123
16
+ ▁IN -4.45251
17
+ Y -4.52708
18
+ M -4.57919
19
+ C -4.62646
20
+ ▁I -4.68235
21
+ A -4.68468
22
+ P -4.69613
23
+ ▁HE -4.71198
24
+ R -4.82144
25
+ O -4.83893
26
+ L -4.88007
27
+ RE -4.88603
28
+ I -4.90352
29
+ U -4.91157
30
+ ER -4.95434
31
+ ▁IT -4.99701
32
+ LY -5.00155
33
+ ▁THAT -5.00795
34
+ ▁WAS -5.04607
35
+ ▁ -5.05486
36
+ ▁S -5.05888
37
+ AR -5.10129
38
+ ▁BE -5.14569
39
+ F -5.18181
40
+ ▁C -5.18836
41
+ IN -5.20029
42
+ B -5.20859
43
+ ▁FOR -5.23485
44
+ OR -5.26974
45
+ LE -5.2702
46
+ ' -5.27032
47
+ ▁HIS -5.28672
48
+ ▁YOU -5.34529
49
+ AL -5.35383
50
+ ▁RE -5.35863
51
+ V -5.36811
52
+ ▁B -5.36912
53
+ G -5.39337
54
+ RI -5.41681
55
+ ▁E -5.41899
56
+ ▁WITH -5.4268
57
+ ▁T -5.49151
58
+ ▁AS -5.50186
59
+ LL -5.50605
60
+ ▁P -5.51409
61
+ ▁HER -5.52391
62
+ ST -5.52898
63
+ ▁HAD -5.53885
64
+ ▁SO -5.56342
65
+ ▁F -5.56472
66
+ W -5.57126
67
+ CE -5.61235
68
+ ▁IS -5.63341
69
+ ND -5.63677
70
+ ▁NOT -5.64685
71
+ TH -5.65822
72
+ ▁BUT -5.65885
73
+ EN -5.67118
74
+ ▁SHE -5.67453
75
+ ▁ON -5.67506
76
+ VE -5.6788
77
+ ON -5.68108
78
+ SE -5.68128
79
+ ▁DE -5.68499
80
+ UR -5.70257
81
+ ▁G -5.70716
82
+ CH -5.71597
83
+ K -5.73117
84
+ TER -5.73588
85
+ ▁AT -5.74333
86
+ IT -5.75281
87
+ ▁ME -5.75999
88
+ RO -5.78792
89
+ NE -5.81237
90
+ RA -5.84735
91
+ ES -5.86915
92
+ IL -5.9164
93
+ NG -5.95522
94
+ IC -5.95622
95
+ ▁NO -5.95875
96
+ ▁HIM -5.9632
97
+ ENT -5.96542
98
+ IR -5.98727
99
+ ▁WE -6.00283
100
+ H -6.00505
101
+ ▁DO -6.01678
102
+ ▁ALL -6.0215
103
+ ▁HAVE -6.04155
104
+ LO -6.04175
105
+ ▁BY -6.0754
106
+ ▁MY -6.07629
107
+ ▁MO -6.07686
108
+ ▁THIS -6.07749
109
+ LA -6.07936
110
+ ▁ST -6.10084
111
+ ▁WHICH -6.10116
112
+ ▁CON -6.11572
113
+ ▁THEY -6.13378
114
+ CK -6.13419
115
+ TE -6.14807
116
+ ▁SAID -6.15974
117
+ ▁FROM -6.16115
118
+ ▁GO -6.16495
119
+ ▁WHO -6.17239
120
+ ▁TH -6.18452
121
+ ▁OR -6.18883
122
+ ▁D -6.18975
123
+ ▁W -6.19439
124
+ VER -6.20494
125
+ LI -6.22095
126
+ ▁SE -6.2297
127
+ ▁ONE -6.23026
128
+ ▁CA -6.23397
129
+ ▁AN -6.24262
130
+ ▁LA -6.24786
131
+ ▁WERE -6.24858
132
+ EL -6.271
133
+ ▁HA -6.28368
134
+ ▁MAN -6.29896
135
+ ▁FA -6.31168
136
+ ▁EX -6.31364
137
+ AD -6.31526
138
+ ▁SU -6.34091
139
+ RY -6.35457
140
+ ▁MI -6.37021
141
+ AT -6.39107
142
+ ▁BO -6.39815
143
+ ▁WHEN -6.40637
144
+ AN -6.41853
145
+ THER -6.42115
146
+ PP -6.43792
147
+ ATION -6.44105
148
+ ▁FI -6.45761
149
+ ▁WOULD -6.47257
150
+ ▁PRO -6.47274
151
+ OW -6.48047
152
+ ET -6.48289
153
+ ▁O -6.48763
154
+ ▁THERE -6.48808
155
+ ▁HO -6.48841
156
+ ION -6.48846
157
+ ▁WHAT -6.49159
158
+ ▁FE -6.49797
159
+ ▁PA -6.50029
160
+ US -6.50081
161
+ MENT -6.50395
162
+ ▁MA -6.50516
163
+ UT -6.53807
164
+ ▁OUT -6.54596
165
+ ▁THEIR -6.54897
166
+ ▁IF -6.55068
167
+ ▁LI -6.55453
168
+ ▁K -6.56374
169
+ ▁WILL -6.5669
170
+ ▁ARE -6.57477
171
+ ID -6.57502
172
+ ▁RO -6.57553
173
+ DE -6.58188
174
+ TION -6.59518
175
+ ▁WA -6.59566
176
+ PE -6.60184
177
+ ▁UP -6.60311
178
+ ▁SP -6.61112
179
+ ▁PO -6.61409
180
+ IGHT -6.63771
181
+ ▁UN -6.64183
182
+ RU -6.64252
183
+ ▁LO -6.64639
184
+ AS -6.64889
185
+ OL -6.65442
186
+ ▁LE -6.66382
187
+ ▁BEEN -6.67806
188
+ ▁SH -6.67835
189
+ ▁RA -6.68003
190
+ ▁SEE -6.69546
191
+ KE -6.70826
192
+ UL -6.71612
193
+ TED -6.71696
194
+ ▁SA -6.72967
195
+ UN -6.73177
196
+ UND -6.73395
197
+ ANT -6.73423
198
+ ▁NE -6.743
199
+ IS -6.75404
200
+ ▁THEM -6.75743
201
+ CI -6.76993
202
+ GE -6.77069
203
+ ▁COULD -6.77503
204
+ ▁DIS -6.78135
205
+ OM -6.79366
206
+ ISH -6.80337
207
+ HE -6.80705
208
+ EST -6.81058
209
+ ▁SOME -6.81704
210
+ ENCE -6.82267
211
+ ITY -6.83585
212
+ IVE -6.84022
213
+ ▁US -6.84091
214
+ ▁MORE -6.84456
215
+ ▁EN -6.84947
216
+ ARD -6.85159
217
+ ATE -6.86067
218
+ ▁YOUR -6.86188
219
+ ▁INTO -6.86252
220
+ ▁KNOW -6.86753
221
+ ▁CO -6.86961
222
+ ANCE -6.8727
223
+ ▁TIME -6.8751
224
+ ▁WI -6.88392
225
+ ▁YE -6.8877
226
+ AGE -6.8955
227
+ ▁NOW -6.90387
228
+ TI -6.90402
229
+ FF -6.90618
230
+ ABLE -6.90917
231
+ ▁VERY -6.92207
232
+ ▁LIKE -6.92869
233
+ AM -6.94687
234
+ HI -6.94986
235
+ Z -6.9511
236
+ ▁OTHER -6.96464
237
+ ▁THAN -6.97308
238
+ ▁LITTLE -6.97622
239
+ ▁DID -6.98152
240
+ ▁LOOK -6.98467
241
+ TY -6.99474
242
+ ERS -6.99918
243
+ ▁CAN -7.00994
244
+ ▁CHA -7.01063
245
+ ▁AR -7.02555
246
+ X -7.02668
247
+ FUL -7.03465
248
+ UGH -7.04608
249
+ ▁BA -7.04937
250
+ ▁DAY -7.0543
251
+ ▁ABOUT -7.05554
252
+ TEN -7.05588
253
+ IM -7.0583
254
+ ▁ANY -7.05892
255
+ ▁PRE -7.06131
256
+ ▁OVER -7.06494
257
+ IES -7.07832
258
+ NESS -7.08111
259
+ ME -7.08369
260
+ BLE -7.08645
261
+ ▁M -7.09032
262
+ ROW -7.09219
263
+ ▁HAS -7.0976
264
+ ▁GREAT -7.10444
265
+ ▁VI -7.10527
266
+ TA -7.10781
267
+ ▁AFTER -7.10952
268
+ PER -7.1132
269
+ ▁AGAIN -7.11478
270
+ HO -7.11545
271
+ SH -7.11555
272
+ ▁UPON -7.12675
273
+ ▁DI -7.13004
274
+ ▁HAND -7.13211
275
+ ▁COM -7.13273
276
+ IST -7.13438
277
+ TURE -7.13771
278
+ ▁STA -7.14628
279
+ ▁THEN -7.15775
280
+ ▁SHOULD -7.15836
281
+ ▁GA -7.16774
282
+ OUS -7.16944
283
+ OUR -7.16965
284
+ ▁WELL -7.17122
285
+ ▁ONLY -7.17628
286
+ MAN -7.17899
287
+ ▁GOOD -7.18494
288
+ ▁TWO -7.18746
289
+ ▁MAR -7.18755
290
+ ▁SAY -7.19636
291
+ ▁HU -7.20214
292
+ TING -7.20818
293
+ ▁OUR -7.21813
294
+ RESS -7.22296
295
+ ▁DOWN -7.22319
296
+ IOUS -7.24748
297
+ ▁BEFORE -7.24835
298
+ ▁DA -7.2508
299
+ ▁NA -7.25861
300
+ QUI -7.26388
301
+ ▁MADE -7.26918
302
+ ▁EVERY -7.26921
303
+ ▁OLD -7.27794
304
+ ▁EVEN -7.28005
305
+ IG -7.28081
306
+ ▁COME -7.28172
307
+ ▁GRA -7.28522
308
+ ▁RI -7.29145
309
+ ▁LONG -7.29314
310
+ OT -7.29691
311
+ SIDE -7.30065
312
+ WARD -7.31155
313
+ ▁FO -7.31391
314
+ ▁WHERE -7.31479
315
+ MO -7.31992
316
+ LESS -7.32883
317
+ ▁SC -7.32891
318
+ ▁MUST -7.3305
319
+ ▁NEVER -7.33078
320
+ ▁HOW -7.34345
321
+ ▁CAME -7.34599
322
+ ▁SUCH -7.34721
323
+ ▁RU -7.35962
324
+ ▁TAKE -7.35962
325
+ ▁WO -7.36853
326
+ ▁CAR -7.37945
327
+ UM -7.37964
328
+ AK -7.39205
329
+ ▁THINK -7.40939
330
+ ▁MUCH -7.40963
331
+ ▁MISTER -7.4201
332
+ ▁MAY -7.43389
333
+ ▁JO -7.44387
334
+ ▁WAY -7.4456
335
+ ▁COMP -7.45547
336
+ ▁THOUGHT -7.45606
337
+ ▁STO -7.46284
338
+ ▁MEN -7.46377
339
+ ▁BACK -7.46526
340
+ ▁DON -7.46548
341
+ J -7.46605
342
+ ▁LET -7.48391
343
+ ▁TRA -7.4958
344
+ ▁FIRST -7.49608
345
+ ▁JUST -7.49723
346
+ ▁VA -7.49963
347
+ ▁OWN -7.51114
348
+ ▁PLA -7.51664
349
+ ▁MAKE -7.5218
350
+ ATED -7.52921
351
+ ▁HIMSELF -7.53339
352
+ ▁WENT -7.54
353
+ ▁PI -7.55717
354
+ GG -7.55994
355
+ RING -7.5606
356
+ ▁DU -7.56341
357
+ ▁MIGHT -7.56862
358
+ ▁PART -7.57009
359
+ ▁GIVE -7.58394
360
+ ▁IMP -7.58817
361
+ ▁BU -7.59192
362
+ ▁PER -7.59583
363
+ ▁PLACE -7.60502
364
+ ▁HOUSE -7.60934
365
+ ▁THROUGH -7.62521
366
+ IAN -7.62855
367
+ ▁SW -7.63782
368
+ ▁UNDER -7.64373
369
+ QUE -7.64402
370
+ ▁AWAY -7.64512
371
+ ▁LOVE -7.64681
372
+ QUA -7.65031
373
+ ▁LIFE -7.66029
374
+ ▁GET -7.66648
375
+ ▁WITHOUT -7.67412
376
+ ▁PASS -7.682
377
+ ▁TURN -7.69236
378
+ IGN -7.69649
379
+ ▁HEAD -7.69697
380
+ ▁MOST -7.70515
381
+ ▁THOSE -7.7168
382
+ ▁SHALL -7.71795
383
+ ▁EYES -7.71861
384
+ ▁COL -7.74121
385
+ ▁STILL -7.74208
386
+ ▁NIGHT -7.74542
387
+ ▁NOTHING -7.76816
388
+ ITION -7.76824
389
+ HA -7.76993
390
+ ▁TELL -7.76996
391
+ ▁WORK -7.77374
392
+ ▁LAST -7.77545
393
+ ▁NEW -7.78277
394
+ ▁FACE -7.78386
395
+ ▁HI -7.79126
396
+ ▁WORD -7.8012
397
+ ▁FOUND -7.80479
398
+ ▁COUNT -7.80516
399
+ ▁OB -7.80527
400
+ ▁WHILE -7.80765
401
+ ▁SHA -7.81719
402
+ ▁MEAN -7.83619
403
+ ▁SAW -7.83676
404
+ ▁PEOPLE -7.83766
405
+ ▁FRIEND -7.8565
406
+ ▁THREE -7.86814
407
+ ▁ROOM -7.87585
408
+ ▁SAME -7.88751
409
+ ▁THOUGH -7.89248
410
+ ▁RIGHT -7.89619
411
+ ▁CHILD -7.89778
412
+ ▁FATHER -7.90382
413
+ ▁ANOTHER -7.90587
414
+ ▁HEART -7.91276
415
+ ▁WANT -7.92235
416
+ ▁TOOK -7.94253
417
+ OOK -7.94851
418
+ ▁LIGHT -7.96145
419
+ ▁MISSUS -7.9786
420
+ ▁OPEN -7.98527
421
+ ▁JU -7.9894
422
+ ▁ASKED -7.99096
423
+ PORT -8.0012
424
+ ▁LEFT -8.00187
425
+ ▁JA -8.02979
426
+ ▁WORLD -8.03992
427
+ ▁HOME -8.04939
428
+ ▁WHY -8.0618
429
+ ▁ALWAYS -8.06564
430
+ ▁ANSWER -8.0739
431
+ ▁SEEMED -8.088
432
+ ▁SOMETHING -8.08951
433
+ ▁GIRL -8.09335
434
+ ▁BECAUSE -8.10615
435
+ ▁NAME -8.10768
436
+ ▁TOLD -8.11746
437
+ ▁NI -8.12176
438
+ ▁HIGH -8.12178
439
+ IZE -8.13805
440
+ ▁WOMAN -8.14522
441
+ ▁FOLLOW -8.15124
442
+ ▁RETURN -8.17126
443
+ ▁KNEW -8.17579
444
+ ▁EACH -8.18032
445
+ ▁KIND -8.18787
446
+ ▁JE -8.19007
447
+ ▁ACT -8.20004
448
+ ▁LU -8.20535
449
+ ▁CERTAIN -8.21049
450
+ ▁YEARS -8.2275
451
+ ▁QUITE -8.22848
452
+ ▁APPEAR -8.23444
453
+ ▁BETTER -8.24887
454
+ ▁HALF -8.25101
455
+ ▁PRESENT -8.25108
456
+ ▁PRINCE -8.263
457
+ SHIP -8.26654
458
+ ▁ALSO -8.27005
459
+ ▁BEGAN -8.27496
460
+ ▁HAVING -8.28474
461
+ ▁ENOUGH -8.28608
462
+ ▁PERSON -8.28836
463
+ ▁LADY -8.29498
464
+ ▁WHITE -8.3115
465
+ ▁COURSE -8.31653
466
+ ▁VOICE -8.3193
467
+ ▁SPEAK -8.32799
468
+ ▁POWER -8.35211
469
+ ▁MORNING -8.35582
470
+ ▁BETWEEN -8.3565
471
+ ▁AMONG -8.35997
472
+ ▁KEEP -8.36216
473
+ ▁WALK -8.3686
474
+ ▁MATTER -8.37061
475
+ ▁TEA -8.37863
476
+ ▁BELIEVE -8.37922
477
+ ▁SMALL -8.3814
478
+ ▁TALK -8.38939
479
+ ▁FELT -8.39477
480
+ ▁HORSE -8.39834
481
+ ▁MYSELF -8.40051
482
+ ▁SIX -8.40069
483
+ ▁HOWEVER -8.40304
484
+ ▁FULL -8.40399
485
+ ▁HERSELF -8.40714
486
+ ▁POINT -8.41248
487
+ ▁STOOD -8.41336
488
+ ▁HUNDRED -8.41404
489
+ ▁ALMOST -8.42802
490
+ ▁SINCE -8.43626
491
+ ▁LARGE -8.44207
492
+ ▁LEAVE -8.44699
493
+ ▁PERHAPS -8.4576
494
+ ▁DARK -8.46885
495
+ ▁SUDDEN -8.46906
496
+ ▁REPLIED -8.47536
497
+ ▁ANYTHING -8.48241
498
+ ▁WONDER -8.48792
499
+ ▁UNTIL -8.48926
500
+ Q -9.78832
data/lang_bpe_500/words.txt ADDED
The diff for this file is too large to render. See raw diff