GlobalTokenizer

GlobalTokenizer tokenize all languages of the earth.

usage

from transformers import AutoModelForMaskedLM, AutoTokenizer
model_path = "JuanbaiAI/GlobalTokenizer"

tokenizer = AutoTokenizer.from_pretrained(model_path)
print(tokenizer("Hello, 世界"))

corpus:

Wikipedia, Wikisource, SlimRedPajama, baidubaike, daizhige, Wanjuan.

languages(200+) coverage:

• English:33%
• Chinese:33%
• Global (other langauges exclude English/Chinese):33%

train

• algorithm: BBPE
• code: https://huggingface.co/learn/nlp-course/chapter6/8

benchmark

Corpus		UnicodeTokenizer		Bloom		GlobalTokenizer
	num of bytes	avg bytes /tokens	num of tokens	avg bytes /tokens	num of tokens	avg bytes /tokens	num of tokens
RedPajama	4.2E+10	2.731478	1.54E+10	4.370739	9.6E+09	3.96642	1.06E+10
baidubaike	4.97E+09	3.980441	1.25E+09	4.127816	1.2E+09	3.922987	1.27E+09
daizhige	2.29E+09	3.473577	6.59E+08	3.244604	7.05E+08	3.174714	7.21E+08
TextBook-cn	1.1E+09	3.480186	3.16E+08	3.299681	3.33E+08	3.241768	3.39E+08
WebText-cn	4.2E+10	4.177369	1.01E+10	4.381168	9.6E+09	4.281016	9.82E+09
ab	1237796	6.056468	204375.9	2.37433	521324.3	3.014965	410550.7
ace	1574258	3.166403	497175.5	2.920242	539084.7	2.865722	549340.7
af	1.24E+08	2.842203	43798967	3.089724	40290184	3.024023	41165548
als	49452211	2.957404	16721491	2.899396	17056040	2.794853	17694029
alt	2910477	5.346916	544328.2	2.811496	1035206	3.755865	774915.3
am	10720027	5.450505	1966795	1.651675	6490396	2.368297	4526471
ami	2261859	2.511871	900467.9	2.999494	754080.3	2.741438	825063
an	29573020	2.638144	11209781	3.678932	8038480	3.001739	9851963
ang	1789811	3.09878	577585.8	2.648479	675788.2	2.541121	704339.2
anp	3355979	6.35562	528033.3	8.648999	388019.4	3.51896	953684.9
ar	1.79E+09	4.512798	3.98E+08	6.1454	2.92E+08	4.111721	4.37E+08
ary	4672257	4.115167	1135375	4.571392	1022064	3.667588	1273932
arz	3.17E+08	4.658518	68097110	5.049221	62827827	5.280836	60072224
as	38558469	7.573861	5090992	8.281984	4655704	3.34377	11531436
ast	2.71E+08	2.864269	94682610	3.76335	72062514	3.128894	86674857
av	2573436	5.750956	447479.7	2.826744	910388.9	3.639347	707114.7
avk	7984474	2.768189	2884367	2.775794	2876465	2.650919	3011964
awa	1269824	6.413487	197992.8	7.927757	160174.4	3.513209	361442.8
ay	1756131	2.819794	622787	2.824251	621804	2.463072	712983.9
az	2.51E+08	3.781892	66373457	2.594857	96736449	2.82334	88907897
azb	46739926	4.962435	9418748	3.655348	12786724	4.257964	10977061
ba	1.23E+08	5.780001	21210345	2.623517	46729559	4.177275	29348274
ban	6715876	3.355344	2001546	3.239442	2073158	2.977388	2255627
bar	21799389	2.91939	7467105	2.870996	7592972	2.713014	8035119
bat-smg	3348765	3.503523	955827.9	2.320313	1443238	2.480687	1349935
bcl	11369234	2.875691	3953567	3.320823	3423619	3.084119	3686380
be	3.11E+08	5.476524	56838635	3.017893	1.03E+08	4.068214	76514692
be-x-old	1.14E+08	5.694565	20075035	2.982457	38330337	4.088079	27963890
bg	5.27E+08	4.992976	1.06E+08	3.51584	1.5E+08	4.492216	1.17E+08
bh	5880458	5.962781	986193.8	7.715468	762164.8	3.517409	1671815
bjn	3323032	3.253983	1021220	3.749782	886193.4	3.105171	1070161
blk	8028430	6.456261	1243511	1.551035	5176175	3.901145	2057968
bn	3.44E+08	8.168696	42098026	9.874329	34826264	3.375484	1.02E+08
bo	38851784	5.471347	7100954	2.369802	16394527	3.550876	10941464
bpy	6568483	7.031357	934170	6.442238	1019596	3.316048	1980816
br	49768597	2.596823	19165184	2.847565	17477598	2.618958	19003205
bs	1.17E+08	3.034931	38645253	2.676185	43825689	2.806575	41789603
bxr	3078699	5.557568	553965.1	2.724922	1129830	3.588642	857900.8
ca	1.12E+09	2.739052	4.1E+08	4.175102	2.69E+08	3.151386	3.56E+08
cdo	1982914	2.605206	761135.1	2.549999	777613.7	2.208196	897979
ce	88393330	4.908585	18007906	2.941855	30046803	5.186398	17043298
ceb	8.28E+08	2.780197	2.98E+08	3.081311	2.69E+08	3.753617	2.21E+08
ckb	42964544	5.585014	7692826	2.731714	15728052	3.510231	12239804
co	5794731	2.716548	2133123	3.019771	1918931	2.767274	2094021
crh	3654461	3.631442	1006339	2.507428	1457454	2.738573	1334440
cs	1.1E+09	3.254687	3.38E+08	2.60566	4.22E+08	2.818314	3.91E+08
csb	2055233	3.303751	622090.8	2.390176	859867	2.315591	887563.2
cv	29640641	4.05758	7305005	2.837606	10445649	3.770825	7860520
cy	1.14E+08	2.799797	40541764	2.564199	44266735	3.051617	37196248
da	3.39E+08	2.976211	1.14E+08	2.994664	1.13E+08	3.050977	1.11E+08
dag	9026986	2.625832	3437762	3.41197	2645681	3.140065	2874777
de	6.08E+09	3.265201	1.86E+09	3.293052	1.85E+09	3.295854	1.85E+09
diq	7616839	3.015742	2525693	2.618003	2909408	2.480188	3071074
dsb	1931937	3.149291	613451.5	2.481379	778573.9	2.486934	776834.9
dty	2521250	7.573179	332918.3	8.289923	304134.3	3.395063	742622.5
dv	5283133	7.507441	703719.5	1.082612	4879986	2.04329	2585601
dz	2583520	5.389404	479370.3	2.35514	1096971	3.561721	725357.2
el	7.41E+08	5.303681	1.4E+08	2.740865	2.7E+08	3.639245	2.04E+08
eml	1689575	2.63297	641699.2	2.679583	630536.4	2.404491	702674.7
en	1.3E+10	2.784381	4.67E+09	4.317653	3.01E+09	4.016363	3.24E+09
eo	3E+08	2.934561	1.02E+08	2.982057	1.01E+08	2.910416	1.03E+08
es	3.72E+09	2.865625	1.3E+09	4.372629	8.5E+08	3.423013	1.09E+09
et	2.68E+08	3.519661	76075834	2.766091	96801265	2.704236	99015440
eu	2.8E+08	3.412258	82006870	4.233857	66093076	2.948036	94920346
ext	2761099	2.848068	969463.7	3.367813	819849.2	2.909613	948957.6
fa	7.93E+08	4.328803	1.83E+08	4.326995	1.83E+08	4.276889	1.85E+08
fat	1124684	2.635375	426764.4	2.885461	389776.2	2.563495	438730.7
ff	1087702	3.073717	353871.9	2.283122	476410	2.181656	498567
fi	7.14E+08	3.875986	1.84E+08	2.826093	2.53E+08	2.847687	2.51E+08
fiu-vro	2434159	3.292221	739366.9	2.463464	988104.1	2.407473	1011085
fo	8857239	3.002534	2949922	2.697937	3282967	2.633042	3363881
fr	4.66E+09	2.812367	1.66E+09	4.357305	1.07E+09	3.327677	1.4E+09
frp	1914046	2.802151	683063	3.104856	616468.5	2.716365	704635.1
frr	5317694	2.855094	1862529	2.727262	1949829	2.649299	2007208
fur	2421238	2.630931	920297	3.050013	793845.2	2.810208	861586.7
fy	76002257	2.748455	27652722	2.854965	26621080	2.831785	26838989
ga	34136733	2.907793	11739740	2.763848	12351159	2.647309	12894878
gag	1331866	3.32828	400166.4	2.67359	498156.4	2.659286	500836
gan	1508844	3.566718	423034.3	3.216688	469067.6	3.075602	490585
gcr	1345482	2.66169	505499.1	2.859282	470566.5	2.564873	524580.3
gd	7190137	2.708183	2654967	2.723874	2639673	2.507579	2867363
gl	2.95E+08	2.91577	1.01E+08	3.951435	74610154	3.256364	90535697
glk	2382997	4.282207	556488.1	3.740417	637094	3.520784	676837.1
gn	3806548	3.111337	1223444	2.798959	1359987	2.494229	1526142
gom	11306217	6.703585	1686593	6.073611	1861531	3.33379	3391400
gor	2101154	3.124766	672419.7	3.949286	532033.9	3.423681	613712
gpe	1141261	2.811199	405969.5	4.171368	273594	3.974151	287171
gu	39554078	7.279748	5433441	8.638276	4578932	3.18546	12417069
gur	1068954	2.313831	461984.5	2.921246	365924	2.644545	404211
gv	3347095	2.782664	1202838	2.629385	1272957	2.491743	1343275
ha	43131815	2.643041	16319013	2.953407	14604086	2.807367	15363795
hak	1878558	2.424355	774869.2	2.560305	733724.2	2.09133	898260
he	1.5E+09	4.460416	3.36E+08	2.26048	6.63E+08	3.514656	4.26E+08
hi	2.38E+08	6.332328	37558795	9.035875	26321147	3.568483	66648667
hif	2715682	2.700324	1005687	3.324587	816848	3.056977	888355.3
hr	3.14E+08	2.997707	1.05E+08	2.677169	1.17E+08	2.779891	1.13E+08
hsb	7437491	3.240744	2294995	2.492049	2984488	2.576124	2887086
ht	21993952	2.672347	8230202	2.976064	7390281	2.873894	7653015
hu	9.67E+08	3.478646	2.78E+08	2.690766	3.59E+08	2.782708	3.47E+08
hy	5.09E+08	5.823873	87346563	2.294067	2.22E+08	3.956965	1.29E+08
hyw	27450541	5.907639	4646618	2.240724	12250745	3.683785	7451722
ia	8237640	2.872221	2868039	3.820258	2156304	3.441554	2393581
id	6.02E+08	3.171613	1.9E+08	4.526109	1.33E+08	3.31139	1.82E+08
ie	3019044	2.769531	1090092	3.798603	794777.4	3.400945	887707.4
ig	34663540	2.788113	12432617	3.572043	9704121	2.880903	12032178
ilo	7352572	2.660928	2763161	3.001911	2449297	2.8403	2588660
inh	1279524	5.404367	236757.4	2.922974	437747.4	3.828947	334171.3
io	17106040	2.648579	6458572	3.071699	5568918	2.878832	5942006
is	63906249	3.209898	19909118	2.661164	24014398	2.652901	24089191
it	3.05E+09	2.906538	1.05E+09	3.41333	8.93E+08	3.293968	9.25E+08
ja	4.03E+09	4.420311	9.12E+08	3.207393	1.26E+09	3.529492	1.14E+09
jv	40261285	3.217514	12513166	3.222019	12495670	2.891515	13923939
ka	2.4E+08	8.898958	26968063	2.879661	83338866	4.77234	50287207
kaa	2913134	3.610466	806858.3	2.580794	1128774	2.572442	1132439
kab	2580584	2.713354	951067.8	2.518844	1024511	2.306864	1118654
kbd	1304580	5.821687	224089.7	2.34828	555547	3.118805	418294.8
kbp	1806400	3.173719	569174.6	2.053971	879467.2	2.129424	848304.7
kk	1.81E+08	6.057511	29839073	2.795574	64655964	4.276387	42267107
km	35567417	8.744804	4067263	2.495309	14253715	2.932378	12129205
kn	2.2E+08	9.743682	22624591	9.264455	23794905	3.269786	67419352
ko	1.09E+09	4.188618	2.59E+08	2.060378	5.27E+08	2.865548	3.79E+08
koi	1888392	4.277403	441480.9	2.810405	671928.8	3.239775	582877.5
krc	2022144	5.998891	337086.3	3.09196	654000.8	4.020692	502934.3
ks	1094458	3.276181	334065.2	3.128696	349812.8	2.783333	393218.5
ksh	2009928	2.806068	716279.2	2.734781	734950.4	2.657422	756344.9
ku	22938233	3.011483	7616923	2.604819	8806074	2.522653	9092899
kv	3690978	4.57134	807417.1	2.772082	1331482	3.439921	1072983
kw	2711398	2.640414	1026884	2.64011	1027002	2.545511	1065168
ky	63947035	5.787278	11049588	2.961551	21592413	4.034032	15851891
la	4.28E+08	3.151159	1.36E+08	3.296501	1.3E+08	3.134588	1.37E+08
lad	2754531	2.871327	959323.3	3.365023	818577.1	3.048921	903444.7
lb	50515020	2.947684	17137189	3.09739	16308900	2.929474	17243714
lez	3864848	5.783752	668225	2.998299	1289014	3.824453	1010562
lfn	5207546	2.493424	2088512	3.404739	1529499	3.007587	1731470
lg	3708097	3.12612	1186166	3.009781	1232015	2.576595	1439146
li	18919316	2.827532	6691106	2.869629	6592949	2.722327	6949687
lij	6255409	2.686963	2328060	2.764908	2262429	2.464669	2538033
lld	13866243	2.570061	5395297	3.028007	4579330	3.179828	4360690
lmo	19142356	2.505112	7641316	3.033964	6309355	2.74959	6961896
ln	1122138	2.82705	396928.9	3.03398	369856.7	2.58079	434804.1
lo	5646554	7.447939	758136.4	1.612251	3502279	2.910082	1940342
lt	2E+08	3.406909	58560670	2.625878	75978739	2.659482	75018710
lv	1.3E+08	3.446303	37645920	2.589252	50106832	2.619658	49525246
mai	6180231	7.075233	873502.1	7.641407	808781.8	3.416389	1808995
map-bms	2377123	3.182913	746838.9	3.934144	604228.8	3.132915	758757.6
mdf	1725294	4.607865	374423.7	3.031015	569213.3	3.428923	503159.2
mg	22117304	2.868733	7709783	2.830482	7813972	2.795544	7911627
mhr	6902162	4.916981	1403740	3.066632	2250730	3.901038	1769314
min	25691303	3.246645	7913186	3.440872	7466509	3.782731	6791734
mk	2.76E+08	5.178571	53362784	3.540047	78061941	4.591763	60182317
ml	2.13E+08	10.76792	19737657	8.952587	23739904	3.170749	67029451
mn	41521786	5.443966	7627121	2.750759	15094662	3.709353	11193807
mni	2208525	6.843953	322697.3	1.090132	2025925	2.764841	798789
mnw	13765461	5.687051	2420492	1.661704	8283943	3.330587	4133044
mr	95291716	7.609766	12522293	8.816335	10808540	3.347845	28463595
mrj	3283618	4.500504	729611.2	2.582179	1271646	3.173599	1034667
ms	2.1E+08	3.193519	65804920	4.299357	48879231	3.278002	64108953
mt	18686521	2.879751	6488937	2.590851	7212503	2.472526	7557665
mwl	11521563	2.844074	4051077	3.572892	3224716	2.995551	3846225
my	85497205	8.523657	10030577	1.67488	51046763	3.759618	22740930
myv	4600620	5.171224	889657.9	3.152488	1459362	3.787347	1214734
mzn	5419390	4.135287	1310523	3.992757	1357305	3.829374	1415216
nah	1191779	3.676918	324124.4	2.495853	477503.7	2.294145	519487.2
nap	3188122	2.527682	1261283	2.929708	1088205	2.642238	1206599
nds	48106879	2.942096	16351225	3.016395	15948466	2.987598	16102194
nds-nl	8354427	2.883845	2896975	2.871939	2908985	2.806673	2976630
ne	37548833	7.989947	4699509	9.557693	3928650	3.460884	10849493
new	20517810	7.302558	2809675	6.637754	3091077	3.412097	6013255
nia	1086670	2.666779	407484.2	2.791992	389209.5	2.556508	425060.2
nl	1.46E+09	2.980084	4.92E+08	3.117232	4.7E+08	3.19817	4.58E+08
nn	1.35E+08	2.921273	46127799	2.97112	45353896	3.012606	44729344
no	5.92E+08	2.99826	1.98E+08	3.018857	1.96E+08	3.069458	1.93E+08
nqo	3508645	4.800661	730867	1.01531	3455738	2.313888	1516342
nrm	1507257	2.643664	570139.5	3.056702	493099.1	2.62216	574815
nv	3304031	3.214593	1027823	2.34207	1410731	2.970995	1112096
oc	64043588	2.800003	22872683	3.548549	18047826	3.003005	21326497
olo	1724315	3.46881	497091.3	2.630054	655619.6	2.532276	680934.9
om	1982849	3.146444	630187.3	2.669185	742866.9	2.485367	797809.2
or	34054918	7.503656	4538443	8.050034	4230407	3.019334	11278949
os	5572799	3.930341	1417892	2.741069	2033075	3.398942	1639569
pa	84036831	6.039642	13914207	7.968192	10546536	3.347806	25102062
pam	4277038	2.979156	1435654	3.518991	1215416	3.234084	1322488
pap	2435005	2.619218	929668.8	3.336065	729903.4	2.959935	822654.9
pcd	3145572	2.547703	1234670	3.264388	963602.5	2.748718	1144378
pcm	1160762	2.454551	472902	3.379737	343447.4	3.187808	364125.4
pfl	1971214	3.069015	642295.3	2.853351	690841.8	2.743806	718423.1
pl	1.8E+09	3.372125	5.34E+08	2.596431	6.93E+08	2.862203	6.29E+08
pms	21712699	2.483721	8742006	2.994908	7249872	2.768085	7843943
pnb	1.33E+08	3.850482	34610271	4.844648	27507933	3.869398	34441083
ps	53312545	3.863741	13798167	3.120696	17083543	3.5782	14899263
pt	1.64E+09	2.850676	5.76E+08	4.279796	3.84E+08	3.262223	5.03E+08
qu	7688106	3.070997	2503456	2.76082	2784719	2.523484	3046623
rm	10483970	2.82811	3707058	3.396868	3086364	3.049769	3437627
ro	5.48E+08	2.959287	1.85E+08	2.898551	1.89E+08	2.946163	1.86E+08
roa-tara	4003095	2.822698	1418180	2.927105	1367595	2.66504	1502077
ru	9.48E+09	5.401585	1.75E+09	3.537656	2.68E+09	4.602799	2.06E+09
rue	6346106	5.189003	1222991	3.239176	1959173	4.218571	1504326
rw	6623388	3.115312	2126076	3.386084	1956062	2.580994	2566216
sa	2.56E+08	9.344079	27376241	6.500521	39351579	3.328375	76856060
sah	29330820	6.091066	4815383	2.686451	10918055	3.771827	7776289
sat	15428584	6.501258	2373169	1.095737	14080552	3.694191	4176444
sc	7711996	2.789137	2765011	3.222008	2393537	2.871126	2686053
scn	10223816	2.814459	3632604	2.91792	3503802	2.710677	3771684
sco	24287944	2.814675	8629040	3.556268	6829617	3.322475	7310197
sd	17591997	3.820602	4604510	3.189349	5515859	3.323255	5293605
se	1816006	3.639336	498993.8	2.519246	720853	2.374435	764816.1
sh	2.66E+08	3.109088	85677639	2.810773	94770818	3.086501	86304614
shi	1359828	2.603696	522268.4	2.388864	569236.3	2.228096	610309.3
shn	8163231	6.818837	1197159	1.563793	5220148	3.548365	2300561
si	54229127	6.861562	7903321	1.605319	33780901	3.228928	16794776
simple	1.57E+08	2.604897	60227038	4.105208	38216143	3.746763	41872201
sk	2.42E+08	3.185025	75896153	2.660064	90874192	2.743562	88108506
skr	9978607	3.390208	2943361	4.619296	2160201	3.69874	2697840
sl	5.1E+08	2.776806	1.84E+08	2.65154	1.92E+08	2.679793	1.9E+08
smn	2813872	3.460772	813076.5	2.543944	1106106	2.433844	1156143
sn	4979456	3.124034	1593919	3.120518	1595715	2.655855	1874898
so	7940363	2.992185	2653701	2.708393	2931762	2.508277	3165664
sq	1.17E+08	2.979585	39248921	2.73659	42734026	2.815558	41535465
sr	7.84E+08	4.795495	1.64E+08	3.16455	2.48E+08	4.331577	1.81E+08
stq	2884429	2.993801	963467.2	2.787199	1034884	2.690831	1071947
su	19806020	3.211327	6167549	3.37651	5865827	3.11505	6358171
sv	9.91E+08	3.20815	3.09E+08	2.957759	3.35E+08	3.228023	3.07E+08
sw	35936177	2.91445	12330346	3.787084	9489141	2.725329	13185994
szl	7347967	3.358912	2187603	2.544334	2887973	2.958125	2483995
szy	6192815	2.928042	2115002	3.039297	2037581	2.753526	2249049
ta	3.06E+08	10.13499	30165058	10.02704	30489830	3.312023	92306924
tay	1232811	2.576051	478566.2	2.76919	445188.3	2.525171	488209
tcy	4611006	8.512955	541645.8	6.638766	694557.7	3.144578	1466335
te	2.52E+08	8.549729	29447150	9.063892	27776717	3.237454	77766408
tg	49825647	5.281643	9433740	2.829083	17611941	3.940163	12645580
th	3.96E+08	8.271261	47823543	2.862464	1.38E+08	4.089545	96724945
tk	7383654	3.667693	2013160	2.523091	2926431	2.4631	2997708
tl	45797527	2.775242	16502176	3.24525	14112173	3.02519	15138727
tly	1070456	3.351578	319388.7	2.374073	450894.2	2.408261	444493.3
tn	1708110	2.639858	647046.2	3.149103	542411.6	2.677334	637989.2
tr	5.87E+08	3.482104	1.69E+08	2.764463	2.12E+08	2.937163	2E+08
trv	2706664	2.665858	1015307	2.623241	1031802	2.404338	1125742
tt	1.29E+08	4.933108	26178434	2.920663	44216357	4.736173	27266963
tum	5459856	3.069905	1778510	3.047064	1791842	2.854557	1912681
tw	4118530	2.631451	1565118	2.860678	1439704	2.522578	1632667
tyv	6528290	5.510621	1184674	2.975143	2194278	3.877014	1683845
udm	2982821	5.038611	591992.7	2.958814	1008114	3.557863	838374.3
ug	17741860	6.217129	2853706	2.574344	6891799	3.036012	5843805
uk	2.3E+09	5.424781	4.24E+08	3.209132	7.16E+08	4.452019	5.16E+08
ur	1.68E+08	4.031675	41577728	5.905548	28384813	4.152323	40369661
uz	2.1E+08	3.270747	64285816	2.768941	75936117	2.790661	75345110
vec	19276543	2.667171	7227336	3.076717	6265297	3.048724	6322824
vep	6423002	3.340101	1922996	2.645025	2428333	2.570651	2498590
vi	7.47E+08	2.828246	2.64E+08	4.607117	1.62E+08	3.510764	2.13E+08
vls	6985406	2.779891	2512834	2.846496	2454037	2.700319	2586882
vo	6582571	2.806364	2345587	2.703518	2434817	2.661834	2472946
wa	9550310	2.458642	3884384	2.794199	3417906	2.460766	3881031
war	1.05E+08	3.290553	31784461	3.186653	32820783	3.690181	28342365
wo	2094574	2.400075	872712	2.73538	765734.2	2.446062	856304.7
wuu	15985963	3.970011	4026680	4.025088	3971581	3.612661	4424983
xh	2505472	5.03599	497513.3	2.826526	886413.9	2.368852	1057674
xmf	12948576	8.046115	1609295	2.743687	4719408	4.326371	2992942
yi	26905047	4.673416	5757040	2.147134	12530677	3.193248	8425605
yo	8861465	2.830447	3130765	3.656377	2423564	2.580695	3433751
zea	2620396	2.786056	940539.5	2.82913	926219.7	2.65124	988366.2
zh	5.03E+09	3.687222	1.36E+09	3.552345	1.42E+09	3.42038	1.47E+09
zh-classical	10098073	3.569372	2829090	3.35671	3008324	3.177208	3178285
zh-min-nan	47410572	2.36058	20084289	2.552651	18573074	2.519768	18815450
zh-yue	64950815	3.802575	17080745	3.706028	17525721	3.302348	19668068
zu	3792429	3.822076	992243.3	3.093608	1225892	2.477525	1530733

test

from transformers import AutoModelForMaskedLM, AutoTokenizer
model_path = "JuanbaiAI/GlobalTokenizer"

tokenizer = AutoTokenizer.from_pretrained(model_path)
print(tokenizer("Hello, 世界"))

doc = [
    "                                                          ",
    "ننصحك أن تستخدم كلمة سر فريدة لا تستخدمها على أي موقع شبكي آخر.",
    "แก๊สมีสกุล หรือ แก๊สมีตระกูล (ในอดีตเรียกว่า แก๊สเฉื่อย หรือบางครั้งใช้ชื่อว่า aerogens) เป็นกลุ่มของธาตุทางเคมีที่มีสมบัติคล้ายกัน ภายใต้ภาวะมาตรฐานสำหรับอุณหภูมิและความดันธาตุเหล่านี้ต่างไม่มีกลิ่น ไม่มีสี เป็นแก๊สอะตอมเดี่ยวซึ่งไม่มีความว่องไวต่อปฏิกริยาเคมี แก๊สมีสกุลที่เกิดในธรรมชาติทั้งหกธาตุ ได้แก่ ฮีเลียม (He), นีออน (Ne), อาร์กอน (Ar), คริปทอน (Kr), ซีนอน (Xe) (ในภาพตามลำดับ) และเรดอน (Rn)  โอกาเนสซอน (Og) เป็นธาตุสังเคราะห์มีความเป็นกัมมันตรังสีสูงมาก แม้ว่า IUPAC จัดโอกาเนสซอนเป็นแก๊สมีสกุลหรือหมู่ที่ 18 มันอาจไม่เฉื่อยทางเคมีเหมือนธาตุอื่นในหมู่เดียวกัน และถูกทำนายว่าจะสลายและแผ่กัมมันตรังสีเนื่องจากปรากฏการณ์สัมพัทธ์ เนื่องจากครึ่งชีวิตที่สั้นเพียง 0.7 ไมโครวินาทีของไอโซโทปตัวเดียว ทำให้คุณสมบัติทางเคมีของโอกาเนสซอนยังไม่มีการศึกษามากนัก     首先8.88设置 st。art_new_word=True 和 output=[açaí]，output 就是最终� no such name",
    "ᠠᠩᠬ᠎ᠠ ᠳᠤᠮᠳᠠᠳᠤ ᠶᠢᠨ ᠮᠣᠩᠭᠤᠯ ᠬᠡᠯᠡᠨ ᠤ ᠬᠢᠴᠢᠶᠡᠯ ᠤᠨ ᠪᠡᠯᠡᠳᠭᠡᠮᠵᠢ ᠮᠢᠨᠣ ᠳᠤᠰᠬᠠᠢ ᠰᠡᠳᠤᠪ ᠤᠨ ᠭᠤᠣᠯ ᠠᠭᠤᠯᠭ᠎ᠠ ᠨᠢ ᠠᠩᠬ᠎ᠠ ᠳᠤᠮᠳᠠᠳᠤ ᠶᠢᠨ ᠮᠣᠩᠭᠤᠯ ᠬᠡᠯᠡᠨ ᠤ ᠬᠢᠴᠢᠶᠡᠯ ᠳᠤ ᠳᠠᠭᠠᠯᠳᠤᠭᠤᠯᠤᠨ ᠵᠣᠬᠢᠶᠠᠭᠰᠠᠨ ᠺᠣᠣᠷᠰᠸᠠᠢᠷ ᠪᠣᠯᠤᠨ᠎ᠠ᠃ ᠲᠤᠰ ᠺᠣᠣᠷᠰᠸᠠᠢᠷ ᠲᠤ ᠪᠠᠨ ᠰᠤᠷᠤᠭᠴᠢ ᠶᠢᠨ ᠰᠤᠷᠬᠤ ᠤᠷᠮ᠎ᠠ ᠪᠠᠬ᠎ᠠ ᠶᠢ ᠳᠠᠳᠠᠬᠤ᠂ ᠬᠢᠴᠢᠶᠡᠯ ᠤᠨ ᠭᠤᠣᠯᠳᠠᠯᠭ᠎ᠠ ᠬᠦᠴᠢᠷᠳᠡᠯᠭᠡ ᠶᠢ ᠳᠠᠭᠤᠯᠵᠤ ᠬᠢᠴᠢᠶᠡᠯᠯᠡᠭᠡ ᠶᠢᠨ ᠶᠠᠪᠤᠴᠠ ᠪᠠᠨ ᠪᠦᠷᠢᠨᠵᠢᠬᠦᠯᠵᠤ᠂ ᠮᠡᠳᠡᠯᠭᠡ ᠶᠢᠨ ᠥᠷᠭᠡᠳᠭᠡᠯ ᠢᠶᠠᠷ ᠳᠠᠮᠵᠢᠨ ᠵᠢᠭᠠᠨ ᠰᠤᠷᠭᠠᠬᠤ ᠴᠢᠨᠠᠷ ᠴᠢᠨᠰᠠᠭ᠎ᠠ ᠪᠠᠨ ᠳᠡᠭᠡᠭᠰᠢᠯᠡᠭᠦᠯᠬᠦ ᠬᠡᠰᠡᠨ ᠵᠣᠷᠢᠯᠭ᠎ᠠ ᠲᠠᠢ᠃ ᠲᠤᠰ ᠬᠥᠮᠦᠨ ᠤ ᠬᠤᠪᠢ ᠶᠢᠨ ᠬᠡᠪ ᠴᠢᠨᠠᠷ᠂ ᠮᠡᠳᠡᠯᠭᠡ ᠴᠢᠳᠠᠪᠤᠷᠢ ᠶᠢᠨ ᠮᠡᠬᠦᠰ ᠠᠴᠠ ᠪᠣᠯᠵᠤ ᠪᠡᠯᠡᠳᠭᠡᠮᠵᠢ ᠶᠢᠨ ᠳᠤᠮᠳᠠ ᠠᠰᠠᠭᠤᠳᠠᠯ ᠣᠷᠤᠰᠢᠵᠤ ᠪᠠᠢᠭ᠎ᠠ ᠨᠢ ᠯᠠᠪᠳᠠᠢ᠃ ᠠᠩᠬᠠᠷᠤᠯ ᠶᠢᠨ ᠬᠠᠨᠳᠤᠭᠤᠯᠤᠭᠰᠠᠨ ᠲᠠ ᠪᠦᠬᠦᠨ ᠢᠯᠡᠭᠦᠬᠡᠨ ᠰᠠᠨᠠᠯ ᠵᠥᠪᠯᠡᠯᠭᠡ ᠥᠭᠬᠦ ᠶᠢ ᠬᠦᠰᠡᠵᠤ ᠪᠠᠢᠨ᠎ᠠ᠃    的输出คุณจะจัดพิธีแต่งงานเมื่อไรคะ탑승 수속해야pneumonoultramicroscopicsilicovolcanoconiosis",
    "四色定理是一个著名的数学定理：如果在平面上划出一些邻接的有限区域，那么可以用四种颜色来给这些区域染色，使得每两个邻接区域染的颜色都不一样。被称为邻接的两个区域是指它们有一段公共的边界，而不仅仅是一个公共的交点。四色问题最早是由英国数学家法兰西斯·古德里在1852年提出的。人们发现，要证明宽松一点的“五色定理”很容易，但四色问题却出人意料地异常困难。1976年，数学家凯尼斯·阿佩尔和沃夫冈·哈肯借助电子计算机首次得到一个完全的证明，四色问题也终于成为四色定理。这是首个主要借助计算机证明的定理。",
    """est 𗴂𗹭𘜶𗴲𗂧, ou "phiow-bjij-lhjij-lhjij", ce que l'on peut traduire par « pays-grand-blanc-élevé » (白高大夏國). """,
    "蜂蜜とはミツバチが花の蜜を採集し、巣の中で加工、貯蔵したものをいう。約8割の糖分と約2割の水分によって構成され、ビタミンとミネラル類などの栄養素をわずかに含む。味や色は蜜源植物によって様々である。  本来はミツバチの食料であるが、しばしば他の生物が採集して食料としている。人類も「蜂蜜の歴史は人類の歴史」ということわざがあるように、古来、食用、薬用など様々な用途に用いている。人類は初め、野生のミツバチの巣から蜂蜜を採集していたが、やがてミツバチを飼育して採集すること（養蜂）を身に付けた。人類による蜂蜜の生産量は、世界全体で年間約120万tと推定される。",
    "[2]: Different from the embedding model, reranker uses question and document as input and directly output similarity instead of embedding. To balance the accuracy and time cost, cross-encoder is widely used to re-rank top-k documents retrieved by other simple models. For example, use bge embedding model to retrieve top 100 relevant documents, and then use bge reranker to re-rank the top 100 documents to get the final top-3 results.    ",
    "오스트레일리아청개구리(Australian green tree frog)는 오스트레일리아와 뉴기니 원산의 청개구리과에 속한 청개구리의 일종이다. 미국과 뉴질랜드에도 외래종으로 흘러들어갔는데, 뉴질랜드에서는 멸종된 것으로 여겨진다. 학명 명명자 존 화이트의 이름을 따 화이트청개구리(White's tree frog), 특유의 생김새에서 유래한 시무룩청개구리(dumpy tree frog) 등의 별명이 있으며, 학명은 리토리아 카에룰레아(Litoria caerulea)이다. 형태학적으로 오스트레일리아청개구리속의 다른 청개구리, 특히 예쁜청개구리와 왕청개구리와 많이 닮았다. ",
    "Элой была придумана студией Guerrilla Games, которая хотела создать сильного и героического женского персонажа в рамках своей совершенно новой игры. Лицо героини было создано на основе внешности нидерландской актрисы Ханны Хукстры; роль персонажа при озвучивании на английском языке и в сценах захвата движения исполняет американская актриса Эшли Бёрч. ",
    "網上瘋傳一張照片，指在港鐵的機場快綫列車有床蝨蹤影。. 通訊事務管理局公布，香港寬頻 (01310.HK)於1至4月期間發生三宗涉及固定和流動電訊服務用戶的錯誤收帳事件。. 三宗事件共影響超過1萬名客戶",
    "하는데 카운터가 어디에 있어요ꆃꎭꆈꌠꊨꏦꏲꅉꆅꉚꅉꋍꂷꂶꌠلأحياء تمارين تتطلب من [MASK] [PAD] [CLS][SEP]" "def normalize(text, maxlen=0, isolate_digits=False):     text = unicodedata.normalize('NFC', text)     if maxlen > 0:         if isolate_digits:             regex = '\d|[^\n\d]{,%d}\n{1,100}|[^\n\d]{1,%d}' % (maxlen, maxlen)         else:             regex = '.{,%d}\n{1,100}|.{1,%d}' % (maxlen, maxlen)     else:         if isolate_digits:             regex = '\d|[^\n\d]*\n+|[^\n\d]+'         else:             regex = '.*\n+|.+'     return [t.encode() for t in re.findall(regex, text)]",
    "ꃛꏢꄧꍏꇩꇭꍣꄈ꒜ꀉꒉꊰꏁꃢꌠꇩꏤꑭꊂꁧꎁ꒜ꈿꅉꇬꄉꅇꄜꌠ",
]
for i,line in enumerate(doc):
    print(i,len(tokenizer.tokenize(line)))