m3_tokenizer / persianToken
Aleereza's picture
Create persianToken
d2755b5
raw
history blame contribute delete
662 Bytes
[35953, 35954, 35955, 35956, 35957, 35958, 35959, 35960, 35961, 35962, 35963, 35964, 35965, 35966, 35967, 35968, 35969, 35970, 35971, 35972, 35973, 35974, 35975, 35976, 35977, 35978, 35981, 35986, 35987, 35990, 35991, 35995, 35997]
[', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ', ']
[30112, 30202, 30156, 30172, 30162, 30204, 30171, 30159, 30195, 30177, 30198, 30343, 30256, 30295, 30138, 30241, 30421, 30218, 30321, 30265, 30270, 30726, 30240, 30585, 30367, 30360, 30905, 30611, 30624, 30839, 30912, 30818, 30372]