creating transformer for tamil language
Author: AswiN-7
Pull request: no

hi im trying to create a new transformer for tamil language
dataset is downloaded from oscar-corpus 

       from tokenizers import ByteLevelBPETokenizer
       tokenizer = ByteLevelBPETokenizer()
       tokenizer.train(files = path, vocab_size = 200000, min_frequency = 2, special_tokens=['
', '<pad>', '
', '<unk>', '<mask>'])
       
       import os
       os.mkdir('./tamberto')
       tokenizer.save_model('tamberto')

after executing this code tokenizer creates the two files 
`megres.txt`  `vocab.json`
but the contents of the file are showing like junks

     {"
":0,"<pad>":1,"
":2,"<unk>":3,"<mask>":4,"!":5,"\"":6,"#":7,"$":8,"%":9,"&":10,"'":11,"(":12,")":13,"*":14,"+":15,",":16,"-":17,".":18,"/":19,"0":20,"1":21,"2":22,"3":23,"4":24,"5":25,"6":26,"7":27,"8":28,"9":29,":":30,";":31,"<":32,"=":33,">":34,"?":35,"@":36,"A":37,"B":38,"C":39,"D":40,"E":41,"F":42,"G":43,"H":44,"I":45,"J":46,"K":47,"L":48,"M":49,"N":50,"O":51,"P":52,"Q":53,"R":54,"S":55,"T":56,"U":57,"V":58,"W":59,"X":60,"Y":61,"Z":62,"[":63,"\\":64,"]":65,"^":66,"_":67,"`":68,"a":69,"b":70,"c":71,"d":72,"e":73,"f":74,"g":75,"h":76,"i":77,"j":78,"k":79,"l":80,"m":81,"n":82,"o":83,"p":84,"q":85,"r":86,"s":87,"t":88,"u":89,"v":90,"w":91,"x":92,"y":93,"z":94,"{":95,"|":96,"}":97,"~":98,"¡":99,"¢":100,"£":101,"¤":102,"¥":103,"¦":104,"§":105,"¨":106,"©":107,"ª":108,"«":109,"¬":110,"®":111,"¯":112,"°":113,"±":114,"²":115,"³":116,"´":117,"µ":118,"¶":119,"·":120,"¸":121,"¹":122,"º":123,"»":124,"¼":125,"½":126,"¾":127,"¿":128,"À":129,"Á":130,"Â":131,"Ã":132,"Ä":133,"Å":134,"Æ":135,"Ç":136,"È":137,"É":138,"Ê":139,"Ë":140,"Ì":141,"Í":142,"Î":143,"Ï":144,"Ð":145,"Ñ":146,"Ò":147,"Ó":148,"Ô":149,"Õ":150,"Ö":151,"×":152,"Ø":153,"Ù":154,"Ú":155,"Û":156,"Ü":157,"Ý":158,"Þ":159,"ß":160,"à":161,"á":162,"â":163,"ã":164,"ä":165,"å":166,"æ":167,"ç":168,"è":169,"é":170,"ê":171,"ë":172,"ì":173,"í":174,"î":175,"ï":176,"ð":177,"ñ":178,"ò":179,"ó":180,"ô":181,"õ":182,"ö":183,"÷":184,"ø":185,"ù":186,"ú":187,"û":188,"ü":189,"ý":190,"þ":191,"ÿ":192,"Ā":193,"ā":194,"Ă":195,"ă":196,"Ą":197,"ą":198,"Ć":199,"ć":200,"Ĉ":201,"ĉ":202,"Ċ":203,"ċ":204,"Č":205,"č":206,"Ď":207,"ď":208,"Đ":209,"đ":210,"Ē":211,"ē":212,"Ĕ":213,"ĕ":214,"Ė":215,"ė":216,"Ę":217,"ę":218,"Ě":219,"ě":220,"Ĝ":221,"ĝ":222,"Ğ":223,"ğ":224,"Ġ":225,"ġ":226,"Ģ":227,"ģ":228,"Ĥ":229,"ĥ":230,"Ħ":231,"ħ":232,"Ĩ":233,"ĩ":234,"Ī":235,"ī":236,"Ĭ":237,"ĭ":238,"Į":239,"į":240,"İ":241,"ı":242,"Ĳ":243,"ĳ":244,"Ĵ":245,"ĵ":246,"Ķ":247,"ķ":248,"ĸ":249,"Ĺ":250,"ĺ":251,"Ļ":252,"ļ":253,"Ľ":254,"ľ":255,"Ŀ":256,"ŀ":257,"Ł":258,"ł":259,"Ń":260,"à®":261,"à¯":262,"à¯į":263,"Ġà®":264,"à¯ģ":265,"à®¿":266,"à®ķ":267,"à®¤":268,"à®°":269,"à®¾":270,"à®Ł":271,"à®©":272,"à®®":273,"à®²":274,"à®ª":275,"à¯Ī":276,"à®¯":277,"à®³":278,"à®±":279,"à®µ":280,"Ġà®ª":281,"Ġà®ķ":282,"Ġà®µ":283,"à®ķà®³":284,"Ġà®®":285,"à®ļ":286,"Ġà®ħ":287,"à¯ĩ":288,"à®¨":289,"à®£":290,"Ġà®¤":291,"Ġà®ļ":292,"Ġà®ĩ":293,"à¯Ĩ":294,"Ġà®¨":295,"à¯ĭ":296,"Ġà®İ":297,"à¯į.":298,"à¯Ĭ":299,"à®Ļ":300,"à®"
     µà®°":307,"à¯į,":308,"Ġà®Ĩ":309,"à¯ģ.":310,"Ġà®ĩà®°":311,"Ġà®Ĵ":312,"Ġà®ħà®¤":313,"à®¸":314,"à®ªà®¤":315,"âĢ":316,"à®µà®¤":317,"à®Łà®¤":318,"Ġà®µà®°":319,"Ġà®Ĵà®°":320,"à®Łà®®":321,"à®±à®¤":322,"Ġà®ĩà®¨":323,"à®©à®°":324,"à®ķà®ª":325,"à®ķà®®":326,"Ġà®ħà®µà®°":327,"Ġà®ĩà®¤":328,"Ġà®°":329,"à®¤à®¤":330,"Ġà®ķà®Ł":331,"à®¤à®²":332,"ÂŃ":333,"à®°à®®":334,"à®ķà®¤":335,"à®ķà®µ":336,"..":337,"Ġà®ı":338,"à®Łà®©":339,"à¯ģ,":340,"à®µà®©":341,"Ġà®¨à®Ł":342,"Ġà®ħà®°":343,"à®¯à®¤":344,"à®ľ":345,"à®ķà®°":346,"à®¯à®²":347,"à®³à®¤":348,"à®¯à®®":349,"Ġà®ħà®®":350,"à®¤à®®":351,"à®·":352,"Ġà®ªà®Ł":353,"Ġà®®à®±":354,"à®Łà®°":355,"Ġà®īà®³":356,"à®¯à®°":357,"Ġà®ĩà®²":358,"à®©à®¤":359,"Ġ1":360,"ĠâĢ":361,"Ġà®ħà®©":362,"Ġà®ħà®Ł":363,"Ġà®¤à®®":364,"Ġà®Ł":365,"à®¤à®©":366,"à®ŀ":367,"Ġà®ªà®²":368,"à®³à®°":369,"Ġà®ĩà®©":370,"à®°à®¤":371,"Ġà®ľ":372,"Ġ2":373,"à¯Ī.":374,"Ġà®ħà®±":375,"Ġà®ķà®£":376,"Ġà®µà®

How to change this encoding property??


Comments:
cc @SaulLu 

Hi @AswiN-7,

Could you tell me more about what you mean when you say "How to change this encoding property??". :blush: 

I would like to comment on your impression that the tokens produced by BPE are showing like junks. This type of output is indeed surprising but well expected by a byte-level BPE tokenizer! 

I'll link you to an [old comment](https://github.com/huggingface/tokenizers/issues/203#issuecomment-605105611) that details why the vocabulary tokens look like this. Don't hesitate if you need more info!

what i think about this problem is
maybe the model saves the text as ascii ensure =true this will make clash with utf-8 encoding where Tamil language format is utf-8 encoded
is there anyway to change this tokenizer.save_model function? 

> maybe the model saves the text as ascii ensure =true this will make clash with utf-8 encoding where Tamil language format is utf-8 encoded

Why do you think there would be this problem? :slightly_smiling_face: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.