Package: dl-translate
DL TranslateA deep learning-based translation library built on Huggingfacetransformers💻GitHub Repository📚Documentation🐍PyPi project🧪Colab Demo/Kaggle DemoQuickstartInstall the library with pip:pip install dl-translateTo translate some text:importdl_translateasdltmt=dlt.TranslationModel()# Slow when you load it for the first timetext_hi="संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"mt.translate(text_hi,source=dlt.lang.HINDI,target=dlt.lang.ENGLISH)Above, you can see thatdlt.langcontains variables representing each of the 50 available languages with auto-complete support. Alternatively, you can specify the language (e.g. "Arabic") or the language code (e.g. "fr" for French):text_ar="الأمين العام للأمم المتحدة يقول إنه لا يوجد حل عسكري في سوريا."mt.translate(text_ar,source="Arabic",target="fr")If you want to verify whether a language is available, you can check it:print(mt.available_languages())# All languages that you can useprint(mt.available_codes())# Code corresponding to each language acceptedprint(mt.get_lang_code_map())# Dictionary of lang -> codeUsageSelecting a deviceWhen you load the model, you can specify the device:mt=dlt.TranslationModel(device="auto")By default, the value will bedevice="auto", which means it will use a GPU if possible. You can also explicitly setdevice="cpu"ordevice="gpu", or some other strings accepted bytorch.device().In general, it is recommend to use a GPU if you want a reasonable processing time.Choosing a different modelBy default, them2m100model will be used. However, there are a few options:mBART-50 Large:  Allows translations across 50 languages.m2m100: Allows translations across 100 languages.nllb-200(New in v0.3): Allows translations across 200 languages, and is faster than m2m100 (On RTX A6000, we can see speed up of 3x).Here's an example:# The default approvalmt=dlt.TranslationModel("m2m100")# Shorthandmt=dlt.TranslationModel("facebook/m2m100_418M")# Huggingface repo# If you want to use mBART-50 Largemt=dlt.TranslationModel("mbart50")mt=dlt.TranslationModel("facebook/mbart-large-50-many-to-many-mmt")# Or NLLB-200 (faster and has 200 languages)mt=dlt.TranslationModel("nllb200")mt=dlt.TranslationModel("facebook/nllb-200-distilled-600M")Note that the language code will change depending on the model family. To find out the correct language codes, please read the doc page on available languages or runmt.available_codes().By default,dlt.TranslationModelwill download the model from the huggingface repo formbart50,m2m100, ornllb200and cache it. It's possible to load the model from a path or a model with a similar format, but you will need to specify themodel_family:mt=dlt.TranslationModel("/path/to/model/directory/",model_family="mbart50")mt=dlt.TranslationModel("facebook/m2m100_1.2B",model_family="m2m100")mt=dlt.TranslationModel("facebook/nllb-200-distilled-600M",model_family="nllb200")Notes:Make sure your tokenizer is also stored in the same directory if you load from a file.The available languages will change if you select a different model, so you will not be able to leveragedlt.langordlt.utils.Splitting into sentencesIt is not recommended to use extremely long texts as it takes more time to process. Instead, you can try to break them down into sentences with the help ofnltk. First install the library withpip install nltk, then run:importnltknltk.download("punkt")text="Mr. Smith went to his favorite cafe. There, he met his friend Dr. Doe."sents=nltk.tokenize.sent_tokenize(text,"english")# don't use dlt.lang.ENGLISH" ".join(mt.translate(sents,source=dlt.lang.ENGLISH,target=dlt.lang.FRENCH))Batch size during translationIt's possible to set a batch size (i.e. the number of elements processed at once) formt.translateand whether you want to see the progress bar or not:# ...mt=dlt.TranslationModel()mt.translate(text,source,target,batch_size=32,verbose=True)If you setbatch_size=None, it will compute the entiretextat once rather than splitting into "chunks". We recommend loweringbatch_sizeif you do not have a lot of RAM or VRAM and run into CUDA memory error. Set a higher value if you are using a high-end GPU and the VRAM is not fully utilized.dlt.utilsmoduleAn alternative tomt.available_languages()is thedlt.utilsmodule. You can use it to find out which languages and codes are available:print(dlt.utils.available_languages('mbart50'))# All languages that you can useprint(dlt.utils.available_codes('m2m100'))# Code corresponding to each language acceptedprint(dlt.utils.get_lang_code_map('nllb200'))# Dictionary of lang -> codeOffline usageUnlike the Google translate or MSFT Translator APIs, this library can be fully used offline. However, you will need to first download the packages and models, and move them to your offline environment to be installed and loaded inside a venv.First, run in your terminal:mkdirdltcddlt
mkdirlibraries
pipdownload-dlibraries/dl-translateOnce all the required packages are downloaded, you will need to use huggingface hub to download the files. Install it withpip install huggingface-hub. Then, run inside Python:importshutilimporthuggingface_hubashubdirname=hub.snapshot_download("facebook/m2m100_418M")shutil.copytree(dirname,"cached_model_m2m100")# Copy to a permanent folderNow, move everything in thedltdirectory to your offline environment. Create a virtual environment and run the following in terminal:pipinstall--no-index--find-linkslibraries/dl-translateNow, run inside Python:importdl_translateasdltmt=dlt.TranslationModel("cached_model_m2m100",model_family="m2m100")AdvancedIf you have knowledge of PyTorch and Huggingface Transformers, you can access advanced aspects of the library for more customization:Saving and loading: If you wish to accelerate the loading time the translation model, you can usesave_objand reload it later withload_obj. This method is only recommended if you are familiar withhuggingfaceandtorch; please read the docs for more information.Interacting with underlying model and tokenizer: When initializingmodel, you can pass in arguments for the underlying BART model and tokenizer withmodel_optionsandtokenizer_optionsrespectively. You can also access the underlyingtransformerswithmt.get_transformers_model().Keyword arguments for thegenerate()method: When runningmt.translate, you can also givegeneration_optionsthat is passed to thegenerate()method of the underlying transformer model.For more information, please visit theadvanced section of the user guide.Acknowledgementdl-translateis built on top of Huggingface's implementation of two models created by Facebook AI Research.The multilingual BART finetuned on many-to-many translation of over 50 languages, which isdocumented hereThe original paper was written by Tang et. al from Facebook AI Research; you canfind it hereand cite it using the following:@article{tang2020multilingual,
    title={Multilingual translation with extensible multilingual pretraining and finetuning},
    author={Tang, Yuqing and Tran, Chau and Li, Xian and Chen, Peng-Jen and Goyal, Naman and Chaudhary, Vishrav and Gu, Jiatao and Fan, Angela},
    journal={arXiv preprint arXiv:2008.00401},
    year={2020}
}The transformer model published inBeyond English-Centric Multilingual Machine Translationby Fan et. al, which supports over 100 languages. You can cite it here:@misc{fan2020englishcentric,
     title={Beyond English-Centric Multilingual Machine Translation}, 
     author={Angela Fan and Shruti Bhosale and Holger Schwenk and Zhiyi Ma and Ahmed El-Kishky and Siddharth Goyal and Mandeep Baines and Onur Celebi and Guillaume Wenzek and Vishrav Chaudhary and Naman Goyal and Tom Birch and Vitaliy Liptchinsky and Sergey Edunov and Edouard Grave and Michael Auli and Armand Joulin},
     year={2020},
     eprint={2010.11125},
     archivePrefix={arXiv},
     primaryClass={cs.CL}
 }Theno language left behindmodel, which extends NMT to 200+ languages. You can cite it here:@misc{nllbteam2022language,
    title={No Language Left Behind: Scaling Human-Centered Machine Translation}, 
    author={NLLB Team and Marta R. Costa-jussà and James Cross and Onur Çelebi and Maha Elbayad and Kenneth Heafield and Kevin Heffernan and Elahe Kalbassi and Janice Lam and Daniel Licht and Jean Maillard and Anna Sun and Skyler Wang and Guillaume Wenzek and Al Youngblood and Bapi Akula and Loic Barrault and Gabriel Mejia Gonzalez and Prangthip Hansanti and John Hoffman and Semarley Jarrett and Kaushik Ram Sadagopan and Dirk Rowe and Shannon Spruit and Chau Tran and Pierre Andrews and Necip Fazil Ayan and Shruti Bhosale and Sergey Edunov and Angela Fan and Cynthia Gao and Vedanuj Goswami and Francisco Guzmán and Philipp Koehn and Alexandre Mourachko and Christophe Ropers and Safiyyah Saleem and Holger Schwenk and Jeff Wang},
    year={2022},
    eprint={2207.04672},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}dltis a wrapper with usefulutilsto save you time. For huggingface'stransformers, the following snippet is shown as an example:fromtransformersimportMBartForConditionalGeneration,MBart50TokenizerFastarticle_hi="संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"article_ar="الأمين العام للأمم المتحدة يقول إنه لا يوجد حل عسكري في سوريا."model=MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")tokenizer=MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")# translate Hindi to Frenchtokenizer.src_lang="hi_IN"encoded_hi=tokenizer(article_hi,return_tensors="pt")generated_tokens=model.generate(**encoded_hi,forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"])tokenizer.batch_decode(generated_tokens,skip_special_tokens=True)# => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire en Syria."# translate Arabic to Englishtokenizer.src_lang="ar_AR"encoded_ar=tokenizer(article_ar,return_tensors="pt")generated_tokens=model.generate(**encoded_ar,forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])tokenizer.batch_decode(generated_tokens,skip_special_tokens=True)# => "The Secretary-General of the United Nations says there is no military solution in Syria."Withdlt, you can run:importdl_translateasdltarticle_hi="संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"article_ar="الأمين العام للأمم المتحدة يقول إنه لا يوجد حل عسكري في سوريا."mt=dlt.TranslationModel()translated_fr=mt.translate(article_hi,source=dlt.lang.HINDI,target=dlt.lang.FRENCH)translated_en=mt.translate(article_ar,source=dlt.lang.ARABIC,target=dlt.lang.ENGLISH)Notice you don't have to think about tokenizers, condition generation, pretrained models, and regional codes; you can just tell the model what to translate!If you are experienced withhuggingface's ecosystem, then you should be familiar enough with the example above that you wouldn't need this library. However, if you've never heard of huggingface or mBART, then I hope using this library will give you enough motivation tolearn more about them:)