Package: langdist
langdistis a Python project for experimentingCharacter-level Multilingual Language Modeling, which is to see how learning a character-level language model in one language helps learning another character-level language model in a different language. The project is stillunder developmentand can offer limited functionality.FeaturesDownload and preprocess multilingual parallel corpora ([Multilingual Bible Parallel Corpus](http://christos-c.com/bible/))Train amonolingual language model- This is a language model trained in one languageTrain abilingual language model- This is a language model that is trained on top of another language model (the parameters are initialized using another language model’s parameters)Generate texts using a trained language modelInstallationThis repository can run on Ubuntu 14.04 LTS & Mac OSX 10.x (not tested on other OSs)Tested only on Python 3.5langdistdepends on [NumPy and Scipy](https://www.scipy.org/install.html), Python packages for scientific computing. You might need to have them installed prior to installinglangdist.You can installlangdistby:`pip install langdist`This installslangdistpackage to your Python, as well aslangdistcommand and add it to yourPATH.langdistalso depends ontensorflowpackage. In default, it tries to install the CPU-only version oftensorflow. If you want to use GPU, you need to installtensorflowwith GPU support by yourself. (C.f. [Installing Tensorflow](https://www.tensorflow.org/install/))UsageAfter installing,langdist –helpwill print help of how to uselangdistcommand.### 1. Download and preprocess a corpuslangdistimplements a command to download and preprocess a corpus from [Multilingual Bible Parallel Corpus](http://christos-c.com/bible/). The following command will download an English corpus and save it to./en_corpus.pkl.`langdistdownload-bibleen en_corpus.pkl`Note thatenhere is the language code of English. Specifying an invalid language code will raise an error message that shows the valid language codes.### 2. Fit an encoder on the characters used in corporaYou need to fit an encoder to the character used in corpora before you train a language model on them. Note that the same encoder will be used when you train a new language model on top of another language model (multilingual language model). Therefore, you need to fit an encoder to all the corpora you will train multilingual language models on.The following command will fit an encoder to English, French, and Japanese corpora and save it to./en_fr_ja_encoder.pkl:`langdistfit-encoderen_fr_ja_encoder.pkl en_corpus.pkl fr_corpus.pkl ja_corpus.pkl`Note thatxx_corpus.pklis a pickle file of a corpus, which can be generated bylangdist download-biblecommand. You can also create a list of texts by yourself and save it to a pickle file. (Each element of the list would correspond to a segment such as sentence, paragraph, article, etc. depending on your purpose.)### 3. Train a language model from the scratch (monolingual language model)The following command will train a French language model and save it to./fr_modeldirectory:`langdist train fr_corpus.pkl en_fr_ja_encoder.pkl fr_model--patience=819200--logpath=fr.log`Note that using an encoder that was not fit to the corpus will throw an exception.–patienceoption specifies how many iterations you want to keep training and–logpathoption specifies the path to the log file that records the progress of the training (no log file will be created if you don’t specify the option).During the training, various stats are dumped topath_to_model_dir/tensorboard.logdirectory. You can visualize them usingtensorboardbytensorboard –logdir=path_to_model_dir/tensorboard.log. The model is saved every time after computing validation perplexity and is available to use before finishing the training.Check the output oflangdist –helpto know what other options are available for training a language model.### 4. Train a new language model on of another language model (multilingual language model)The following command will train an English language model on top of the French language model we have trained and save it tofr2en_modeldirectory:`langdist retrain fr_model en_corpus.pkl fr2en_model--patience=819200--logpath=langdist.log`Note that you don’t have to specify the path to an encoder because the model infr_modelincludes it. If the encoder that was used when trainingfr_modelwas not fit to characters inen_corpus.pkl, it will throw an exception.During the training, various stats are dumped topath_to_model_dir/tensorboard.logdirectory. You can visualize them usingtensorboardbytensorboard –logdir=path_to_model_dir/tensorboard.log. The model is saved every time after computing validation perplexity and is available to use before finishing the training.Check the output oflangdist –helpto know what other options are available for training a language model.### 5. Generate texts using a trained language modelOnce you have trained a language model, the following command will generate texts using the trained language model:`langdist generate fr2en_model--sample-num=50`–sample-numoption decides the number of texts to generate. Note that each text is independently generated (sampled) by the language model.Check the output oflangdist –helpto know what other options are available for training a language model.### Uselangdistfrom Pythonlangdistcan be used as a normal python package by importinglangdistpackage, which is installed to your Python environment bypip install langdist. Readinglangdist/cli.pyis a good way to figure out how to use the package.TODO: Add a link to the blog postBilingual Character-level Neural Language Modeling