Package: word2word
word2wordEasy-to-use word translations for 3,564 language pairs.This is the official code accompanyingour LREC 2020 paper.SummaryA large collection of freely & publicly available bilingual lexiconsfor 3,564 language pairs across 62 unique languages.Easy-to-use Python interface for accessing top-k word translations and
for building a new bilingual lexicon from a custom parallel corpus.Constructed using a simple approach that yields bilingual lexicons with
high coverage and competitive translation quality.UsageFirst, install the package usingpip:pipinstallword2wordORgitclonehttps://github.com/kakaobrain/word2word
pythonsetup.pyinstallThen, in Python, download the model and retrieve top-5 word translations
of any given word to the desired language:fromword2wordimportWord2worden2fr=Word2word("en","fr")print(en2fr("apple"))# out: ['pomme', 'pommes', 'pommier', 'tartes', 'fleurs']Supported LanguagesWe provide top-k word-to-word translations across all available pairs
fromOpenSubtitles2018.
This amounts to a total of 3,564 language pairs across 62 unique languages.The full list is providedhere.MethodologyOur approach computes top-k word translations based on
the co-occurrence statistics between cross-lingual word pairs in a parallel corpus.
We additionally introduce a correction term that controls for any confounding effect
coming from other source words within the same sentence.
The resulting method is an efficient and scalable approach that allows us to
construct large bilingual dictionaries from any given parallel corpus.For more details, see the Methodology section ofour paper.Building a Bilingual Lexicon on a Custom Parallel CorpusTheword2wordpackage also provides interface for
building a custom bilingual lexicon using a different parallel corpus.
Here, we show an example of building one from
theMedline English-French dataset:fromword2wordimportWord2word# custom parallel data: data/pubmed.en-fr.en, data/pubmed.en-fr.frmy_en2fr=Word2word.make("en","fr","data/pubmed.en-fr")# ...building...print(my_en2fr("mitochondrial"))# out: ['mitochondriale', 'mitochondriales', 'mitochondrial',#       'cytopathies', 'mitochondriaux']When built from source, the bilingual lexicon can also be constructed from the command line as follows:pythonmake.py--lang1en--lang2fr--dataprefdata/pubmed.en-frIn both cases, the custom lexicon (saved todatapref/by default) can be re-loaded in Python:fromword2wordimportWord2wordmy_en2fr=Word2word.load("en","fr","data/pubmed.en-fr")# Loaded word2word custom bilingual lexicon from data/pubmed.en-fr/en-fr.pklMultiprocessingIn both the Python interface and the command line interface,makeuses multiprocessing with 16 CPUs by default.
The number of CPU workers can be adjusted by settingnum_workers=N(Python) or--num_workers N(command line).ReferencesIf you use word2word for research, please citeour paper:@inproceedings{choe2020word2word,author={Yo Joong Choe and Kyubyong Park and Dongwoo Kim},title={word2word: A Collection of Bilingual Lexicons for 3,564 Language Pairs},booktitle={Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020)},year={2020}}All of our pre-computed bilingual lexicons were constructed from the publicly availableOpenSubtitles2018dataset:@inproceedings{lison-etal-2018-opensubtitles2018,title="{O}pen{S}ubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora",author={Lison, Pierre  andTiedemann, J{\"o}rg  andKouylekov, Milen},booktitle="Proceedings of the Eleventh International Conference on Language Resources and Evaluation ({LREC} 2018)",month=may,year="2018",address="Miyazaki, Japan",publisher="European Language Resources Association (ELRA)",url="https://www.aclweb.org/anthology/L18-1275",}AuthorsKyubyong Park,Dongwoo Kim, andYJ Choe