Package: eflomal
pip installable eflomalThis is a pip installableeflomalpackage for the python scriptsalign.py,makepriors.pyandmergefiles.py. Installation:pip install eflomaleflomalEfficient Low-Memory AlignerThis is a word alignment tool based onefmaral, with the following main
differences:More compact data structures are used, so memory requirements are much
lower (by orders of magnitude).The estimation of alignment variable marginals is done one sentence at a
time, which also saves a lot of memory at no detectable cost in accuracy.New: User-specified Dirichlet priors, which can be generated by themakepriors.pyscript to allow models to be saved. See below underInput data formatandGenerating priors.Technical details relevant to bothefmaralandeflomalcan be found in
the following article:Östling and Tiedemann (2016)(BibTeX).InstallingTo compile and install the C binary and the Python bindings:make
sudo make install
python3 setup.py installChange theINSTALLDIRparameter in the install step if you want to install somewhere
other than the default/usr/local/bin(e.g.make install -e INSTALLDIR=~/bin).
Note that thealign.pyscript now uses theeflomalexecutable in the same directory
asalign.py, rather than in$PATH.UsingThere are three main ways of usingeflomal:Directly call theeflomalbinary. Note that this requires some
preprocessing.Use thealign.pycommand-line interface, which is partly
compatible with that ofefmaral. Runpython3 align.py --helpfor
instructions.Use the Cython module to call theeflomalbinary, this takes care of
the preprocessing and file conversions necessary. See the docstrings
ineflomal.pyxfor documentation.In addition, there are convenience scripts for aligning and symmetrizing (with
theatoolsprogram fromfast_align) as well as evaluating with data from
the WPT shared task datasets. These work the same way as inefmaral,
please see itsREADMEfor
details.Input data formatWhen used with the-sand-toptions for separate source/target files, thealign.pyinterface expects one sentence per line with space-separated
tokens, similar to most word alignment software.The-ioption assumes afast_textstyle joint source/target file of the
formatsource sentence ||| target sentence
another source sentence ||| another target sentence
...The--priorsoption expects a file generated bymakepriors.py(see below).
This file contains user-specified lexical, HMM and/or fertility distribution
priors. Since the algorithm is asymmetric, HMM and fertility priors can be
stored for both the forward and reverse directions.makepriors.pyhandles
this automatically, see examples below.Note that the default value of the Dirichlet priors (defined ineflomal.casLEX_ALPHA,JUMP_ALPHAandFERT_ALPHA) will beaddedto whatever is
specified in the priors file. This means that integer counts for whatever word
forms you have data on are fine in the priors file.It s possible to use the special<NULL>token in the priors file, in case
you want to encourage certain word forms to remain unaligned.
Currently themakepriors.pyscript does not generate these, and this feature
has not been tested yet.Generating priorsIf you have a large file that you want to use as "training data",en-sv, and
a small file  that you later want to align quickly,en-sv.small, start by
aligning the large file as usual, e.g.:./align.py -i en-sv --model 3 -f en-sv.fwd -r en-sv.revNow you can generate priors based on this large aligned file, stored inen-sv.priors:./makepriors.py -i en-sv -f en-sv.fwd -r en-sv.rev --priors en-sv.priorsAlternatively, you can symmetrizeen-sv.fwdanden-sv.revintoen-sv.symand pass the same file to both-fand-r:atools -c grow-diag-final-and -i en-sv.fwd -j en-sv.rev >en-sv.sym
./makepriors.py -i en-sv -f en-sv.sym -r en-sv.sym --priors en-sv.priorsNow, if you have another file to align,en-sv.small, simply use e.g.:./align.py -i en-sv.small --priors en-sv.priors --model 3 \
    -f en-sv.small.fwd -r en-sv.small.revThis will be much faster than mergingen-svanden-sv.smalland aligning
them jointly, while nearly as accurate (assumingen-sv.smallis much smaller
thanen-sv).Output data formatThe alignment output contains the same number of lines as the input files,
where each line contains pairs of indexes. For instance, if the source input
contains the following:a black catand the target input is the following:kuro nekothe correct output would be:1-0 2-1That is,1-0indicates token 1 of the source (black) is aligned to token 0
of the target (kuro), and2-1that token 2 of the source (cat) is aligned to
token 1 of the target (neko).NULLalignments are not present in the output.Note that the forward and reverse alignments both use source-target order, so
the output can be fed directly toatools(seescripts/align_symmetrize.shfor an example).In case you made a mistake with the direction, you can fix it afterwards withscripts/reverse_moses.py.PerformanceThis is a comparison between eflomal,efmaraland fast_align.The difference between efmaral and eflomal is in part due to different default
parameters, in particular the number of iterations and the number of
independent samplers.Note that all timing figures below include alignments in both directions
(run in parallel) and symmetrization.eflomalLanguagesSentencesAERCPU time (s)Real time (s)English-French1,130,5510.0811,232337English-Inkutitut340,6010.20316144Romanian-English48,6810.29815933English-Hindi3,5300.467316efmaralLanguagesSentencesAERCPU time (s)Real time (s)English-Swedish1,862,4260.1331,719620English-French1,130,5510.085763279English-Inkutitut340,6010.23512246Romanian-English48,6810.28716146English-Hindi3,5300.4839810fast_alignLanguagesSentencesAERCPU time (s)Real time (s)English-Swedish1,862,4260.20511,090672English-French1,130,5510.1533,840241English-Inuktitut340,6010.28747747Romanian-English48,6810.32520817English-Hindi3,5300.672242