Package: simalign
SimAlign: Similarity Based Word AlignerSimAlign is a high-quality word alignment tool that uses static and contextualized embeddings anddoes not require parallel training data.The following table shows how it compares to popular statistical alignment models:ENG-CESENG-DEUENG-FASENG-FRAENG-HINENG-RONfast-align.78.71.46.84.38.68eflomal.85.77.63.93.52.72mBERT-Argmax.87.81.67.94.55.65Shown is F1, maximum across subword and word level. For more details see thePaper.Installation and UsageTested with Python 3.7, Transformers 3.1.0, Torch 1.5.0. Networkx 2.4 is optional (only required for Match algorithm).
For full list of dependencies seesetup.py.
For installation of transformers seetheir repo.Download the repo for use or alternatively install with PyPipip install simalignor directly with pip from GitHubpip install --upgrade git+https://github.com/cisnlp/simalign.git#egg=simalignAn example for using our code:fromsimalignimportSentenceAligner# making an instance of our model.# You can specify the embedding model and all alignment settings in the constructor.myaligner=SentenceAligner(model="bert",token_type="bpe",matching_methods="mai")# The source and target sentences should be tokenized to words.src_sentence=["This","is","a","test","."]trg_sentence=["Das","ist","ein","Test","."]# The output is a dictionary with different matching methods.# Each method has a list of pairs indicating the indexes of aligned words (The alignments are zero-indexed).alignments=myaligner.get_word_aligns(src_sentence,trg_sentence)formatching_methodinalignments:print(matching_method,":",alignments[matching_method])# Expected output:# mwmf (Match): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]# inter (ArgMax): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]# itermax (IterMax): [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]For more examples of how to use our code seescripts/align_example.py.DemoAn online demo is availablehere.Gold StandardsLinks to the gold standars used in the paper are here:Language PairCitationTypeLinkENG-CESMarecek et al. 2008Gold Alignmenthttp://ufal.mff.cuni.cz/czech-english-manual-word-alignmentENG-DEUEuroParl-basedGold Alignmentwww-i6.informatik.rwth-aachen.de/goldAlignment/ENG-FASTvakoli et al. 2014Gold Alignmenthttp://eceold.ut.ac.ir/en/node/940ENG-FRAWPT2003, Och et al. 2000,Gold Alignmenthttp://web.eecs.umich.edu/~mihalcea/wpt/ENG-HINWPT2005Gold Alignmenthttp://web.eecs.umich.edu/~mihalcea/wpt05/ENG-RONWPT2005 Mihalcea et al. 2003Gold Alignmenthttp://web.eecs.umich.edu/~mihalcea/wpt05/Evaluation ScriptFor evaluating the output alignments usescripts/calc_align_score.py.The gold alignment file should have the same format as SimAlign outputs.
Sure alignment edges in the gold standard have a '-' between the source and the target indices and the possible edges have a 'p' between indices.
For sample parallel sentences and their gold alignments from ENG-DEU, seesamples.PublicationIf you use the code, please cite@inproceedings{jalili-sabet-etal-2020-simalign,
    title = "{S}im{A}lign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings",
    author = {Jalili Sabet, Masoud  and
      Dufter, Philipp  and
      Yvon, Fran{\c{c}}ois  and
      Sch{\"u}tze, Hinrich},
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.findings-emnlp.147",
    pages = "1627--1643",
}FeedbackFeedback and Contributions more than welcome! Just reach out to @masoudjs or @pdufter.FAQDo I need parallel data to train the system?No, no parallel training data is required.Which languages can be aligned?This depends on the underlying pretrained multilingual language model used. For example, if mBERT is used, it covers 104 languages as listedhere.Do I need GPUs for running this?Each alignment simply requires a single forward pass in the pretrained language model. While this is certainly
faster on GPU, it runs fine on CPU. On one GPU (GeForce GTX 1080 Ti) it takes around 15-20 seconds to align 500 parallel sentences.LicenseCopyright (C) 2020, Masoud Jalili Sabet, Philipp DufterA full copy of the license can be found in LICENSE.