Package: nmtscore
NMTScoreA library of translation-based text similarity measures.To learn more about how these measures work, have a look atJannis' blog post. Also, read our paper,"NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measures"(Findings of EMNLP).InstallationRequires Python >= 3.8 and PyTorchpip install nmtscoreExtra requirements for the Prism model:pip install nmtscore[prism]UsageNMTScorerInstantiate a scorer and start scoring short sentence pairs.fromnmtscoreimportNMTScorerscorer=NMTScorer()scorer.score("This is a sentence.","This is another sentence.")# 0.4677300455046415Different similarity measuresThe library implements three different measures:# Translation cross-likelihood (default)scorer.score_cross_likelihood(a,b,tgt_lang="en",normalize=True,both_directions=True)# Direct translation probabilityscorer.score_direct(a,b,a_lang="en",b_lang="en",normalize=True,both_directions=True)# Pivot translation probabilityscorer.score_pivot(a,b,a_lang="en",b_lang="en",pivot_lang="en",normalize=True,both_directions=True)Thescoremethod is a shortcut for cross-likelihood.Batch processingThe scoring methods also accept lists of strings:scorer.score(["This is a sentence.","This is a sentence.","This is another sentence."],["This is another sentence.","This sentence is completely unrelated.","This is another sentence."],)# [0.46772973967003206, 0.15306852595255185, 1.0]The sentences in the first list are compared element-wise to the sentences in the second list.The default batch size is 8.
An alternative batch size can be specified as follows (independently for translating and scoring):scorer.score_direct(a,b,a_lang="en",b_lang="en",score_kwargs={"batch_size":16})scorer.score_cross_likelihood(a,b,translate_kwargs={"batch_size":16},score_kwargs={"batch_size":16})Different NMT modelsThis library currently supports four NMT models:small100byMohammadshahi et al. (2022)m2m100_418Mandm2m100_1.2BbyFan et al. (2021)prismbyThompson and Post (2020)By default, the leanest model (small100) is loaded. The main results in the paper are based on the Prism model, which has some extra dependencies (see "Installation" above).scorer=NMTScorer("small100",device=None)# defaultscorer=NMTScorer("small100",device="cuda:0")# Enable faster inference on GPUscorer=NMTScorer("m2m100_418M",device="cuda:0")scorer=NMTScorer("m2m100_1.2B",device="cuda:0")scorer=NMTScorer("prism",device="cuda:0")Which model should I choose?The pageexperiments/results/summary.mdcompares the models regarding their accuracy and latency.Generally, we recommend Prism because it tends to have the highest accuracy. Also, Prism's implementation currently translates up 10x faster on GPU than the other models do, so we highly recommend to use Prism for the measures that require translation (score_pivot()andscore_cross_likelihood()).small100is 3.4x faster forscore_direct()and has 94–98% of Prism's accuracy.Enable caching of NMT outputIt can make sense to cache the translations and scores if they are needed repeatedly, e.g. in reference-based evaluation.scorer.score_direct(a,b,a_lang="en",b_lang="en",score_kwargs={"use_cache":True}# default: False)scorer.score_cross_likelihood(a,b,translate_kwargs={"use_cache":True},# default: Falsescore_kwargs={"use_cache":True}# default: False)Activating this option will create an SQLite database in the ~/.cache directory. The directory can be overriden via theNMTSCORE_CACHEenvironment variable.Print a version signature (à laSacreBLEU)scorer.score(a,b,print_signature=True)# NMTScore-cross|tgt-lang:en|model:alirezamsh/small100|normalized|both-directions|v0.3.0|hf4.26.1Direct usage of NMT modelsThe NMT models also provide a direct interface for translating and scoring.fromnmtscore.modelsimportload_translation_modelmodel=load_translation_model("small100")model.translate("de",["This is a test."])# ["Das ist ein Test."]model.score("de",["This is a test."],["Das ist ein Test."])# [0.8293135166168213]ExperimentsSeeexperiments/README.mdCitation@inproceedings{vamvas-sennrich-2022-nmtscore,title="{NMTS}core: A Multilingual Analysis of Translation-based Text Similarity Measures",author="Vamvas, Jannis  andSennrich, Rico",booktitle="Findings of the Association for Computational Linguistics: EMNLP 2022",month=dec,year="2022",address="Abu Dhabi, United Arab Emirates",publisher="Association for Computational Linguistics",url="https://aclanthology.org/2022.findings-emnlp.15",pages="198--213"}LicenseCode: MIT LicenseData: See data subdirectoriesChangelogv0.3.3Update minimum required Python version to 3.8Require transformers<4.34 to ensure compatibility forsmall100modelm2m100/small100: Stop adding extra EOS tokens when scoring, which is not needed anymorev0.3.2Fix score calculation withsmall100model (account for the fact that the target sequence is not prefixed with the target language, as is the case form2m100).Improve caching efficiencyv0.3.1Implement the distilledsmall100model byMohammadshahi et al. (2022)and use this model by default.Enable half-precision inference form2m100models andsmall100by default; see (/experiments/results/summary.md) for benchmark resultsv0.2.0Bugfix: Provide source language tom2m100models (#2). The fix is backwards-compatible but a warning is now raised ifm2m100is used without specifying the input language.