Package: lambre
Morphological Evaluation of NLGL'AMBRE is a tool to measure the grammatical well-formedness of texts generated by NLG systems. It analyzes the dependency parses of the text using morpho-syntactic rules, and returns a well-formedness score. This tool utilizes the Surface Syntactic Universal Dependency (SUD) project both for extracting rules as well as parsing, and is therefore applicable across languages. See ourEMNLP 2021 paperfor more details.Getting StartedInstall from PyPIpython-mpipinstalllambreTry L'AMBREFor a given input text file,lambrecomputes a morpho-syntactic well-formedness score [0-1]. The following method first downloads the parsers and rule sets for the specified language before computing the document-level score. See the output folder (out) for error visualizations.>>>importlambre>>>withopen("data/txt/ru.txt","r")asrf:...data=rf.readlines()>>>lambre.score("ru",data)0.9962L'AMBRE can also be used from command line. Seelambre --helpfor more options.lambrerudata/txt/ru.txtMorpho-syntactic Ruleslambrecurrently supports two rule sets,chaudhary-etal-2021(seeChaudhary et al., 2020,2021) andpratapa-etal-2021(seePratapa et al., 2021). The former is the default, but the rule set can be specified using--rule-setoption.Visualization ExamplesAlong with the overall L'AMBRE score, we write the erroneous sentences to the output folderout/errors. We provide two visualizations, i) plain text (errors.txt), ii) HTML (errors/*.html). For plain text visualization, we use theipymarkuptool. We usebratandUniversal Dependenciesfor HTML visualizations.Below is a sample run on 1000 example Hindi sentences from theSamanantar corpus.>>>importlambre>>>withopen("examples/hi_sents_1k.txt","r")asrf:...data=rf.readlines()>>>lambre.score("hi",data)0.8821A few erroneous sentences from this corpus (as detected by L'AMBRE):Input sentence: संख्या की स्टाफ प्रशिक्षित में हिंदी/वाले जानने हिंदी
                 (Stenography Hindi in trained persons of No.)
                 
                 तीनवर्षतक अग्रनीत किए| जाने के बाद व्यपगत हुए| आरक्षणों की| संख्या
                 (of after forward No. reservations lapsed carrying for 3 years)Below, we show the visualizations of word order related errors for the above two sentences. We also generate separate files for agreement and case marking (seeexamples/for full HTML outputs).ParserWe provide SUD parsers trained usingStanzatoolkit. See section 4 inour paperfor more details.Supported LanguagesWe currently support the following languages.lambreautomatically downloads the necessary language-specific resources (when available).LanguageCodeLanguageCodeLanguageCodeLanguageCodeCatalancaSpanishesItalianitRussianruCzechcsEstonianetLatvianlvSlovenianslDanishdaPersianfaDutchnlSwedishsvGermandeFrenchfrPolishplUkrainianukGreekelHindihiPortugueseptUrduurEnglishenIndonesianidRomanianroTo manually download rules or parsers for a given language,>>>importlambre>>>lambre.download("ru")# RussianReferenceIf you find this toolkit helpful in your research, consider citing our paper,@inproceedings{pratapa-etal-2021-evaluating,title="Evaluating the Morphosyntactic Well-formedness of Generated Texts",author="Pratapa, Adithya  andAnastasopoulos, Antonios  andRijhwani, Shruti  andChaudhary, Aditi  andMortensen, David R.  andNeubig, Graham  andTsvetkov, Yulia",booktitle="Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",month=nov,year="2021",address="Online and Punta Cana, Dominican Republic",publisher="Association for Computational Linguistics",url="https://aclanthology.org/2021.emnlp-main.570",pages="7131--7150",}We also encourage you to cite the original works for thechaudhary-etal-2021ruleset,Chaudhary et al., 2020andChaudhary et al., 2021.LicenseL'AMBRE is available under MIT License. The code for training parsers is adapted fromstanza, which is available under Apache License, Version 2.0.IssuesFor any issues, questions or requests, please use theGithub Issue Tracker.