Package: tuw-nlp
TUW-NLPNLP utilities developed at TUW informatics.The main goal of the library is to provide a unified interface for working with different semantic graph representations. To represent
graphs we use thenetworkxlibrary.
Currently you can use the following semantic graph representations integrated in the library:4langUD(Universal Dependencies)AMR(Abstract Meaning Representation)SDP(Semantic Dependency Parsing)UCCA(Universal Conceptual Cognitive Annotation)DRS(Discourse Representation Structure)Setup and UsageInstall the tuw-nlp repository from pip:pip install tuw-nlpOr install from source:pip install -e .On Windows and Mac, you might also need to installGraphvizmanually.You will also need some additional steps to use the library:Download nltk resources:importnltknltk.download('stopwords')nltk.download('propbank')Download stanza models for UD parsing:importstanzastanza.download("en")stanza.download("de")4langThe4langsemantic graph representation is implemented in the repository. We use Interpreted Regular Tree Grammars (IRTGs) to build the graphs from UD trees. The grammar can be found in thelexicon. It supports English and German.To use the parser download thealtoparser and tuw_nlp dictionaries:importtuw_nlptuw_nlp.download_alto()tuw_nlp.download_definitions()Also please make sure to have JAVA on your system to be able to use the parser!Then you can parse a sentence as simple as:fromtuw_nlp.grammar.text_to_4langimportTextTo4langtfl=TextTo4lang("en","en_nlp_cache")fl_graphs=list(tfl("brown dog",depth=1,substitute=False))# Then the fl_graphs are the classes that contains the networkx graph objectfl_graphs[0].G.nodes(data=True)# Visualize the graphfl_graphs[0].to_dot()UDTo parseUniversal Dependenciesinto networkx format, we use thestanzalibrary. You can use all the languages supported by stanza:https://stanfordnlp.github.io/stanza/models.htmlFor parsing you can use the snippet above, just use theTextToUDclass.AMRFor parsingAbstract Meaning Representationgraphs we use theamrliblibrary. Models are only available for English.If you want to use AMR parsing, install theamrlibpackage (this is also included in thesetupfile) and download the models:pipinstallamrlibGo to theamrlibrepository and follow the instructions to download the models.Then also download the spacy model for AMR parsing:python-mspacydownloaden_core_web_smTo parse AMR, see theTextToAMRclass.SDPFor parsingSemantic Dependency Parsingwe integrated the Semantic Dependency Parser from theSuParlibrary. Models are only available for English.See theTextToSDPclass for more information.UCCAFor parsingUCCAgraphs we integrated thetupaparser. See our fork of the parserhere. Because of the complexity of the parser, we included a docker image that contains the parser and all the necessary dependencies. You can use the docker image to parse UCCA graphs. To see more go to theservicesfolder and follow the instructions there.UCCA parsing is currently supporting English, French, German and Hebrew. The docker service is a REST API that you can use to parse UCCA graphs. To convert the output to networkx graphs, see theTextToUCCAclass.DRSThe task ofDiscourse Representation Structure (DRS) parsingis to convert text into formal meaning representations in the style of Discourse Representation Theory (DRT; Kamp and Reyle 1993).To make it compatible with our library, we make use of the paper titledTransparent Semantic Parsing with Universal Dependencies Using Graph Transformations. This work first transformes DRS structures into graphs (DRG). They make use of a rule-based method developed using theGREWlibrary.Because of the complexity of the parser, we included a docker image that contains the parser and all the necessary dependencies. You can use the docker image to parse DRS graphs. To see more go to theservicesfolder and follow the instructions there. For parsing we our own fork of theud-boxerrepository. Currently it supports English, Italian, German and Dutch.To convert the output of the REST API (from the docker service) to networkx graphs, see theTextToDRSclass.For more examples you can check ourexperimentsjupyter notebook.Command line interfaceWe provide a simple script to parse into any of the supported formats.
For this you can use thescripts/semparse.pyscript. For usage:usage:semparse.py[-h][-fFORMAT][-cdCACHE_DIR][-cnNLP_CACHE]-lLANG[-dDEPTH][-sSUBSTITUTE][-pPREPROCESSOR][-oOUT_DIR]optionalarguments:-h,--helpshowthishelpmessageandexit-fFORMAT,--formatFORMAT-cdCACHE_DIR,--cache-dirCACHE_DIR-cnNLP_CACHE,--nlp-cacheNLP_CACHE-lLANG,--langLANG-dDEPTH,--depthDEPTH-sSUBSTITUTE,--substituteSUBSTITUTE-pPREPROCESSOR,--preprocessorPREPROCESSOR-oOUT_DIR,--out-dirOUT_DIRFor example to parse a sentence into UCCA graph run:echo"A police statement did not name the man in the boot, but in effect indicated the traveler was State Secretary Samuli Virtanen, who is also the deputy to Foreign Minister Timo Soini."|pythonscripts/semparse.py-fucca-len-cncache/nlp_cache_en.jsonServicesWe also provide services built on our package. To get to know more visitservices.Text_to_4lang serviceTo run a browser-based demo (also availableonline) for building graphs from raw texts, first start the graph building service:python services/text_to_4lang/backend/service.pyThen run the frontend with this command:streamlit run services/text_to_4lang/frontend/demo.pyIn the demo you can parse english and german sentences and you can also try out multiple algorithms our graphs implement, such asexpand,substituteandappend_zero_paths.ModulestextGeneral text processing utilities, contains:segmentation: stanza-based processors for word and sentence level segmentationpatterns: various patterns for text processing tasksgraphTools for working with graphs, contains:utils: misc utilities for working with graphsgrammarTools for generating and using grammars, contains:alto: tools for interfacing with thealtotoolirtg: class for representing Interpreted Regular Tree Grammarslexicon: Rule lexica for building lexicalized grammarsud_fl: grammar-based mapping ofUniversal Dependenciesto4langsemantic graphs.utils: misc utilities for working with grammarsContributingWe welcome all contributions! Please fork this repository and create a branch for your modifications. We suggest getting in touch with us first, by opening an issue or by writing an email to Gabor Recski or Adam Kovacs atfirstname.lastname@tuwien.ac.atCitingIf you use the library, please cite ourpaper@inproceedings{Recski:2021,title={Explainable Rule Extraction via Semantic Graphs},author={Recski, Gabor and Lellmann, Bj{\"o}rn and Kovacs, Adam and Hanbury, Allan},booktitle={{Proceedings of the Fifth Workshop on Automated Semantic Analysisof Information in Legal Text (ASAIL 2021)}},publisher={{CEUR Workshop Proceedings}},address={São Paulo, Brazil},pages="24--35",url="http://ceur-ws.org/Vol-2888/paper3.pdf",year={2021}}LicenseMIT license