Package: constituent-treelib
Constituent Treelib (CTL)A lightweight Python library for constructing, processing, and visualizing constituent trees.DescriptionCTL is a lightweight Python library that offers you a convenient way to parse sentences intoconstituent trees, modify them according to their structure, as well as visualize and export them into variousfile formats. In addition, you can extract phrases according to their phrasal categories (which can be used e.g., as features for various NLP tasks), validate already parsed sentences in bracket notation or convert them back into sentences.CTL is built on top ofbenepar(Berkeley Neural Parser) as well as the two well-known NLP frameworksspaCyandNLTK. Here, spaCy is used for tokenization and sentence segmentation, while benepar performs the actual parsing of the sentences. NLTK, on the other hand, provides the fundamental data structure for storing and processing the parsed sentences.To gain a clearer picture of what a constituent tree looks like, we consider the following example. LetSdenote the sentence..."Isaac Asimov was an American writer and professor of biochemistry at Boston University."This sentence can be parsed into a bracketed tree string representation (shown below in aPenn tree-bankstyle)(S
  (NP (NNP Isaac) (NNP Asimov))
  (VP
    (VBD was)
    (NP
      (NP (DT an) (JJ American) (NN writer) (CC and) (NN professor))
      (PP (IN of) (NP (NN biochemistry)))
      (PP (IN at) (NP (NNP Boston) (NNP University)))))
  (. .))which represents the actual constituent tree. However, since this notation is not really easy to read, we can turn it into a nicer visualization using - guess what - CTL! Once we have parsed and visualized the tree, we can export it to a desired format, here for example as aPNGfile:In case you grew up in theUsenetera, you might prefer the classic ASCII-ART look...S                                                              
        __________________________________|____________________________________________________________   
       |                    VP                                                                         | 
       |           _________|____________________                                                      |  
       |          |                              NP                                                    | 
       |          |                  ____________|________________________________                     |  
       |          |                 |                       PP                    PP                   | 
       |          |                 |                    ___|_______          ____|_____               |  
       NP         |                 NP                  |           NP       |          NP             | 
   ____|____      |    _____________|____________       |           |        |     _____|______        |  
 NNP       NNP   VBD  DT    JJ      NN    CC     NN     IN          NN       IN  NNP          NNP      . 
  |         |     |   |     |       |     |      |      |           |        |    |            |       |  
Isaac     Asimov was  an American writer and professor  of     biochemistry  at Boston     University  .Regardless of which format is considered, the underlying representation[^1] shows three aspects of the structure ofS:Linear order of the words and their part-of-speech:NNP = Isaac,NNP = Asimov,VBD = was, ...Groupings of the words and their part-of-speech into phrases:NP = Isaac Asimov,VP = an American writer and professor,PP = of  biochemistryandPP = at Boston UniversityHierarchical structure of the phrases:S,VP,NPandPPApplicationsConstituent trees offer a wide range of applications including:Analysis and comparison of sentence structures between different languages for (computational) linguistsExtracting phrasal features for certain NLP tasks (e.g.,Machine Translation,Information Extraction,Paraphrasing,Stylometry,Deception DetectionorNatural Language Watermarking)Using the resulting representations as an input to trainGNNsfor specific tasks (e.g.,Chemical–Drug Relation ExtractionorSemantic Role Labeling)FeaturesEasy construction of constituent trees from raw or already processed sentencesConverting parsed constituent trees back into sentencesConvenient export of tree visualizations into variousfile formatsExtraction of phrases according to theirphrasal categoriesManipulation of the tree structure (without inner postag nodes or without token leaves)Multilingual (currently CTL supportseight languages)Automatic NLP pipeline creation (loads and installs the benepar + spaCy models on demand)No API dependency (after downloading the models CTL can be used completely offline)Extensively documented source codeNo Code DemoIn case you just want to play around with CTL, there is a minimally functionalStreamlitapp that will be gradually extended. To run the demo, please first install Streamlit via:pip install streamlit. Afterwards, you can call the app from the command line as follows:streamlit run ctl_app.pyInstallationThe easiest way to install CTL is to use pip, where you can choose between (1) the PyPI[^2] repository and (2) this repository.(1)pip install constituent-treelib(2)pip install git+https://github.com/Halvani/constituent_treelib.gitThe latter will pull and install the latest commit from this repository as well as the required Python dependencies.Non-Python dependencies:CTL also relies on two open-source tools to export constituent trees into various file formats:To export the constituent tree into a PDF, the command line toolwkhtmltopdfis required. Once downloaded and installed, the path to the wkhtmltopdf binary must be passed to the export function.To export the constituent tree into the file formats JPG, PNG, GIF, BMP, EPS, PSD, TIFF and YAML, the software suiteImageMagickis required.QuickstartBelow you can find several examples of the core functionality of CTL. More examples can be found in thejupyter notebook demo.Creating an NLP pipelineTo instantiate aConstituentTreeobject, CTL requires a spaCy-based NLP pipeline that incorporates a benepar component. Although you can set up this pipeline yourself, it is recommended (and more convenient) to let CTL do it for you automatically via thecreate_pipeline()method. Given the desiredlanguage, this method creates the NLP pipeline and also downloads[^3] the corresponding spaCy and benepar models, if requested. The following code shows an example of this:fromconstituent_treelibimportConstituentTree,BracketedTree,Language,Structure# Define the language for the sentence as well as for the spaCy and benepar modelslanguage=Language.English# Define which specific SpaCy model should be used (default is Medium)spacy_model_size=ConstituentTree.SpacyModelSize.Medium# Create the pipeline (note, the required models will be downloaded and installed automatically)nlp=ConstituentTree.create_pipeline(language,spacy_model_size)>>>✔Downloadandinstallationsuccessful>>>Youcannowloadthepackageviaspacy.load('en_core_web_md')>>>[nltk_data]Downloadingpackagebenepar_en3to>>>[nltk_data][..]\nltk_data...>>>[nltk_data]Unzippingmodels\benepar_en3.zip.Define a sentenceNext, we instantiate aConstituentTreeobject and pass it the created NLP pipeline along with a sentence to parse, e.g. the memorable quote"You must construct additional pylons!"^4. Rather than a raw sentence,ConstituentTreealso accepts an already parsed sentence wrapped as a BracketedTree object, or alternatively in the form of an NLTK tree. The following example illustrates all three options:# Raw sentencesentence='You must construct additional pylons!'# Parsed sentence wrapped as a BracketedTree objectbracketed_tree_string='(S (NP (PRP You)) (VP (MD must) (VP (VB construct) (NP (JJ additional) (NNS pylons)))) (. !))'sentence=BracketedTree(bracketed_tree_string)# Parsed sentence in the form of an NLTK treefromnltkimportTreesentence=Tree('S',[Tree('NP',[Tree('PRP',['You'])]),Tree('VP',[Tree('MD',['must']),Tree('VP',[Tree('VB',['construct']),Tree('NP',[Tree('JJ',['additional']),Tree('NNS',['pylons'])])])]),Tree('.',['!'])])tree=ConstituentTree(sentence,nlp)Modified tree structureCTL allows you to modify the structure of the tree by either:Eliminatinginner postag nodes(tree contains now phrasal categories as inner nodes and tokens as leaves)Eliminatingtoken leaves(tree contains now phrasal categories as inner nodes and postags as leaves)without_token_leaves=ConstituentTree(sentence,nlp,Structure.WithoutTokenLeaves)without_inner_postag_nodes=ConstituentTree(sentence,nlp,Structure.WithoutPostagNodes)The result...Modified tree structures offer several benefits. One of them, for example, is saving space when using the visualizations in papers. Eliminating the inner postag nodes (shown on the right) reduces the tree height from level 5 to 4. Another useful application arises from the elimination of token leaves, which will be discussed in more detail in the following section.Extract phrasesOnce we have createdtree, we can now extract phrases according to givenphrasal categoriese.g., verb phrases:phrases=tree.extract_all_phrases()print(phrases)>>>{'S':['You must construct additional pylons !'],>>>'VP':['must construct additional pylons','construct additional pylons'],>>>'NP':['additional pylons']}# Only verb phrases..print(phrases['VP'])>>>['must construct additional pylons','construct additional pylons']As can be seen here, the second verb phrase is contained in the former. To avoid this, we can instruct the method to disregard nested phrases:non_nested_phrases=tree.extract_all_phrases(avoid_nested_phrases=True)print(non_nested_phrases['VP'])>>>['must construct additional pylons']If you want to extract phrases, but are more interested in their postag representation than the actual words/tokens, you can apply the same function to the modified tree...pos_phrases=without_token_leaves.extract_all_phrases()print(pos_phrases)>>>{'S':['PRP MD VB JJ NNS .'],>>>'NP':['JJ NNS'],>>>'VP':['MD VB JJ NNS','VB JJ NNS']}This is especially helpful when investigating the writing style of authors.Export the treeCTL offers you to export a constituent tree into various file formats, which are listed below. Most of these formats result in a visualization of the tree, while the remaining file formats are used for data exchange.ExtensionDescriptionOutputPDFPortable Document FormatVector graphicSVGScalable Vector GraphicsVector graphicEPSEncapsulated PostScriptVector graphicJPGJoint Photographic Experts GroupRaster imagePNGPortable Network GraphicsRaster imageGIFGraphics Interchange FormatRaster imageBMPBitmapRaster imagePSDPhotoshop DocumentRaster imageTIFFTagged Image File FormatRaster imageJSONJavaScript Object NotationData exchange formatYAMLYet Another Markup LanguageData exchange formatTXTPlain-TextPretty-print text visualizationTEXLaTeX-DocumentLaTeX-typesettingThe following example shows an export of the tree into a PDF file:tree.export_tree(destination_filepath='my_tree.pdf',verbose=True)>>>PDF-filesuccessfullysavedto:my_tree.pdfIn the case of raster/vector images, CTL automatically removes unnecessary margins with respect to the resulting visualizations. This is particularly useful if the visualizations are to be used in papers.Available models and languagesCTL currently supports eight languages: English, German, French, Polish, Hungarian, Swedish, Chinese and Korean. The performance of the respective models can be looked up in thebenepar repository.LicenseThe code and thejupyter notebook demoof CTL are released under the MIT License. SeeLICENSEfor further details.CitationIf you find this repository helpful, feel free to cite it in your paper or project:@software{Halvani_Constituent_Treelib_-_2023,author={Halvani, Oren},doi={10.5281/zenodo.7765147},month={3},title={{Constituent Treelib - A Lightweight Python Library for Constructing, Processing, and Visualizing Constituent Trees.}},url={https://github.com/Halvani/constituent-treelib},version={0.0.6},year={2023}Please also give credit to the authors of benepar andcite their work.[^1]: Note, if you are not familiar with the bracket labels of constituent trees, have a look at the followingGistor alternativelythis website.[^2]: It's recommended to install CTL fromPyPI(Python Package Index). However, if you want to benefit from the latest update of CTL, you should use this repository instead, since I will only update PyPi at irregular intervals.[^3]: After the models have been downloaded, they are cached so that there are no redundant downloads when the method is called again. However, loading and initializing the spaCy and benepar models can take a while, so it makes sense to invoke thecreate_pipeline()method only once if you want to process multiple sentences.