Package: node-distance
node-distance: Tree node distances as featuresCompute distance between all nodes of a tree, and estimate an histogram that can be used as features for other models.Toy Text Corpuscorpus="Als Ada Lovelace auf einem Ball den Mathematiker Charles Babbage traf, der sie einlud, die von ihm erfundene „Differenzmaschine“ anzusehen, war sie hellauf begeistert. Die Maschine konnte selbstständig addieren und subtrahieren, doch Ada war klar, dass die Möglichkeiten damit noch lange nicht erschöpft waren. Sie träumte davon, dass eine solche Maschine eines Tages sogar Musik abspielen könnte, und ersann so die Idee eines modernen Computers. 1845 legte sie den ersten Algorithmus zur maschinellen Berechnung der Bernoulli-Zahlen vor und wird daher von vielen als erste Computerprogrammiererin der Welt gefeiert."(Source: DWDS, Wort des Tages, "Algorithmus, der", 27.11.2021, URL:https://www.dwds.de/adt)Extract the graph edges from a dependency tree with SpaCy/Stanza/TrankitWe assume that NodeIDs are numbers[1,2,3,...]starting with 1.
The NodeIDs are equivalent to the TokenIDs in Conll-U.An graph edge is a tuple(ParentID, NodeID)Example, SpaCy:# load the SpaCy modelimportde_dep_news_trfmodel=de_dep_news_trf.load()# extract the edges for each sentenceimportnode_distanceasndall_edges,num_nodes=nd.extract_edges_from_spacy(corpus,model)Example, stanza:# load the stanza modelimportstanzamodel=stanza.Pipeline(lang='de',processors='tokenize,mwt,pos,lemma,depparse',tokenize_pretokenized=False)# extract the edges for each sentenceimportnode_distanceasndall_edges,num_nodes=nd.extract_edges_from_stanza(corpus,model)Example, trankit:# load the trankit modelimporttrankitmodel=trankit.Pipeline(lang='german',gpu=False,cache_dir='./cache')# extract the edges for each sentenceimportnode_distanceasndall_edges,num_nodes=nd.extract_edges_from_trankit(corpus,model)num_nodeswith the number of tokens/nodes in each sentence.all_edgescontains lists of edges for each sentence, e.g.# Edges of the 3rd sentenceedges=all_edges[2]# Edge between the 6th token/node and its parent nodeedge=edges[6]parent_id,node_id=edgeCompute Shortest Paths between NodesCompute node distances and the corresponding token distancesimportnode_distanceasndnodedist,tokendist,indicies=nd.node_token_distances(all_edges,num_nodes,cutoff=25)Histograms as FeaturesThe Distribution of node distances:importnode_distanceasndxobs,pdf,_=nd.nodedist_distribution(nodedist,xmin=1,xmax=12)importmatplotlib.pyplotaspltplt.bar(xobs,pdf);plt.title("distribution of node distances");plt.xlabel("node distance");plt.ylabel("PDF");plt.show();Distribution of token distance vs node distance:importnode_distanceasndxobs,pdf,_=nd.tokenvsnode_distribution(tokendist,nodedist,xmin=-5,xmax=15)importmatplotlib.pyplotaspltplt.bar(xobs,pdf);plt.title("Distribution of token distance vs node distance");plt.xlabel("token distance minus node distance");plt.ylabel("PDF");plt.show();AppendixInstallationThenode-distancegit repois available asPyPi packagepipinstallnode-distance
pipinstallgit+ssh://git@github.com/ulf1/node-distance.gitInstall a virtual environmentIt is recommended to install python packages into a seperate virtual environement. (If your git repo is stored in a folder with whitespaces, then don't use the subfolder.venv. Use an absolute path without whitespaces.)python3-mvenv.venvsource.venv/bin/activate
pipinstall--upgradepip
pipinstall-rrequirements.txt--no-cache-dir
pipinstall-rrequirements-dev.txt--no-cache-dir
pipinstall-rrequirements-demo.txt--no-cache-dirThe usage example and demo notebooks might require spacy, stanza and trankit to be installed. You should download the pretrained models beforehand, e.g. pretrained models for German:python-mspacydownloadde_dep_news_trf
python-c"import stanza; stanza.download(lang='de')"python-c"import trankit; trankit.Pipeline(lang='german', gpu=False, cache_dir='./cache')"Python commandsJupyter for the examples:jupyter labCheck syntax:flake8 --ignore=F401 --exclude=$(grep -v '^#' .gitignore | xargs | sed -e 's/ /,/g')Run Unit Tests:PYTHONPATH=. pytestPublishpythonsetup.pysdisttwineupload-rpypidist/*Clean upfind.-typef-name"*.pyc"|xargsrm
find.-typed-name"__pycache__"|xargsrm-r
rm-r.pytest_cache
rm-r.venvSupportPleaseopen an issuefor support.ContributingPlease contribute usingGithub Flow. Create a branch, add commits, andopen a pull request.AcknowledgementsThe "Evidence" project was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -433249742(GU 798/27-1; GE 1119/11-1).Maintenancetill 31.Aug.2023 (v0.1.0) the code repository was maintained within the DFG project433249742since 01.Sep.2023 (v0.2.0) the code repository is maintained by Ulf Hamster.CitationYou can cite the following paper if you want to use this repository in your research work.@inproceedings{hamster-2022-everybody,
    title = "Everybody likes short sentences - A Data Analysis for the Text Complexity {DE} Challenge 2022",
    author = "Hamster, Ulf A.",
    booktitle = "Proceedings of the GermEval 2022 Workshop on Text Complexity Assessment of German Text",
    month = sep,
    year = "2022",
    address = "Potsdam, Germany",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.germeval-1.2",
    pages = "10--14",
}