How to Find Shortest Dependency Path with spaCy and StanfordNLP | by Xu LIANG | Towards Data Science

Considering the documentation and dependency parsing accuracy, I recommend using spaCy than StanfordNLP.
The content is structured as follows.
What is Shortest Dependency Path (SDP)?
Find Shortest Dependency Path with spaCy
Find Shortest Dependency Path with StanfordNLP
Semantic dependency parsing had been frequently used to dissect sentence and to capture word semantic information close in context but far in sentence distance.
To extract the relationship between two entities, the most direct approach is to use SDP. The motivation of using SDP is based on the observation that the SDP between entities usually contains the necessary information to identify their relationship.[1]
Convulsions that occur after DTaP are caused by a fever.
In the above figure, words in square brackets are marked entities. The red dashed-line arrows indicate the SDP between two entities.
First, install the necessary libraries in the terminal. I add the version number for clearness.
pip install spacy==2.1.4python -m spacy download en_core_web_smpip install stanfordnlp==0.2.0pip install networkx==2.3
First, we print out all dependency labels follow the official tutorial.
import spacyimport networkx as nxnlp = spacy.load("en_core_web_sm")doc = nlp(u'Convulsions that occur after DTaP are caused by a fever.')for token in doc:    print((token.head.text, token.text, token.dep_))# output: (head, current_token, dep_relation)('caused', 'Convulsions', 'nsubjpass')('occur', 'that', 'nsubj')('Convulsions', 'occur', 'relcl')('occur', 'after', 'prep')('caused', 'DTaP', 'nsubjpass')('caused', 'are', 'auxpass')('caused', 'caused', 'ROOT')('caused', 'by', 'agent')('fever', 'a', 'det')('by', 'fever', 'pobj')('caused', '.', 'punct')
We can plot the whole dependency tree by the convenient spaCy visualization tool.
The code below can give the SDP
import spacyimport networkx as nxnlp = spacy.load("en_core_web_sm")doc = nlp(u'Convulsions that occur after DTaP are caused by a fever.')print('sentence:'.format(doc))# Load spacy's dependency tree into a networkx graphedges = []for token in doc:    for child in token.children:        edges.append(('{0}'.format(token.lower_),                      '{0}'.format(child.lower_)))graph = nx.Graph(edges)# Get the length and pathentity1 = 'Convulsions'.lower()entity2 = 'fever'print(nx.shortest_path_length(graph, source=entity1, target=entity2))print(nx.shortest_path(graph, source=entity1, target=entity2))
The edges looks like below.
In [6]: edgesOut[6]:[('convulsions', 'occur'), ('occur', 'that'), ('occur', 'after'), ('caused', 'convulsions'), ('caused', 'dtap'), ('caused', 'are'), ('caused', 'by'), ('caused', '.'), ('by', 'fever'), ('fever', 'a')]
The output is below
3['convulsions', 'caused', 'by', 'fever']
This means the SDP length from ‘convulsions’ to ‘fever’ is 3.
If you don’t want to use networkx library, and only use the spaCy, you can check my another post, Find Lowest Common Ancestor Shortest Dependency Path with spaCy
First, we print out all dependency labels follow the official tutorial.
import stanfordnlpstanfordnlp.download('en')nlp = stanfordnlp.Pipeline()doc = nlp('Convulsions that occur after DTaP are caused by a fever.')doc.sentences[0].print_dependencies()# output: (current_token, head_index, dep_relation)('Convulsions', '0', 'root')('that', '3', 'nsubj')('occur', '1', 'acl:relcl')('after', '7', 'mark')('DTaP', '7', 'nsubj:pass')('are', '7', 'aux:pass')('caused', '3', 'advcl')('by', '10', 'case')('a', '10', 'det')('fever', '7', 'obl')
In order to follow the [(token, children), (token, children),...]format for networkx graph, we need to modify the code according to the source code.
import stanfordnlpstanfordnlp.download('en')nlp = stanfordnlp.Pipeline()doc = nlp('Convulsions that occur after DTaP are caused by a fever.')# Load stanfordnlp's dependency tree into a networkx graphedges = []for token in doc.sentences[0].dependencies:    if token[0].text.lower() != 'root':        edges.append((token[0].text.lower(), token[2].text))graph = nx.Graph(edges)# Get the length and pathentity1 = 'Convulsions'.lower()entity2 = 'fever'print(nx.shortest_path_length(graph, source=entity1, target=entity2))print(nx.shortest_path(graph, source=entity1, target=entity2))
The edges looks like below.
In [19]: edgesOut[19]:[('occur', 'that'), ('convulsions', 'occur'), ('caused', 'after'), ('caused', 'DTaP'), ('caused', 'are'), ('occur', 'caused'), ('fever', 'by'), ('fever', 'a'), ('caused', 'fever'), ('convulsions', '.')]
The output is below
3['convulsions', 'occur', 'caused', 'fever']
Even the SDP length calculated by StanfordNLP is the same with spaCy. But the words in the SDP between two entity should be 'caused', 'by'. So the dependency parsing accuracy of spaCy is better than StanfordNLP.
Check out my other posts on Medium with a categorized view!GitHub: BrambleXuLinkedIn: Xu LiangBlog: BrambleXu