Repository: nltk/nltk

Issue:
Bug in DependencyGraph._tree
Running the dependencygraph doctests produces an error, probably due to recent changes in `DependencyGraph`. It shows up in https://github.com/nltk/nltk/blob/develop/nltk/parse/dependencygraph.py#L265

@dimazest – would you be able to look at this please?

```
Traceback (most recent call last):
  File "dependencygraph.py", line 627, in <module>
    demo()
  File "dependencygraph.py", line 421, in demo
    malt_demo()
  File "dependencygraph.py", line 451, in malt_demo
    tree = dg.tree()
  File "dependencygraph.py", line 281, in tree
    return Tree(word, [self._tree(i) for i in deps])
  File "dependencygraph.py", line 281, in <listcomp>
    return Tree(word, [self._tree(i) for i in deps])
  File "dependencygraph.py", line 265, in _tree
    word = node['word']
KeyError: 'word'
```



Discussion:
There's a related problem for the non-projective dependency parser:

```
Traceback (most recent call last):
  File "nonprojectivedependencyparser.py", line 766, in <module>
    demo()
  File "nonprojectivedependencyparser.py", line 724, in demo
    nonprojective_conll_parse_demo()
  File "nonprojectivedependencyparser.py", line 743, in nonprojective_conll_parse_demo
    for parse_graph in npp.parse(['Cathy', 'zag', 'hen', 'zwaaien', '.'], ['N', 'V', 'Pron', 'Adj', 'N', 'Punc']):
  File "nonprojectivedependencyparser.py", line 465, in parse
    g_graph.connect_graph()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/nltk-3.0.0-py3.4.egg/nltk/parse/dependencygraph.py", line 110, in connect_graph
    node1['deps'].append(node2['address'])
AttributeError: 'collections.defaultdict' object has no attribute 'append'
```

I'll try to have a look by the end of the week.

Thanks. It's one of the few things to resolve before 3.0.2 can be released.

There's another downstream problem with the projective dependency parser:

```
  File "projectivedependencyparser.py", line 557, in <module>
    demo()
  File "projectivedependencyparser.py", line 468, in demo
    projective_rule_parse_demo()
  File "projectivedependencyparser.py", line 487, in projective_rule_parse_demo
    for tree in trees:
  File "projectivedependencyparser.py", line 195, in parse
    dg = DependencyGraph(conll_format)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/nltk-3.0.0-py3.2.egg/nltk/parse/dependencygraph.py", line 71, in __init__
    cell_separator=cell_separator,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages/nltk-3.0.0-py3.2.egg/nltk/parse/dependencygraph.py", line 245, in _parse
    "The graph does'n contain a node "
nltk.parse.dependencygraph.DependencyGraphError: The graph does'n contain a node that depends on the root element.
```

This is holding up 3.0.2.

I just saw the PR. Thanks @dimazest – I hope that solves the problems.

Unfortunately there are still problems with our dependency parsers, which seem to have arisen after #822 was merged. @dimazest I'd be grateful if you would look into it some more please.

right, i've noticed problems with `nltk.parse.nonprojectivedependencyparser.ProbabilisticNonprojectiveParser` but didn't see a clear solution right on spot. I'll have a look later this week.

@longdt219 would you please take a look at these problems soon? They're holding up 3.0.2. I'd rather not have to revert @dimazest's PR #822.

yes I will @stevenbird 

Thanks for #869 @longdt219 

These issues have been resolved in #870 and #872.

We still seem to have a problem stemming from #822:

```
Traceback (most recent call last):
  File "/scratch/jenkins/python/python-2.6.9-x86_64/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/scratch/jenkins/workspace/nltk/TOXENV/py26-jenkins/jdk/oraclejdk8/nltk/test/unit/test_corpora.py", line 137, in test_parsed_sents
    '.'
  File "/scratch/jenkins/python/python-2.6.9-x86_64/lib/python2.6/unittest.py", line 350, in failUnlessEqual
    (msg or '%r != %r' % (first, second))
AssertionError: Tree('fortaleci\xf3', ['hoy', 'considerablemente', Tree('aumento', ['El', Tree('del', [Tree('\xedndice', [Tree('de', [Tree('desempleo', ['estadounidense'])])])])]), '.', Tree('al', [Tree('euro', [Tree('cotizaba', [',', 'que', Tree('a', ['0,9452_d\xf3lares']), Tree('a', [Tree('15.35', ['las', 'GMT'])]), Tree('en', [Tree('mercado', ['el', Tree('de', ['divisas']), Tree('de', ['Fr\xe1ncfort'])])]), Tree('frente_a', [',', Tree('0,9349_d\xf3lares', ['los', Tree('de', [Tree('ma\xf1ana', ['esta'])])])]), 'se'])])])]) != Tree('fortaleci\xf3', [Tree('aumento', ['El', Tree('del', [Tree('\xedndice', [Tree('de', [Tree('desempleo', ['estadounidense'])])])])]), 'hoy', 'considerablemente', Tree('al', [Tree('euro', [Tree('cotizaba', [',', 'que', Tree('a', [Tree('15.35', ['las', 'GMT'])]), 'se', Tree('en', [Tree('mercado', ['el', Tree('de', ['divisas']), Tree('de', ['Fr\xe1ncfort'])])]), Tree('a', ['0,9452_d\xf3lares']), Tree('frente_a', [',', Tree('0,9349_d\xf3lares', ['los', Tree('de', [Tree('ma\xf1ana', ['esta'])])])])])])]), '.'])
```

https://nltk.ci.cloudbees.com/job/nltk/TOXENV=py26-jenkins,jdk=oraclejdk8/lastCompletedBuild/testReport/nltk.test.unit.test_corpora/TestCoNLL2007/test_parsed_sents/

The dependencies doctest is also broken by this PR:

```
cd nltk/test
make dependency.errs
```

@dimazest this is blocking the 3.0.2 release

I'll have a look as soon as I get to my laptop.


Resolving patch:
diff --git a/nltk/parse/dependencygraph.py b/nltk/parse/dependencygraph.py
--- a/nltk/parse/dependencygraph.py
+++ b/nltk/parse/dependencygraph.py
@@ -102,7 +102,7 @@ def add_arc(self, head_address, mod_address):
         self.nodes[head_address]['deps'].setdefault(relation,[])
         self.nodes[head_address]['deps'][relation].append(mod_address)
         #self.nodes[head_address]['deps'].append(mod_address)
-        
+
 
     def connect_graph(self):
         """
@@ -113,7 +113,7 @@ def connect_graph(self):
             for node2 in self.nodes.values():
                 if node1['address'] != node2['address'] and node2['rel'] != 'TOP':
                     relation = node2['rel']
-                    node1['deps'].setdefault(relation,[]) 
+                    node1['deps'].setdefault(relation, [])
                     node1['deps'][relation].append(node2['address'])
                     #node1['deps'].append(node2['address'])
 
@@ -214,17 +214,21 @@ def extract_10_cells(cells):
         lines = (l.rstrip() for l in input_)
         lines = (l for l in lines if l)
 
+        cell_number = None
         for index, line in enumerate(lines, start=1):
             cells = line.split(cell_separator)
-            nrCells = len(cells)
+            if cell_number is None:
+                cell_number = len(cells)
+            else:
+                assert cell_number == len(cells)
 
             if cell_extractor is None:
                 try:
-                    cell_extractor = extractors[nrCells]
+                    cell_extractor = extractors[cell_number]
                 except KeyError:
                     raise ValueError(
                         'Number of tab-delimited fields ({0}) not supported by '
-                        'CoNLL(10) or Malt-Tab(4) format'.format(nrCells)
+                        'CoNLL(10) or Malt-Tab(4) format'.format(cell_number)
                     )
 
             word, lemma, ctag, tag, feats, head, rel = cell_extractor(cells)
@@ -246,6 +250,9 @@ def extract_10_cells(cells):
                 }
             )
 
+            # Make sure that he fake root node has labeled dependencies.
+            if (cell_number == 3) and (head == 0):
+                rel = 'ROOT'
             self.nodes[head]['deps'][rel].append(index)
 
         if not self.nodes[0]['deps']['ROOT']:
@@ -271,7 +278,7 @@ def _tree(self, i):
         """
         node = self.get_by_address(i)
         word = node['word']
-        deps = list(chain.from_iterable(node['deps'].values()))
+        deps = sorted(chain.from_iterable(node['deps'].values()))
 
         if deps:
             return Tree(word, [self._tree(dep) for dep in deps])
@@ -286,7 +293,7 @@ def tree(self):
         node = self.root
 
         word = node['word']
-        deps = chain.from_iterable(node['deps'].values())
+        deps = sorted(chain.from_iterable(node['deps'].values()))
         return Tree(word, [self._tree(dep) for dep in deps])
 
     def triples(self, node=None):
@@ -299,7 +306,7 @@ def triples(self, node=None):
             node = self.root
 
         head = (node['word'], node['ctag'])
-        for i in node['deps']:
+        for i in sorted(chain.from_iterable(node['deps'].values())):
             dep = self.get_by_address(i)
             yield (head, dep['rel'], (dep['word'], dep['ctag']))
             for triple in self.triples(node=dep):
diff --git a/nltk/parse/nonprojectivedependencyparser.py b/nltk/parse/nonprojectivedependencyparser.py
--- a/nltk/parse/nonprojectivedependencyparser.py
+++ b/nltk/parse/nonprojectivedependencyparser.py
@@ -462,8 +462,8 @@ def parse(self, tokens, tags):
                 }
             )
         #print (g_graph.nodes)
-        
-        
+
+
         # Fully connect non-root nodes in g_graph
         g_graph.connect_graph()
         original_graph = DependencyGraph()
@@ -567,8 +567,10 @@ def parse(self, tokens, tags):
 
         logger.debug('Betas: %s', betas)
         for node in original_graph.nodes.values():
-            # deps must be a dictionary 
-            #node['deps'] = []
+            # TODO: It's dangerous to assume that deps it a dictionary
+            # because it's a default dictionary. Ideally, here we should not
+            # be concerned how dependencies are stored inside of a dependency
+            # graph.
             node['deps'] = {}
         for i in range(1, len(tokens) + 1):
             original_graph.add_arc(betas[i][0], betas[i][1])
@@ -701,22 +703,32 @@ def parse(self, tokens):
         # Filter parses
         # ensure 1 root, every thing has 1 head
         for analysis in analyses:
-            root_count = 0
-            root = []
-            for i, cell in enumerate(analysis):
-                if cell == -1:
-                    root_count += 1
-                    root = i
-            if root_count == 1:
-                graph = DependencyGraph()
-                graph.nodes[0]['deps'] = root + 1
-                for i in range(len(tokens)):
-                    node = {'word': tokens[i], 'address': i+1}
-                    node['deps'] = [j+1 for j in range(len(tokens)) if analysis[j] == i]
-                    graph.nodes[i + 1] = node
-#               cycle = graph.contains_cycle()
-#               if not cycle:
-                yield graph
+            if analysis.count(-1) > 1:
+                # there are several root elements!
+                continue
+
+            graph = DependencyGraph()
+            graph.root = graph.nodes[analysis.index(-1) + 1]
+
+            for address, (token, head_index) in enumerate(zip(tokens, analysis), start=1):
+                head_address = head_index + 1
+
+                node = graph.nodes[address]
+                node.update(
+                    {
+                        'word': token,
+                        'address': address,
+                    }
+                )
+
+                if head_address == 0:
+                    rel = 'ROOT'
+                else:
+                    rel = ''
+                graph.nodes[head_index + 1]['deps'][rel].append(address)
+
+            # TODO: check for cycles
+            yield graph
 
 
 #################################################################


Test patch:
diff --git a/nltk/test/dependency.doctest b/nltk/test/dependency.doctest
--- a/nltk/test/dependency.doctest
+++ b/nltk/test/dependency.doctest
@@ -35,30 +35,33 @@ CoNLL Data
     ... .       .       9       VMOD
     ... """
 
-
     >>> dg = DependencyGraph(treebank_data)
-    >>> print(dg.tree().pprint())
+    >>> dg.tree().pprint()
     (will
       (Vinken Pierre , (old (years 61)) ,)
       (join (board the) (as (director a nonexecutive)) (Nov. 29) .))
-    >>> print(list(dg.triples()))
-    [((u'will', u'MD'), u'SUB', (u'Vinken', u'NNP')),
-     ((u'Vinken', u'NNP'), u'NMOD', (u'Pierre', u'NNP')),
-     ((u'Vinken', u'NNP'), u'P', (u',', u',')),
-     ((u'Vinken', u'NNP'), u'NMOD', (u'old', u'JJ')),
-     ((u'old', u'JJ'), u'AMOD', (u'years', u'NNS')),
-     ((u'years', u'NNS'), u'NMOD', (u'61', u'CD')),
-     ((u'Vinken', u'NNP'), u'P', (u',', u',')),
-     ((u'will', u'MD'), u'VC', (u'join', u'VB')),
-     ((u'join', u'VB'), u'OBJ', (u'board', u'NN')),
-     ((u'board', u'NN'), u'NMOD', (u'the', u'DT')),
-     ((u'join', u'VB'), u'VMOD', (u'as', u'IN')),
-     ((u'as', u'IN'), u'PMOD', (u'director', u'NN')),
-     ((u'director', u'NN'), u'NMOD', (u'a', u'DT')),
-     ((u'director', u'NN'), u'NMOD', (u'nonexecutive', u'JJ')),
-     ((u'join', u'VB'), u'VMOD', (u'Nov.', u'NNP')),
-     ((u'Nov.', u'NNP'), u'NMOD', (u'29', u'CD')),
-     ((u'join', u'VB'), u'VMOD', (u'.', u'.'))]
+    >>> for head, rel, dep in dg.triples():
+    ...     print(
+    ...         '({h[0]}, {h[1]}), {r}, ({d[0]}, {d[1]})'
+    ...         .format(h=head, r=rel, d=dep)
+    ...     )
+    (will, MD), SUB, (Vinken, NNP)
+    (Vinken, NNP), NMOD, (Pierre, NNP)
+    (Vinken, NNP), P, (,, ,)
+    (Vinken, NNP), NMOD, (old, JJ)
+    (old, JJ), AMOD, (years, NNS)
+    (years, NNS), NMOD, (61, CD)
+    (Vinken, NNP), P, (,, ,)
+    (will, MD), VC, (join, VB)
+    (join, VB), OBJ, (board, NN)
+    (board, NN), NMOD, (the, DT)
+    (join, VB), VMOD, (as, IN)
+    (as, IN), PMOD, (director, NN)
+    (director, NN), NMOD, (a, DT)
+    (director, NN), NMOD, (nonexecutive, JJ)
+    (join, VB), VMOD, (Nov., NNP)
+    (Nov., NNP), NMOD, (29, CD)
+    (join, VB), VMOD, (., .)
 
 Using the dependency-parsed version of the Penn Treebank corpus sample.
 
@@ -159,21 +162,22 @@ Non-Projective Dependency Parsing
       'dog' -> 'his'
 
     >>> dp = NonprojectiveDependencyParser(grammar)
-    >>> for g in dp.parse(['the', 'man', 'taught', 'his', 'dog', 'to', 'play', 'golf']):
-    ...     print(g)  # doctest: +NORMALIZE_WHITESPACE
-    {0: {'address': 0,
-         'ctag': 'TOP',
-         'deps': 3,
-         'feats': None,
-         'lemma': None,
-         'rel': 'TOP',
-         'tag': 'TOP',
-         'word': None},
-     1: {'address': 1, 'deps': [], 'word': 'the'},
-     2: {'address': 2, 'deps': [1], 'word': 'man'},
-     3: {'address': 3, 'deps': [2, 7], 'word': 'taught'},
-     4: {'address': 4, 'deps': [], 'word': 'his'},
-     5: {'address': 5, 'deps': [4], 'word': 'dog'},
-     6: {'address': 6, 'deps': [], 'word': 'to'},
-     7: {'address': 7, 'deps': [5, 6, 8], 'word': 'play'},
-     8: {'address': 8, 'deps': [], 'word': 'golf'}}
+    >>> g, = dp.parse(['the', 'man', 'taught', 'his', 'dog', 'to', 'play', 'golf'])
+
+    >>> print(g.root['word'])
+    taught
+
+    >>> for _, node in sorted(g.nodes.items()):
+    ...     if node['word'] is not None:
+    ...         print('{address} {word}: {d}'.format(d=node['deps'][''], **node))
+    1 the: []
+    2 man: [1]
+    3 taught: [2, 7]
+    4 his: []
+    5 dog: [4]
+    6 to: []
+    7 play: [5, 6, 8]
+    8 golf: []
+
+    >>> print(g.tree())
+    (taught (man the) (play (dog his) to golf))