nltk/nltk — Issue #134: Summary: MLCorpusView gives different "Unmatched tag" depending on block size
Author: alexrudnick
Created: 2012-01-17 06:26:51+00:00
State: open

NLTK 2.0b9 on Mac OS X 10.6.5 Python 2.6.6

1. Download and install in a nltk corpus folder http://wordnet.princeton.edu/glosstag.shtml
2. Run the attached file with the command "python glosstag_simple.py"

You get a "ValueError: Unmatched tag error" at different tags depending on the block size.

The XMLCorpusView seems to loose track of the end tag nesting. The place where it looses track depends on the block size, so my initial guess it has something to do with the block boundaries.

Another hint is in using the ElementTree directly instead of XMLCorpusView the parser is having trouble with things like:
<wf id="n00001740_wf8" lemma="infer%2" pos="VBN" tag="man">
<id id="n00001740_id.9" lemma="infer" sk="infer%2:31:01::"/>
<id id="n00001740_id.8" lemma="infer" sk="infer%2:31:00::"/>
<id id="n00001740_id.7" lemma="infer" sk="infer%2:32:00::"/>inferred</wf>

The ElementTree parser would attach the text "inferred" as a tail to the last <id /> tag element rather than as the text for the <wf /wf> element

This code can be seen in the file "glosstag.py".

Note that glosstag.py is another unsuccessful attempt to use lazy access. (It works if lazy is set to False).

Migrated from http://code.google.com/p/nltk/issues/detail?id=617