nltk/nltk — Issue #70: XML corpus reader fails on XML comments
Author: alexrudnick
Created: 2012-01-17 06:25:21+00:00
State: open

File K/K2/K21.xml in the BNC contains the following comment string:

<!--<w c5="CRD" hw="10.00" pos="ADJ">10.00</w><c c5="PUN">/</c><w c5="CRD"
hw="8.23/8.54" pos="ADJ">8.23/8.54</w-->

When it encounters the closing tag, the BNC corpus reader raises the
following error and exits.

ValueError: Unmatched tag <w>...</w-->

Migrated from http://code.google.com/p/nltk/issues/detail?id=394

earlier comments

ewan.klein said, at 2009-06-11T12:00:51.000Z:

I should have noticed that the XML above is ill-formed. The BNC corpus reader *does*
handle the comment syntax correctly if the closing tag is amended to: -->
However, it would be nice if there was an option for the corpus reader to deal more
gracefully with ill-formed XML; for example, to have a flag which allows defective
XML tags to be skipped with a warning and file location, rather than terminating the
processing.

StevenBird1 said, at 2009-06-13T07:44:33.000Z:

I think the error is already quite helpful. It might be more helpful if it gave a
line number. Ignoring the defective tag seems prone to cause an even more
cryptic well-formedness or validity error downstream.

ewan.klein said, at 2009-06-13T11:59:51.000Z:

I encountered the problem in the following circumtances: I was trying to construct a
FreqDist of adjectives in the whole BNC, which involves reading > 4000 XML files.
After a couple of hours processing, the program died, with no output apart from the
error message given above -- not very helpful in my view. For diagnosis, I would need
a fileid as well as a line number. I'm not sure what you have in mind as a way of
dealing with the problem once it has been correctly diagnosed. While I could edit the
offending file and fix the ill-formed XML, this doesn't seem to be very good practice
in general. Are you perhaps suggesting the corpus reader should be told to ignore all
of the offending file (3864 lines)?

StevenBird1 said, at 2009-06-13T12:20:44.000Z:

Agreed, a fileid would have been helpful. Since the XML corpus reader can only
process one file at a time, couldn't the calling program easily catch the error and
either ignore it, or print the fileid before raising the same error? That seems more
standard than passing in a flag that tells it whether to silently ignore ill-formed
XML. It also seems like better design than passing the filename around the code
that's parsing an XML stream.

ewan.klein said, at 2009-06-13T12:53:58.000Z:

"Since the XML corpus reader can only process one file at a time..." But once I've
defined a BNC corpus reader instance, I can call e.g. the tagged_words() method on it
without supplying a fileid.

StevenBird1 said, at 2009-06-13T13:34:05.000Z:

I see -- do you think you could attach an example of a good and a bad BNC file
please, to permit a possible solution to be tested?

cmsmcq@blackmesatech.com said, at 2009-09-04T20:08:54.000Z:

Why do you believe the string given is ill-formed? It looks well-formed to me: there is no requirement that
the contents of an XML comment contain well-formed content, only that they do not contain the string "--".
For any string s, if "" is an XML comment, then "" is an XML comment.

So whatever the merits of the issue regarding fileids, the reader in question appears to have a bug in its XML
handling. Or am I missing something here?