Repository: nltk/nltk

Issue:
XML corpus reader fails on XML comments
File K/K2/K21.xml in the BNC contains the following comment string:

&lt;!--&lt;w c5=&quot;CRD&quot; hw=&quot;10.00&quot; pos=&quot;ADJ&quot;&gt;10.00&lt;/w&gt;&lt;c c5=&quot;PUN&quot;&gt;/&lt;/c&gt;&lt;w c5=&quot;CRD&quot;
hw=&quot;8.23/8.54&quot; pos=&quot;ADJ&quot;&gt;8.23/8.54&lt;/w--&gt;

When it encounters the closing tag, the BNC corpus reader raises the
following error and exits.

ValueError: Unmatched tag &lt;w&gt;...&lt;/w--&gt; 

<p>Migrated from http://code.google.com/p/nltk/issues/detail?id=394</p><hr/><h2>earlier comments</h2>
<p><strong>ewan.klein said, at 2009-06-11T12:00:51.000Z:</strong></p>

<p>I should have noticed that the XML above is ill-formed. The BNC corpus reader *does*
handle the comment syntax correctly if the closing tag is amended to: </w>-->
However, it would be nice if there was an option for the corpus reader to deal more
gracefully with ill-formed XML; for example, to have a flag which allows defective
XML tags to be skipped with a warning and file location, rather than terminating the
processing.</p><p><strong>StevenBird1 said, at 2009-06-13T07:44:33.000Z:</strong></p>

<p>I think the error is already quite helpful.  It might be more helpful if it gave a
line number.  Ignoring the defective </w--> tag seems prone to cause an even more
cryptic well-formedness or validity error downstream.
</p><p><strong>ewan.klein said, at 2009-06-13T11:59:51.000Z:</strong></p>

<p>I encountered the problem in the following circumtances: I was trying to construct a
FreqDist of adjectives in the whole BNC, which involves reading > 4000 XML files.
After a couple of hours processing, the program died, with no output apart from the
error message given above -- not very helpful in my view. For diagnosis, I would need
a fileid as well as a line number. I'm not sure what you have in mind as a way of
dealing with the problem once it has been correctly diagnosed. While I could edit the
offending file and fix the ill-formed XML, this doesn't seem to be very good practice
in general. Are you perhaps suggesting the corpus reader should be told to ignore all
of the offending file (3864 lines)? </p><p><strong>StevenBird1 said, at 2009-06-13T12:20:44.000Z:</strong></p>

<p>Agreed, a fileid would have been helpful.  Since the XML corpus reader can only
process one file at a time, couldn't the calling program easily catch the error and
either ignore it, or print the fileid before raising the same error?  That seems more
standard than passing in a flag that tells it whether to silently ignore ill-formed
XML.  It also seems like better design than passing the filename around the code
that's parsing an XML stream.


</p><p><strong>ewan.klein said, at 2009-06-13T12:53:58.000Z:</strong></p>

<p>"Since the XML corpus reader can only process one file at a time..." But once I've
defined a BNC corpus reader instance, I can call e.g. the tagged_words() method on it
without supplying a fileid.




</p><p><strong>StevenBird1 said, at 2009-06-13T13:34:05.000Z:</strong></p>

<p>I see -- do you think you could attach an example of a good and a bad BNC file
please, to permit a possible solution to be tested?
</p><p><strong>cmsmcq@blackmesatech.com said, at 2009-09-04T20:08:54.000Z:</strong></p>

<p>Why do you believe the string given is ill-formed?  It looks well-formed to me:  there is no requirement that 
the contents of an XML comment contain well-formed content, only that they do not contain the string "--". 
For any string s, if "<!--" followed by s followed by ">" followed by "-->" is an XML comment, then "<!--" || s 
|| "-->" is an XML comment.

So whatever the merits of the issue regarding fileids, the reader in question appears to have a bug in its XML 
handling.  Or am I missing something here?</p>



Discussion:
Whether the comment is well-formed or not is a red herring.  The problem is that _VALID_XML_RE in xmldocs.py is happy with XML fragments like:

``` xml
<w>xx </w><w>yy </w><!-- <w>zz<w/>
```

and then the fact that zz is supposed to be part of a comment gets lost.  It happens that in this case the contents of the comment is almost but not quite valid XML so that read_block() tries to process it and fails, when it shouldn't be trying to interpret the comment at all.

I've just committed a patch that fixes this: if a fragment contains <!-- then it also has to contain a corresponding -->.

But, this all raises the question: why are we trying to parse XML with regular expressions?  Wouldn't it make more sense to use a real SAX XML parser?  

@rmalouf Shall we pull this fix, do you think?

Sure, go ahead (if you're asking me...)!  It's probably not an ideal way of handling XML, but at least the BNC reader will work.

is this fix in the current development branch? it seems to be not there https://github.com/nltk/nltk/blob/develop/nltk/corpus/reader/xmldocs.py#L206


Resolving patch:
diff --git a/nltk/corpus/reader/xmldocs.py b/nltk/corpus/reader/xmldocs.py
--- a/nltk/corpus/reader/xmldocs.py
+++ b/nltk/corpus/reader/xmldocs.py
@@ -209,7 +209,7 @@ def handle_elt(self, elt, context):
           ((<!--.*?-->)                         |  # comment
            (<![CDATA[.*?]])                     |  # raw character data
            (<!DOCTYPE\s+[^\[]*(\[[^\]]*])?\s*>) |  # doctype decl
-           (<[^>]*>))                              # tag or PI
+           (<[^!>][^>]*>))                         # tag or PI
           [^<]*)*
         \Z""",
         re.DOTALL|re.VERBOSE)
@@ -245,9 +245,9 @@ def _read_xml_fragment(self, stream):
         """
         fragment = ''
 
+        if isinstance(stream, SeekableUnicodeStreamReader):
+            startpos = stream.tell()
         while True:
-            if isinstance(stream, SeekableUnicodeStreamReader):
-                startpos = stream.tell()
             # Read a block and add it to the fragment.
             xml_block = stream.read(self._BLOCK_SIZE)
             fragment += xml_block


Test patch: