Repository: nltk/nltk

Issue:
downloader breaking on unicode issues
At the tip of master right now, on Python 2.7.3, trying to run nltk.download() and then doing 'u' for update, we get a break due to a unicode issue...

```
>>> import nltk
>>> nltk.download()
NLTK Downloader
---------------------------------------------------------------------------
    d) Download   l) List    u) Update   c) Config   h) Help   q) Quit
---------------------------------------------------------------------------
Downloader> u

Will update following packages (o=ok; x=cancel)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alex/nltk/nltk/downloader.py", line 645, in download
    self._interactive_download()
  File "/home/alex/nltk/nltk/downloader.py", line 961, in _interactive_download
    DownloaderShell(self).run()
  File "/home/alex/nltk/nltk/downloader.py", line 1000, in run
    self._simple_interactive_update()
  File "/home/alex/nltk/nltk/downloader.py", line 1047, in _simple_interactive_update
    print('  [ ] %s %s' % (pid.ljust(20, '.'), name))
TypeError: must be char, not unicode
```

It also breaks, in a different way, on Python 3.2.3:
    TypeError: unorderable types: Package() < Package()

Working on a fix for both!



Discussion:
The surface thing that's going wrong in Python 2.7 is that ljust doesn't work when called on a str with a unicode argument. Like so:

```
>>> "foo".ljust(20, u".")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: must be char, not unicode
```

... and all of our string literals are now unicode. The deeper thing going on is that the strings coming from the server in the index XML are str, not unicode ...

Under Python 2.x xml_tree.attrib values are bytestrings for ascii data and unicode for non-ascii data (I think this is a some kind of optimization) so it seems that wrapping values of xml.attrib to compat.text_type (in Package.fromxml) should do the trick.

By the way, I also missed @compat.python_2_unicode_compatible to nltk.downloader.Package (for fixing its `__repr__`).


Resolving patch:
diff --git a/nltk/downloader.py b/nltk/downloader.py
--- a/nltk/downloader.py
+++ b/nltk/downloader.py
@@ -183,6 +183,7 @@
 # Directory entry objects (from the data server's index file)
 ######################################################################
 
+@compat.python_2_unicode_compatible
 class Package(object):
     """
     A directory entry for a downloadable package.  These entries are
@@ -254,12 +255,17 @@ def __init__(self, id, url, name=None, subdir='',
     def fromxml(xml):
         if isinstance(xml, compat.string_types):
             xml = ElementTree.parse(xml)
+        for key in xml.attrib:
+            xml.attrib[key] = compat.text_type(xml.attrib[key])
         return Package(**xml.attrib)
 
+    def __lt__(self, other):
+        return self.id < other.id
+
     def __repr__(self):
         return '<Package %s>' % self.id
 
-
+@compat.python_2_unicode_compatible
 class Collection(object):
     """
     A directory entry for a collection of downloadable packages.
@@ -288,9 +294,14 @@ def __init__(self, id, children, name=None, **kw):
     def fromxml(xml):
         if isinstance(xml, compat.string_types):
             xml = ElementTree.parse(xml)
+        for key in xml.attrib:
+            xml.attrib[key] = compat.text_type(xml.attrib[key])
         children = [child.get('ref') for child in xml.findall('item')]
         return Collection(children=children, **xml.attrib)
 
+    def __lt__(self, other):
+        return self.id < other.id
+
     def __repr__(self):
         return '<Collection %s>' % self.id
 
@@ -685,12 +696,12 @@ def show(s, prefix2=''):
                             show('Downloaded collection %r with errors' %
                                  msg.collection.id)
                         else:
-                            show('Done downloading collection %r' %
+                            show('Done downloading collection %s' %
                                  msg.collection.id)
 
                     # Package downloading messages:
                     elif isinstance(msg, StartPackageMessage):
-                        show('Downloading package %r to %s...' %
+                        show('Downloading package %s to %s...' %
                              (msg.package.id, download_dir))
                     elif isinstance(msg, UpToDateMessage):
                         show('Package %s is already up-to-date!' %
@@ -1585,10 +1596,10 @@ def show(s):
             self._show_progress(None)
             return # halt progress.
         elif isinstance(msg, StartCollectionMessage):
-            show('Downloading collection %r' % msg.collection.id)
+            show('Downloading collection %s' % msg.collection.id)
             self._log_indent += 1
         elif isinstance(msg, StartPackageMessage):
-            show('Downloading package %r' % msg.package.id)
+            show('Downloading package %s' % msg.package.id)
         elif isinstance(msg, UpToDateMessage):
             show('Package %s is up-to-date!' % msg.package.id)
         #elif isinstance(msg, StaleMessage):


Test patch: