Repository: pypa/pip

Issue:
Handle encoding gzip/deflate - causing python 3.x tests failing with UnicodeDecodeError
There seems to be an issue here - potentially a threading issue on py3k when running the tests we get

```
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.1/threading.py", line 516, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.1/threading.py", line 469, in run
    self._target(*self._args, **self._kwargs)
  File "/var/lib/hudson/jobs/pip_python3.1/workspace/tests/tests_cache/test_ws/.virtualenv/lib/python3.1/site-packages/pip-1.0.1-py3.1.egg/pip/index.py", line 239, in _get_queued_page
    page = self._get_page(location, req)
  File "/var/lib/hudson/jobs/pip_python3.1/workspace/tests/tests_cache/test_ws/.virtualenv/lib/python3.1/site-packages/pip-1.0.1-py3.1.egg/pip/index.py", line 324, in _get_page
    return HTMLPage.get_page(link, req, cache=self.cache)
  File "/var/lib/hudson/jobs/pip_python3.1/workspace/tests/tests_cache/test_ws/.virtualenv/lib/python3.1/site-packages/pip-1.0.1-py3.1.egg/pip/index.py", line 445, in get_page
    inst = cls(u(resp.read()), real_url, headers)
  File "/var/lib/hudson/jobs/pip_python3.1/workspace/tests/tests_cache/test_ws/.virtualenv/lib/python3.1/site-packages/pip-1.0.1-py3.1.egg/pip/backwardcompat.py", line 58, in u
    return s.decode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte
```

Running under pdb showed the page it was downloading was the pip homepage ie doing something like

``` python
resp = urlopen('http://www.pip-installer.org/en/latest/')
s = resp.read()
s.decode('utf-8')
```



Discussion:
Note this patch wallpapers over the failure, but it'd be good to get to the root cause here 

https://gist.github.com/1004525

OK so saw some mention on the web a. that it might be gzipped, and 0x1f 0x8b is the start of a gzip encoded file

 I tried a tcpdump and that looked ok

GET /en/latest/ HTTP/1.1
Host: www.pip-installer.org
User-Agent: Python-urllib/3.2
Connection: close
Accept-Encoding: identity

HTTP/1.1 200 OK
Server: nginx/0.8.54
Content-Type: text/html
Last-Modified: Sun, 29 May 2011 12:11:33 GMT
Content-Length: 29095
Date: Thu, 02 Jun 2011 15:35:46 GMT
X-Varnish: 319516731 318923294
Age: 357612
Via: 1.1 varnish
Connection: close

GET /en/latest/index.html HTTP/1.1
Host: www.pip-installer.org
User-Agent: Python-urllib/3.2
Connection: close
Accept-Encoding: identity

HTTP/1.1 200 OK
Server: nginx/0.8.54
Content-Type: text/html
Last-Modified: Sun, 29 May 2011 12:11:33 GMT
Content-Encoding: gzip
Content-Length: 9081
Date: Thu, 02 Jun 2011 15:46:04 GMT
X-Varnish: 319517972 318923136
Age: 358332
Via: 1.1 varnish
Connection: close

...........].r.........#."..l.mY..-..U|Q$9...)-H6I.@.....I..........q!)[..l.j...H.}..............N..$......7'G...v.

Simple reproducer now:

```
python piptest.py 
Traceback (most recent call last):
  File "piptest.py", line 10, in <module>
    getpiphome()
  File "piptest.py", line 8, in getpiphome
    s = u(resp.read())
  File "/Users/pnasrat/Development/pip/pip/backwardcompat.py", line 61, in u
    return s.decode('utf-8')
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1: invalid start byte
```

``` python
#!/usr/bin/env python
from pip.download import urlopen, geturl
from pip.backwardcompat import u

def getpiphome():
    resp = urlopen('http://www.pip-installer.org/en/latest/index.html')
    url = geturl(resp)
    s = u(resp.read())

getpiphome()
```

Looks like bad caching on pip-installer.org can reproduce with netcat and

```
GET /en/latest/index.html HTTP/1.1
Host: www.pip-installer.org
```


Resolving patch:
diff --git a/pip/backwardcompat.py b/pip/backwardcompat.py
--- a/pip/backwardcompat.py
+++ b/pip/backwardcompat.py
@@ -37,7 +37,7 @@ def any(seq):
 console_encoding = sys.__stdout__.encoding
 
 if sys.version_info >= (3,):
-    from io import StringIO
+    from io import StringIO, BytesIO
     from functools import reduce
     from urllib.error import URLError, HTTPError
     from queue import Queue, Empty
@@ -91,6 +91,7 @@ def console_to_str(s):
     reduce = reduce
     cmp = cmp
     raw_input = raw_input
+    BytesIO = StringIO
 
 try:
     from email.parser import FeedParser
diff --git a/pip/index.py b/pip/index.py
--- a/pip/index.py
+++ b/pip/index.py
@@ -3,6 +3,7 @@
 import sys
 import os
 import re
+import gzip
 import mimetypes
 import threading
 import posixpath
@@ -10,11 +11,12 @@
 import random
 import socket
 import string
+import zlib
 from pip.log import logger
 from pip.util import Inf
 from pip.util import normalize_name, splitext
 from pip.exceptions import DistributionNotFound
-from pip.backwardcompat import (WindowsError,
+from pip.backwardcompat import (WindowsError, BytesIO,
                                 Queue, httplib, urlparse,
                                 URLError, HTTPError, u,
                                 product, url2pathname)
@@ -442,7 +444,15 @@ def get_page(cls, link, req, cache=None, skip_archives=True):
 
             real_url = geturl(resp)
             headers = resp.info()
-            inst = cls(u(resp.read()), real_url, headers)
+            contents = resp.read()
+            encoding = headers.get('Content-Encoding', None)
+            #XXX need to handle exceptions and add testing for this
+            if encoding is not None:
+                if encoding == 'gzip':
+                    contents = gzip.GzipFile(fileobj=BytesIO(contents)).read()
+                if encoding == 'deflate':
+                    contents = zlib.decompress(contents)
+            inst = cls(u(contents), real_url, headers)
         except (HTTPError, URLError, socket.timeout, socket.error, OSError, WindowsError):
             e = sys.exc_info()[1]
             desc = str(e)


Test patch: