Repository: pypa/pip

Issue:
Fix file corruption on download
I am getting a lot of invalid hash errors with pip 1.5.4.

For instance, when I try to install Twisted from a local proxy, it claims that md5 hash is invalid:

```
$ pip install Twisted==13.2.0 --index-url=http://pypi.example.com/proxy
Downloading/unpacking Twisted==13.2.0
  http://pypi.example.com/proxy/Twisted/13.2.0 uses an insecure transport scheme (http). Consider using https if pypi.example.com has it available
  http://pypi.example.com/proxy/Twisted/ uses an insecure transport scheme (http). Consider using https if pypi.example.com has it available
  Downloading Twisted-13.2.0.tar.bz2 (unknown size): 2.7MB downloaded
  Hash of the package http://pypi.example.com/proxy/packages/source/T/Twisted/Twisted-13.2.0.tar.bz2#md5=83fe6c0c911cc1602dbffb036be0ba79 (from http://pypi.example.com/proxy/simple/Twisted/) (876f102623d87f451c86d260f49c3ec7) doesn't match the expected hash 83fe6c0c911cc1602dbffb036be0ba79!
Cleaning up...
Bad md5 hash for package http://pypi.example.com/proxy/packages/source/T/Twisted/Twisted-13.2.0.tar.bz2#md5=83fe6c0c911cc1602dbffb036be0ba79 (from http://pypi.example.com/proxy/simple/Twisted/)
Storing debug log for failure in /opt/puppet/.pip/pip.log
```

But if I download the file directly from my proxy, it is just fine:

```
$ wget http://pypi.example.com/proxy/packages/source/T/Twisted/Twisted-13.2.0.tar.bz2#md5=83fe6c0c911cc1602dbffb036be0ba79 -O remote.tar.bz2
--2014-03-26 10:35:09--  http://pypi.example.com/proxy/packages/source/T/Twisted/Twisted-13.2.0.tar.bz2
Resolving pypi.example.com... 192.168..129.187
Connecting to pypi.example.com|192.168.129.187|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2704819 (2.6M) [text/html]
Saving to: `remote.tar.bz2'

100%[====================================================================================================================================================================================================>] 2,704,819   --.-K/s   in 0.09s

2014-03-26 10:35:10 (27.9 MB/s) - `remote.tar.bz2' saved [2704819/2704819]

$ md5sum remote.tar.bz2
83fe6c0c911cc1602dbffb036be0ba79  remote.tar.bz2
```

And the file download by pip is really corrupted:

```
$ sudo md5sum /tmp/pip-gvpeUI-unpack/Twisted-13.2.0.tar.bz2
876f102623d87f451c86d260f49c3ec7  /tmp/pip-gvpeUI-unpack/Twisted-13.2.0.tar.bz2
```

My change has fixed the problem, but I don't know what was the exact reason that was causing the error.



Discussion:

Resolving patch:
diff --git a/pip/download.py b/pip/download.py
--- a/pip/download.py
+++ b/pip/download.py
@@ -447,7 +447,30 @@ def resp_read(chunk_size):
                 # Special case for urllib3.
                 try:
                     for chunk in resp.raw.stream(
-                            chunk_size, decode_content=False):
+                            chunk_size,
+                            # We use decode_content=False here because we do
+                            # want urllib3 to mess with the raw bytes we get
+                            # from the server. If we decompress inside of
+                            # urllib3 then we cannot verify the checksum
+                            # because the checksum will be of the compressed
+                            # file. This breakage will only occur if the
+                            # server adds a Content-Encoding header, which
+                            # depends on how the server was configured:
+                            # - Some servers will notice that the file isn't a
+                            #   compressible file and will leave the file alone
+                            #   and with an empty Content-Encoding
+                            # - Some servers will notice that the file is
+                            #   already compressed and will leave the file
+                            #   alone and will add a Content-Encoding: gzip
+                            #   header
+                            # - Some servers won't notice anything at all and
+                            #   will take a file that's already been compressed
+                            #   and compress it again and set the
+                            #   Content-Encoding: gzip header
+                            #
+                            # By setting this not to decode automatically we
+                            # hope to eliminate problems with the second case.
+                            decode_content=False):
                         yield chunk
                 except IncompleteRead as e:
                     raise ChunkedEncodingError(e)
@@ -582,7 +605,30 @@ def unpack_http_url(link, location, download_cache, download_dir=None,
     # let's download to a tmp dir
     if not temp_location:
         try:
-            resp = session.get(target_url, stream=True)
+            resp = session.get(
+                target_url,
+                # We use Accept-Encoding: identity here because requests
+                # defaults to accepting compressed responses. This breaks in
+                # a variety of ways depending on how the server is configured.
+                # - Some servers will notice that the file isn't a compressible
+                #   file and will leave the file alone and with an empty
+                #   Content-Encoding
+                # - Some servers will notice that the file is already
+                #   compressed and will leave the file alone and will add a
+                #   Content-Encoding: gzip header
+                # - Some servers won't notice anything at all and will take
+                #   a file that's already been compressed and compress it again
+                #   and set the Content-Encoding: gzip header
+                # By setting this to request only the identity encoding We're
+                # hoping to eliminate the third case. Hopefully there does not
+                # exist a server which when given a file will notice it is
+                # already compressed and that you're not asking for a
+                # compressed file and will then decompress it before sending
+                # because if that's the case I don't think it'll ever be
+                # possible to make this work.
+                headers={"Accept-Encoding": "identity"},
+                stream=True,
+            )
             resp.raise_for_status()
         except requests.HTTPError as exc:
             logger.fatal("HTTP error %s while getting %s" %


Test patch:
diff --git a/tests/unit/test_download.py b/tests/unit/test_download.py
--- a/tests/unit/test_download.py
+++ b/tests/unit/test_download.py
@@ -109,6 +109,7 @@ def test_unpack_http_url_bad_cache_checksum(mock_unpack_file):
         # despite existence of cached file with bad hash, downloaded again
         session.get.assert_called_once_with(
             "http://www.example.com/somepackage.tgz",
+            headers={"Accept-Encoding": "identity"},
             stream=True,
         )
         # cached file is replaced with newly downloaded file
@@ -151,6 +152,7 @@ def test_unpack_http_url_bad_downloaded_checksum(mock_unpack_file):
         # despite existence of downloaded file with bad hash, downloaded again
         session.get.assert_called_once_with(
             'http://www.example.com/somepackage.tgz',
+            headers={"Accept-Encoding": "identity"},
             stream=True,
         )
         # cached file is replaced with newly downloaded file