File download weirdness

I noticed this issue building conda recipes.  Conda uses requests to download files from the internet.

The file that is being fetched is: https://dakota.sandia.gov/sites/default/files/distributions/public/dakota-6.5-public.src.tar.gz
(link found here: https://dakota.sandia.gov/download.html)

Downloading with curl -O
filesize: 78MB
md5: 02c46e904d40bba6b308065db34c1ad7

Downloading with urllib2 (from the standard library):
filesize: 78MB
md5: 02c46e904d40bba6b308065db34c1ad7

Downloading with requests-2.12.1 (supplied with conda)
filesize: 248MB
md5: 41e4268140d850756812510512d8eee8
tar -tf doesn't indicate any corruption.

I'm not sure what is different with this particular URL, but the other  files I tried with requests worked.  I don't know where the extra 170MB is coming from?

code used to download files:
```python
def download_file(url, fn):
    r = requests.get(url, stream=True)
    with open(fn, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024): 
            if chunk:
                f.write(chunk)

def download_urllib2(url, fn):
    f = urllib2.urlopen(url)
    with open(fn, 'wb') as fh:
        for x in iter(lambda: f.read(1024), b''):
            fh.write(x)
```

My best guess is that this is related to compression. Can you print the response headers from curl and from Requests please?

curl
```
HTTP/1.1 200 OK
Date: Thu, 01 Dec 2016 14:44:30 GMT
Server: Apache/2.2.15 (Red Hat)
X-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Last-Modified: Tue, 15 Nov 2016 17:15:23 GMT
ETag: "3000ae5-4d831c8-5415a1a05c0c0"
Accept-Ranges: bytes
Content-Length: 81277384
Cache-Control: max-age=1209600
Expires: Thu, 15 Dec 2016 14:44:30 GMT
Content-Disposition: attachment
Content-Type: application/x-gzip
Content-Encoding: x-gzip
```

Requests:
```
[('Accept-Ranges', 'bytes'),
 ('Cache-Control', 'max-age=1209600'),
 ('Connection', 'Keep-Alive'),
 ('Content-Disposition', 'attachment'),
 ('Content-Encoding', 'gzip'),
 ('Content-Length', '81277384'),
 ('Content-Type', 'application/x-gzip'),
 ('Date', 'Thu, 01 Dec 2016 14:35:58 GMT'),
 ('ETag', '"3000ae5-4d831c8-5415a1a05c0c0"'),
 ('Expires', 'Thu, 15 Dec 2016 14:35:58 GMT'),
 ('Keep-Alive', 'timeout=30, max=2500'),
 ('Last-Modified', 'Tue, 15 Nov 2016 17:15:23 GMT'),
 ('Server', 'Apache/2.2.15 (Red Hat)'),
 ('X-Content-Type-Options', 'nosniff'),
 ('X-Frame-Options', 'SAMEORIGIN')]
```

From urllib2
```
[('accept-ranges', 'bytes'),
 ('cache-control', 'max-age=1209600'),
 ('connection', 'close'),
 ('content-disposition', 'attachment'),
 ('content-encoding', 'x-gzip'),
 ('content-length', '81277384'),
 ('content-type', 'application/x-gzip'),
 ('date', 'Thu, 01 Dec 2016 14:36:09 GMT'),
 ('etag', '"3000ae5-4d831c8-5415a1a05c0c0"'),
 ('expires', 'Thu, 15 Dec 2016 14:36:09 GMT'),
 ('last-modified', 'Tue, 15 Nov 2016 17:15:23 GMT'),
 ('server', 'Apache/2.2.15 (Red Hat)'),
 ('x-content-type-options', 'nosniff'),
 ('x-frame-options', 'SAMEORIGIN')]
```


Yup, we definitely think it's 81mb too. Do you want to diff the file output to see what is different in them?

Oh, nevermind, I think I know what's going on here.

By default, Requests will decompress the file. This is because it's been sent with Content-Encoding: gzip. The remote peer really shouldn't be saying `Content-Encoding: gzip` *and* `Content-Type: application/x-gzip`: that implies that the file is a gzipped gzip file, which is presumably not what is meant.

The way to deal with this, if you can't coerce the server to be less stupid, is to use `r.raw.read()` in a loop instead.

Thank you for the tip.