[resolved] incomplete file download ? file received capped at 1024MB then stopped.

the link to the questions is here [requests response.iter_content() gets incomplete file ( 1024MB instead of 1.5GB )?](http://stackoverflow.com/questions/23645212/requests-response-iter-content-gets-incomplete-file-1024mb-instead-of-1-5gb)

pasted from stackoverflow:

i have been using this code snippet to download files from a website, so far files smaller than 1GB are all good. but i noticed a 1.5GB file is incomplete

```
# s is requests session object
r = s.get(fileUrl, headers=headers, stream=True)

start_time = time.time()
with open(local_filename, 'wb') as f:
    count = 1
    block_size = 512
    try:
        total_size = int(r.headers.get('content-length'))
        print 'file total size :',total_size
    except TypeError:
        print 'using dummy length !!!'
        total_size = 10000000

    for chunk in r.iter_content(chunk_size=block_size):

        if chunk:  # filter out keep-alive new chunks

            duration = time.time() - start_time
            progress_size = int(count * block_size)
            if duration == 0:
                duration = 0.1
            speed = int(progress_size / (1024 * duration))
            percent = int(count * block_size * 100 / total_size)
            sys.stdout.write("\r...%d%%, %d MB, %d KB/s, %d seconds passed" %
                            (percent, progress_size / (1024 * 1024), speed, duration))

            f.write(chunk)
            f.flush()
            count += 1
```

using latest requests 2.2.1 python 2.6.6, centos 6.4 
the file download always stops at 66.7% 1024MB, what am i missing ?
the output:

```
file total size : 1581244542
...67%, 1024 MB, 5687 KB/s, 184 seconds passed
```

it seems the generator returned by iter_content() thinks all chunks are retrieved and there is no error. btw the exception part did not run, because the server did return the content-length in response header.

edit: after chaning my post url to something like

```
https://xxx.com/yyyfiles.php?userid=xxx&tutid=yyy&sessionid=zzz
```

the files downloads fine.


Can you print out the headers on the response? I'm particularly interested in `r.headers['Content-Encoding']`.


didn't see `Content-Encoding` in response headers
`r.encoding` is None

 request header:
i tried posting `"Range"`, still not working. ( of course because there is no `content-range` in the response header )

```
{'Referer': 'https://xxx.com/yyy.html', 'Accept-Language': 'en-US,en;q=0.5', 'Accept-Encoding': 'gzip, deflate', 'Host': 'xxx.com', 'Accept': '*/*', 'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:28.0) Gecko/20100101 Firefox/28.0', 'DNT': '1', 'Connection': 'keep-alive', 'X-Requested-With': 'XMLHttpRequest', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache', 'Range': 'bytes=0-1584577123', 'Cookie': 'PHPSESSID=xxxxxxxx; __utma=.....utmz=....1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8'}
```

response header:

```
 CaseInsensitiveDict({'content-length': '1584577123', 'content-disposition': 'attachment; filename=xxx.zip', 'content-transfer-encoding': 'binary', 'expires': 'Tue, 03 Jun 2014 08:03:05 GMT', 'server': 'nginx/1.4.1', 'connection': 'keep-alive', 'content-description': 'File Transfer', 'pragma': 'public', 'cache-control': 'max-age=1728000', 'date': 'Wed, 14 May 2014 08:03:05 GMT', 'content-type': 'application/octet-stream'})
```


Well, that's incredibly weird. I was wondering if the content was compressed, but apparently not.

Can you try to confirm that there are actually the full set of bytes in the file? (Wireshark can probably prove that).


i don't have enough skill set to "confirm", i'm just a entry level python user. 

edit: finally i successfully downloaded the full file now. the site is a https site, i used session to keep the cookies.   when i manually download the files using downThemAll firefox addon, the address shows as 

```
https://xxx.com/yyyfiles.php?userid=xxx&tutid=yyy&sessionid=zzz
```

previsouly i was using a different link, after using the above link, the full file is get without stopping at 1GB. i didn't change the code above. and now it works ...

what is the reason behind this.


Hmm, not sure. Try hitting the old URL and printing `r.history`.


Closing for inactivity.


hi,
I'm a beginner in python. Can you please help me in writing a script to download a file using the requests module.


@naveenjindal I'm afraid the issue tracker is not the place to ask for programming assistance, it's the place to report bugs. I recommend resources like reddit.com/r/learnpython. Alternatively, try googling it: I'm sure you'll find an example.


okay. Thank you very much.


@oglops : I too facing same issue, Could you please add or update your code