iter_content not chunking?

I have code which is practically identical to the quickstart code:

```
        with open(self.filename, 'wb') as fd:
            for chunk in self.req.iter_content(self.chunk_size):
                fd.write(chunk)
                if self.progressbar:
                    self.in_queue.put(len(chunk))
```

where the queue is read in another thread by the GUI object and updates the progress bar.  The problem that I have noticed is that despite the chunk_size being 1024, this code seems to wait until until the code is done and then puts the entire file size in the queue, defeating the point of providing the user with continuous progress on feedback.


@coyotlinden Do you make the original requests call with `stream=True`?


Yes.

```
req = requests.get(url, stream=True)
```


@coyotlinden And that `req` is the same as the one that is called `self.req` elsewhere? How big is the file? Is the URL publicly available?


- Same `req`? Yes.  The `req` is passed to a subclass of `threading.Thread` where it is referred to as `self.req` .
- How big is the file?  90,728,799 bytes
- Is the URL public?  Yes.  http://download.cloud.secondlife.com/Viewer_4/Second_Life_4_0_5_315117_i386.dmg 


Are you absolutely sure that `self.chunk_size` is 1024? Testing that locally I get a series of 1024-byte chunks, until a final 351-byte chunk. Can you throw a print or a logging statement in there to confirm what `self.chunk_size` is?


I was using the Wing IDE debugger to check the value, but will put in a print statement after lunch.


Given:

```
    def run(self):
        with open(self.filename, 'wb') as fd:
            for chunk in self.req.iter_content(self.chunk_size):
                fd.write(chunk)
                print "chunk_size: %s chunk length: %s" % (self.chunk_size, len(chunk))
                if self.progressbar:
                    self.in_queue.put(len(chunk))
```

I get:

`chunk_size: 1024 chunk length: 90728799`


Are you sending custom headers or anything?


Nope.


Ok, when I run this I see repeatedly 1024-byte sized chunks:

``` python
import requests


CHUNK_SIZE = 1024


def run():
    req = requests.get('http://download.cloud.secondlife.com/Viewer_4/Second_Life_4_0_5_315117_i386.dmg', stream=True)
    with open('out.bin', 'wb') as fd:
        for chunk in req.iter_content(CHUNK_SIZE):
            fd.write(chunk)
            print "chunk_size: %s chunk length: %s" % (CHUNK_SIZE, len(chunk))


run()
```

Can you try running that and see what your output is?


Given:

```
        """"
        with open(self.filename, 'wb') as fd:
            for chunk in self.req.iter_content(self.chunk_size):
                fd.write(chunk)
                print "chunk_size: %s chunk length: %s" % (self.chunk_size, len(chunk))
                if self.progressbar:
                    self.in_queue.put(len(chunk))"""
        CHUNK_SIZE = 1024
        req = requests.get('http://download.cloud.secondlife.com/Viewer_4/Second_Life_4_0_5_315117_i386.dmg', stream=True)
        with open('out.bin', 'wb') as fd:
            for chunk in req.iter_content(CHUNK_SIZE):
                fd.write(chunk)
                print "chunk_size: %s chunk length: %s" % (CHUNK_SIZE, len(chunk))   
```

I _do_ see the correct output.  I'll try and do some incremental swapping back to my previous and see if I can't isolate the problem.


This is very strange.  Just changing the chunk size variable from a local constant to the passed in value breaks it:

```
#CHUNK_SIZE = 1024
        req = requests.get('http://download.cloud.secondlife.com/Viewer_4/Second_Life_4_0_5_315117_i386.dmg', stream=True)
        with open('out.bin', 'wb') as fd:
            for chunk in req.iter_content(self.chunk_size):
                fd.write(chunk)
                print "chunk_size: %s chunk length: %s" % (self.chunk_size, len(chunk))
```

Gets `chunk_size: 1024 chunk length: 90728799`


@coyotlinden What is `type(self.chunk_size)`?


Ah ha!  `chunk type: <type 'str'> chunk_size: 1024 chunk length: 90728799`


Fixed with `self.chunk_size = int(chunk_size)`


=D There we go!


I'd be inclined to say that we should maybe `TypeError` on stupid inputs there.


I'd +1 that change!

This now works _perfectly_:

```
    def run(self):
        with open(self.filename, 'wb') as fd:
            for chunk in self.req.iter_content(self.chunk_size):
                fd.write(chunk)
                if self.progressbar:
                    self.in_queue.put(len(chunk))                       
            self.in_queue.put(-1)
```

Many thanks!


> I'd be inclined to say that we should maybe TypeError on stupid inputs there.

I'd be inclined to agree.


I threw together a quick check and a couple tests for non-int chunk_sizes in PR #3365


Resolved by #3365.