"Pushable" streaming upload requests

I have an interesting situation which doesn't seem to be covered by the `requests` API. For a project I'm working on, I have a server which receives a multipart upload from a user in chunks in a handler which looks and works something like this:

```
class MyUploadHandler(BaseUploadHandler):

    def on_file_started(self, file_name):
        pass

    def on_file_chunk(self, chunk):
        pass

    def on_file_finished(self, file_size):
        pass
```

What I'd like to do is something like this:

```
import requests

class MyUploadHandler(BaseUploadHandler):

    def on_file_started(self, file_name):
        self.request = requests.put("http://secondary.com/upload/url", stream=True)

    def on_file_chunk(self, chunk):
        self.request.raw.write(chunk)

    def on_file_finished(self, file_size):
        self.request.finish()
```

It seems that this is currently not possible in the API, though I could be wrong. If I have to rewrite my code to incorporate a [file-like object](http://docs.python-requests.org/en/latest/user/advanced/#streaming-uploads) or a [generator](http://docs.python-requests.org/en/latest/user/advanced/#chunk-encoded-requests), I'd probably have to get my hands dirty with threading and this would make everything 1,000,000 times harder in my application than they should be.

Is what I'm talking about currently possible in `requests`, and if not, can the feature be easily added?


**tl;dr** It can not be easily added as far as I know and likely won't be considered by Kenneth given we're in a feature freeze.

Something about the API you show in the second snippet is oddly attractive but not at all what is within the realm of possibility for requests. I think a generator is your best bet in this instance but without knowing exactly what you're doing (beyond this) I can't speak as to the need for threading (although frankly I'm unsure of how that would be necessary for a generator).

With respect to the feature freeze please see #1165. As for adding this, all we do in requests is prepare a request to be sent using urllib3. You may be able to do something like this using urllib3 but the initial API will not be as simple as ours. It's a great library though and I think it has more functionality than what requests exposes to be frank. @shazow could definitely confirm or deny the ability to do this there though.


Since I haven't really done work like this in Python before (ie: I'm pretty new and confused by generators) I'm not sure where to start when converting my push to a pull.

Essentially, the data is pushed to me in my handler. I need to essentially write a generator which blocks while reading from a queue of some sort. (If you'd like to help me work this out, you can check out my [question on SO](http://stackoverflow.com/questions/15557841/streaming-upload-request)) As new data becomes available and the `on_file_chunk` method is called, the handler should provide data to the generator which should be able to provide data to the request for the request body. Clear as mud, right?

Essentially, I need something like this:

```
import requests

class MyUploadHandler(BaseUploadHandler):

    def on_file_started(self, file_name):
        self.request = requests.put("http://secondary.com/upload/url", data=self.generate_data())

    def generate_data(self):
        # block until data is available. when it is, yield it
        ...    

    def on_file_chunk(self, chunk):
        # send the data to the generator so that it can be returned to the request
        ...

    def on_file_finished(self, file_size):
       # inform the generator that it's 'done'
```

This part, crossing over between data being pushed to data being pulled, is what's stumping me currently. It's okay that you might not implement the functionality, but I'm totally in the dark on how to make this happen. 

Any ideas? Feel free to jump in [on SO](http://stackoverflow.com/questions/15557841/streaming-upload-request).


In reference to this feature request, what do you guys consider a file-like object? For example, which methods are critical for `requests` to treat an object passed as `data` as a file-like object?

For example, what objects should I implement in this class:

```
class FlieLike(object):
    pass

pseudo_file = FileLike()

requests.put("http://outgoing.com/put", data=pseudo_file)
```

Is `__iter__` enough? Should I instead implement `next`?


Actually, `__iter__` and `next` make it an iterable, not a file-like object. Is there any reason you can't block until all the chunks are there and then just build a generator from them?


File-like objects require at least (I think) a `read` method and I think we also look for a `seek` method so we can determine `Content-Length`. We don't determine that (or chunk) the upload when the object has the `__iter__` method but is not any sort of string, list or dict.

You could build an object that is a generator easily.

``` python
from collections import Iterator

class MyGenerator(Iterator):
    def __init__(self, connection):
        self.connection = connection

    def __iter__(self):
        while True
            chunk = self.connection.read()  # blocks
            if chunk:  # e.g., if there's something to send
                yield chunk
            else:  # what to do otherwise
                break  # you don't necessarily have to do this
```

Then you would do something roughly like:

``` python
conn = incoming_data()
response = requests.put(url, data=MyGenerator(conn))
```

The `read` should be a blocking method, not a non-blocking `read`, and that should save you. You might need to define (depending on your version of python) either `__next__` or `next`. For that, you can just check out a similar generator I wrote [here](https://github.com/sigmavirus24/github3.py/blob/develop/github3/structs.py#L5). (It isn't exactly the same and isn't simple enough to just show you it for the above purposes, it does a lot more work than you want to do, otherwise I just would have pointed you in that direction.)


@Lukasa I can't wait until all the chunks are there, because this is a file upload server and it's imperative that only the current chunk from the client stays in memory at any one given time. If they were all being saved, writing a generator or a file-like object would be a cakewalk. 

@sigmavirus24 I can't reliably provide `Content-Length`, as I'm accepting uploads from users and I can't know for certain the size of the files they're uploading. They could easily upload a 4GB file and lie to me about the content length being 1024B. 

Your example makes a lot more sense of the problem than I've seen so far, so kudos on that! Essentially, it would seem that I'd have to implement a blocking queue and a thread to make this happen:

```
from collections import Iterator

import requests

class MyUploadHandler(BaseUploadHandler):

    is_running = True    

    def setup_request(self):
        self.request = requests.put("http://myurl.com/put", data=MyGenerator(self.queue, self))

    def on_file_started(self, file_name):
        self.queue = Queue(maxsize=1)
        self.upload_thread = Thread(target=self.setup_request, args=(self))
        self.upload_thread.start()

    def on_file_chunk(self, chunk):
        self.queue.put(chunk)

    def on_file_finished(self, filesize):
        self.is_running = False

class MyGenerator(Iterator):

    def __init__(self, queue, upload_handler):
        self.queue = queue
        self.upload_handler = upload_handler

    def __iter__(self):
        while self.upload_handler.is_running:
            yield queue.get()
```

Something like the above would seem to be able to solve my problem, right? Essentially, when a new file comes down the pipe, an upload request is generated in another thread so that it won't block the main thread. Immediately after being created, it attempts to get from the queue, which blocks because it's empty. Then, when the upload handler receives a chunk, it puts it into the queue, which is read by the request. The put will block if there's more than one chunk in the queue, so this will manage memory well. This will continue until the upload is done. Once it's done, we simply set `is_running` to `False`, which breaks the iterator's loop and the thread (presumably) exits gracefully.

The only problem I can see so far is trying to handle the case in which a request failed. We'd have to check in the `on_file_chunk` method and break the whole process if the request failed. Does this seem to work? Like I said, I'm  a bit new at this whole generator thing, but I understand threading fairly well. 


Sorry for the confusion. I don't believe that with the chunked encoding you have to provide `Content-Length`. I meant that was one thing having a file-like object was used for.

I don't quite know python Threads so well, but I'm going to take a few shots in the dark. If there's an issue when you start the thread (when the request is made), you'll need to catch any exceptions from the call and set `is_running` to False. I don't know how Threads raise exceptions (i.e. if an exception was raised from requests how would that affect the running of the thread) so I don't know if there's a better way of doing it.

I am also not 100% sure that you need the Threads. Can you not wait for the first chunk (at least) and then start the put request? Doing that you'd have the first one so your generator would return a chunk immediately and then it would get caught on the read from your client(s). This would have the added benefit that if something went wrong on either end, you wouldn't have to worry about cleaning up threads.


I'd really like to not have to worry about threading if possible, but I'm not sure how to facilitate it. I guess I'm not understanding your logic. Essentially, I need to have the iterator block until another chunk is ready, and if the request is running in the same thread as my handler, if anything becomes blocked, nothing happens and the whole program hangs. 

Can you give me an example of what you're talking about? It's definitely possible for me to start with the first chunk, I'm just not sure about how the implementation would look like for the generator. 


I guess I'm thinking of something like:

``` python
class MyUploadHandler(BaseUploadHander):
     def generate_data(self):
          # yields data
          pass

     def on_file_started(self):
          self.response = requests.put(url, data=self.generate_data())
```

Honestly, I wouldn't think the issue of `generate_data` blocking would be negative since that's what you want. And if the `put` request fails then you're process will hang either way. You say you can not keep the entire file in memory so assuming there are two threads, if one hangs the other still can't proceed unless it goes on to try a second put request with the rest of the data. If that's your plan/design then yes you absolutely must use Threads, otherwise I think you might be able to use a simpler design like above.


@sigmavirus24 In your example, how then would I pass data received in `on_file_chunk` to `generate_data`?

Since I imagine that the call to `requests.put` blocks until completed, I would imagine that this code would essentially immediately hang. Looks like I'm working with threads.