Async mode seems to have disappeared

Hey,
I'm trying to find some kind of documentation about how to issue multiple concurrent HTTP requests using this library, but all the Googling and searching I'm doing points to outdated versions of this library. Is there a chance you can add back the docs? Even if they just mention the features are deprecated and point users in the correct location, it's probably useful.

Best,
Kevin


Hi Kevin, thanks for raising this issue!

I'd have to say that [grequests](https://github.com/kennethreitz/grequests) is likely to be the right way to go about this. It might be worth grequests to the documentation, in fact. I'll have a think about it. =)


There is also https://pypi.python.org/pypi/requests-futures/. I haven't used it, so no promises.


Sure... the grequests documentation is fairly lacking though, https://github.com/kennethreitz/grequests

I keep getting an error using grequests with requests version 1.2.2, I'll raise an issue on the other repo.


I'm adding this to my list of documentation changes I'd like to make.


I've added mentions of grequests and requests-futures to the documentation in #1413. Let me know if this is not sufficient. =)


Is there any desire from the community to have async support added to requests?  It's something I am looking for an would be willing to contribute back


We have some unofficial plans to unveil something around async within the next year, @Lukasa and I were discussing this at PyCon.

It's something that will take some time and a lot of energy to introduce, and would no longer be a "tacked on" bit of functionality like the original "requests.async" (now known as "grequests") was. 


I added some comments here, which I will re-iterate: https://github.com/kennethreitz/requests/issues/1390#issuecomment-224797206

---

We have some unofficial plans to unveil something around async within the next year, @Lukasa and I were discussing this at PyCon.

It's something that will take some time and a lot of energy to introduce, and would no longer be a "tacked on" bit of functionality like the original "requests.async" (now known as "grequests") was.


@kennethreitz @Lukasa It's something that I seriously need so am happy to contribute.  Would you be willing to take help contributing to it?  I am CTO of my company and have been using python for years.


Also depends on the type of help you want, don't mind taking a back seat on the design if that's a concern


So, the problem with having an async mode is that we need to decide _how_ to have the async mode. _In principle_ the way to do that is coroutine based, but _in practice_ asyncio coroutines are not quite compatible with non-asyncio event loops at this time (specifically, Twisted).

However, they are very, very close. There's some work going on to achieve the remaining bit of standardisation we need. If we can pull that off upstream then we can look very hard at having a requests that is async-first.


@Lukasa I seem to recall it sounding like defereds (perhaps with a slight (upcoming) alteration) would be the best implementation approach for us?


@kennethreitz The answer to that is _maybe_. The problem with using Deferreds is that Deferreds don't make things magically async: they're literally just fancy callback containers. We can almost certainly _use_ Deferreds to build an abstraction layer on top of multiple concurrency models if that's a route we want to go.

This is a problem that really needs to be approached _very_ carefully, because as long as Python 2.7 is around we cannot unconditionally expect that the standard library will have an event loop inside it that we can use. That represents a concern: we need a way to execute _without_ an event loop. This is why Deferreds are attractive: they can execute in just such a manner because they're _literally_ just callback holders.

So one approach here would be to write an entirely callback-based HTTP/1.1 + HTTP/2 client library that uses Deferreds (probably on top of @njsmith's [h11](https://github.com/njsmith/h11) and @python-hyper's [h2](https://github.com/python-hyper/h2)).

We could then try to think about how we can integrate that into the various different event loops, though again, I'm not 100% certain of how we'd do that best. _Probably_ we'd have to write (sigh) an event-loop abstraction layer that basically defines certain functions as returning deferreds ("data_from_network"), and that then plugs into the relevant event loops. In practice, though I've said "event loops", we really only need two implementations to get started: one that implements the Twisted/asyncio "protocol" notion, and one that is synchronous and blocking (so that we continue to work in the absence of an event loop). In fact, we could even just start with the second one.

This is one of those things that it'd be really interesting to get someone like @glyph to weigh in on, in part because @glyph is strongly incentivised to help us get this right (no more treq!) and in part because he's a lot smarter than me. Certainly, however, I'd love to have a version of requests that you can either use synchronously in its current form without noticing the difference _or_ that you can use on top of ${EVENT_LOOP}. Requests provides enough goodness that we should really reimplement as little of it as possible on different platforms.


Oh, and @shazow, this is something you might care about too, in part because we might be able to make urllib3 do most of the heavy lifting here.


One of urllib3 v2.0 wishlist items is async by default (without py2 support).


@Lukasa @kennethreitz Guess the question is exactly what to support.  In a ideal world it would be a universal design that's elegant to support both, in practice if you wanted to support both a event loop model and a twisted model, then maybe it would be better to support two models one for each.  Alternatively I think a lot of people are using (Gevent/Asyncio/Trollius) so maybe work on a implementation that supports that then see how the model can be altered to include twisted?

Above you mention "In practice, though I've said "event loops", we really only need two implementations to get started: one that implements the Twisted/asyncio "protocol" notion, and one that is synchronous and blocking (so that we continue to work in the absence of an event loop)"

My initial thoughts are is there a reason to majorly reimplement or alter the existing requests library aka ("one that is synchronous and blocking") and just maybe cleanly refactor things under the hood so the can be shared between the "synchronous" implementation and the "async" implementation?  Or is there a motivating factor beyond that I am not aware of?

@smorin 


Async without py2 support might be a good option for Requests too — would help push the community forward, a bit. We shouldn't do it purely for political reasons, but if it would make the implementation a lot cleaner, I think that could be a feature, not a bug. 


@smorin the idea is that async is going to be a lot more important in the future than it is now (as is most likely should be, especially for io-bound tasks), and Requests both wants to stay ahead for the long-term future and help people build better software.

Nothing would change about the current API, except it may be secretly powered by the new asynchronous methods under the hood, just in a blocking fashion. 


> My initial thoughts are is there a reason to majorly reimplement or alter the existing requests library aka ("one that is synchronous and blocking") and just maybe cleanly refactor things under the hood so the can be shared between the "synchronous" implementation and the "async" implementation

Well, we don't _have_ to: we can do async without it by just always deferring to threads, but that's kinda gross and sad for various different reasons and is also rarely what people _actually_ want.

If we want to do _actual_ async then at a certain point the code _has_ to diverge. Right now the baseline code YOLOs bytes into and out of sockets like nobody's business: that's fine, but it doesn't work _at all_ if you throw it into an asyncio codebase, and the only way to make it work is source level changes. The net result is that you can't support _both_ an event loop _and_ a non-blocking model in the same source code: while you can retrofit a synchronous API on top of an event loop, it still _requires_ the event loop, which Python 2.7 doesn't give us, and requests' "no external dependencies" model makes tricky for us to provide.

> Async without py2 support might be a good option for Requests too — would help push the community forward, a bit. We shouldn't do it purely for political reasons, but if it would make the implementation a lot cleaner, I think that could be a feature, not a bug.

That is _substantially_ easier. At that point, we can say that the asyncio event loop is the standard event loop to which we need to program. We can assert that it will always be possible to `import asyncio`, which means we can rely on having the event loop and relevant primitives in place. At that point Deferreds aren't necessary either: the synchronous APIs basically just look like this (with namespacing, obviously):

``` python
def get(*args, **kwargs):
    return asyncio.run_until_complete(async_get(*args, **kwargs))
```


It's quite possible that we shouldn't/won't do this at all, given the other priorities of the project. This is definitely evaluation phase :)


One use-case I have actually is requests-async to be used in AWS Lambda under Python 2.7, but would be okay using Trollius for that which is basically equivalent to Asyncio anyway.  So the underlying async implementation can be py3 primary with a py2 solution being a `import trollius as asyncio` but would simplify the implementation.


> One use-case I have actually is requests-async to be used in AWS Lambda under Python 2.7, but would be okay using Trollius for that which is basically equivalent to Asyncio anyway.

Well........not quite.

You see, a Python 3-only codebase will feel free to use `yield from`, which you cannot use with Trollius in Python 2 because it's literally a syntax error. So if that's a use-case we were going for we'd need to be somewhat careful.


Good call


Just wanted to let y'all know that I saw the mention, and I am digesting the thread; it may be a couple of days before I'm caught up enough on PyCon to reply, but I am definitely interested in this :).


It's at least an interesting exercise to try and imagine what the core of requests would look like if written in the style of h2/h11. Even if in the end it turns out to not be the best approach -- it's certainly more complex than any of the existing examples, so I guess we'd at least learn something :-). (And hey @Lukasa, what's this backsliding to concrete I/O APIs as soon as the problem gets more complicated? ;-))

Wildly sketching, since I have no idea how the internals of requests actually look in any kind of detail: I guess the basic I/O operations that requests needs to do are to request a connection (possibly with TLS and possibly with its own hacks to the TLS handshake logic), to do reads and writes on a connection, maybe to set socket options, and to close a connection. And while in h2/h11 the basic state object is a Connection that manages a single socket, for requests it would be a Session that manages a socket pool.

And we'd want to make it possible to wrap this in different APIs like:
- `blocking_session.get(...)` -> returns a `BlockingResponse`
- `await coroutine_session.get(...)` -> returns a `CoroutineResponse`
- `twisted_session.get(...)` -> returns a `Deferred` that when fired returns a `TwistedResponse`

(the different response classes would differ in the API they exposed for streaming reads of the response body)

Since the user is going to be calling methods on individual response wrapper objects, probably we want to let each of these wrappers handle the I/O coupling for each connection. So maybe the API would look something like this:

```
class BlockingSession:
    def __init__(self, ...):
        self._byoio_session = BYOIOSession(...)

    def get(self, ...):
        byoio_response = self._byoio_session.get(...)
        return BlockingResponse(byoio_response)

    def close(self):
        self._byoio_session.close()
        for todo_item in self._byoio_session.io_to_do():
            _do_io(todo_item)

class BlockingResponse:
    def __init__(self, byoio_response):
        self._byoio_response = byoio_response
        # Handle whatever I/O is needed to initiate the request
        self._do_io()

    @property
    def status_code(self):
        while not self._byoio_response.got_headers:
            self._do_io()
        return self._byoio_response.status_code

    def _do_io(self):
        for todo_item in self._byoio_response.io_to_do():
            _do_io(todo_item)

    # ...

def _do_io(todo_item):
    if type(todo) is MakeConnection:
        try:
            sock = socket.create_connection(todo.address, timeout=todo.timeout)
        except TimeoutError:
            todo.connect_timed_out()
        else:
            # Argument passed here is an opaque (to the BYOI/O layer) object
            # that acts as a handle to the socket. Life cycle: passed in to connect_succeeded,
            # passed back out with SendData or RecvData todo items and eventually will get
            # passed back out in a CloseConnection todo item. (But that might happen in a
            # different response object entirely or at session close.)
            todo.connect_succeeded(sock)
    elif type(todo) is SendData:
        sock = todo.socket_handle
        sock.sendall(todo.data, timeout=timeout)
    # ...

```

I dunno, maybe this is only interesting to me :-). Certainly it would be a lot of work, but the result might be rather nice...


Heh @njsmith, this is a problem because you don't know what Requests looks like internally. =)

The internals of Requests itself already _are_ very, very close to the design of h2/h11: that is, they almost entirely twiddle data around in in-memory objects. Unlike in aiohttp where it seems like every single function/class in the library touches or handles I/O in some way, the vast majority of requests is function calls.

The functions that either directly or transitively do I/O or that would need to be adjusted in some way are the following:
- `requests.api.*`: all of these transitively wrap `Session.request`.
- `requests.sessions.Session.request`: this transitively calls `Session.send`, which actually does do I/O
- `requests.sessions.SessionRedirectMixin.resolve_redirects`: this both reads and writes (using `Session.send`), and so is clearly an I/O based function, but mostly just uses the primitives we'd have to work out at the lower level.
- `requests.sessions.Session.send`: This is the _only_ top-level requests function that actually does I/O (excepting file reads/writes which don't count because Unix).
  `requests.adapters.HTTPAdapter.send`: This does the bulk of the low-level I/O, though it calls one helper function it defines on itself that is I/O-adjacent, namely...
- `requests.adapters.HTTPAdapter.get_connection`: This obtains a connection object from urllib3's connection pool. This is going to be different in concurrent and synchronous modes because the way we wait for connections to show up will be different.

In this case, for requests, I'd ideally like to touch _only_ those functions. That ensures that the rest of Requests goes on functioning exactly as it does today. This reduces the churn in Requests and reduces the odds of us introducing bugs in terrible places.

From my perspective, the easiest way to do this is to just write `@asyncio.coroutine` on top of all of them and throw some `yield from`s in there, then wrap it all in `asyncio.run_until_complete` if people want to use the blocking APIs.

However, if we decide we need a separate blocking backend this _all_ gets way harder. In that kind of model, your approach may work slightly better, because it may be quite painful (and indeed impossible?) to use `Deferred`s in such a way that the top-level user of requests doesn't have to think about them in synchronous code. In that kind of model, we actually do need to write things quite differently, and at _that_ point I'm willing to discuss tearing the whole library apart to rebuild it.

Otherwise, I'm extremely reluctant to do a big rewrite: there's just no way that we'll persist all the useful functionality that the library provides in its current form. =(


I don't think I understand what you have in mind when you talk about only touching those functions. Even if we narrow the scope strictly to asyncio, then that still intrinsically requires touching almost every public API function (for asyncio users `Session.get` becomes a coroutine, `Response.iter_chunked` becomes an async iterator, ...), so somehow one has to thread that change down from the API level to the low level functions you mention, right? Or do you mean that just sprinkling `yield from` everywhere doesn't really count as "touching" them?

Another thought: if you're able to use `yield from` (which would require dropping 2.6 unfortunately, but since you mentioned it...), then that doesn't necessarily have to be tied to asyncio. It shouldn't be too hard to write your own coroutine runner that exposes an h2 style API to the outside. (Very roughly it could look like the API thing I sketched, where the leaf coroutines yield those "io to-do objects" and then someone on the outside either services them synchronously or hands them off to some event loop.)


If we drop Python 2 support then being tied to asyncio isn't any more of a problem than being tied to httplib is today; its there, it's the de facto event loop abstraction, and it covers the primary use case. 

As far as I can tell the biggest advantage of moving to the model you proposed is that it's relatively extensible to systems that don't maintain the semantics of asyncio (e.g. curio). That's somewhat valuable, but I'm not sure it's worth it to drastically tear up the code. 

With that said, we should consider both options. We're so early on right now that there is literally no reason to decide on one over the other: I remain open to doing whatever provides the best API to all Requests users. 


Ah, right, I'm assuming that requests dropping support for python 2 is still like... 3-5 years away (is that wrong?), so the primary advantage of the model I was talking about is that it would allow a single codebase to simultaneously support a non-blocking api and a py2 with blocking io and no external dependencies api. (Plus it would be easier to test ;-))

Certainly it would be nice to have non-blocking support in the "real" requests codebase in less than 3-5 years (and also a pony).


> Ah, right, I'm assuming that requests dropping support for python 2 is still like... 3-5 years away (is that wrong?)

Heh, that's part of what this issue is about. =D

I think this whole issue is going to be part of my list of "things I want to deal with in the next year". It's my genuine belief that Requests needs its own async story. Right now I'm mostly interested in soliciting opinions about how we could do it (which is why I also tagged @glyph and @shazow).

However, from my perspective, all options @kennethreitz will entertain are on the table. If @kennethreitz is willing to consider dropping Python 2 support at some time soon, I'm willing to consider that approach, even if we don't end up doing it.