Consider Requests' Inclusion in Python 3.5's Standard Library

There's a lot to this, but I'll keep it simple...

Would the Python community, as a whole, benefit from Requests being added into the standard library? 

Would love to hear your thoughts and opinions on the subject!


Yes, because it simplifies the entire process and does not sacrifice the performance. So yes.


- What kind of impact would this have on Requests' ability to evolve and grow?
- Does Requests' release frequency jive with that of Python's? A big difference here might be a good indication that stdlib isn't the right home for Requests.


Let's CC some people whose opinions we care strongly about:

@shazow @kevinburke @dstufft @alex @sigmavirus24


What happened to "the standard lib is where things go to die"?


The release cadence question is a good one; I'd be concerned about losing request's ability to evolve freely. That said, perhaps a requests core library would be a good fit.


My 2 $CURRENCY's worth:

I would be disinclined to do it. I think being outside the standard library has given us the freedom to make choices that benefit our users without being stuck behind core dev's policies for version support and release. It allows us to respectfully disagree with the priorities of core dev. And it allows us to make decisions that are ideological, which has been the lifeblood of this project for more than three years.

I think the reality is that if this module enters the standard library the current core team will move on from it. I certainly have little interest following it into the quagmire that is core dev. The most likely to steward requests in the stdlib is @sigmavirus24, and he's just one man. That loss of direction will inevitably lead to an erosion of the library's interface over time, and I think that would be a tragic thing.

The only thing that being in the standard library gives us is our time back. That's a good reason to put it there, if that's what you think it needs, but I don't think we should pretend that it will make the library or the Python ecosystem any better.


I don't think you actually _can_ add requests to the stdlib without first adding chardet and urllib3 or removing your dependency on them. There's also the thing where Python doesn't want to ship a CA Bundle so you'd need to modify it so that it uses the system platform bundle as Python does naturally now.

Besides that, I'm not sure. It would certainly make it easier to get requests, however part of my work on pip is basically to make it so it's easy to get things like requests without needing to add it to the stdlib. On top of that it's also somewhat confusing to have multiple ways to make HTTP requests in the Python stdlib. The unification of urllib and urllib2 was a good thing in the Python stdlib and I don't think that re-adding that with urllib.request and "requests" is a good idea either. Finally I don't think it would actually help many people, this would only go into 3.5+ so anyone who wants to use requests would have to either use the version that is on PyPI or making 3.5 their minimum supported Python version which I don't think is a realistic thing to happen in any near future.


While I think having Requests in the standard library would help new users I don't know that it would help the Python community as a whole. Experienced users know to use Requests and know how to install it.

I'd be especially concerned with the chilling effect it would have on new development as others would be disinclined to contribute, knowing that they wouldn't see their contributed changes in an easily-usable release for a long time. 

What about the middle ground of making Python distributions ship with it installed by default?


No it wouldn’t.

requests is absolutely unsuitable for stdlib inclusion for the many reasons stated before me.  The urllib3 dependency alone is a complete showstopper; we don’t want it to got to die in the stdlib.

What _would_ be useful is to add something _basic_ and similar to requests built on top of stdlib’s current http resources that allows users to do simple get/post requests to https without practicing blood magic.

Also kind reminder that it would be added to Python 3 only. :)  (and not earlier than Python 3.6).

Are you getting tired of maintaining it Kenneth? ;)


I don't think we can even begin to discuss this question without someone saying what becomes of httplib, urllib, and friends. Adding requests without addressing the complexity of choice is, I think, worse than the answer "ignore teh stdlib, just use requests". It's a regression to the days of urllib, urllib2.


> I think the reality is that if this module enters the standard library the current core team will move on from it. I certainly have little interest following it into the quagmire that is core dev. The most likely to steward requests in the stdlib is @sigmavirus24, and he's just one man. That loss of direction will inevitably lead to an erosion of the library's interface over time, and I think that would be a tragic thing.

I would wander into the stdlib to try to help, but given the fact that exactly one of I don't know how many previous patchsets I've submitted has been accepted and one other _reviewed_ makes me wary of wanting to bother with that process. I know the core devs are entirely swamped by more important things. I also know someone else has decided randomly that they want to maintain httplib/http but they're clearly not suited for the job (yet) and I don't have the patience to work on httplib when patches that both @Lukasa and I sit around, unreviewed, and not cared about (when they fix pressing issues with the library).

I'd probably end up just forking requests to continue using it.

> requests is absolutely unsuitable for stdlib inclusion for the many reasons stated before me. The urllib3 dependency alone is a complete showstopper; we don’t want it to got to die in the stdlib.

It's always been a contention of @kennethreitz (and therefore, the project as a whole) that urllib3 is an implementation detail. Many of requests' biggest features are handled entirely by urllib3, but it doesn't mean they couldn't be reimplemented with care into truly dependency-less library.

Regarding the chardet dependency: it's been nothing but a headache to us (and to me specifically). It used to have separate codebases for py2 and py3 until I got it into a single codebase library (which has only in the last several months been merged back into chardet proper). The library is slow and a huge memory hog (which angers many people to the point of yelling at us here on the issue tracker). It's not entirely accurate and Mozilla's universalchardet that it is modeled after has all but been abandoned by Mozilla. So removing chardet would probably be a net positive anyway.

Regarding whether we should do this or not, I'm frankly unconcerned. Whatever would be in the stdlib would end up being requests in API only. The Python 3 adoption rate is slow enough that I don't think people will be meaningfully affected by this for the next N years (where N is the globally unknown number of years for 3.5 to be used in production by corporations).

And like I said, I'd probably end up just forking requests or using urllib3 directly at that point.


I discussed this at length with Guido the other day — chardet would have to be included first. I think that urllib3 and requests could be included into the http package together. 

However, I'm very inclined to agree with @hynek and @dstufft. Perhaps requests is fine just the way it is :)


Either way, if you have an opinion that you'd like to share, you are welcome to share it here (or anytime really). 

:sparkles: :cake: :sparkles:


Also, adding a new http module to the stdlib with no asyncio-story seems
bonkers to me.

On Sun Jan 25 2015 at 1:15:51 PM Kenneth Reitz notifications@github.com
wrote:

> Either way, if you have an opinion that you'd like to share, you are
> welcome to share it here (or anytime really).
> 
> [image: :sparkles:] [image: :cake:] [image: :sparkles:]
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2424#issuecomment-71384152
> .


> I don't think we can even begin to discuss this question without someone saying what becomes of httplib, urllib, and friends

This is my issue. I think the current confusing state of affairs has to be sorted out first.


:+1: :metal:


Just to be clear, my above comment about re-implementing urllib3 for inclusion in the stdlib shouldn't be taken lightly. The amount of work necessary to do that would be immense because urllib3 is the product of (likely 10 or more) years of developer work.


I too have spoken with Guido about tossing urllib3 into the stdlib some years ago with the conclusion that it's not a great idea, but I'm fairly neutral about it at this point.

urllib3's API has been mostly-stable and pretty much completely backwards compatible for several years now. Its' pace is possibly even slower than that of the stdlib today, with the vast majority of changes being minor fixes or security improvements (with occasional backwards-compatible feature additions like granular timeouts/retries). If somebody really wanted to try and get urllib3 into the standard library, I don't think it's a terrible idea—it's just not the _best_ idea.

(I'm not speaking for requests, as it moves at a very different pace with different goals than urllib3.)

The best idea, in my opinion, would be for the PSF to hire (or maybe Kickstart or something) 1-3 developers to build out a brand new http library on top of asyncio with HTTP/2 support with heavy inspiration from urllib3, requests, and hyper. I'd be happy to see as much code taken verbatim as possible but laid out in a consistent, modular, and reusable manner. Ideally target Python 4 or something, and get rid of all the urllibs and httplibs. I expect this would be 6-9mo of hard work, but possibly more.

The very worst part about urllib3, which I'd love to see replaced if somebody attempts to rewrite it per @sigmavirus24's suggestion, is that it depends on httplib. urllib3's functionality is substantially limited with lots of code spent working around shortcomings of httplib. Though if HTTP/2 support would be taken seriously in this goal, then the scope of re-implementing HTTP/1.1 would be a very comforting fraction of the work required.

Many PyCons ago, a bunch of us met up and whiteboarded a layout of a brand new http library that refactors all the pieces into the ideal arrangement we could imagine at the time. I'd be happy to dig up these notes if anyone is going to attempt this.


+1 @shazow

Again, if anyone finds themselves with the time and inclination to take on that fairly large project, I've sketched out a putative API design that might make a good starting point.


Yes because the only way I will ever allow requests as a dependency is if its in stdlib.

That said, urllib3 contains the features that people really want, so having that in stdlib is enough for me


Do you not use any dependencies? 


@dstufft this is in projects that generally don't, where everyone can't be bothered to figure out how to use urllib. (people aren't adding it as a dep because of ssl/etc generally, but out of laziness)


@dstufft also multi-version deps basically make it hard to use things in libraries. You probably want to use requests in your project and if we require it then there's a potential for a world of hurt if API changes happen in versions.


While I appreciate some people wanting to develop dependencies without dependencies on other software that hasn't changed its API in years, this isn't the place for the discussion.


@sigmavirus24 I disagree. requests has had its API change in the past. APIs change, thats why we have versioning, thats why dependencies are complex. This is a perfect case for that discussion because requests is a dependency in a lot of projects.

If you move into the stdlib the API must be stable.


@dcramer the API broke exactly once, in 1.0. APIs do change, but requests' API isn't planning any changes either. The only change we've had is adding the `json` parameter which doesn't break anything. You can keep trying to accuse us of breaking the API too much, but when projects like OpenStack have had requirements defined as `>=1.2.3` for a long time, I think that says a lot about the stability of requests. The API has been stable, precisely because after we cut 1.0 we rejected all new additions to the API (with the obvious recent exception of adding a `json` param) and we've been very strict about not breaking the API. If you're not a consumer of requests, you wouldn't know this though. So I don't take your ignorance personally.


Further if the stdlib API is supposedly so stable, explain why argparse broke it's public API between Python 3.3 and 3.4?


@sigmavirus24 you're now purely turning this into an API debate. I was just pointing out the reason I dont include it is because it can change, and everyone uses it, and everyone uses different versions. It's great that you guys never change your API but I have no desire or time to follow it or assume thats true.


You know Python changes it's API too, quite often actually, sometimes in very major ways, perhaps you've heard of Python 3?


Well i'm leaving this debate. I made my opinions clear. Not sure what you're all bitching about.