Raibek/requests-github-issues huggingface dataset:
Getting cookies doesn't work in some cases

Steps to reproduce:

```
import requests
from pyquery import PyQuery

def obtainCookies(username, password):
    tokenUrl = 'https://news.ycombinator.com/newslogin'
    loginUrl = 'https://news.ycombinator.com/y'
    query = PyQuery(requests.get(tokenUrl).content)
    token = query('input[name="fnid"]').val()

    payload = {
        'fnid' : token,
        'u' : username,
        'p' : password
    }

    return requests.post(loginUrl, data=payload).cookies

print len(obtainCookies('some_user', 'some_password')) 
```

This code will return 0, yet the cookie is clearly being sent (can be seen in Chrome dev tools) AND the login is successful.

Why is this broken?


Chrome dev tools can only see requests made from inside Chrome, which means you don't know that the cookie is being returned in this case. Wireshark can tell you that. Could you check with Wireshark to see if the cookie is actually being returned?


How would you suggest to do that, given that the connection is TLS encrypted? 

Regardless, the login request is definitely successful, as I can see the username being embedded in the HTML of the response object (HN only does that for authenticated users). And I can see the cookie is definitely being set in Chrome when I login via the same page. See more sample code below (you can even run it, provided no one changes the password):

```
import requests
from pyquery import PyQuery

tokenUrl = 'https://news.ycombinator.com/newslogin'
loginUrl = 'https://news.ycombinator.com/y'
query = PyQuery(requests.get(tokenUrl).content)
token = query('input[name="fnid"]').val()

payload = {
        'fnid' : token,
        'u' : 'requests_bug',
        'p' : '123456'
}

r = requests.post(loginUrl, data=payload)
print r.text[r.text.index('user?id=requests_bug') - 31 : r.text.index('user?id=requests_bug') + 70]
print len(r.cookies)
```

Result:

```
>>> print r.text[r.text.index('user?id=requests_bug') - 31 : r.text.index('user?id=requests_bug') + 70]
<span class="pagetop"><a href="user?id=requests_bug"><font color=#3c963c>requests_bug</font></a>&nbsp
>>> print len(r.cookies)
0
```

So, in short, I see no reason why the cookie wouldn't be returned in this case, unless requests drops it.


Ah, yes, I think this is related to a problem we've been having in v1.1.0 with cookie persistence: see #1228.

The cookies _are_ there. If you run your code in an interactive interpreter, you can do the following:

``` python
>>> r.history
(<Response [302]>,)
>>> len(r.history[0].cookies)
1
```

Hopefully v1.1.1 should make this a bit easier, but for now this will have to do.


Yes, this looks to be the same problem. Good to know it will be fixed as part of the next release.
Documentation for iter_lines doesn't indicate stream=True is required

This is the current description of `iter_lines`:

> Iterates over the response data, one line at a time. This avoids reading the content at once into memory for large responses.

The "avoids reading the content at once into memory" part seems to imply that this has streaming behavior, but that's not currently the case without specifying stream=True on the request.

Maybe this is just a reading comprehension fail on my part, but it took longer than I would have liked to get the desired behavior.


Feel free to submit a pull request to make it more explicit. We'll be happy to accept it.


Here's the pull request: https://github.com/kennethreitz/requests/pull/1263

Thanks for the quick response!
Adding default kwargs in Session.send. Fixes #1258

This ensures that the proper defaults are passed along to the associated adapters send() method.

As per the related issue - class attributes previously set on the Session object are not being properly passed along to the adapter when calling Session.send()


Nope - this doesn't work. On running the unit tests I see now that hook() lands up with an unexpected keyword argument


It gets an unexpected keyword argument unrelated to your change.


Scratch that, thought you'd only added the proxies argument as a default.
Should Session.send() ignore Session.proxies?

My test code:

``` python
import requests
s = requests.Session()
s.proxies = {'http': 'myproxy:3128'}
r = requests.Request(method='GET', url='http://www.google.com')
s.send(r.prepare())
```

In the above case, the proxy is ignored completely.

It looks to me like in Session.request() the following code is run:

``` python
proxies = merge_kwargs(proxies, self.proxies)
```

But in the send() function this is not happening. I am not sure if this is a bug - or if always passing proxies to the send() function explicitly is the intended behavior.

I am relatively new in using the requests library - my apologies if this has been brought up before or if i'm missing something obvious.


That should be a simple one line fix using `setdefault` on the arguments to send


@sigmavirus24 something like the following:

``` python
 def send(self, request, **kwargs):
        """Send a given PreparedRequest."""
        adapter = self.get_adapter(url=request.url)

        kwargs.setdefault('stream', self.stream)
        kwargs.setdefault('cert', self.cert)
        kwargs.setdefault('verify', self.verify)
        kwargs.setdefault('proxies', self.proxies)

        r = adapter.send(request, **kwargs)
        return r
```


Exactly what I was thinking of. :)


I have opened [Pull Request 1259](https://github.com/kennethreitz/requests/pull/1259) related to this issue.


See #1267
Give hooks a way to retry requests in the same session.

I am talking to an app with a custom auth module that I've written. The auth object installs a hook, and if the response fails and indicates that the auth object is expired, I get a new auth object.

I'd like to be able to retry the exact same request with the same settings as the previous request but with a new auth object from my response hook. This means I'd like to be able to make the request as part of the same session (ideally) but to also have access to the initial request parameters such as verify.


This is a duplicate of the PR #1190 which was merged. We really have to cut a new release. If that PR suits your needs (which I suspect it does), feel free to close this. I don't want to close it preemptively.


Also related, #1261


Closing due to no response


Yeah, no problem. Looks like #1190 will do for me.
Allowing @ in proxy passwords and usernames.

Well, I don't have much to say, except that, the change that I made, is to allow for @ in the passwords when someone is using a proxy (like me!). Someone may change it to be more readable.

Test script:

``` python
import requests
proxy = {'http':'http://test:test@123@myproxy.com:80'} # imaginary proxy, I used a real one in my case
print requests.get('http://google.com', proxies=proxy)
```

Before:

``` python
Traceback (most recent call last):
  File "<pyshell#2>", line 1, in <module>
    requests.get('http://google.com', proxies=proxy)
  File "C:\Python2.7 For Chintoo\lib\site-packages\requests-1.1.0-py2.7.egg\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Python2.7 For Chintoo\lib\site-packages\requests-1.1.0-py2.7.egg\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python2.7 For Chintoo\lib\site-packages\requests-1.1.0-py2.7.egg\requests\sessions.py", line 279, in request
    resp = self.send(prep, stream=stream, timeout=timeout, verify=verify, cert=cert, proxies=proxies)
  File "C:\Python2.7 For Chintoo\lib\site-packages\requests-1.1.0-py2.7.egg\requests\sessions.py", line 374, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python2.7 For Chintoo\lib\site-packages\requests-1.1.0-py2.7.egg\requests\adapters.py", line 209, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='123@netmon.iitb.ac.in', port=None): Max retries exceeded with url: http://google.com/ (Caused by <class 'socket.gaierror'>: [Errno 11003] getaddrinfo failed)
```

Now:

``` python
<Response [407]>
```


This change has been made in urllib3, not in Requests. We vendor in a copy of urllib3 that we keep in sync with the upstream version, so any change in urllib3 should be made [there](https://github.com/shazow/urllib3).

Thanks for creating the PR though!


@LukaTes just saw that, I was going to delete this request...
url-authentication

- When credentials are included in a URL they are extracted and passed
  to the server using HTTP Basic Auth.

I've tested this by hand and it works.  I'm not quite sure how I
should test this in `test_requests.py`.  I would normally mock this
kind of thing, since I don't have a persistent service that requires
credentials running to use from a test, which seems to be the existing
pattern.  I'm reluctant to add a dependency on a mocking library and
jump through all those hoops if that technique is not already used.


:sparkles: :cake: :sparkles:
Attach original Request object to Response object.

Allows for quickly creating Requests similar to those already made, or for
examining Request object manipulation by third party hooks or authentication
methods.

This is particularly useful when using requests with something like oauthlib: https://github.com/requests/requests-oauthlib

There appeared to be an instance of the original request object passed along as far as `Session.send()`, but it was never used after that. So, I grabbed it and attached it to the Response object.


First off, thanks for the contribution :)

That being said: Request objects are only for creating PreparedRequest objects, and shouldn't be used in this way.


That's the idea; use the returned Request object to create a PreparedRequest object, and issue a second request using a new or an existing Session.  With this, the developer has access, client-side, to what was actually sent to the server, whether doing low-level stuff with Request objects, or simply using `.get()`.


All responses currently should have a PreparedRequest object attached them already. 


Ah, I see now... `dir()` does indeed show the existence of a PreparedRequest on the Response (which, incidentally, this commit completely overwrites with the Request object...) It wasn't in the documentation so I completely missed it. Got it! Thanks!
Documentation doesn't mention Session.request

Neither http://docs.python-requests.org/en/latest/user/advanced/#session-objects or http://docs.python-requests.org/en/latest/api/#request-sessions talks about Session.request - which I only found once I cloned the source locally. 

Originally I was using session.send(preparedRequest) which wasn't persisting cookies or headers. I thought this was a bug / weird issue so I cloned the source and that's when I saw Session.request.

It would be good to clarify the intent of .request versus .send in regards to cookie persistence, headers, etc. etc.

Happy to put together a pull request to update the docs.


Hm, just noticed https://github.com/kennethreitz/requests/issues/1201
I'm using the latest sources, maybe the docs haven't been updated? 


Cookie persistence is no longer an issue. That has been fixed in master, so if you're still seeing that issue, let us know. I'm, baffled though that Session#request was lost in the docs. I'm guessing it's doc-string was accidentally deleted which is why it no longer shows up. If you don't want to send a PR, I'll get to this tonight.
'PreparedRequest.headers' keys are a mixture of bytes and (unicode) strings in Python 3.x

In Python 3.x `PreparedRequest.headers` keys end up being a mixture of byte strings and unicode strings, because in [`PreparedRequest.prepared_headers`](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L326) the headers received from the user are byte encoded but later code in the request preparation (e.g. [`PreparedRequest.prepare_body`](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L331)) will add or modify unicode string keys.

The problem is visible, for example, when the user tries to create a `PreparedRequest` with a given content type with the `'Content-Type'` key as a unicode string, which then in [`PreparedRequest.prepare_body`](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L383) is ignored and a default `Content-Type` header added because the key already in `headers` is a byte string, but a unicode string literal is being used for the existence test.

I tried to use always unicode strings or always byte strings, for the header keys, but that caused unit tests to fail either on Python 2.x or Python 3.x, so clearly a better thought solution is required.


Yes, I've wondered about this before. I might work on a PR that has Requests take control of encoding and decoding strings, though it might end up being sufficiently complex that Kenneth won't want to merge it. =)


#1338 is the most recent pull request that should address this issue, I'm going to close this and move the discussion over there. =)
stream_decompress corrupts output in case of decompress error

A `zlib.Decompress` object consumes at least one byte without raising an error and without returning any output:

```
>>> zlib_mode = 16 + zlib.MAX_WBITS
>>> any(zlib.decompressobj(zlib_mode).decompress(chr(i)) for i in xrange(256))
False
```

So if there is an zlib.error in `stream_decompress`, there will be at least one byte missing in the output string.

Additionally, there are two variations of the deflate compression in urllib3, while requests is only handling one of them: https://github.com/shazow/urllib3/blob/master/urllib3/response.py#L25


[This comment](https://github.com/kennethreitz/requests/issues/465#issuecomment-15106811) is also interesting in this context.


With this https://github.com/shazow/urllib3/pull/159 merged, `stream_decompress` could be dropped entirely.


Fixed with #1299
proof of concept async requests via futures

this is a proposal/proof of concept. by no means is it expected to be merged.

it provides a FuturesSession which returns concurrent.futures.Future objects to calls in place of requests.Response. this provides a near seamless async requests api.

an example:

```
session = FuturesSession()
future = session.get('http://www.foo.com/bar/')
... do other stuff ...
# wait for response or return immediately if it's already back
response = future.result()
... continue as normal ...
```

as currently implemented it requires no external dependancies and uses the python3 futures, tested with python 3.3.0. if desired support could be added to 2.x releases by either using the backport of futures or without dependancies using methods similar to https://github.com/ross/python-asynchttp.

feedback solicited. 
## best,

-rm


Very cool. I think it's worth you putting this into its own repository, much like Kenneth has with [grequests](https://github.com/kennethreitz/grequests).


+1 to @Lukasa's comment :)


I'll third @Lukasa's comment.


just threw up a new package, i've registered it in pypi, but haven't built and uploaded it:
    https://github.com/ross/requests-futures

i added the necessary pieces to support it pre 3.3 using the backport of concurrent.futures.

would appreciate anything you all would be able to do in terms of making it easy for people to find it if/when they go to look for async functionality with requests.


@ross I tweeted about it and put it in the wiki: https://github.com/kennethreitz/requests/wiki

If others come asking about similar items, I'll be sure to point them in your direction. :) Thanks


thanks :)
Please provide a RELEASE tag

Please provide a release tag for the repository, so we can clone and use sub-repositories that would use the latest released version.

Currently you have tags with versions, but there is no way of getting the latest release using a single command.

It think it shoudn't be hard to modify the tagging on a new release, even as a script. Thanks.


I'm not sure I follow. You mean you'd like to have two tags point to the latest release, e.g.,

```
1.1.0 -> fakesha
RELEASE -> fakesha
```

Then when Kenneth releases 1.1.1:

```
1.1.0 -> fakesha0
1.1.1 -> fakesha1
RELEASE -> fakesha1
```

Or am I missing the concept?


We used to do this, but like the simplicity of doing it this way now.

Explicit is better than implicit :)


Are you against because it is hard to make the tagging or for other reasons? If I come up would provide a release utility script that does all the tagging stuff when you release a new version, it would be ok to accept this? Thanks.


Other reasons. As the Zen of Python states, "Explicit is better than implicit". If you are forcing a download of "RELEASE", you are implicitly selecting a version. This makes resolving dependencies hard, it means you don't know what version of Requests you're developing against, and it means you shout at us when we make breaking changes. All bad. =)


@Lukasa 
"Explicit is better than implicit" does not really apply to this at all. If someone downloads RELEASE then it's because he wants RELEASE and he knows that he will get RELEASE and what RELEASE means. Let me summarize with another quote - "All problems in computer science can be solved by another level of indirection"


More simply put, this is @kennethreitz's project and he can do as he pleases and he doesn't need to explain his actions to anyone. If he would rather not do this then that is his call.


@piotr-dobrogost: I respectfully disagree, but happily this is both a) a matter of opinion and b) not something about which our opinions matter at all. =)


@sigmavirus24 <3
New "proxies" argument option

I'm currently working on integrating SOCKS proxies into urllib3 (see: https://github.com/shazow/urllib3/pull/68#issuecomment-14672574), and this is a minor change that I think could be useful. Idea was obtained from here: https://github.com/kennethreitz/requests/pull/1192#issuecomment-13564585

I feel the `proxies` keyword argument to Session() should accept a plain string URL in addition to the current dict-of-URLs option (`{"http": "...", "https": "..."}`). In the case of a string being passed, that proxy would be used regardless of the destination URL scheme.

Original:
`requests.get(url, proxies={"http": "http://1.1.1.1:8080"})`
(Proxy would only be used for `"http://website.com"`).

Proposed:
`requests.get(url, proxies="http://1.1.1.1:8080")`
(Proxy would be used for `"https://website.com"` and `"http://website.com"`).

Both would be permitted.

It's very common for users to only specify one proxy. After SOCKS proxies become supported, this will become even more common because SOCKS proxies can handle every kind of connection (an FTP destination, HTTP, HTTPS, etc.). So if a user is using a SOCKS proxy, it doesn't really make sense to specify multiple proxies in a dict. Use of a SOCKS5 proxy would look as follows: `requests.get(url, proxies="socks5://1.1.1.1:1080")`

This would not break the old API in any way, just add a new way of setting a proxy. The only disadvantage is that it creates a bit of a naming mismatch, since `proxies` is plural. I think it would be sillier to create a new `proxy` parameter, though (unless `proxy` simply became an alias for `proxies`).


Creating a new `proxy` parameter is right out because of Requests [feature freeze](http://docs.python-requests.org/en/latest/dev/todo/#feature-freeze).

As for adding a global way of specifying proxies for all schemes, I'm +0 on it. It's not a huge change, and could potentially go into 1.2.0, but it _does_ change the API slightly, so it's possible Kenneth won't be keen on it.


Creating a new "proxy" parameter certainly isn't necessary, was just a possible suggestion. It's probably not worth the change.

As for the API change (via a new string parameter), I personally think it would be sensible since it would be useful for most of the cases in which someone will need to use a proxy with requests.


Only a slight improvement, but I typically do:

```
requests.get(url, proxies=dict.fromkeys(['http', 'https'], "http://1.1.1.1:8080"))
```


@dhagrow That seems a bit unintuitive.


Well, I think the typical way to configure a proxy is via `http(s)_proxy` environment variables so I don't think any API change is necessary. (I assume `requests` is dealing with them, if not this would be a huge design bug...)

It even would break the standard Unix workflow that you specify both, http **and** https, so -1 from me.


@schlamar actually I believe urllib3 handles getting it via the environment variables for us, so there really shouldn't be an API change necessary.

I'm also :-1: on this
Cannot install as current user

This is a very interesting issue.
I need requests on the place I work, and I do not have administration rights. For that reason, I install it using `python setup.py install --user` .
Actually, I can log in on two different computers: one is where I actually do my work, and has Python version 2.7, the other is the server from where I need to run my program, which has Python 2.4.3

I managed to install it on the working computer (Python v2.7), but it won't work on the server (Python v2.4.3). The error I get is: `python setup.py install --user
Traceback (most recent call last):
  File "setup.py", line 6, in ?
    import requests
  File "/home/S12/tomoiac2/bin/kennethreitz-requests-3e72f23/requests/__init__.py", line 52
    from . import utils
         ^
SyntaxError: invalid syntax`

Any ideas on why this happens and how it can be solved?

Thank you very much


Requests does not support Python 2.4. In this case, I suspect Python 2.4 does not allow relative imports, on which Requests relies.


Just FYI, Requests only supports 2.6+
Use session cookies when following redirects

A fix for issue #1228.

I have a nagging feeling, that there'd be a better way to do this, but here goes.


Excited to see this getting fixed!

My immediate reaction here is perhaps `cookies` shouldn't need to be passed in? `SessionRedirectMixin` is extending `Session`, so we should be able to just reference `self.cookies` directly.

Do you think your code will still work properly if we do that?


Sure - I attached a commit. You could just do `cookiejar = self.cookies`, but that would make Session.send persist the cookies across calls. I looked at issue #1201 but wasn't sure if that was desirable.


Yeah, I'm not sure which would be desirable at this time either. Need to think about it more first.

Thanks so much for your work on this! 


Hmm, would you mind doing a rebase so i can merge?


Rebased.


:+1: looks good to me. Sorry I was sick and couldn't review this earlier.


:sparkles: :cake: :sparkles:
FTP possible ?

Is it possible to download a file over FTP with requests?
I have tried it but it does not work.

Thanks.


Hey, I have topical knowledge!

The answer is that, strictly speaking, no. Requests is an HTTP library. However, after the release of v1.0.0, I made a [proof of concept of FTP in Requests](https://github.com/Lukasa/requests-ftp). Feel free to use it. Bear in mind it's barely functional. If you encounter problems, raise an issue and I'll see what I can do.


All right, that is what I was thinking.
Thank you for your quick answer.
Removing traling and leading spaces, and newline from headers

Hello, 

currently Requests has the [same issue](http://bugs.python.org/issue17322) (see the full discussion) than the standard library.

```
>>> import requests
>>> url = "http://www.w3.org/"
>>> headers = {'foo': 'bar', ' fooLead': 'bar', 'fooTrail ': 'bar', 'foo\nlish': 'bar'}
>>> r = requests.get(url, headers=headers)
>>> r.text
u'<html><body><h1>400 Bad request</h1>\nYour browser sent an invalid request.\n</body></html>\n'
```

Quick summary on the constraints on HTTP header `field-name`:
- Leading spaces are deprecated and interpreted by conformant servers as continuation line. 
- Trailing spaces are forbidden and might create security issues
- Newline break the full thing. 

Bonus: Check also if we use only the allowed set of characters.

Would you like me to provide a patch and associated tests for fixing it?


I don't know if policing this is our job. It's not clear to me that we should police that your headers meet the RFCs.


Policing, nope.
But giving the chance to have a more predictable behavior, yes. At least, if not removing the forbidden characters, a warning could be raised so people are given the opportunity to fix it. Instead of running into circles to see that empty space they had forgotten ;)


I still don't like requests performing this check either honestly. This is never going to affect the headers that we create and users can send whatever they like, especially by modifying a PreparedRequest object. That would also be the only place to add this check if @kennethreitz wants to add it, but I don't see a real need for it.


I don't really understand the resistance or friction to improve and make a better predictable product. I will fix things on my side. I'm closing.
Cut a release for v1.2.0

/cc @sigmavirus24 @slingamn 

Anything pending for a new release? A ton of stuff has happened.

_Note_ I updated the title to reflect what is generally believed to be the correct version we should cut. --Ian


A damn lot has happened. Did you want multidicts ready for this? I'm still trying to think of a good way to handle the merging of keyword arguments. Also, I'm missing one error class from Flask, but my branch isn't really ready for a merge at all.

Also pinging the jet-lagged @Lukasa 


I'm not sitting on anything that needs doing. =) I've only been doing small stuff while I was on holiday so that I wouldn't hold up any releases.


What about waiting for HTTPS proxy support: https://github.com/shazow/urllib3/issues/139? This might be worth a 1.2, though.
Changes for requests would be relatively small, see #1209.


I don't think it's worth blocking 1.1.1 for that, because shazow's time is always at a premium and it could take quite some time to make it in to urllib3.


Merge #1223 and I'll be happy. :) Also, we need to figure out what has since 1.1.0. A git log would definitely work. (And it's really what the Linux Kernel does)


Merged :)


I'm :+1: for a 1.1.1 release then. Just filter out all the Merge commits from GitHub and use the gitlog for the changelog ;) (It's the lazy way&trade;)


Yeah, and re @schlamar's concern: https://github.com/shazow/urllib3/pull/56#issuecomment-14412603 would indicate we'd have to wait longer than I think you'd like to.


Hi all; any updates on this?

I've got a few users running into #1189, so we're all eagerly waiting for this release =)


Just a side note #1261 is blocking this release in my opinion although, not anything that's too difficult.


Tonight, I'm going to put together a changelog as best as I can. I post it here, and after @Lukasa wraps up #1252 I think we should cut 1.2 tonight.


#1252 is no longer blocking this release. See the discussion there.


I'm working on the changelog. I'll have a pull request ready for tonight. I'm leaving the exact date open to Kenneth.
Initial docs patch for migrating from pre 1.0

Hopefully this helps with issues like #1196. I know when 1.0 first dropped I went straight to the docs looking for this and I was surprised it didn't exist.


Thanks for this. I've been working a little bit on something like this on the side, but it's way better if you do it than me. =D This is a great place to start from. I've made one small note in the diff which is worth doing in this PR. I'm +1 on this or something like it.


I updated the patch based on the feedback. I never actually used streaming responses myself so I didn't actually realize that the logic was inverted. Good catch.


Thanks so much for this!

My only feedback is this:

`s = requests.Session()`

Instead of:

`with requests.session() as s:`


I don't know exactly what the stance on this is, but I can rebase these commits if you prefer.


:sparkles: :cake: :sparkles:
Add ability to pass `response` to `HTTPError()`

Just a little refactoring, but it seems nicer to me to be able to pass
the response when constructing the `HTTPError` instance instead of
constructing it and then changing the member variable.


Fixed test failing on Python 3 and force-pushed.


I'm +1 on this. It definitely seems worth adding to allow for extra information and it isn't so much a change in the API although @Lukasa and @kennethreitz may disagree.


Well, it doesn't really add any extra info as `response` was attached to `HTTPError` and before this. It's just that you couldn't've do it with the initialization of the exception, only attaching afterwards like `instance.response = response`.


I'm +0 on this. As you said @dmedvinsky, it's a minor refactor. I have no objection to it being in the library, and it's an internal API really, so not too problematic. I'm happy to go with whatever @kennethreitz feels is right here.


Seems legit. :100: 


I'll merge this if you do a rebase so it'll merge :)


There you go. :twisted_rightwards_arrows: 


grr, still no luck. I'll do it manually :)


Thanks again!


Well, if pull requests are making it so fast in here that it hard to keep them rebased, I wouldn't say “no luck”. I call it a win. :-)
Documentation:  response object status/reason

The response header [documentation](http://docs.python-requests.org/en/latest/user/quickstart/#response-headers) claims that the headers dict will contain a 'status' entry, containing both the return status and the reason, for example '200 OK'.

In my tests, no such entry exists in the headers dict, although the same data is available with a combination of (assuming a response object called 'r') r.status_code and r.reason.


Wow. That's not been the case for...a while. Want to submit a docs PR to remove it? I'd do it, but I'm preparing to get on a plane.


Fixed by #1234. Thanks!


Note that now the `response.reason` attribute is _entirely undocumented_.


This change did not cause that situation to arise. You're right though, it should at least be in the API docs (even though `response.reason` isn't hugely useful).


> (even though response.reason isn't hugely useful)

[Someone finds it useful](http://stackoverflow.com/q/21583195/100297); I was looking into creating a pull request, but the API docs are generated from the source. Digging a little further.


Yeah, I can't imagine how it would be useful to them, but whatever they need. =)

The fix is to put a comment above [this line](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L544) in the format of the other comment blocks in that function, e.g.:

``` python
        #: The reason string returned by the server, e.g.
        #: "Not Found" or "OK".
        self.reason = None
```
IOError when accessing a Response object after socket closes

If the server closes the connection immediately after writing the data, an attempt to use the Response object causes an IOError.

``` python
r = requests.get(url)
  # Server closes connection
print(r.content) # Throws IOError
```


Why is this an issue? We can only tell you exactly what happened, which is exactly an error with IO.


My assumption is that the get() call sends a GET request and captures the output, storing it in the Response object.
If that is the case, then I should be able to access that Response as long as it stays in scope, regardless of the connection status. But accessing the content of the response raises the IOError. I can access the status_code and headers without any problem.
There doesn't appear to be any way to use requests with servers that write their data and close the socket.


There's no chance you would provide a website that does this so we could test/reproduce it, is there?


I finally got a look at the code for the server I'm connecting to. The error was on that side.
Fixes __getstate__ for session pickling

As noted in #1088, sessions cannot be pickled currently. I ran into this when implementing some multiprocessing functionality for a project. 

Here, I'm just checking for `__attrs__` prior to trying to iterate over it. It does occur to me that if `__attrs__` is removed permanently, maybe the `__getstate__` method should just be stubbed out to preserve pickling functionality.


:-1: Why not just add `__attrs__` back to the Session object? You also have the code you describe in the PR commented out.

Fix it up please?


Whoops. I committed while I was testing the later possibility of stubbing out the method. I amended back to the way I originally had it.

I'm fine with adding `__attrs__` back to the session object, but 42b0295 removed it for some reason. 

My personal feeling is to program to the interface and not add an unused attribute.

Your call, since you have a better idea of the ongoing needs of the lib.


Yeah, I'd say just re-add the attribute. It was probably deleted by accident. Otherwise we're just going to consistently return `{}`, which is useless.


Yeah, let's add it back :)


Changed `__getstate__` back to its original form and added back `__attrs__` to the Session object.


:+1:


Thank you sir :)
NameError in virtualenv

Hello,

I have just attempted to install requests into a virtualenv environment and am getting the error:
    NameError: name 'CERT_NONE' is not defined

The full stacktrace is below. I tried to install requests with the command pip install -U requests in my virtual environment. An admittedly quick Google search didn't turn up anything, but I am assuming its user error. Am I doing it wrong somehow? 

This is on Red Hat Enterprise Linux 5 with Python 2.7.3. I have tried virtualenv with and without site packages, same error either way. If there is any more information I can provide, feel free to let me know.

Thanks,

Brian

Full Stack trace from error:

249   File "requests/packages/**init**.py", line 3, in <module>
250
251     from . import urllib3
252
253   File "requests/packages/urllib3/**init**.py", line 16, in <module>
254
255     from .connectionpool import (
256
257   File "requests/packages/urllib3/connectionpool.py", line 45, in <module>
258
259     from .util import get_host, is_connection_dropped, ssl_wrap_socket
260
261   File "requests/packages/urllib3/util.py", line 293, in <module>
262
263     def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=CERT_NONE,
264
265 NameError: name 'CERT_NONE' is not defined
266
267 ----------------------------------------
268
269 Command python setup.py egg_info failed with error code 1 in /home/bneal/pyv/build/requests
270
271 Exception information:
272 Traceback (most recent call last):
273   File "/home/bneal/pyv/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/basecommand.py", line 107, in main
274     status = self.run(options, args)
275   File "/home/bneal/pyv/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/commands/install.py", line 256, in run
276     requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
277   File "/home/bneal/pyv/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/req.py", line 1042, in prepare_files
278     req_to_install.run_egg_info()
279   File "/home/bneal/pyv/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/req.py", line 236, in run_egg_info
280     command_desc='python setup.py egg_info')
281   File "/home/bneal/pyv/lib/python2.7/site-packages/pip-1.2.1-py2.7.egg/pip/util.py", line 612, in call_subprocess
282     % (command_desc, proc.returncode, cwd))
283 InstallationError: Command python setup.py egg_info failed with error code 1 in /home/bneal/pyv/build/requests


Duplicate of #1221. This seems to be related to your not having SSL installed or properly configured/linked in Python. The fix is being worked out in shazow/urllib3 and will be imported as soon as it's ready.


The fix has been on requests master for a long time. (#1128)


Ah @t-8ch sorry for the confusion. I must have misread @Lukasa's response to #1221 last night (or had some rather wild dreams of what it said). Thanks for the correction.


You could have read it just right. This one _is_ a duplicate, but both are fixed on master. :smile:


_shrug_ I'm too tired for life :-P
Adding max_retries as an argument

Related ticket: #1208

This pull request adds support for `max_retries` as an argument to `requests.request`. I think that since we already support `timeout` as a top-level argument then it makes sense to support `max_retries` too. 

I've added a unit test, but this is a very difficult thing to test for. I can't see any easy way to simulate an unreliable connection.

Please let me know what you think.


There is nothing wrong with this PR, but it won't be accepted. Requests is in [feature freeze](http://docs.python-requests.org/en/latest/dev/todo/#feature-freeze), and that goes doubly for changes that affect the API. As @sigmavirus24 pointed out in #1208, there are plenty of ways to specify this value.


Thanks, I'll send a docs PR.


Basically, max_retries is a feature of urllib3, not of requests. Requests has always tried to discourage automatic retries as much as possible.

Thanks for the contribution though, it was a good one. Keep 'em coming! 


Uh, @kennethreitz, did you intend to merge this? Your comment seems like you weren't going to, but then you kinda did...


oh geeze


Reverted.
Make documentation for old versions of requests available

I can't keep up with the progress that requests is making in one of my projects. On the one hand, it's great that requests is constantly being developed. On the other hand, I can no longer find the documentation which applies to the version I'm actually developing with (0.14.2), which is inconvenient.


I know I can view it in the github code viewer; what I'm asking is that old doc versions be included on readthedocs.


They used to be included, but RTD's version picker got absurd after 10+ versions.


Yeah, I've run into that issue with their UI as well. You've inspired me to open a ticket on the subject: rtfd/readthedocs.org#344


0.14.2's docs are now included on RTD.
Not sure if minor spelling mishap

Checked it out, couldn't see pycon as a syntax highlighter (not trying to be a smart ass either :sparkles:)


pycon works on GitHub, but not sphinx if I remember correctly... then again 
GitHub uses pygments and so does sphinx... I'm honestly not sure now
On Sun, Feb 24, 2013 at 04:21:06PM -0800, Rhys Elsmore wrote:

> Checked it out, couldn't see pycon as a syntax highlighter (not trying to be a smart ass either :sparkles:)
> You can merge this Pull Request by running:
> 
>   git pull https://github.com/rhyselsmore/requests develop
> 
> Or you can view, comment on it, or merge it online at:
> 
>   https://github.com/kennethreitz/requests/pull/1216
> 
> -- Commit Summary --
> - Not sure if minor spelling mishap
> 
> -- File Changes --
> 
> ```
> M README.rst (2)
> ```
> 
> -- Patch Links --
> 
> https://github.com/kennethreitz/requests/pull/1216.patch
> https://github.com/kennethreitz/requests/pull/1216.diff


pycon = python console :)


The things you learn. Ta

On 25/02/2013, at 11:23 AM, Kenneth Reitz notifications@github.com wrote:

pycon = python console :)

—
Reply to this email directly or view it on
GitHubhttps://github.com/kennethreitz/requests/pull/1216#issuecomment-14020693.
Requests doesn't compute Content-Length correctly when transmitting OAuth params in request body. Change for issue #1159 regressed fix for issue #1096.

The change for #1159 regressed the fix for #1096. Recomputing Content-Length for requests that transmit OAuth params in the request body is broken again. Here's a test case:

```
[mlm@matt-mmf-macbook.local:~]
$ ipython
Python 2.7.3 (default, Sep 15 2012, 19:45:36) 
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import requests

In [2]: from requests_oauthlib import OAuth1

In [3]: oauth = OAuth1(client_key=u'a-client-id', client_secret=u'a-client-secret', callback_uri=u'http://example.com/oauth_callback', signature_type=u'BODY')
In [4]: req = requests.Request(method='POST', url='http://example.com/request_token', headers={ 'Accept': 'application/x-www-form-urlencoded', 'Content-Type': 'application/x-www-form-urlencoded' }, auth=oauth).prepare()

In [5]: req.__dict__Out[5]: 
{'body': u'oauth_nonce=119221505668693451461361716039&oauth_timestamp=1361716039&oauth_version=1.0&oauth_signature_method=HMAC-SHA1&oauth_consumer_key=a-client-id&oauth_callback=http%3A%2F%2Fexample.com%2Foauth_callback&oauth_signature=azH8mCrSMYvEpQ7oonDIoIq38oQ%3D',
 'headers': {'Accept': 'application/x-www-form-urlencoded',
  'Content-Length': '0',
  'Content-Type': u'application/x-www-form-urlencoded'},
 'hooks': {'response': []},
 'method': 'POST',
 'url': u'http://example.com/request_token'}

In [6]: resp = requests.request(method='POST', url='http://example.com/request_token', headers={ 'Accept': 'application/x-www-form-urlencoded', 'Content-Type': 'application/x-www-form-urlencoded' }, allow_redirects=False, auth=oauth)

In [7]: resp.request.__dict__
Out[7]: 
{'body': u'oauth_nonce=40755579898176978971361716049&oauth_timestamp=1361716049&oauth_version=1.0&oauth_signature_method=HMAC-SHA1&oauth_consumer_key=a-client-id&oauth_callback=http%3A%2F%2Fexample.com%2Foauth_callback&oauth_signature=Ti910zdRt%2BC5UxLG2dJ5eJ%2BkZ90%3D',
 'headers': {'Accept': 'application/x-www-form-urlencoded',
  'Accept-Encoding': 'gzip, deflate, compress',
  'Content-Length': '0',
  'Content-Type': u'application/x-www-form-urlencoded',
  'User-Agent': 'python-requests/1.1.0 CPython/2.7.3 Darwin/11.4.2'},
 'hooks': {'response': []},
 'method': 'POST',
 'url': u'http://example.com/request_token'}

In [9]: 
```


I didn't like that pull request anyway. Like we said on that issue, the user should go way out of their way to break things. @kennethreitz do you mind if I revert and push directly to requests? It's really not much of a difficult fix.


@sigmavirus24 sure, this once :)


Or you could revert it yourself ;)


Anyway this can now be closed thanks to be62645dd56580dd7576032b348cf79d880851d8


Thanks!


Thanks for reporting it! I'm sorry it happened in the first place.
Status Code Descriptions

I think being able to return a description for HTTP status codes might be nice. Something you could print out along with the status code, in case you had to.

I'm thinking a use case as something along these lines:

``` python
r = requests.get(url, timeout=20)

if r.status_code != requests.codes.ok:
    return 'HTTP {code}: {desc}'.format(code=r.status_code,
            desc=requests.status_codes.description[r.status_code])
```

With something like this added to request_codes:

``` python
description = {
    # ...
    403: 'Access Denied',
    404: 'File Not Found',
    405: 'Method Not Allowed',
    406: 'Not Acceptable'
    # ...
}
```

And other descriptions mostly grabbed from [Wikipedia.](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)

I think it'd be a nice addition, especially if we could roll it into `r`, as `r.status_description` or something similar. I can write up some code and submit a PR, if you think this would go well in Requests.

And thanks for the awesome library. I dunno how I ever lived without Requests :smile: 


Sorry, should've written the code right away, rather than duping the issue with the PR. See #1213
Not possible to specify max_retries in v1.X?

In older versions of requests (pre v1.0), I was able to do:

```
requests.get('http://nonexistentdomainfoobar.com', config={"max_retries":10})
```

as far as I can tell, this isn't possible in v.1.0+. `HTTPAdapter.max_retries` uses `DEFAULT_RETRIES` and there's no way to change this.

Would it be possible to restore this feature? If not, perhaps a note in the FAQ informing users that this isn't possible and they'll have to write a loop themselves?


Why not do this?

``` pycon
>>> import requests
>>> from requests.adapters import HTTPAdapter
>>> s = requests.session()
>>> h = HTTPAdapter()
>>> h.max_retries = 10
>>> h.max_retries
10
>>> s.mount('http://', h)
>>> s.mount('https://', h)
>>> s.get('http://nonexistentdomainfoobar.com')
```

It isn't an "easy" config option, but it certainly is still possible to configure it. That aside, it's definitely worth adding to the docs.


Hmm, i wonder if we can just modify session.adapters directly 


Why not:

``` python
import requests
s = requests.Session()
url = 'http://testurl.com/'
h = s.get_adapter(url)
h.max_retries = 10
```

This method uses the recommended method of interacting with Transport Adapters on sessions.

If you really really wanted to, you could take advantage of the fact that the adapters are a Python dictionary (which I'm pretty sure is just an implementation detail) and do this:

``` python
import requests
s = requests.Session()

for _, adapter in s.adapters.iteritems():
    adapter.max_retries = 10
```


@Wilfred, I'm not sure why you submitted a pull request. Also I'm not sure why this issue stayed open when there were two sufficient solutions for fixing this.


@sigmavirus24 The two solutions given are not documented anywhere. It would be nice to at least document how to do this.

That being said, it's a shame that a simple GET to a URL becomes ~5 lines of code in requests v1.0, when it was just one line before.


Well we also got rid of `pre_request` hooks so that to do something equivalent looks like: https://gist.github.com/sigmavirus24/5049172 but that's the beauty of semantic versioning.

We've been behind on updating the docs, but I'll add this to the list of things that need to be added.
URL-encoding not handled for proxy username and password

Proxy authentication information is specified by the `proxies` dictionary passed to `requests.get()` and other methods, like so:

```
requests.get(..., proxies = {'http': 'http://user:pass@myproxy.mydomain.com'})
```

Because the username and password are specified as part of a URL, they _must_ be URL-encoded by the caller; otherwise consider what would happen if the user's password or username contained an `@` or a `:`, etc.

So, for example, if my proxy password is `p@ssword!!!`, I need to write my call like this:

```
requests.get(..., proxies = {'http': 'http://user:p%40ssword%21%21%21@myproxy.mydomain.com'})
```

However, this functionality is currently broken, because `utils.get_auth_from_url` does not _decode_ the URL-encoded username and password. The result is that the raw URL-encoded username and password are sent in the `Proxy-Authorization` header to the proxy, but of course the proxy expects the _actual_, decoded username and password, so authentication fails.

It is a one-line fix to perform URL decoding in `utils.get_auth_from_url`. It would also be a good idea to add this to the `test_get_auth_from_url` test case.

I would submit a pull request myself, but the fix I developed was written on company time, so I would need like 7 signatures to release it you. Argh.


That's ok, I'll work on it. =)


See #1210 for the PR.


Fixed. =) Thanks for raising the issue!


Thanks for fixing :+1:
Default value of allow_redirects in HEAD requests

In version 1.1.0 allow_redirects for HEAD defaults to True.
According to the docs [1], though, the default should be False.

[1] http://docs.python-requests.org/en/latest/user/quickstart/#redirection-and-history


Here's the test case:

```
>>> import requests
>>> requests.__version__
'1.1.0'
>>> requests.head('http://httpbin.org/redirect/2')
<Response [200]>
>>> _.history
(<Response [302]>, <Response [302]>)
```


I know what the issue is. Also if you instantiate a Session first, this doesn't happen. Thanks!


Thanks.