Raibek/requests-github-issues huggingface dataset:
get_redirect_location return error hostname

when I visited site http://bitsnoop.com/chano-dominguez-hecho-a-mano-200-q18406062.html like this:

```
In [3]: r = get('http://bitsnoop.com/chano-dominguez-hecho-a-mano-200-q18406062.html', allow_redirects=False)
In [4]: r.status_code
Out[4]: 302
In [5]: r.raw.get_redirect_location()
Out[5]: 'http://127.0.0.1/'
```

I got the error redirect address. So I test the website with curl:

```
$ curl -v http://bitsnoop.com/chano-dominguez-hecho-a-mano-200-q18406062.html
* Hostname was NOT found in DNS cache
*   Trying 31.7.59.14...
*   Trying 2a02:29b8:1925::2...
* Immediate connect fail for 2a02:29b8:1925::2: Network is unreachable
* Connected to bitsnoop.com (31.7.59.14) port 80 (#0)
> GET /chano-dominguez-hecho-a-mano-200-q18406062.html HTTP/1.1
> User-Agent: curl/7.35.0
> Host: bitsnoop.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Content-Type: text/html; charset=utf-8
< Keep-Alive: timeout=300
< Location: http://bitsnoop.com/chano-dominguez-hecho-a-mano-2002-q18406062.html
< Transfer-Encoding: chunked
< Date: Sat, 13 Feb 2016 01:29:09 GMT
< Connection: keep-alive
* Server Hampsterbox/2.0 is not blacklisted
< Server: Hampsterbox/2.0
<
* Connection #0 to host bitsnoop.com left intact
```

Obviously the right redirect address is http://bitsnoop.com/chano-dominguez-hecho-a-mano-2002-q18406062.html. It's really confusing;)


`r.raw.get_redirect_location()` is not actually part of the Requests API, and is not a supported way to procure that information.

The proper way (as it stands) is to simply look at the value of the `Location` header:

``` python
>>> r.headers['Location']
'http://127.0.0.1/'
```


@jaron92: It looks like this server is sending a phony redirect based on the `User-Agent` header the request used. 

This works for me:

``` python
url = 'http://bitsnoop.com/chano-dominguez-hecho-a-mano-200-q18406062.html'
headers = {'User-Agent': 'curl/7.43.0'}

r = requests.get(url, headers=headers, allow_redirects=False)
```

``` python
   >>> r.headers['location']
'http://bitsnoop.com/chano-dominguez-hecho-a-mano-2002-q18406062.html'
```

This is why you were seeing differing results with cURL. 


Thx.

It helps me a lot;)


@jaron92 sure thing! :cake:
Add ability to pass CA certificates by fileobj

Sometimes requests has no ability to open file with CA because is no such file, e.g. when CA is embedded in the app.

requests should allow to specify fileobj (which can be a file or io wrapper on top of string and bytes) for that.


Unfortunately that's extremely difficult for requests to do. 

The 'path to a file' approach is actually the API used by OpenSSL. It's also possible to pass an in-memory serialized representation (as read from a file object), but that API is not exposed by the standard library so we can't rely on that being present. 


We could create a tmpfile automatically when passed a file-like-object, but we would likely want to ensure its deletion after execution is complete.

So, definitely possible, but probably not the best idea.


Creating intermediate file can be handled by user of requests.

After checking Python's code I agree that it's, at least, a limitation in Python.
Sad that OpenSSL relies on file being present in FS.


@Lukasa is there a way to query presense of this API in runtime? Could you give me some links to read more about it?


Is it possible to verify by using system's CA without shipping cacert.pem nor certifi?


@Kentzo: for a discussion about the system cert bundle, see #2966. It is possible to query for this API at runtime because it's only exposed by PyOpenSSL.


Python 2.7.9+ and 3.5's implementation of `load_verify_locations()` in `Modules/_ssl.c` support passing of a `cadata` argument, which contains the serialized trust store. This is also exposed in the `ssl.create_default_context()` function.

@Lukasa I think the original issue is orthogonal to #2966 and this issue should be re-opened because of the discovery made by @dsully.
Use time.monotonic() for tracking elapsed time where available


Hey @scop I was wondering if you could provide detail about why this is desirable and why we should change the existing behaviour as well as a description of when/where monotonic will be available and where/when it won't.


Most of the requested details should be answered by the `time.monotonic` docs: https://docs.python.org/3/library/time.html#time.monotonic

In a nutshell, this change makes `r.elapsed` unaffected by system clock changes where `time.monotonic` is available. There are no changes to behavior on setups where it isn't. It was introduced in Python 3.3. In 3.3 and 3.4 it wasn't available on all platforms, but in 3.5 it is.


Have you run into issues with the elapsed attribute that this would help with?


No, but I have elsewhere with similar things, just thought it'd be pretty much a no-brainer to fix in requests as well.


I don't think this is anything we need to be concerned about, and it adds unnecessary complexity to one of our simplest (and least used) features. I understand the thought process behind this PR, and if we were all using 3.5, then I'd probably accept it (with a few improvements).
Added pprint() function to Response and PreparedRequest objects for human-reading goodness

This feature is aimed at making debugging/following HTTP traffic easier on a developer. Many times in the past, I would send Requests' traffic through Burp proxy in order to have a visual representation of my HTTP traffic. With this pull request, a developer can simply call pprint() on their HTTP requests to see a Burp-like view of the traffic.

Example usage:

```
import requests

r = requests.get("http://www.praetorian.com")
r.request.pprint()
r.pprint(body=False)
```


This already exists (to a degree) in the requests-toolbelt: https://toolbelt.readthedocs.org/en/latest/dumputils.html


My major problem with the dump utility is that it is not as granular as pprint(), which gives a developer the ability to dump a single request or a response without the body attached (good for when the response is a large static page). The dump util returns a lot more data than is usually needed while monitoring Requests traffic.


@7uice Presumably we could add flags to `dump` to resolve that problem though?

Generally speaking I'm in favour of keeping this kind of code out of requests: it becomes one more thing we have to maintain.


@Lukasa It looks like this could be accomplished using utils.dump by adding a `print_body=True` flag and wrapping the following lines in a conditional statement:

`line 78: bytearr.extend(prefix + _coerce_to_bytes(request.body))`

`line 106: bytearr.extend(response.content)`

This would allow a user to print Request & Response while not flooding their screen with data returned in the response. If it's preferred to keep this out of Requests, I can move the code over to the utils.dump code.


I think that would be better for now. =) At some time that may make its way over here some time in the future. =)


Sounds good :) Closing this for now and moving the feature into utils.dump.
Multipart mixed responses generate warnings

If you read a multipart/mixed response, the connectionpool issues a couple of warnings about defects in the message. I'm not sure what the expected, correct behavior is, but these warnings seem spurious.

Stick this perl script somewhere:

```
#!/usr/bin/perl

print "Server: Some Server Name\r\n";
print "Content-Type: multipart/mixed; boundary=36eeb8c4e26d842a\r\n";
print "Content-Length: 178\r\n";
print "\r\n\r\n";
print "--36eeb8c4e26d842a\r\n";
print "Content-Type: text/plain\r\n";
print "\r\n";
print "7\r\n";
print "--36eeb8c4e26d842a\r\n";
print "Content-Type: text/plain\r\n";
print "\r\n";
print "9\r\n";
print "--36eeb8c4e26d842a\r\n";
print "Content-Type: text/plain\r\n";
print "\r\n";
print "11\r\n";
print "--36eeb8c4e26d842a--\r\n";
```

Read it with requests (naturally, you'll have to change the URI to wherever you put the script):

```
import requests, logging

logging.basicConfig(level=logging.WARNING)
logging.getLogger("requests").setLevel(logging.DEBUG)

headers = {'accept': "multipart/mixed"}
r = requests.get("http://localhost:8124/cgi-bin/mpm.pl", headers=headers)

print(r)
```

The following errors are displayed:

```
DEBUG:requests.packages.urllib3.connectionpool:"GET http://localhost:8124/cgi-bin/mpm.pl HTTP/1.1" 200 178
WARNING:requests.packages.urllib3.connectionpool:Failed to parse headers (url=http://localhost:8888/http://localhost:8124/cgi-bin/mpm.pl): [StartBoundaryNotFoundDefect(), MultipartInvariantViolationDefect()], unparsed data: ''
Traceback (most recent call last):
  File "/home/ndw/.virtualenvs/pyapi/lib/python3.4/site-packages/requests-2.8.0-py3.4.egg/requests/packages/urllib3/connectionpool.py", line 390, in _make_request
    assert_header_parsing(httplib_response.msg)
  File "/home/ndw/.virtualenvs/pyapi/lib/python3.4/site-packages/requests-2.8.0-py3.4.egg/requests/packages/urllib3/util/response.py", line 58, in assert_header_parsing
    raise HeaderParsingError(defects=defects, unparsed_data=unparsed_data)
requests.packages.urllib3.exceptions.HeaderParsingError: [StartBoundaryNotFoundDefect(), MultipartInvariantViolationDefect()], unparsed data: ''
```

It took me quite a while to work out that they were spurious (because in real life, the server side that is generating the multipart/mixed is more complicated!)


Ah, can you raise this issue on the urllib3 repo! We'd love to merge a fix at the project! It's [here](https://github.com/shazow/urllib3).
Problem with Content-Lenght is missing

I'm playing clonk rage but i can't play online because this happend 
Content-Lenght is missing pls help me
PS: i don't speak english so well


@victor1969 can you please share the code you are using so we can understand what you're asking?


where i get the code?


@victor1969 is this an error you saw while playing a game? Why did you decide to post about it here?


i'm just trying to play online Clonk rage but says on the server list
Internet server on league.clonkspot.org
Invalid server response: Content-Length is missing!


@victor1969 unfortunately, that has nothing to do with project. There's probably somewhere else online for you to report this, though!


ok 
:(
URL-encoded boolean parameter passed as ?foo=True instead of ?foo=1

When encoding a boolean into a URL encoded parameter:

```
payload = {'foo': True}
resp = requests.post(path, params=payload)
```

Requests currently generates: `?foo=True` .. I would expect the boolean to be encoded as a 1 or 0, e.g. `?foo=1` , as some clients may not understand `True` as a boolean. 

I did some google searching, and I am not even sure if there are rules for how a boolean should be encoded, I didn't find any specific guidelines, just that a lot of URL encoding libraries seem to always encode booleans as 1's and 0's.

Apologies if this is known and/or expected behavior.  I'm not a web guru by any means, but at the least, Requests does seem to be handling it differently than other libraries I've used in the past.


There are no rules for how to encode Boolean types, because urlencoding is performed on strings, not on typed objects. For that reason, Reuests coerces all objects passed to the urlencoding logic to strings (except for `None`, which it handles specially).

This is not considered a bug by the requests team: while it can be surprising if you come from a non-requests background, I'd argue that in aggregate our behaviour is far less surprising than converting objects into entirely different representations. 


@FlyingLotus1983 if you want a `1` there, you should provide a `1` :)
Requests default to environment CA even when there's a session-set CA

If I set a CA via an environment variable, such as:

``` console
$ export REQUESTS_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
```

(Where the CA bundle actually does exist)

And then I prepare a session like this:

``` python
import requests

session = requests.Session()
session.verify = '/path/to/custom/ca.pem'
```

(Again, assume this 2nd CA path exists)

Executing a `session.get` against an HTTPS endpoint signed by the 2nd CA but not the first will result in an SSL verification error. This is because the request is defaulting to using the environment CA rather than the session CA. This sort of contradicts my expectations, where the session is much closer to the request in the stack than the process's environment.

The source of this issue is the merging happening here:

https://github.com/kennethreitz/requests/blob/3a77b59df222920391359f052cab9319dda31ac7/requests/sessions.py#L625

...where the conditional validates whether the verification is set at the request level (either `True` or `None`) before choosing to set the env CA, but doesn't look at the session (`self.verify`) (which likely has, and in my case does have, a more accurate CA setting).


I believe this is related to https://github.com/kennethreitz/requests/issues/2018 which is scheduled for 3.0.0


As an aside, you can disable using the environment by setting `trust_env=False`.


Closing to centralise on #2018.


@sigmavirus24 That's the workaround I intend to use, although it's a little troublesome since I'm using a library which is itself using `requests`, and it doesn't expose `trust_env`, so I'm having to patch it in. It's also troublesome because it disables proxy configuration via the environment.

Thanks though, I didn't see the other issue!
Added tests module

Hello, I want to propose some tests changes again :)

What is done:
1. All tests are moved to`tests` module.
2. All tests related to `requests.utils` are separated in `tests.test_utils` module.
3. Fixtures `httpbin` and `httpbin_secure` were slightly refactored and moved to `tests.conftest` module.
4. `ThreadPool` was removed, because it was not used.
5. All compatibility code was moved to `tests.compat`.
6. More parametrization was added.

I want to ask you if I'm going in the right way or not. Also I noticed, that `requests.utils` has ~45% coverage by unit tests, so, would you mind if I'll add more unit tests i this PR? If everything is ok


Pinging @kennethreitz again for review on test code.

Thanks for this @Stranger6667!


@Stranger6667 more tests for `utils` are more than welcome!


:sparkles: :cake: :sparkles:


Thank you, @Lukasa @kennethreitz ! :)
SSL Get Server Certificate Error

I'm getting certificate verify failed errors for a few websites including Google. I've just tried with a new Digital Ocean droplet and still having the same issues so it's not just my server.

I did the following, create a new Ubuntu 14.04 droplet on Digital Ocean. Installed python-pip (python and OpenSSL were already installed). Upgraded requests to the latest version (pip install requests --upgrade) and installed security (pip install requests[security]). Just to confirm, the droplet originally came with requests version 2.2.1, hence the reason for upgrading to 2.9.1. I also installed the pip modules: certifi and urllib3.

Before upgrading requests and installing the other modules, I ran `import requests` and `requests.get("https://www.google.co.uk")`. This worked fine and returned 200.

After upgrading and installing the other packages, I got the following:
`Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:510: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
`

I tried it on https://github.com and it worked perfectly. I tried it on https://google.com and https://themoviedb.org and both has certificate verify failed errors even though they both have valid certificates.


You'll need to read certifi/python-certifi#26.
Changes to support sending a complete response object when 407 errors…

… is found while doing the CONNECT to proxy. Added a way to stablish a connection using connect over an already connected socket

Trying to connect to a "https" web page using a proxy uses a "HTTP CONNECT" verb to tell the proxy that we want to connect to a https destination.
With "basic authentication" the "CONNECT" is sent with the appropriate headers so the proxy can authenticate the request and do the request to the https web server. (That headers are setup by the requests library)
With "NTLM" we use a plugin (requests-ntlm) that is set as a hook and it is called on every request response. For example: with HTTP, the requests library sends a "GET" to a web page, as a response we get a HTTP-407 that is handled by the plugin hook to do the NTLM negotiation and authentication
In a HTTPS scenario, the first connection to the proxy is being done with the "CONNECT". At that time, requests does not set any authentication header (because it is handled by a plugin) and we get an 407 authentication Error.

This Patch tries to avoid raising an exception on "Proxy Authentication" errors and it sends back a complete http response so it can be handled by the requests-ntlm plugin and reuse an already connected and authenticated socket to do the connection


@angelcaido19 Sorry, but this includes changes in urllib3. You'll need to propose this patch over there first, I'm afraid. =(


@Lukasa Hey thanks. Help me with this: Is this "https://github.com/shazow/urllib3" the oficial urllib3 repo?
Thanks again!!


Ah, sorry I didn't link it: yeah it is!
Consider simplifying Session.resolve_redirects

Currently, the `Session.resolve_redirects` method requires a `Response` to be passed in, as well as a `Request` (the response's originating request). When using this method directly, this is a bit confusing, especially considering that the desired/required `Request` is an attribute of the obviously required `Response`.

There must have been some reason behind why I made this design decision, but I'm not seeing it currently.


Adding this to 3.0.0 because it would be a fairly significant breaking change. I was also never quite certain why that was required, but I'm not certain it is actively harmful either :/


Yeah, this would be a 3.0.0 change. 

@sigmavirus24 I was just trying to use it directly while debugging an infinitely redirecting URL, and I had to look at the source code to figure out how to do so. I just really need to spend some time figuring out why I wrote it that way, there must have been a reason.


Done! https://github.com/kennethreitz/requests/blob/proposed/3.0.0/requests/sessions.py#L88
Any idea if it would be possible to use something like oscrypto instead of the bundled SSL lib for HTTPS?

Any idea if it would be possible to use something like [oscrypto](https://github.com/wbond/oscrypto) instead of the bundled SSL lib for HTTPS?

I need to use this in a Sublime Text 3 plugin where the bundled version of SSL is pretty old and I can't connect to some hosts supporting only TLS1.2. Using the above should fix this issue.

Any idea how this can be used for SSL?


@danechitoaie Let's split this into two questions.

First, the problem with OpenSSL. If you can use a third-party library, you should be able to install `PyOpenSSL`, `ndg-httpsclient`, and `pyasn1`. If the `cryptography` library that backs PyOpenSSL is linked against a newer version of OpenSSL than the bundled one (or if you're on Windows or OS X, where `cryptography` is statically linked against a modern OpenSSL).

Using oscrypto instead is extremely tricky. Requests cannot add support for this in the mainline. You could in principle add support for this in urllib3, but that won't happen quickly. PyOpenSSL is probably your best bet here.


I can't use PyOpenSSL as that requires compilation and it would not work as a SublimeText3 plugin as they have their own version of python bundled that executes the plugins code. So you can't use pip, easy_install, etc... you can only use "pure" python code that doesn't have external dependencies.
Oscrypto would work as it's just Python code (nothing to compile) that uses the native OS SSL lib trough CFFI. 

I understand that it can't be added to requests, I'm looking at maybe some pointers or ideas on how/if this could be implemented, even if it means I have to change some code in requests or urllib3 (private fork). I know it doesn't sound good but it's kind of the only option.


@danechitoaie PyOpenSSL and oscrypto both use the exact same backing library. PyOpenSSL has not required compilation for more than a year now.


Hmm, then I need to look more into PyOpenSSL then. Maybe I can just use that. Thanks for the input.
Enhanced form-data serialization

Would be great to include the functionality of `requests_toolbelt.utils.formdata.urlencode` in Requests. 

http://toolbelt.readthedocs.org/en/latest/formdata.html

Thoughts? 

/cc @Lukasa @sigmavirus24 


No objection from me, but the tool belt is @sigmavirus24's baby so he may have a stronger position,


To be clear, I mean making that the default behavior. 


I think I'm the only contributing author of that code and both requests and the toolbelt are Apache-2.0.

If we want to move that into requests by default, we absolutely can and I'm happy to do that. We'll leave it in the toolbelt for people who are using requests prior to whatever version we ship that in.


> I think I'm the only contributing author of that code and both requests and the toolbelt are Apache-2.0.

I only bring this up because copyright is a huge PITA ;)


That's true, but I think that definitionally requests can use that code by reproducing the license: any other contributors have by definition made that code available under Apache 2.0.

Otherwise, I'm happy to do this. I'll happily do the (probably cursory) code review. 

This is another great use of the toolbelt by the way: letting ideas marinate and get tested outside the core codebase. 


> This is another great use of the toolbelt by the way: letting ideas marinate and get tested outside the core codebase. 

My thoughts exactly.

> Otherwise, I'm happy to do this. I'll happily do the (probably cursory) code review. 

I think you did the original CR too ;)


> I think you did the original CR too ;)

Oh god in that case we should do a really thorough code review!


In an effort to clean up the issue tracker to only have issues that are still relevant to the project we've done a quick pass and decided this issue may no longer be relevant for a variety of potential reasons, including:

* Applies to a much older version, unclear whether the issue still applies.
* Change requires a backwards incompatible release and it's unclear if the benefits are worth the migration effort from the community.
* There isn't a clear demand from the community on the change landing in Requests.

If you think the issue should remain open, please comment so below or open a new issue and link back to the original issue. Again, thank you for opening the issue and for the discussion, it's much appreciated.
Remove int type on HTTPAdapter's max_retries argument doc

The argument description says `max_retries` can be an int, or a `urllib3.util.Retry` object. However, the type declared in the doc restricts it to int only. Passing a Retry object causes a warning in PyCharm code inspection.

The behaviour described for `max_retries` is accurate: HTTPAdapter passes `max_retries` to `Retry.from_int()`, which just returns the input if `max_retries` is a Retry object.


Thanks!
First line of uploading files disappears.

requests 2.9.1 
Executing of this code (from samples in documentation):

`import requests

url = 'http://192.168.88.164:8000/'
files = {'file': ('conf.txt', 'some, data,to send\r\nnanother,row,to,send\r\nyet another,row,to,send\r\n')}

r = requests.post(url, files=files)
print r.text`

Results in correctly uploaded file, but without first line.
I.e. part 'some, data,to send\r\n' is missed.

Same with actoual file upload:
`import requests

url = 'http://192.168.88.164:8000/'
files = {'file': ('conf.txt', open('conf.txt', 'rb'))}

r = requests.post(url, files=files)
print r.text`

First line disappears from uploaded content.
The same file uploaded to the same server with curl:
`curl -v -X POST -F file=@conf.txt http://192.168.88.164:8000`
do not such an issue.

Issue seems to be in incorrect encoding - extra \r\n where missed somewhere.


Sorry, but I can't reproduce this using a publicly available website. If I run this code:

``` python
import requests
url = 'http://http2bin.org/post'
files = {'file': ('conf.txt', 'some, data,to send\r\nnanother,row,to,send\r\nyet another,row,to,send\r\n')}
r = requests.post(url, files=files)
print r.content
```

then the terminal prints:

```
{
  "args": {}, 
  "data": "", 
  "files": {
    "file": "some, data,to send\r\nnanother,row,to,send\r\nyet another,row,to,send\r\n"
  }, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Connection": "keep-alive", 
    "Content-Length": "211", 
    "Content-Type": "multipart/form-data; boundary=83f1175700aa4a2b8a49111a87fce425", 
    "Host": "http2bin.org", 
    "User-Agent": "python-requests/2.9.1", 
    "Via": "1.1 http2bin.org"
  }, 
  "json": null, 
  "origin": "87.115.225.178", 
  "url": "http://http2bin.org/post"
}
```

As you can see, all the data is present. The actual HTTP request on the wire looked like this:

```
POST /post HTTP/1.1
Host: http2bin.org
Content-Length: 211
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.9.1
Connection: keep-alive
Content-Type: multipart/form-data; boundary=83f1175700aa4a2b8a49111a87fce425

--83f1175700aa4a2b8a49111a87fce425
Content-Disposition: form-data; name="file"; filename="conf.txt"

some, data,to send
nanother,row,to,send
yet another,row,to,send

--83f1175700aa4a2b8a49111a87fce425--
```

This all seems correct to me. Are you sure your server is handling this appropriately?


Indeed, issue was on server-side.
When requests send string, there is no 'Content-Type' field, and it confused server so it ate first line of uploaded file.

Thanks for great library!
Sorry for hesitate.


No worries @amdei
request question

requests.get('xxx', params={"id":111})

requests.get('xxx', params=json.dumps{"id":111})

These two are the same ？


@rfyiamcool No. The first case does this: `xxx?id=111`. The second case does this: `xxx?%7B%22id%22:%20111%7D`.

In future, please ask questions on Stack Overflow: the GitHub issues page is strictly for bugs.
JSON content type for PUT method

According to the [doc](http://docs.python-requests.org/en/latest/api/), you can specify to send a request with JSON content type for POST method. But there is nothing like this for PUT or PATCH methods.

Is there a reason? Can this be implemented?


If you look at the documentation for [`requests.put`](http://docs.python-requests.org/en/latest/api/#requests.put) and [`requests.patch`](http://docs.python-requests.org/en/latest/api/#requests.patch) they take `**kwargs` that are passed to [`requests.request`](http://docs.python-requests.org/en/latest/api/#requests.request).

This means that `json=` is already a valid parameter for those methods.
JSON-ROA data ignored

Hi,
I'm trying to read JSON-ROA responses from an API that look like this:

``` json
{
    "_json-roa": {
        "filled_with": "lots of data"
    },
    "created_at": "2015-11-05T10:33:37.249Z",
    "creator_id": "a41b963a-e50e-40ad-9c37-dc5dfcc9a1de",
    "id": "c97e9d4c-aaff-415a-a996-4a7e79ce0c81",
}
```

A call to `r.json()` recognizes `created_at`, `creator_id`, and `id` correctly but ignores `_json-roa`.
`r.text` shows likewise only the last three nodes.

If I open and read the same JSON not as a server response but as a static file, parsing it is not problem.

Any ideas?

Best,
birk


If `r.text` only shows the last three nodes, that means that only the last three nodes are present. This means that for some reason the server is choosing not to send you that data.

You'll need to consult with your API documentation or server operator to determine why that data is absent.


Thanks for the quit response. If I do the same request in the shell via curl I do get all the data from the server.


So in this case the problem is likely to be one of user-agent filtering. Try using `curl -v` to see what headers curl sends, and then set the user-agent header to the same thing using requests.


The header it was … I used `'accept': 'application/json'` but it has to be either `'accept': '*/*'` or correctly `'accept': 'application/json-roa+json'`.
Why retry doesn't work on sesson.post method

I've pasted requests.packages.urllib3.util.Retry object to max_retries argument of HTTPAdapter
But i found, it only works on session.get. How could i make it work on session.post ?


When you say it doesn't work on post, what does that mean? What are you expecting to see and what actually happens?


it means. when i use session.post to access url = 'https://httpbin.org/status/500', it doesn't retry and just return an 500 error, not maxretry error.


@demonguy That's not how retries work by default. Generally speaking, retries only affect actual connection issues that prevent us from sending the request or retrieving the response altogether. Normally, we do not issue retries on 500 errors. You can change that behaviour by initializing your `Retry` object with `status_forcelist=range(500, 600)`, as documented on the [Retry object API document](https://urllib3.readthedocs.org/en/latest/helpers.html#urllib3.util.retry.Retry).


I am doing what you said, did you read my original post? i past "requests.packages.urllib3.util.Retry" object.

And of course i passed status_forcelist, but it only works on get, not on post


If you look at the API documentation, you'll notice that it says this:

> method_whitelist (iterable) –
> Set of uppercased HTTP method verbs that we should retry on.
> By default, we only retry on methods which are considered to be indempotent (multiple requests with the same parameters end with the same state).

POST is not considered IDEMPOTENT. You'll need to extend the method whitelist with POST to allow max_retries to function there.


@demonguy in the future **questions** are to be asked on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests). Please do not ask a question in an issue again.


Sorry. I search for the whole Request document, i should expect it's in urllib3 documents.
Really sorry. 
And i did search Google and stackoverflow, maybe i use wrong keyword so i didn't get answer from that.
Sorry
Response encoding defaulting to ISO-8859-1

Hello,

For the last month I've been getting reported issues on badly encoded data (ie: https://github.com/eskerda/pybikes/issues/109)

I've searched through the issues and also tried to pinpoint it to a particular change, but only one if found is from 2014.

Has something changed recently?

Can be reproduced as:

``` python
import requests
import re

rgx = r'setEstacao\((.+)\)'
resp = requests.get('http://www.ciclosampa.com.br/estacoes.php')

print "Uses default encoding %r" % resp.encoding
print re.findall(rgx, resp.text)[0]
print ""
resp.encoding = 'UTF-8'
print "Uses set encoding %r" % resp.encoding
print re.findall(rgx, resp.text)[0]
```

```
Uses default encoding 'ISO-8859-1'
-23.572208,-46.644283,"1","EstaÃ§Ã£o Hcor","Rua Desembargador Eliseu Guilherme 14","5","11"

Uses set encoding 'UTF-8'
-23.572208,-46.644283,"1","Estação Hcor","Rua Desembargador Eliseu Guilherme 14","5","11"
```

Sorry if this has already been reported or resolved. Looked for issues but most changes on this are too old to be relevant.


This has not changed: requests has behaved like this for an extremely long time. See #2086 for a discussion about it.

If this used to work but has now stopped working, it's likely that the server used to send a `charset` field in the Content-Type header but no longer does.

If you know you're going to be handling HTML, you should look to extract the encoding from the HTML body. You can handle that automatically by using [this utility function from the requests toolbelt](https://toolbelt.readthedocs.org/en/latest/deprecated.html#requests_toolbelt.utils.deprecated.get_encodings_from_content).
Stop falling back to distutils in setup.py

distutils has been deprecated for years now and pip is refusing to install
projects using it. As such, _some_ users are reporting problems installing
requests. Let's avoid this entirely by not falling back to distutils at all.


/cc @dstufft (re: https://github.com/pypa/pip/issues/3384#issuecomment-173202320) although I'm starting to suspect that that is a version of requests packaged by the distro that is causing wichert's problems.


Still, we just don't need this fallback now.
Return request & response with TooManyRedirects

I'm using requests to fetch data from websites that I don't maintain, and sometimes websites have broken cyclic redirects, but still return useful responses.  So I would like to grab the last cut off response, similar to the browser.

The problem is that `TooManyRedirects` doesn't throw the ending response state for me to inspect, so this PR adds that.


@Lukasa I'm +0.5 on this. The reason I'm not a full +1 is that we might start leaking sockets this way.


@sigmavirus24 We shouldn't leak sockets here: the redirect logic consumes all the data from the connection, which should return it back to the connection pool, and that happens before we raise `TooManyRedirects`.


Got a few small notes, but this looks really good! :cake:


> Got a few small notes, but this looks really good! :cake:

Yay!  Thanks a lot for the fast response! @sigmavirus24 @Lukasa 


LGTM. @Lukasa thoughts?


LGTM.
SSL error: bad handshake

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dev-django1/.virtualenvs/logic_srv/local/lib/python2.7/site-packages/requests/api.py", line 68, in get
    return request('get', url, *_kwargs)
  File "/home/dev-django1/.virtualenvs/logic_srv/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, *_kwargs)
  File "/home/dev-django1/.virtualenvs/logic_srv/local/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
    resp = self.send(prep, *_send_kwargs)
  File "/home/dev-django1/.virtualenvs/logic_srv/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, *_kwargs)
  File "/home/dev-django1/.virtualenvs/logic_srv/local/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno bad handshake] [('SSL routines', 'SSL3_GET_SERVER_CERTIFICATE', 'certificate verify failed')]

I do not know why it is coming in requests 2.6.0 while it is not coming in requests==2.2.1. This problem start arising yesterday when I hit it to google api on link https://www.googleapis.com/oauth2/v1/tokeninfo. I can't use verify=False as many stackoverflow thread is advising. And I'm pretty sure I have used ssl certificate properly in my web site too. 


What OS are you using?


Yeah im using linux. Btw I fixed the error by removing and installing virtualenv package, not sure why this happened but i got it running. Thank you for your reply :)
Cut a release to match urllib3-1.14?

Hi, we noticed that urllib3-1.14 was released right at the end of 2015.  Are there plans to release a new version of requests bundling that?

Downstream in Fedora we had a bug filed requesting urllib3-1.14, but with the way things stand now, we can't update to that until we have a matching requests release.  https://bugzilla.redhat.com/show_bug.cgi?id=1295402


@ralphbean At some point, sure. We weren't generally planning on racing to new urllib3 versions, especially as really the only change in that urllib3 release is the SOCKS proxy support.

Is there a particular rush to get urllib3 1.14 into Fedora?


I think Nir Magnezi in that ticket was having issues setting up an OpenStack dep that may have pinned version 1.14 of urllib3.  I'll ask for more details.


:+1: for this. 
There was another release in 1.14 as well: "Fixed AppEngine handling of transfer-encoding header and bug in Timeout defaults checking." Together with this patch: https://github.com/sigmavirus24/requests-toolbelt/pull/119 and the use of requests_toolbelt, requests will actually work on appengine. 

So it would be much appreciated if requests got updated to urllib3 1.14.  


So to be clear, we _will_ cut such a release. I'm just trying to work out how time-sensitive this release is.


Thank you :). Looking forward to when it will be. 


As the one doing the changes to get urllib3/requests working on appengine, I would say "not high priority" on that count, as it's not blocked on this urllib3 import yet. Once we get https://github.com/sigmavirus24/requests-toolbelt/pull/119 submitted, and that branch integrated into master, then I'll come pushing on this thread again to get appengine functionality unblocked.


Okay that went quicker than expected, based on extrapolation of past activity on my pull request. :) My changes are now integrated into a requests-toolbelt branch, conditioned against a hypothetical 2.10 version of requests containing the latest urllib3 functionality. I assume once requests cuts this release, @sigmavirus24 can then integrate the branch into mainline requests-toolbelt.

I'll leave it to you guys to determine the time-sensitivity of appengine functionality.


+1 - looking forward to the appengine fix


+1 for SOCKS support. [requesocks](https://github.com/dvska/requesocks) is horribly out of date.


+1 for SOCKS support as well.


+1 for SOCKS support especially for using with tor directly without tsocks/proxychains in middle.


+1 for SOCKS support, especially with the recent release of Docker 1.11.0 with SOCKS support, and the fact that docker-py can't support SOCKS until requests does.
how can i use keystore file in request?

i got a keystore file, it contains much x.509 certificates.
how to import them, while request something? 


@vinsia At this time requests can only pass X509 certificates to OpenSSL from the filesystem: either a file of PEM-encoded certificates or a directory of them. If you serialize your keystore file out to PEM-encoding, you should be able to use the `verify` argument to use them.
SSL-Client-Certificate and Subject Alternative Names

I have a (Puppet-)generated SSL-certificate with a CN and several SANs. A stripped output of openssl-output looks like this:

``` bash
# openssl x509 -in cert.crt -text
Certificate:
    ...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: puppet.domain.com
        ...
        X509v3 extensions:
            ...
            X509v3 Subject Alternative Name:
                DNS:srv01.domain.com, DNS:srv02.domain.com, DNS:srv03.domain.com
        ...
```

Now when i use this certificate with requests, it always picks the CN to do the actual request. Meaning my remote always receives "puppet.domain.com" to verify the client-certificate against. Since the remote IS puppet.domain.com, and the client is srv01.domain.com, the verification always fails.

Is there any way, to have requests use any of the SANs to do its request?

Or is it my remote the actually only uses the CN to do verification?


Sorry...I'm confused. When you say "to do the actual request", what do you mean?


I meant when i do request.post(url, cert()..) that requests always picks the CN.

But i think thats not the problem here. Im quite certain that the remote (apache) or maybe even the application (foreman) has to look at the SANs to actually do the verification since the client-certificate is sent as is, isnt it?


Correct: requests doesn't touch or extract any information at all from the client certificate. If the remote peer is failing to validate it, that's a configuration error with the remote peer.


Great, thx :-)
HTTPAdapter ignored

Hi,
i'm trying to force ssl.PROTOCOL_TLSv1 as written in documentation:
http://docs.python-requests.org/en/latest/user/advanced/#example-specific-ssl-version

```
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
import ssl

class MyAdapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        self.poolmanager = PoolManager(num_pools=connections,
                                       maxsize=maxsize,
                                       block=block,
                                       ssl_version=ssl.PROTOCOL_TLSv1)

import requests
s = requests.Session()
s.mount('https://', MyAdapter())

requests.get('https://services.mailup.com/')
```

but this gives me 

```
requests.exceptions.SSLError: [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:600)
```

so i tried to patch requests/packages/urllib3/poolmanager.py with:

```
class PoolManager(RequestMethods):
    proxy = None

    def __init__(self, num_pools=10, headers=None, **connection_pool_kw):
        RequestMethods.__init__(self, headers)
        connection_pool_kw['ssl_version'] = ssl.PROTOCOL_TLSv1
        self.connection_pool_kw = connection_pool_kw
        self.pools = RecentlyUsedContainer(num_pools,
                                           dispose_func=lambda p: p.close())
```

and i get:

```
>>> import requests
>>> requests.get('https://services.mailup.com/')
<Response [403]>
```

Any ideas?

Thanks,
Matteo


@swmerko You have to route your request through the Session object. In your code example you create a Session object `s`, but then call `requests.get`. That's not how Session objects work. Change your last line to `s.get()` instead of `requests.get()` and your problem should be solved.


Is there a way to define globally requests with my Adapter?


No, requests deliberately maintains very little global state. Under the covers, each requests.\* call creates a new Session to execute the request. If you really wanted to do this (and I strongly do not recommend this approach), you could monkeypatch `Session.__init__()` to mount your adapters over the default ones.


Thank you very much!
Python 3.5 : AttributeError: 'socket' object has no attribute 'getpeercert'

Traceback:

  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/requests/api.py", line 61, in get
    r = yield from request('get', url, *_kwargs)
  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/requests/api.py", line 49, in request
    r = yield from session.request(method=method, url=url, *_kwargs)
  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/requests/sessions.py", line 465, in request
    resp = yield from self.send(prep, *_send_kwargs)
  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/requests/sessions.py", line 579, in send
    r = yield from adapter.send(request, *_kwargs)
  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/requests/adapters.py", line 364, in send
    timeout=timeout
  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/urllib3/connectionpool.py", line 515, in urlopen
    body=body, headers=headers)
  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/urllib3/connectionpool.py", line 309, in _make_request
    yield from self._validate_conn(conn)
  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/urllib3/connectionpool.py", line 730, in _validate_conn
    yield from conn.connect()
  File "/opt/python-virtual-env/py35/lib/python3.5/site-packages/yieldfrom/urllib3/connection.py", line 295, in connect
    match_hostname(sock.getpeercert(),
AttributeError: 'socket' object has no attribute 'getpeercert'

sock instance is of type socket.socket


It is a mistake using "New Pull Request" from rdbhost:master. Sorry.
Rename {Missing,Invalid}Schema to *Scheme

As @merwok pointed out at https://github.com/kennethreitz/requests/pull/512#commitcomment-1296187, schemes are what they're called, not schemas. I got tripped up by this at https://github.com/CenterForOpenScience/osf.io/pull/4745#issuecomment-169038417, where "InvalidSchema" made me think I was hitting a database error, when in fact it was a configuration error in the format of a URL.

This PR is a redo of #2960 against `proposed/3.0.0` instead of `master`.


\o/ Thanks @whit537! :sparkles: :cake: :sparkles:


Huzzah, thanks! :-)

@Lukasa++


@Lukasa please add release notes for this. I think it may be a good idea to have some shim for this as people may be catching `InvalidSchema` or `MissingSchema` exceptions and they won't be able to catch those anymore in 3.0


I'm away from the laptop for a bit, but I will do.


> I think it may be a good idea to have some shim for this as people may be catching `InvalidSchema` or `MissingSchema` exceptions and they won't be able to catch those anymore in 3.0

If you add a shim then you're basically backwards-compatible, right? (That's what I was getting at with https://github.com/kennethreitz/requests/pull/2960#issuecomment-169070702.)


@whit537 yeah, it isn't _necessary_ for 3.0 though and there's no way to tell people to move to those new messages unless we forcibly break things for them though.

What we could do is add `InvalidScheme` and `MissingScheme` as sub-classes of their now defunct relatives so that people can start (with 2.10.0) using those exceptions so they're future proofed for 3.0.0. I'm not sure if @Lukasa would agree that there is value in that though.


I'm open to that plan.
Rename {Missing,Invalid}Schema to *Scheme

As @merwok pointed out at https://github.com/kennethreitz/requests/pull/512#commitcomment-1296187, schemes are what they're called, not schemas. I got tripped up by this at https://github.com/CenterForOpenScience/osf.io/pull/4745#issuecomment-169038417, where "InvalidSchema" made me think I was hitting a database error, when in fact it was a configuration error in the format of a URL.


Thanks @whit537. Unfortunately, this is a breaking change so we can't accept it until 3.0.0: can you adjust this PR to point to the proposed/3.0.0 branch instead please?


Sure thing. Closing this PR ...


It occurs to me that it'd be possible to implement this in a backwards-compatible way, but since that would involve more than simply renaming the error classes (I guess you'd make an `InvalidScheme` class that subclasses `InvalidSchema`) it would take more time and might deserve a test or two. I can't commit to that work so I'm leaving this PR closed.
Persistent Connection suffer from delay of Nagle

Not sure if this is a bug from urllib3 or Requests, the problem is that there is an extra delay introduced for all subsequent requests other than the first one when reusing a session object.
The test are performed in Ubuntu 14 to a http server running on the localhost

``` python
with requests.Session() as session:
    session.get(url) # this takes <1ms
    session.get(url) # this takes around 40ms
    session.get(url) # this takes around 40ms
```

Similar case was observed in httplib2 with Python 2.6(https://code.google.com/p/httplib2/issues/detail?id=91), and was fixed in Python 2.7. 


There should not be a Nagle's algorithm delay, because by default we disable Nagle's algorithm. Can you please answer the questions from our [CONTRIBUTING.md](https://github.com/kennethreitz/requests/blob/master/CONTRIBUTING.md), specifically:

> Tell us **what version of Requests you're using**, and **how you installed it**.


version: 2.9.1
installed by: sudo pip install requests --upgrade
python version: 2.7.3
os: Ubuntu 14


So [we disable Nagle's algorithm by default](https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connection.py#L97), which means that your diagnosis of this cannot be right.

Are you familiar with wireshark or tcpdump?


Let me investigate more, maybe it has something to do with server side. I will close it first.


To be clear, if the server-side has delayed ACKs turned on it may cause this problem.