Raibek/requests-github-issues huggingface dataset:
API feature - POST request to HTML element by id

Right now, the requests.post() function only appears to send post requests to HTML elements by their name. Are there any plans of creating something like a requests.post_to_name() and requests.post_to_id(), so that the user can make post requests to HTML elements via their id instead, similar to the API used in Splinter for accessing HTML elements?


@aewhatley requests is an _HTTP_ client and has nothing to do with _HTML_. POST requests are made to URLs, not HTML elements. As such, we would not be adding any such methods because they wouldn't make sense as part of our API.

Thanks for the suggestion but I think you're a little confused about how requests works.
HTTPS verification for Let's Encrypt certificates

I've been trying to get requests to work with sites using Let's Encrypt SSL certificates but getting a very unhelpful error:

```
  File "E:\anaconda27\lib\site-packages\httplib2\__init__.py", line 1609, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "E:\anaconda27\lib\site-packages\httplib2\__init__.py", line 1351, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "E:\anaconda27\lib\site-packages\httplib2\__init__.py", line 1272, in _conn_request
    conn.connect()
  File "E:\anaconda27\lib\site-packages\httplib2\__init__.py", line 1059, in connect
    raise SSLHandshakeError(e)
httplib2.SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
```

I've replicated this in Python 3.4 and 3.5, with lib/ssl.py failing.


@yhorian1 Your bug is in httplib2. This is not httplib2.
Discard 0-byte chunks from request body iterators

Previously, an empty chunk in a request body iterator would be
faithfully translated to a 0-length chunk on the wire, which the server
would interpret as the end of the transfer. That is, a request like

```
def data():
    yield 'foo'
    yield ''
    yield 'bar'
requests.put(url, data=data())
```

... would cause the server to receive a chunk for 'foo', receive an
empty chunk, and close the connection before receiving 'bar'.

Further, if `Connection: keep-alive` is set, the server will attempt (and
fail) to parse the next chunk, serve a 400, and close the connection.

Now, the empty chunk will be ignored, and the server should receive all
of the actual bytes emitted from the iterable.


This is a breaking change that cannot be merged for 2.x but has been fixed for 3.0.0 in #2631 by @neosab.
getting unexpected output

So I am creating a script to interact with an API for https://haveibeenpwned.com/API/v2 (link to the API docs) so when I run this from the example on the site in python3 this works as expected.

``` #!/usr/bin/python3

import requests
import pprint

headers = {'User-agent': 'Have I been pwn module'}

API = 'https://haveibeenpwned.com/api/v2/breachedaccount/foo@bar.com'

request = requests.get(API, headers=headers)
res = request.json()
pprint.pprint(res)
```

but when changing out the foo@bar.com email address to my personal email, I should get `404   Not found — the account could not be found and has therefore not been pwned` status code but get this instead

```
  Traceback (most recent call last):
  File "/home/jay/.PyCharm50/config/scratches/scratch_4", line 11, in <module>
    res = request.json()
  File "/usr/lib/python3.5/site-packages/requests/models.py", line 808, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.5/site-packages/simplejson/__init__.py", line 505, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/python3.5/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Process finished with exit code 1
```

Hope its not me being silly?


Can you let us know what your email address is? Do you feel comfortable with that?


@Lukasa I would prefer not to but if you under the API var put your email address you should hit the same issue I am having as long as you are not in any of the breach lists


@L1ghtn1ng So in both Python 3 and Python 2 I find that the response has no body if there have been no breaches. As a result the JSON decoders fail to decode correctly. You should check the status code (`res.status_code`) before using `r.json()`.
Docs: clarify the precedence of `auth=` over `netrc`

This closes #2062 by clarifying in the docs which auth
header takes precedence:
1st auth=
2nd .netrc
3rd headers=

This precedence order is already tested in test_requests.py,
in the test_basicauth_with_netrc method. Perhaps we should
add further tests for non-basic auth schemes.


Thanks! :sparkles: :cake: :sparkles:
Add json parameter to put methods

I enjoy both `requests.post(json=payload)` and `session.post(json=payload)`. I would like to have the same for both `requests.put()` and `session.put()`.

In a cursory examination of the code, it looks like a simple change. I would be happy to create a pull request for it. Is there a reason the convenience wasn't already propagated to PUTs?


> Is there a reason the convenience wasn't already propagated to PUTs?

It was. =) All our keyword arguments work on _all_ verbs, which means `json=` works on PUT as well as POST. The only thing we didn't do was add it as an explicit keyword argument so that it appears in the documentation. Would you like to make that change?


Sure enough! I had tried it but when my tests failed, I thought that was the cause. But, as is usually the case, it was my code. :)

The lack of an explicit parameter doesn't bother me, so I'll pass for now.

Thanks for the quick response!
How to POST a very Big file with 'chunk' mode?

I want to Post to a very big file to the server,  will requests use the "chunk" mode ?
if
>>> f = open("very big file")
>>> requests.post(url, data=f)
From the `PreparedRequest.prepare_body` method, file length can get from `util.super_len`, 
then the `'Content-Length'` will be a number, which make chunk  mode False.
Is these any way to use the chunk mode when post a big file ?
Thank you!


Requests will not chunk the file, but neither will it read it all into memory. Instead, it will "stream" it: that is, send it from the file directly to the network. Do you really specifically need chunked-transfer-encoding here? What does chunked transfer encoding get you that the streaming method does not?


Thank you @Lukasa . My mistake, I thought requests has to read it all into memory before send to network.In fact requests just "stream" it, which is cool.Thank you!


@jchluo in the future, please do **not** use the **issue** tracker for **questions**. Use [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).
COC?

Hi,

I'm just wondering what your code of conduct is for this project?

Thanks,
Alison


Hi @alison985!

Currently our relatively minor code of conduct is [defined here](http://docs.python-requests.org/en/latest/dev/contributing/). There is interest amongst some of the maintainers in moving to something based on the [Contributor Covenant](http://contributor-covenant.org/), given that both myself and @sigmavirus24 maintain personal projects that use it, but I'm unwilling to change that by fiat: I'd want consensus from the whole team.

Regardless, I believe that the Contributor Covenant _implicitly_ applies to this project, and certainly that's the standard to which the maintainers hold themselves.
Add response.closed attribute

I'm able to explictly close a response by calling `response.close()`, but as far as I could see, the only way to check if the response is closed or not is calling `response.raw.closed`. This feels like violating Demeter's Law, so I would suggest adding a `response.closed` attribute.

If you agree, I'd be happy to send a pull request with this change.


@vitorbaptista Out of interest, why do you care if the response object is closed?


I have a class that internally uses requests, but not through contexts (I have to keep the response around for a while, so I can't use contexts). I want to write a test to ensure that the response is closed after the object is destroyed.

Right now, I'm testing it using `response.raw.closed`, which is OK, but as `response.close()` already exists, it seems like `response.closed` should exist as well.


So, the reason I asked is that `response.close()` doesn't do what you think it does. Because we pool our connections, the method doesn't necessarily close anything at all: it just returns the socket connection (that may or may not still be open) to the connection pool.

It should be totally safe to repeatedly call `response.close()`: the method should be idempotent. For that reason, a `closed()` property isn't really necessary: just call `close` instead of checking!


hmmm, interesting... but still, I need to check in the tests that the response was closed. `response.raw.closed` isn't ideal, but works fine. I could also use mocking to check that `response.close()` was called, but using `response.raw.closed` is better IMHO.

In the end, I created this issue just because it was a bit surprising for `response` to have `close()` but not `closed`. It looked like an easy change. There're other easy ways to do what I wanted to do, though, so it doesn't add much.

I'll close this issue for now. If you think this change is worthwhile, please reopen and I'd be happy to write a pull request solving it.

Thanks for your help :+1:
Unicode decode error with PATCH request

In requests 2.9.0:

```
>>> requests.patch('https://example.org', data=bytes('ô', 'utf-8'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/combine/virtualenv/lib/python3.4/site-packages/requests/api.py", line 133, in patch
    return request('patch', url,  data=data, **kwargs)
  File "/usr/local/combine/virtualenv/lib/python3.4/site-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/combine/virtualenv/lib/python3.4/site-packages/requests/sessions.py", line 454, in request
    prep = self.prepare_request(req)
  File "/usr/local/combine/virtualenv/lib/python3.4/site-packages/requests/sessions.py", line 388, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/local/combine/virtualenv/lib/python3.4/site-packages/requests/models.py", line 296, in prepare
    self.prepare_body(data, files, json)
  File "/usr/local/combine/virtualenv/lib/python3.4/site-packages/requests/models.py", line 447, in prepare_body
    body = self._encode_params(data)
  File "/usr/local/combine/virtualenv/lib/python3.4/site-packages/requests/models.py", line 84, in _encode_params
    return to_native_string(data)
  File "/usr/local/combine/virtualenv/lib/python3.4/site-packages/requests/utils.py", line 700, in to_native_string
    out = string.decode(encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)
```

This seems to be a problem with all strings containing Unicode characters. It works without problems in 2.8.1. 


This is a duplicate of #2933 and #2930, and is fixed in #2931. Thanks for the report, but in future please check both open and **closed** issues before filing a bug, as the issue may already be known. A new release of requests will be made on Monday that contains the fix for this issue.

Thanks! :sparkles: :cake: :sparkles:
Release 2.9.1

This release is scheduled for Monday. I'm aiming to set this up now so that it's an easy release to push on Monday.

@shazow, what are the odds that we can get a urllib3 patch release 1.13.1 containing the fix for shazow/urllib3#761 by Monday? I'd owe you lots of :custard: if we could get it! :heart:


Note: please don't merge this until we're ready to release on Monday! We may land other patches between now and then, and we'll want to make sure the changelog is updated!

For those who are interested: the reason we're enforcing a delay between 2.9.0 and 2.9.1 is to try to land as many regression fixes as possible in a single release, rather than rapidly releasing several versions as issues become apparent to us. Now that requests is a widely used project it becomes important to be as reliable as possible in our release management to avoid making people's lives too hard.


@Lukasa are you thinking about making an exception to [our release document](https://github.com/kennethreitz/requests/blob/master/docs/community/release-process.rst#hotfix-releases) in this case?


@sigmavirus24 Yeah, good question, but I think I am, given that the urllib3 release would itself be a patch release. Any concerns there? @ralphbean @eriol?


@Lukasa Could do. Could try to sneak in SOCKS support too, if you wanna.

Also I noticed some of your changelist includes an update note to the urllib3 version, some don't. Dunno if that's intentional.


@shazow I'd rather leave SOCKS out for now: makes 1.13.1 very clearly a simple bugfix release.

Yeah, the missing changelist notes are in error, we should try to fix it.


Sounds good, I'll push out v1.13.1 before monday (maybe today).


@Lukasa yeah, if it's just a couple small bug-fixes, that's fine by me. I think we've been bitten in the past by urllib3 having a lot more than just bug fixes when we pulled it in.


:+1: here.  I just finished packaging 2.9.0.  Bumping to 2.9.1/1.13.1 shouldn't be an issue.  Thanks!


@ralphbean Frankly I don't recommend releasing 2.9.0, it had enough nasty bugs that going straight to 2.9.1 is going to be the better approach.


:+1: from me. I will skip 2.9.0 since 2.9.1 is going to be released soon, thanks!


urllib3 v1.13.1 is live.


> Frankly I don't recommend releasing 2.9.0, it had enough nasty bugs that going straight to 2.9.1 is going to be the better approach.

Acknowledged.  I attached a comment to let people know on the 1.9.0 [Fedora updates](https://bodhi.fedoraproject.org/updates/python-requests).


\o/ Thanks @shazow!


@Lukasa is this still a WIP? I believe you wanted to cut this today


Heh, was just waiting for you. No longer a WIP, should be good to go.


:sparkles: :cake: :sparkles: 

:sparkler: :shipit:
Handle bytes and unicode URLs for netloc

Spotted this on my local machine running tests on Python 3. Annoyingly, we didn't see this on the CI rig because the Jenkins box doesn't have a netrc file, so it never gets that far in the function.

However, tests preparing bytes URLs fail here on Python 3 because the splitter is the wrong type. This change ensures that we use the appropriate type.

Feels like this is safe for 2.9.1.


Woot! :+1: 

Thanks @Lukasa (also YAY OUR CI IS WORKING AGAIN)
Problem with bytes body in request 2.9 in Py 3.5

Hello,

Our library suddenly stops to works in Python 3 for our users after `requests` 2.9 release, for instance:
https://github.com/Azure/azure-storage-python/issues/89

I investigated the stack trace, and I found that when the body of the requests contains "real" bytes ("real" stands for OSImage, PDF,... not something readable), the code breaks.

Especially, `requests` 2.9 in `requests.models#_encode_params` calls a new method `requests.utils#to_native_string` which tries to convert my bytes data using an ASCII encoder which is not possible if data are "real" bytes.

This is the interesting part of the stacktrace extracting from the previously mentioned bug

```
self.response = self.session.request(self.method, self.uri, data=request_body, headers=self.headers, timeout=self.timeout)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 454, in request
    prep = self.prepare_request(req)
  File "/usr/local/lib/python3.5/site-packages/requests/sessions.py", line 388, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 296, in prepare
    self.prepare_body(data, files, json)
  File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 447, in prepare_body
    body = self._encode_params(data)
  File "/usr/local/lib/python3.5/site-packages/requests/models.py", line 84, in _encode_params
    return to_native_string(data)
  File "/usr/local/lib/python3.5/site-packages/requests/utils.py", line 700, in to_native_string
    out = string.decode(encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
```

What do you think? Tell me if I can help for anything.
Thank you,


This is already fixed in #2931. In the future, please check open _and_ closed issues before filing new issues.


Sorry, yes I forgot to check closed issues.
Thank you for the report.
PyPI doesn't know about python 3 support

PyPI doesn't seem to know that requests supports Python 3 (3.3, 3.4, & 3.5!).  As a result, [caniusepython3](https://caniusepython3.com/) thinks `requests` is a Python 3 blocker and `requests` is in red on the [Python 3 Wall of Superpowers](https://python3wos.appspot.com/).

Requests is awesome and it should get the credit it deserves.


Annoyingly, our trove classifiers appear to have disappeared again. I've re-uploaded them. Thanks!
Uploading packages via twine fails

Using latest 2.9.0

```
$ twine upload dist/*.whl
Uploading somepkg-15.12.16-py3-none-any.whl
ConnectionError: ('Connection aborted.', BadStatusLine("''",))
$
```

rollback to 2.8.1 solves the issue.


@sigmavirus24 You're probably best placed to look at this, but my suspicion is that there's an unexpected interaction between the toolbelt's MultipartEncoder and the fix we put in for calculating the actual length of file-like objects that are not seeked to the beginning. That said, I can't find anything in that.


Yeah, so this is a bug in requests-toolbelt and requests interaction.


Closing as the bug is actually over at https://github.com/sigmavirus24/requests-toolbelt/issues/117. Thanks for bringing this to our attention @nicorevin ! :cake:
make setup.py upload happy

distutils brokes when we use tuple based meta values
canonic way is define classifiers as list not a tuple


@heni We've been using a tuple here since [literally the start of the project](https://github.com/kennethreitz/requests/commit/0477018761c67152cdcc0b83d56f27e701e65b9e), which _used_ distutils. I do not believe this can possibly be the problem you're experiencing.


I want to redistribute this package into private repository of our project and with tuple for classifiers I have broken next chain of commands
  pip3 install --download=. --no-binary=:all: requests
  tar xf requests-2.9.0.tar.gz
  cd requests-2.9.0
  python3 setup.py bdist_wheel upload -r private-repo


Distutils does not work with wheel, you have to use setuptools.


Also `setup.py upload` most likely is _not_ performing certificate validation. Use `twine` instead.
Prepare 2.9.0 release.

This should contain the set of changes required for us to push 2.9.0 out the door.


@sigmavirus24 Can you confirm for me that you're happy with these release notes?


You also need to bump `__version__` and `__build__`. :)


@sigmavirus24 I intentionally did not do that, so that we have a single commit entitled `v2.9.0`, rather than a merge commit that hits it. =D


Got it. Well the rest of this looks excellent to me. Feel free to do that and ping me for the merge. :)
Refactor default params

Specified the default argument for params that have a default in the docstring
so that the default is easier to see from the code. Modified the docstring in
api.py to match the docstring in sessions.py.


I also noticed that there may be a problem in test_requests.py. Lines 524-5 reference the httpbin_ca_bundle fixture, but there is no fixture by that name. If I am reading this correctly, this means that the test is passing in None instead of a cert path. Does this make sense?


Hey @bsamek! Thanks for the PR.

I don't think we want to change the default parameters to the `request` method. We want to be able to discern between something that's a session setting and a request setting which we cannot do if we change that signature. We use [`merge_setting`](https://github.com/bsamek/requests/blob/master/requests/sessions.py#L49..L53) to make that determination and I don't think we want to break this.

Please revert that as the docstring changes _are_ desirable.


Reverted the changes to the parameters. Kept the docstring changes.

Should I submit an issue regarding my earlier questions about the test?


There absolutely is such a fixture: https://github.com/kevin1024/pytest-httpbin/blob/master/pytest_httpbin/plugin.py#L40


Thanks again @bsamek !


Ohhhhhhhhh I see. Thanks @sigmavirus24 !


Anytime @bsamek :) Thanks for the PR!
Connections not recycled when stream=True and body not consumed

Calling `response.close()` does not seem to release the connection back to the pool. I'm only using the headers or even just `status_code`, so the content doesn't get downloaded, I call `response.close()` but then the next request opens up a new connection. 

Is there a reason for this? It seems that the documentation is insufficient at the very least, if this is intended behavior.


If you don't consume all the data from the underlying connection, the connection actually _cannot_ be re-used. This is because requests has no way of signalling to the operating system that it doesn't care about the data already received except by closing the socket and throwing the connection away.

Documenting this limitation is possible, but given that it's a reality of how TCP connections function it's hard to justify I think.
[DRAFT] WIP in finding a solution to data encoding check

Use existing method `to_native_string()` as a check.

This is a first attempt to resolve https://github.com/kennethreitz/requests/issues/2638, so marking this PR as [DRAFT]. I'd appreciate feedback on this. 


Yeah, so how we approach this is basically dependent on whether we think it's acceptable for `to_native_string()` to go in `encode_params`. I personally think that's the best place for it, because it allows us to avoid duplicating the large amount of knowledge `encode_params` has about how this variable is structured. However, IIRC @sigmavirus24 originally objected to that idea so I'd like to hear from him.


This should be its own function that we can test thoroughly because I think there's a lot of room to do this wrong.

If we have similar checking in a lot of places, make it one function that we run often and that we test well. Our `_encode_*` methods are too large as it is and barely tested at all
Wrong content type on request.head() only

Using request.head()  I want to get the content-type of a page before proceeding. Unfortunately for me, the request header content-type returns text/html. But if after completing the requests using request.get()
the content type is application/octet-stream.

I have suspected the web server but using urllib2 with a head request it return application/octet-stream

Here is my code and the exact url to reproduce the same error:

``` py
>>> import requests
>>> u = "http://holo-pack.com.tw/download.phpx?soft=ea70"
>>> r = requests.head(u)
>>> r.headers['content-type']
'text/html'
```

With request.get() the header is "application/octet-stream" so with urllib2 as seen here: pect https://filippo.io/send-a-head-request-in-python/


Please review the [CONTRIBUTING](https://github.com/kennethreitz/requests/blob/master/CONTRIBUTING.md#good-bug-reports) file that GitHub prompted you to review prior to filing this bug.

You've not told us what version of requests you're using or how you've installed it.


So in spite of not having the information here, I noticed that the blog post registers a redirect handler. It looks like your head request gets redirected. As such you need to pass `allow_redirects=True` to a `HEAD` request to make sure you follow them too with requests. When you follow it all the way to the bottom you get the content-type you expect.
requests/auth: Handle an empty 'qop' attribute in an Authenticate challenge

Some malfunctioning HTTP servers may return a qop directive with no token, as
opposed to correctly omitting the qop directive completely. For example:

```
header: WWW-Authenticate: Digest realm="foobar_api_auth", qop="",
        nonce="a12059eaaad0b86ece8f62f04cbafed6", algorithm="MD5",
        stale="false"
```

Prior to this patch, requests would respond with a 'None' Authorization header.
While the server is certainly incorrect, this patch updates requests to be
more tolerant to this kind of shenaniganry. If we receive an empty string for
the value of the qop attribute, or some tokens that are not recognized (that
is, not 'auth' or 'auth-int'), we instead treat that as if the qop attribute
was simply not provided.

Closes #2916


Thanks for this!

I think this change is more aggressive than we need actually. Is there any reason you didn't take the simpler path and just change the conditional to look for falsy values of `qop`?


Just got a bit trigger happy on trying to handle unrecognized values in the `qop` attribute. A much simpler PR is now up. :smile: 


\o/

Given that `None` is falsy, I think we can simplify this a little bit further to simply check `if not qop`.


Good point. Simpler patch incoming.


I pulled this down and ran the tests locally. All seems well on this end.
Upgrade urllib3 to overcome Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]

We upgrade our production systems to requests==2.8.1 today and ran into <pre>  File "/home/xxxxxx/penv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/home/xxxxxx/penv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 353, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 979, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 811, in send
    self.sock.sendall(data)
  File "/home/xxxxxx/penv/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 220, in sendall
    sent = self._send_until_done(data[total_sent:total_sent+SSL_WRITE_BLOCKSIZE])
  File "/home/xxxxxx/penv/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 206, in _send_until_done
    return self.connection.send(data)
  File "/home/xxxxxx/penv/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1271, in send
    self._raise_ssl_error(self._ssl, result)
  File "/home/xxxxxx/penv/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1187, in _raise_ssl_error
    _raise_current_error()
  File "/home/xxxxxx/penv/local/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
Error: [('SSL routines', 'SSL3_WRITE_PENDING', 'bad write retry')]</pre>

Digging into the error lead us to https://www.bountysource.com/issues/27417596-ssl3_write_pending-error-from-urllib3-contrib-pyopenssl-sendall and then to https://github.com/shazow/urllib3/pull/719

The fix is on the master branch of urllib3 and hence wondering if there is a possibility of upgrading urllib3 to latest version.


Per our [release policy](http://docs.python-requests.org/en/latest/community/release-process/), we upgrade urllib3 on major releases, and only to active releases of urllib3. There has been no release of urllib3 since the last release of requests, and until such a release happens we cannot bring in urllib3. Note that "upgrading urllib3 to the latest version" will not help because the latest release does not contain that fix either.

Before you open a similar issue on urllib3, please note that I am also a core developer on urllib3. We are working towards releasing a new version _soon_, but I have no concrete timeline for when that will be. Please be patient. =)


Sure I understand. We will be watchful on urllib3 release as well. Until we will figure out a workaround.


FYI, Urllib 1.13 has been released


As has requests 2.9.0.


That's great! :-) Thank you
set_default_verify_paths not implemented for SSLContext class

It is possible to end up in a situation where set_default_verify_paths() is called on the SSLContext but the SSLContext instance is an instance of the implementation of SSLContext implemented in requests/packages/urllib3/util/ssl_.py This implementation does not provide the set_default_verify_paths() method. Thus resulting in a traceback for the user as follows:

response = requests.get(url)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 69, in get
    return request('get', url, params=params, *_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, *_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, *_send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, *_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 367, in send
    timeout=timeout
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 782, in _validate_conn
    conn.connect()
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 250, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 282, in ssl_wrap_socket
    context.set_default_verify_paths()
AttributeError: 'SSLContext' object has no attribute 'set_default_verify_paths'


@rjschwei Where did you install requests from? urllib3 (and by extension requests) have _never_ called `set_default_verify_paths` in any version we shipped, at least according to a quick search of our git history. The only place we've ever called it is inside the PyOpenSSL contrib module, which of course could not produce the traceback you've provided.


Sorry, I took the liberty of glancing at your profile and noted that you're a SUSE developer. This breakage is caused by a patch introduced by [the SUSE packaging process](https://build.opensuse.org/package/view_file/Cloud:OpenStack:Master/python-requests/no-default-cacert.patch?expand=1). You'll probably want to chase up with whomever is packaging Requests for SUSE (we don't have a relationship with that person at this time).


Thanks for the quick response, was just chasing that avenue, sorry for not doing my homework before filing the issue.


Thanks for the apology @rjschwei, it happens to all of us. =) No harm done, and I hope you get the problem resolved!
Support arguments whose values are None

The docs say: `Note that any dictionary key whose value is None will not be added to the URL’s query string.`

Is there a chance that this could be controlled by the user to include None values as well? Perhaps with the help of a flag.


@rohithpr What would a `None` value be turned into?


@Lukasa - `null` perhaps?

```
>>> json.dumps({'val':None})
'{"val": null}'
```


@rohithpr The only place we turn dictionaries into JSON is the `json=` keyword argument, and that is not put into the URL's query string. Query strings in URLs are not JSON, they're urlencoded key-value pairs.


I should ask: what is your _actual_ problem? What are you trying to achieve that you believe you cannot with requests?


No, I didn't mean to say that it should follow JSON. I just said that None could be converted to null along the lines of JSON.


So again, I'm inclined to ask about what exactly you're trying to do. My reading of this issue is that you want the `None` value turned into _something_, but you don't care what it is. That's an unusual request: ordinarily I'd expect someone to have a specific service they're talking to that makes it natural to represent the data in a certain way (namely, to have `None` in there somewhere).

What's the reasoning here?


I'd like to pass a `null` value for some arguments in a PUT (or any other) request.

```
data = { 'p1': 'hello', 'p2': None}
requests.put(url, data)
```

On doing this, the server gets only 'p1' and not 'p2'.

In the project that I'm working on key:null is used to delete a key from a record while updating the database. I don't have control over the server side code and I'm unable to pass null values using requests. (The ORM used on the server side is Vogels for DynamoDB)


If you want `null` that's something you should control by looping over the parameters you are passing and swapping `None` for `'null'`. This is not something requests is going to change.

``` py
def convert_None_to_null(params):
    new_params = {}
    for (key, value) in params.items():
        if value is None:
            value = 'null'
        new_params[key] = value
    return new_params
```


I see.

So, the use of `null` in this fashion is specific to your server: it is not _generally_ true about the URL query string. Requests tries to restrict itself to the subset of behaviour that is universally understood in the query string. Implementation-specific variances have to handle that themselves: otherwise, requests will grow a million different feature flags that subtly change its behaviour in all sorts of weird ways. This kind of combinatorial expansion of the API and the possible configurations of requests would make it a nightmare to use, maintain, and debug.

@sigmavirus24 has provided you a code snippet that should do what you're looking for (with some small edits). I hope you find that helpful!


@Lukasa @sigmavirus24 - Thanks! :)
requests.packages.urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:646)

Mac OS X 10.11.1
Python 3.5.0 from https://www.python.org/downloads/
requests (2.8.1)

``` python
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 0.9.8zg 14 July 2015'
```

URL https://api.telegram.org/


That version of OpenSSL is absolutely _ancient_. I suspect it's the cause of your problem.

I have OS X 10.11.2 Beta, Python 3.5.0 linked against a newer version of OpenSSL (1.0.2d), and the same version of requests, and I can reach that URL fine. I highly encourage you to use a newer version of OpenSSL.

You can get this by running `pip install -U requests[security]`.


@Lukasa So maybe I need to install a newer version of OpenSSL using homebrew and use it to replace the old OpenSSL shipped with OS.


@caizixian It's an option, but if you `pip install -U requests[security]` that will grab PyOpenSSL, which includes its own copy of OpenSSL on OS X.


@Lukasa I had following errors when installing requests[security], both related to cffi.
1. `c/_cffi_backend.c:14:10: fatal error: 'ffi.h' file not found`
   
   so I `brew install pkg-config libffi`, which solved this problem.
2. After that, I tried to install requests[security] again, and I got `Failed building wheel for cffi` . 
   
   Despite this error, pip said that all things were installed. So I checked `ssl.OPENSSL_VERSION` again, still the same.

FYI, my Xcode is 7.1.1


`Failed building wheel for cffi` is fine, you don't mind. You shouldn't check `ssl.OPENSSL_VERSION` because that's the standard library and doesn't get changed by the packages you just installed. Instead, you should try running `python -c "from cryptography.hazmat.backends.openssl.backend import backend;print(backend.openssl_version_text())"`. That should print the newer OpenSSL version, and that's the one that will be used by requests.


```
pip install cryptography
pip install pyopenssl
pip install requests[security]
```

Problem solved.

```
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  5 2015, 21:12:44) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from cryptography.hazmat.backends.openssl.backend import backend
>>> print(backend.openssl_version_text())
OpenSSL 1.0.2e 3 Dec 2015
```


caizixian, you saved my life!!!!!
Thank you so much
response.request.headers does not contain the Hosts Header

if you iterate over the headers of the response objects request, you do not get the complete set. The Host Header is missing. 

``` python
sending_headers = {"Accept-Encoding": "identity"}
method = "GET"
url = "http://192.168.205.65/poster.php"
resp = requests.request(method, url, headers = sending_headers)
rh = resp.request.headers #original request might be lost in redirects?

for header in rh:
    print("%s: %s" % (header, rh[header]))
```

Observe the output:

```
Accept-Encoding: identity
Accept: */*
User-Agent: python-requests/2.8.1
Connection: keep-alive
```

However, on the wire, you will see:

```
GET /poster.php HTTP/1.1
Host: 192.168.205.65
Accept-Encoding: identity
User-Agent: python-requests/2.8.1
Connection: keep-alive
Accept: */*
```


This is because, ordinarily, requests does not set the `Host` header at all: `httplib` does. We generally don't get in the way of that because httplib wants to set it.

In general, requests does not (and indeed cannot) claim that the headers variables (on both requests and responses) represent the full set of headers that were on the wire. Some or all may be added or removed by lower layers of the stack, and requests cannot control that.
Feature Request: Format reponse as HTTP Response Message Format

http://www.tcpipguide.com/free/t_HTTPResponseMessageFormat.htm

A helper method to format a response as shown in the URL above would be useful.


Requests is currently in a feature freeze. As such we aren't going to be implementing this. That said, if you wanted to implement this yourself, you would probably get 90% of the way there with [dump_response](https://toolbelt.readthedocs.org/en/latest/dumputils.html#requests_toolbelt.utils.dump.dump_response) from the [toolbelt](/sigmavirus24/requests-toolbelt).
Does iter_content sometimes emit empty chunks

I've seen a lot of code on stackoverflow that guards the loop-body with an `if chunk:` when using `iter_content`. Is this really necessary, I don't see it anywhere in the documentation.

If it is, could we fix it? Or if not, document it.

``` python
for chunk in r.iter_content():
  if chunk: # <- why this?
    output.write(chunk)
```


`iter_content` should not emit empty chunks. Historically it _could_, however with the current logic that should not be possible. If anyone encounters this problem I'd consider it a bug.


Perfect, that's what I'd hoped for! Thanks
Set 'Transfer-Encoding: chunked' if data is a file with length 0

Fixes #2215 

This is the first time I contribute to the project so I am not sure at all I put that code where it belongs. I know I have to add tests for this, but wanted to show you this beforehand.

Did I do this right?

Thank you


@BraulioVM This is a good first pass! I have a draft of this fix sitting around on my laptop, so when I get home I'll show you how I approached it and let you take the ideas you like best. 


@Lukasa Thanks!

I have just seen #2866, which won't allow me to implement the tests I want, so I might as well think of a solution for that issue.


So I think this change is not quite right. What we want is a simple change around line 437. Right now we test `length` for whether it is `None`: we should just broaden that to say `if length`. That way, we'll fall through to chunked encoding for zero-length files.


I see. What I did didn't make any sense because file objects are _streams_. However, this change won't just affect files, it will affect any stream with length 0. Things will keep working, but I thought we just wanted to set 'Transfer-Encoding: chunked' if `data` was a file object with `super_len(data) == 0`.


@BraulioVM That's actually intentional: sending a chunked zero-length stream is fine, there won't be a problem there. That way, the code is clearer. =)

So, this change looks good to me, but I'll let @sigmavirus24 review it.


Is there anything else I should do regarding this PR?


@BraulioVM Nope, just waiting for @sigmavirus24 to get enough time to swing by and take a look at it. =)


Woops! Sorry. I lost track of this with recent goings on IRL. This looks great to me! Thanks @BraulioVM
Add basic CONTRIBUTING.md file

It's beginning to get frustrating to have to keep dealing with questions on this issue tracker. Let's see if we can dissuade them.


That looks great!


One addition to what makes a good bug report for us.


Updated. =)


I'm not confident people will read this before filing a bug, but we can at least point them to it when closing a bug now.


More importantly, because GitHub prompts people to look at it first we can be sure that anyone who doesn't do what this says definitely didn't care.
How can I detect encoding after I streamed only few pieces of response?

```
    size = 0
    start = time.time()
    for chunk in r.iter_content(1024):
        if time.time() - start > max_time:
            r.reason = 'timeout riched'
            raise ValueError(r.reason)
        size += len(chunk)
        if size > max_size:
            r.reason = 'response too large'
            raise ValueError(r.reason)
```


When you say "encoding", what kind of encoding do you mean?


After stream ends (finish, or I interrupt), I want to use r.content (decoded response). Who can I do this?


If you use `iter_content` manually, it does not save into `r.content`. You should simply save off the data directly yourself.


Okay but I want to use request decode system for my saved data to obtain a result similar to r.text


The easiest way to emulate it is to add `chardet` to your list of dependencies, and then do:

``` python
r = requests.get(url)
if r.encoding:
    text = content.decode(r.encoding)
else:
    encoding = chardet.detect(content)['encoding']
    text = content.decode(encoding)
```

That'll get you text decoded exactly as requests does it.
Ability to use requests as a web server

Hi, I was wondering if a wrapper was existing around requests so that it could be used as a web server ?


@cyberval When you say "used as a web server", what does that mean exactly? What would requests do?


Sorry it's not very clear. I want to access requests through a web server rather than the CLI, something like:
http://mydomain/requests?get=http://foo.com&authuser=a&authpass=b

Because of thrid party tool limitation that can only perform a GET or POST request and can't execute a script...


I'm afraid I'm not aware of any such project, however it should be an easy enough thing to write.
capath support in requests

We are currently using pycurl, but we would like to migrate to requests in order to independent of NSS.
However, we could see that we can only provide a ca_bundle instead of a capath. In urllib2.urlopen you can provide cafile and capath options, however that is not the case in the requests module. 

Also we tried to use urllib2, but this only supports GET and POST requests, we are also interested in HEAD, PUT and DELETE. That is why we would like to use requests.

 Could you let me know if there are plans to support capath in the requests module? 

Thanks a lot,
Maria


Support for directories of CAs is in the current master, and will be released as part of the 2.9.0 release. See also #2858 and #2659.
Specify minimum pyOpenSSL version

urllib3 requires "set_tlsext_host_name" which was only added in
pyOpenSSL 0.13. As some distributions (e.g. Ubuntu 12.04) still ship an
older version enforce the correct minimum version during installation.

I will submit a similar pull requests to the urllib3 project.


It's mind-boggling to me that anyone is shipping a PyOpenSSL that old. Amusingly, the reason we need set_tlsext_host_name is to enable SNI, which is literally the point of using PyOpenSSL with requests, so any earlier version of PyOpenSSL wouldn't do what we need it to do anyway. What a world we live in. Thanks for spotting this!


_reserves comments on Long Term Support versions of distributions harming user security in the interested of perceived stability_
Failure while formatting an error for incorrect SSL CA file

Requests fail while trying to format exception related to non-existent CA file.

Code to reproduce:

```
import requests
requests.get('https://google.com', verify='/some/some')
```

results in:

```
  File "/Users/traut/.p/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/Users/traut/.p/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 348, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/Users/traut/.p/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 318, in _raise_timeout
    if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
TypeError: __str__ returned non-string (type Error)
```

Versions:
Python 2.7.10
Requests 2.8.1


This is definitely a duplicate but I don't have the time to find the exact bug right now. The fix is already merged in urllib3 iirc but @Lukasa can correct me.


Yup, this is a duplicate of shazow/urllib3#556.


cool. thanks for looking into it
sending post request data nested object

Unable to send post data from nested object, some of the data is truncated

``` python
payload = {
   "config": {
      "data" : "test"
   }
}

r = request.post('http://httpbin.org/post', data=payload)
print(r.text)

{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "config": "data"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip, deflate", 
    "Content-Length": "11", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "python-requests/2.8.1"
  }, 
  "json": null, 
  "origin": "xxx.xxx.xxx.xxx", 
  "url": "http://httpbin.org/post"
}

```


What are you expecting the result to be here?


the post data that send become config : data instead of the full object


I'm sorry, I wasn't clear enough.

When you use the `data` keyword, we encode the data using HTTP form-encoding. This is not capable of representing nested data structures, only flat ones. So my question is: what are you trying to achieve, exactly? Because what you're asking requests to do does not make any sense.


If I urlencode the payload  it appear like this

```
config=%7B%27data%27%3A+%27test%27%7
```

which is what should I get on the server


So you literally want to represent the config as a urlencoded JSON string?


Ah sorry shouldn't use urlencode, in php they have http_build_query that convert the payload become like this 

```
config%5Bdata%5D=test
```


@Wiryono That does not match what the urlencoded payload you copied decodes to. In fact, your urlencoded payload corresponds to `config={'data': 'test'}`, which is a JSON-serialized representation of the nested data structure. To get that, you'd want to use:

``` python
import json

payload = {
   "config": json.dumps({
      "data" : "test"
   })
}

r = request.post('http://httpbin.org/post', data=payload)
print(r.text)
```

That will work appropriately.


Ok, so to represent that in requests you want to use:

``` python
payload = {
   "config[data]": 'test',
}
```

There is no consistent, standardised way to represent nested data in form-encoding, and PHP just happily invented one. Requests doesn't guess what you want here, you need to come up with a conversion function yourself I'm afraid.


See also https://github.com/sigmavirus24/requests-toolbelt/issues/45 (which I hope to have time to tackle soon).


The solution is non-obvious, but all you have to do is change the `data` argument to `json`:

```python
r = request.post('http://httpbin.org/post', json=payload)
```

It's not clear to me why the two arguments exist when they seem to be functionally identical. At least, a warning should be given to the user when trying to send a nested dictionary with the `data` argument (see #5058).

They're not functionally identical in the slightest, otherwise both would just work. Please read the documentation.