Raibek/requests-github-issues huggingface dataset:
Refactoring exception hierarchy

This was mentioned in #2230 as a project for requests 3.0, but it probably should get its own thread.

I wrote a lot of words about what I'd like exception hierarchy to look like, here:

https://gist.github.com/kevinburke/b98e053a4bf9835c67bb


I am in principle a strong +1 on this work. It's most definitely a breaking change, however.


I'm starting to collect issues and features for a potential 3.0 release.


Since it seems like a 3.0 release is drawing closer I'm just raising awareness to this issue which seems rather interesting. 


Thanks @jonathan-s. Those of us interested in this were already aware of requests 3.0 drawing close.


Ah, sorry I missed that the 3.0.0 milestone had already been added that's keeping track of the issues for 3.0.0.
Content returned still encoded

When I try the following, the content returned in the response is still encoded. Is this an issue with the requests module?

result = requests.get('http://aceabio.com')

result.content:

'\xff\xfe<\x00!\x00D\x00O\x00C\x00T\x00Y\x00P\x00E\x00 \x00h\x00t\x00m\x00l\x00 \xx......./\x00h\x00t\x00m\x00l\x00>\x00'


`result.content` is always a bytestring, it does not decode the text from whatever text encoding it uses.

The server has not set a text encoding in the headers, so you have three options:
1. Set the encoding yourself. I can see that it's UTF-16, so use that:
   
   ``` python
   result.encoding = 'utf-16'
   result.text
   ```
2. Decode it yourself
   
   ``` python
   result.content.decode('utf-16')
   ```
3. Use `requests.utils.get_encodings_from_content`. This also won't work, for two reasons. Firstly, `get_encodings_from_content` assumes an 8-bit encoding is being used for the meta tag, which is untrue. Second, even if it was true, the upstream server has _lied_ about the text encoding it's using:
   
   ```
   <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
   ```

This is not a bug in requests, this is the upstream server not giving us enough information to work with.
requests will improperly raise errors as ConnectionError

Let's say the server closes the request without sending any data. It's possible that the server began acting on the request (Twilio did this for several years), however requests will raise this as a ConnectionError.

Deep down this gets raised as a BadStatusLine error which is caught by urllib3 and turned into a ProtocolError which is caught & re-raised as a ConnectionError.

See also shazow/urllib3#470.


I tried to produce a test case to demonstrate this, but couldn't figure out how to do this in a nice way. Count this as another vote for https://github.com/kennethreitz/httpbin/issues/153, maybe https://github.com/kevinburke/hamms can get ported/implemented on httpbin.


Sorry @kevinburke, it's not clear to me what the bug is here.


A ConnectionError implies that the server never received/began acting on
data. This way all ConnectionErrors are safe to retry.

imagine you send a POST request to a server and then wait for a response
for 30 seconds, at which point the server closes the connection without
sending data, headers, anything. Requests will raise a ConnectionError. Is
that appropriate?

On Wednesday, September 17, 2014, Cory Benfield notifications@github.com
wrote:

> Sorry @kevinburke https://github.com/kevinburke, it's not clear to me
> what the bug is here.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2230#issuecomment-55998776
> .

## 

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com


Ohhh, yes, that makes sense, but it's a misreading of what 'ConnectionError' means. It doesn't mean 'an error occurred while connecting', it means 'an error occurred that affects the connection'.

We may want to refactor the exception hierarchy to allow your use case (it sounds good to me), but it's a ton of work and requires a move to 3.0.


Can we close this in deference to #2233?


Suits me. =)
Can't get ETag for some resource

I am trying to send get requests to this url: http://data.sfgov.org/resource/rqzj-sfat.json, which will return a etag in the response header, but I just couldn't get it using requests' get method. This is what I did:

```
>>> r = requests.get('http://data.sfgov.org/resource/rqzj-sfat.json')
>>> print r.headers.get('etag')
>>> None
```

Every other fields are in the response header, except for 'etag'. Not sure if it's a bug or I missed something.


Just dug a little bit into the requests' source code, and found requests will set a default header if no header is specified. It's the `Accept-Encoding` header somehow make the server stop sending `ETag`, so closing.
Send delete with data

Hello. 

How I may send delete request with data?

This string don't work to send data: session.request('delete', url, data=data)


@temi4 the following works for me:

``` python
import requests

s = requests.Session()
r = s.delete('https://httpbin.org/delete', data={'foo': 'bar'})
assert {'foo': 'bar'} == r.json()['form']
r2 = s.request('DELETE', 'https://httpbin.org/delete', data={'fiz': 'buz'})
assert {'fiz': 'buz'} == r.json()['form']
```

Could you describe how that is failing to work? Could you maybe show more code than what you have already shown? Is the URL one that you can share?


@sigmavirus24, thanks
I send this 

``` python
data = {  'Reason': u'123' }
request = self.session.request('delete', self.url + '/goodhits/clients/'+client_id+'?token='+token, data=data)
```

at answer I get

```
ERROR_PARAMETER_REASON_CAN_NOT_BE_EMPTY
```

how I may look send or not data at request? 

Then I print header

```
print(request.request.headers)
```

I look this
http://c2n.me/iUQk1B


Okay, so we agree that this is not a bug in how requests handles `DELETE` requests. In that case this belongs on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests) with as much data as you can provide, e.g., 
- how your server app is handling these requests
- What happens when you do:

``` python
import requests
r = requests.delete(url, data={'Reason': '123'})
print(r.request.body)
```

Frankly this seems to be an issue with the server you're running. Perhaps it isn't looking for `Reason` but instead is expecting `reason` or `REASON`, but that is entirely opaque without more information. Even so that's not something for `requests` to solve, that's something you should figure out, potentially with the help of StackOverflow.
Fix errors reported by frosted in tests


@blueyed thanks but we do not follow `pep8` in requests. 


@sigmavirus24 
It's not about pep8, but comes from frosted (and maybe other type (not style) checkers).


Oh right, frosted is the fork of pyflakes. Regardless, these changes are far from critical and are entirely unnecessary. `io` is not a builtin (it is a module) and shadowing it in a local variable is not harmful because the meaning is not being misconstrued. Also both the `url` and `f` are being used. Why they require `# noqa` comments on the same line is baffling. Pyflakes handles these just fine and I'm sure Pylint would as well.
Force specific proxy/proxies / skip looking at the environment

`requests` merges the proxies from the environment unconditionally (https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L439-443).

The use case would be `pip --proxy localhost:3128 install example`, which should make `requests` use the specified proxy always.

I could imagine `PipSession` to override `merge_environment_settings` and discard changes from the environment, but there should be cleaner way to indicate that only the explicitly provided proxies should be used.

Another idea might be to set `PipSession.trust_env = False`, but that would also affect CA bundles and seems even more dirty.
And while pip could unset the environment variables (in case a proxy is provided via its options), this requires to know the implementation details about looking them up.

I would have assumed that `requests` take proxy information from the environment only if no proxies are provided through the API.


> I would have assumed that requests take proxy information from the environment only if no proxies are provided through the API.

That's exactly what happens. You'll find this works a bit better if you read through to `merge_environment_settings`:

``` python
if self.trust_env:
    # Set environment's proxies.
    env_proxies = get_environ_proxies(url) or {}
    for (k, v) in env_proxies.items():
        proxies.setdefault(k, v)
```

This call to `setdefault` does _not_ override any proxy passed to the functional API (e.g `requests.get(url, proxies=proxies)`. It only provides the value if no previous value was present.

Now, we _do_ override things explicitly set on the `Session` with the environment. That is a bug already tracked under #2018.


Yes, you're right. `pip` uses the `Session`.
Fix doc: scheme is not required with proxy URLs

`prepend_scheme_if_needed` is used to add `http` if necessary.


There's also a part in the API migration docs, which would be obsolete then (or needs clarification - see comments below): https://github.com/kennethreitz/requests/blame/master/docs/api.rst#L238-249


I've tested it using the following code:

```
requests.get("http://example.org", proxies={'http': 'localhost:3128'})
```

But when adding a slash/path it fails:

```
requests.get("http://example.org", proxies={'http': 'localhost:3128/'})


File "…/pip/pip/_vendor/requests/adapters.py", line 237, in get_connection
  proxy_manager = self.proxy_manager_for(proxy)
File "…/pip/pip/_vendor/requests/adapters.py", line 147, in proxy_manager_for
  **proxy_kwargs)
File "…/pip/pip/_vendor/requests/packages/urllib3/poolmanager.py", line 265, in proxy_from_url
  return ProxyManager(proxy_url=url, **kw)
File "…/pip/pip/_vendor/requests/packages/urllib3/poolmanager.py", line 216, in __init__
  'Not supported proxy scheme %s' % proxy.scheme
AssertionError: Not supported proxy scheme localhost
```


I guess it's only required really, if urlparse (for Python 3.4.1 in this case) might fail:

```
# in requests/utils.py(640)prepend_scheme_if_needed()
ipdb> url
'localhost:3128/'
ipdb> urlparse(url)
ParseResult(scheme='localhost', netloc='', path='3128/', params='', query='', fragment='')
```

`prepend_scheme_if_needed` is not guaranteed to work, and in case of no scheme, an Exception might get thrown for consistency, instead of silently fixing `host:port`, but not `host:port/`.

OTOH the documentation might get changed to only make the scheme a requirement, if a path is also provided, assuming that `prepend_scheme_if_needed` behaves consistent across python versions. There are no tests for it yet.


Interestingly, `$HOST:$PORT` even fails to get handled properly with Python 2.6 (http://ci.kennethreitz.org/job/requests-pr/433/).


It boils down to that `MissingSchema` is supposed to be raised, but that broke for `host:port` type URLs in https://github.com/kennethreitz/requests/commit/b149be5d864cc7f65ac22113db3d0e4d2ed2b49e.
I've added a PR with failing tests to address/discuss this: https://github.com/kennethreitz/requests/pull/2222.


I've added the commit from this PR also to #2221, which fixes and simplifies `prepend_scheme_if_needed`.

I'm closing this PR in favor of #2221.

The previous comment is not really related, because it isn't related to the proxy url.
Use `base_url` session param to construct relative URLs

This bundles the [URLObject](https://github.com/zacharyvoase/urlobject) library, and uses it for constructing relative URLs.


:-1: This is not appropriate for requests. This is perfectly reasonable for an extension, but not for requests core. This is also available in https://github.com/dstufft/slumber


Ah. When I submitted this to https://github.com/requests/requests-oauthlib/pull/147, there was some thought that perhaps this should go in requests core, but apparently not.


Yeah I just found that myself and saw @Lukasa's [comment](https://github.com/requests/requests-oauthlib/pull/147#issuecomment-55255020). That's exactly how we feel and further this already exists elsewhere. Finally, URLObject's (un)license would prohibit us from actually being able to vendor it in this case. I don't believe it's compatible with Apache v2. [rfc3986](/sigmavirus24/rfc3986) would be vendorable but even so, we won't accept this functionality in requests.


Thanks for thinking of us anyway @singingwolfboy and say hi to Ned for me. =D
Bug on upload file, file name with Latin characters:  requests version  2.3.2 e 2.4.0 

Hello friends. 

I spent a few days trying to upload files using the requests, but had issues with the **version 2.4.0** and python **version 3.2.2.** 

I have a web application that receives a file via a html form, this application sends the received file to a web service, **but when the file name contains Latin characters**, for example: "ç, ã, á, ú" web service can not receive the file name that is not given a cgi.FieldStorage. 

After several attempts trying to encode utf-8 which is already the default string sent did not succeed. I managed to solve the problem only when I made a downgrade to version 1.2.3 of requests, so it worked perfectly. Does anyone know what might be happening to most current versions of requests? An important detail: must do everything in memory, then pass the file name and bytes of it.

I did the following test and it still fails:

``` python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Test upload file
Pyramid-based application

namefile = self.post['filename_0'].filename #str
input_file = self.post['filename_0'].file #bytes
input_file.seek(0) #cursor position 0
binary_content = input_file.read() #read bytes file

r = requests.post(url = "http://webservide.net", 
                         files = {'file': (filename, content)}
                          )
Response(r.text)
"""
import requests

f = open("Atenção 1ª.txt", "rb")

filename = f.name

print(filename)

content = f.read()

#print(content)


r = requests.post(url = "http://webservice.net", 
                         files = {'file': (filename, content)}
                          )

print(r.text)

```

**This even works with version 1.2.3**
 with latest versions like 2.3.2 and 2.4.0 code does **not work.** 

_It will be a bug in the file upload?_


Hi there, thanks for reporting this issue!

When you say it does not work, what exactly to you mean? In what way does this fail?


@Lukasa this is a duplicate of https://github.com/kennethreitz/requests/issues/2117


Good spot! In that case, let's close this. =)
Documentation error

Hi there! Apologies that this probably isn't the ideal place to file this, but I think there's a documentation error at http://requests.readthedocs.org/en/latest/user/quickstart/#make-a-request.

Where it reads "Now, we have a Request object called r", I think it should be "Now, we have a Response object called r", per [the code](https://github.com/kennethreitz/requests/blob/359659cf4b9dbeeef1ed832501dc1f99b0f0beac/requests/api.py#L19) and a test like this:

>>> requests.get('https://api.github.com/events')
<Response [200]>


Great spot, thank you!

It should be fixed now. =)


:thumbsup:! you're welcome--have a good one!
Fix #2207

@Lukasa wrote the fix in #2207 

With this, the expected behaviour is restored (a `TooManyRedirects` error);

``` pycon
>>> import requests
>>> r = requests.get('http://www.dx.com/p/-268479')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./requests/api.py", line 59, in get
    return request('get', url, **kwargs)
  File "./requests/api.py", line 48, in request
    return session.request(method=method, url=url, **kwargs)
  File "./requests/sessions.py", line 451, in request
    resp = self.send(prep, **send_kwargs)
  File "./requests/sessions.py", line 583, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "./requests/sessions.py", line 583, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "./requests/sessions.py", line 111, in resolve_redirects
    raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.
```


I'm an obvious +1. :cake: :stars:


should we add a test for this ? 


> should we add a test for this ?

I'm wont to add a test that will cause our tests to slow-down and may eventually break the tests (if the domain in the original bug fixes its redirects).
Changes the fallback text encoding to cp1252

In my use of Requests, [I noticed at some point](https://stackoverflow.com/questions/11588458/how-to-handle-encodings-using-python-requests-library) that text that was encoded as cp1252 was coming back as ISO-8859-1. While in many cases, that's a fine fallback encoding, cp1252 is a superset of ISO-8859-1, making it right even more often. 

Walking through the scenarios, just to make sure I'm not missing something:
1. Text is encoded as cp1252, but is misidentified as ISO-8859-1. Result: Characters from the cp1252 superset will be mojibake.
2. Text is encoded as cp1252 and is identified as cp1252. Result: All good.
3. Text is encoded as ISO-8859-1, but is misidentified as cp1252. Result: All good. Superset in cp1252 encoding won't cause a problem.
4. Text is encoded as ISO-8859-1 and is identified as ISO-8859-1. Result: All good.

So, I think this can only do good, but encodings have never been my friend, and I'd be only slightly surprised if I missed something. 


Thanks for this!

Unfortunately you've stumbled into a thorny problem. Most of the background is in #2086. The short position here is that we used to fall back to ISO-8859-1 because the spec told us to, but that spec note got removed. For that reason we plan to remove the fallback entirely, not change it to another encoding. The encoding fallback always presented a problem (it was often wrong), so we'd really like to get rid of it entirely.

Sorry about that!


Good lord, that's well above my pay grade and beyond any pay grade I aspire to. I summarily rescind my suggestion and hope that a sane solution can be bartered.


@mlissner it's above all of our paygrades ;)


@mlissner Trust me, this is one of those situations where you'd really rather not know. Get out, get out now. This way lies madness. Dive not into the murky world of HTTP, for monsters lurk in the depths.


All hail the dragon slayers.
TooManyRedirects not working in 2.4.0

hi,

if i try page which has permanent redirect to it self, requests (2.4.0) cant raise TooManyRedirects exception. In 2.3.0 works fine.

try:

```
import requests
requests.get('http://www.dx.com/p/-268479')
```


Hurrah! As I predicted in #2095, our redirect cache has introduced a bug.

Specifically, if a website sets a permanent redirect to itself we will spin in an infinite loop forever on line 535 of `sessions.py`, trying to find the "real" url.

My proposed fix would be to change the logic where we add to the redirect cache (L138 of `sessions.py`) from this:

``` python
if resp.is_permanent_redirect:
    self.redirect_cache[req.url] = prepared_request.url
```

to this:

``` python
# Cache the URL unless it's a redirect to self.
if resp._is_permanent_redirect and req.url != prepared_request.url:
    self.redirect_cache[req.url] = prepared_request.url
```


Don't be too happy @Lukasa. =P
problems with get_unicode_from_response

It looks like there are some problems with the `get_unicode_from_response()` function from requests.utils. The docstring says:

```
Returns the requested content back in unicode.

:param r: Response object to get unicode content from.

Tried:

1. charset from content-type

2. every encodings from ``<meta ... charset=XXX>``

3. fall back and replace all unicode characters
```

but here is the code:

``` python
def get_unicode_from_response():
    tried_encodings = []

    # Try charset from content-type
    encoding = get_encoding_from_headers(r.headers)

    if encoding:
        try:
            return str(r.content, encoding)
        except UnicodeError:
            tried_encodings.append(encoding)

    # Fall back:
    try:
        return str(r.content, encoding, errors='replace')
    except TypeError:
        return r.content
```

Step 1 and step 3 of the docstring are there, but it looks like step 2 is missing. It looks like the intention was to call `get_encodings_from_content` if the result of `get_encoding_from_headers` didn't pan out, but it's not actually doing that.

A more minor issue is the `tried_encodings` list doesn't appear to be serving any benefit, so I'm not sure it's worth its memory / GC cost.

Please advise if you'd accept a pull request or have feedback otherwise. Thanks!

P.S. In researching this issue I noticed that a tag for v2.4.0 is missing: https://github.com/kennethreitz/requests/tags

Here is a reference to this code in the most recent tag available:
https://github.com/kennethreitz/requests/blob/v2.3.0/requests/utils.py#L346..L376


> Step 1 and step 3 of the docstring are there, but it looks like step 2 is missing.

Good catch.

> A more minor issue is the tried_encodings list doesn't appear to be serving any benefit, so I'm not sure it's worth its memory / GC cost.

You're right that it's unnecessary, but it's not incurring any great amount of memory or going to cause a big GC issue (unless you're doing millions of requests an hour or something really incredible, in which case GC is likely the least of your concerns).

> P.S. In researching this issue I noticed that a tag for v2.4.0 is missing: https://github.com/kennethreitz/requests/tags

That's already tracked in #2190. Thanks, though


Looks like it's time to close this issue.


Good catch @ibnIrshad
2.4.0 request refused raises different exception (breaking change)

(related: #2190)

in 2.3.0 the following code worked:

``` python
server_is_down = True
while server_is_down:
    try:
        requests.get(somehost)
    except requests.ConnectionError:
        sleep(0.1)
        continue
    server_is_down = False
```

in 2.4.0 this instead raises some urllib3 error:
`ProtocolError: ('Connection aborted.', error(111, 'Connection refused'))`

this may just be another instance of #1572


Why the hell isn't this wrapped? It looks like it should have raised a MaxRetryError which we'd have wrapped, but it just doesn't. Odd.


@kevinburke Did retry logic change to not raise MaxRetryErrors when `retries=0`?


@Lukasa `retries=False` does not raise `MaxRetryError`s, not sure if the same is true for `retries=0`.


I'm pretty sure this is new: we used to get MaxRetryErrors in our default case. That's going to break a lot of our logic because we don't catch urllib3 HTTPErrors. We probably should treat them like MaxRetryErrors.


@Lukasa I think this is fixed in #2193 since this is very related to #2192.


I actually think this is exactly fixed by #2193. =P
HISTORY.rst contains character unprocessable by ascii library

Hi

Tried to install today, and got an ascii error on illegal character in HISTORY.rst.

Traceback (most recent call last):
  File "setup.py", line 33, in <module>
    history = f.read()
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 472: ordinal not in range(128)

Turns out it is the line just before 'Session.merge_enfiroment_settings' under 2.4.0 (2014-08-29). It is not a normal "-" but some other kind of horizontal line.

Creating an empty HISTORY.rst solves the problem.


Thanks for this! It's a known issue, and will be fixed by #2199.
Open README and HISTORY as utf-8 encoded files (fixes #2196)

This is a fix for #2196 where if you tried running setup.py with some LANG it would throws an exception when reading HISTORY with non-ascii characters.

Try it with

```
LANG=C python setup.py
```


The build broke but that's pypy specific and unrelated to this change

LGTM! :cake:


Are you sure the upload to PyPI won't fail?


@sigmavirus24 Why do we think it might?


Nevermind, I asked @r1chardj0n3s and he assuaged my concerns.
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:598)

Hi there! When i'm trying to execute the code:

``` python
import requests
requests.get('https://www.atarisupport.com/')
```

i get the error:

``` bash
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 722, in _validate_conn
    conn.connect()
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connection.py", line 229, in connect
    ssl_version=resolved_ssl_version)
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\util\ssl_.py", line 123, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Python34\lib\ssl.py", line 364, in wrap_socket
    _context=self)
  File "C:\Python34\lib\ssl.py", line 578, in __init__
    self.do_handshake()
  File "C:\Python34\lib\ssl.py", line 805, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:598)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\requests\adapters.py", line 359, in send
    timeout=timeout
  File "C:\Python34\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 543, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:598)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python34\lib\site-packages\requests\api.py", line 55, in get
    return request('get', url, **kwargs)
  File "C:\Python34\lib\site-packages\requests\api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python34\lib\site-packages\requests\sessions.py", line 448, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python34\lib\site-packages\requests\sessions.py", line 554, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python34\lib\site-packages\requests\adapters.py", line 417, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: EOF occurred in violation of protocol (_ssl.c:598)
```

Any suggestions?


So if you do

```
$ openssl s_client -connect atarisupport.com:443
```

The command will hang, I suspect this is strongly related to the SSLError.


`atarisupport.com` does not respond to the SSL client hello. We emit the first portion of the SSL handshake, and in return the remote website sends us an immediate TCP FIN and tears the connection down.

If you use the [SSLAdapter](http://toolbelt.readthedocs.org/en/latest/user.html#ssladapter) from the requests toolbelt and force the handshake to TLSv1, we are able to establish the connection. Looks like a buggy upstream server to me. =)
UnicodeDecodeError when reading HISTORY.rst (v2.4.0)

Whe installing requests-2.4.0 using pip in python3.4, line 32-33 in setup.py

```
with open('HISTORY.rst') as f:
    history = f.read()
```

is throwing a UnicodeDecodeError caused by the elongated dash on line 18 of HISTORY.rst

Offending line in HISTORY.rst

```
$ head -18 HISTORY.rst | tail -1
- Refactored settings loading from environment ��� new `Session.merge_environment_settings`.
```

Output from running the setup directly:

```
$ python setup.py install
Traceback (most recent call last):
  File "setup.py", line 33, in <module>
    history = f.read()
  File "/usr/lib/python3.4/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 472: ordinal not in range(128)
```


Thanks! The fix you've proposed in #2197 will definitely work, but I briefly wonder whether we shouldn't try to fix the underlying problem instead. We may want to be able to use non-ascii characters in future.


Sounds good. I'll use 2.3.0 for the time being. Feel free to close the pr if you decide to go another route.


Thanks for reporting this @whereskenneth
Add betterssl extra

Fixes #1995 


+1


Could we use another name which doesn't sound like it will solve all problems?


So, I'm going to raise a question!

I was the one who opened the original ticket, but I wonder if it makes sense to do this anymore. On Python 3.4+ and Python 2.7.9+ it's not better anymore. On Python 3.2 and 3.3 it's better, but not in ways that couldn't be reproduced without extra dependencies.

Either way, this PR looks good to me, the only question is if we still need/want it just for Python 2.6 and Python <=2.7.8.


I don't think it will hurt to have it for 2.6 or <= 2.7.8.


I agree that it won't hurt us to have, but I agree with @t-8ch that we should avoid the slightly prejudicial name. Maybe just call it `pyopenssl`?


I really have no opinion on the name. I would think `with_sni` (i.e., `pip install requests[with_sni]` reads well) would be most descriptive of the use case for the largest number of users, but `pyopenssl` is fine too. I'll wait for @kennethreitz to decide.


I'd rather give it a more generic/rememberable name. 

`security`


Updated to `security`. Now, let's document it!


I fear this generic and positive name will lead to people enabling it that don't need it. Triggering compilation of pyopenssl, breakage when missing development tools etc. And on Python3 it will be ignored anyways. 


@t-8ch false


urllib3 uses ndg-httpsclient. Importing this on py3 will raise a syntax error
and abort the import -> PyOpenSSL is not used.
In requests < 2.4.0 The syntax error will even bubble up and just crash
everything.


we can fix that in setup.py


You mean: requests[betterssl] on Py3 will just be a noop?


@t-8ch I don't understand your concerns in this statement:

> In requests < 2.4.0 The syntax error will even bubble up and just crash everything.

The extra does not ship with anything `<= 2.4.0`. How does this affect those people?


Just an example for downsides which can arise when using the pyopenssl backend
Capture and re-raise urllib3 ProtocolError

Fixes #2192 


LGTM. :cake:


+1, though the code block is the same as lines 403-405... could just catch both in one go?


I added a failing test for this in #2194


@kevinburke are you okay with me pulling #2194 in here to keep things simple?


Yep

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Fri, Aug 29, 2014 at 2:26 PM, Ian Cordasco notifications@github.com
wrote:

> @kevinburke https://github.com/kevinburke are you okay with me pulling
> #2194 https://github.com/kennethreitz/requests/pull/2194 in here to
> keep things simple?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/2193#issuecomment-53931737
> .


I think a test against `http://localhost:0/` would also make sense. IIRC `0` means in UNIX that the OS should assign the program a free port.


> +1, though the code block is the same as lines 403-405... could just catch both in one go?

I was concerned others might disagree that it was simply a `ConnectionError`. If anyone is terribly concerned, I can merge the two sets of lines.


+1 Thanks for getting to this quickly.


Is a release coming soon with this? It's not super annoying for me (just giving users a huge traceback instead of a nice error message), but for others it is a critical API break. 


@asmeurer hopefully 2.4.1 will be out in a relatively short period of time.


@sigmavirus24 should we cut it?


@kennethreitz I think we should. This also forced OpenStack to pin to 2.3.0 in some (if not all) projects.


This also is preventing requests upgrade in pip

On Sunday, September 7, 2014, Ian Cordasco notifications@github.com wrote:

> @kennethreitz https://github.com/kennethreitz I think we should. This
> also forced OpenStack to pin to 2.3.0 in some (if not all) projects.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/2193#issuecomment-54752715
> .

## 

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com


Actually wait. We should fix https://github.com/kennethreitz/requests/issues/2207 first. Let me get a PR in


Fixed in #2210
[regression] urllib3.exceptions.ProtocolError not wrapped

``` py
>>> requests.__version__
'2.4.0'
>>> requests.get('http://localhost:1')
# ... stacktrace
requests.packages.urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))
```


Yup I'm hitting this too. Is this going to be fixed quick or should I fix my code? :)


Same issue here.


I'll take a look at this this weekend. =) Thanks for reporting folks!


Looking at it right now @Lukasa 


Ok, suits me. =) If you get there before me I'll happily accept not doing the work. =D


Thanks for the quick turnaround in looking at it guys!


Ditto. This passes on `2.3.0` but fails with `2.4.0`.

``` python
def test_connect_fail(self):
       try:
            requests.get("http://localhost:62123")
            assert False, "The connect() request should be refused."
        except ConnectionError as e:
            assert isinstance(e, ConnectionError)
```


Thanks all for the reports! Please note that a provisional fix is ready in #2193: further +1s and 'me too's are not needed. =)


Is this a related issue ?

  File "build/bdist.linux-x86_64/egg/pyVim/connect.py", line 435, in __GetServiceVersionDescription
    sock = requests.get(url, verify=False)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, *_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, *_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, *_send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, *_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 385, in send
    raise SSLError(e)
SSLError: [Errno 1] _ssl.c:510: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol


No, it is a broken server configuration (or you are talking to something which is not talking SSL back to you).
Take a look at http://toolbelt.readthedocs.org/en/latest/user.html#ssladapter
Document Timeout behaviour

Adds better documentation of `timeout` behavior
- "How long to wait for the server to send data before giving up, as a float, or a (connect timeout, read timeout) tuple." was as short as I could get to accurately describe the behavior.
- Sphinx puts the description of timeouts on the line below the parameter name, but this is not true for other parameters. I'm curious why this is but I can't see any differences.
- We don't use the `stream` parameter anywhere in the `adapters.send` method, so I removed the "Behavior" section that described the behavior of requests incorrectly. May also want to remove the parameter description from the documentation, or state that it's not actually used anywhere.


Also, does requests use british/canadian or US spellings? I see "behavior" spelled both ways in the docs


This change only affects documentation, the test failure looks to be global & related to a bad configuration, see https://github.com/kennethreitz/requests/issues/2189


I think I can safely speak for Cory (and myself) in saying we prefer the King's English (British spelling). We have accepted both though.
Change request to make keep alive more human

Not sure if this is the right way & design to do this:
- I added a new property to request and session
- I used existing merge function to assure that if something is set in request parameter, it will overides sessions´s ones.
- I keep it disable by deafault as https://github.com/kennethreitz/requests/issues/2169#issuecomment-53496634 stated.

More info in https://github.com/kennethreitz/requests/issues/2169 thread.


Thanks for this!

This is not normally the way we'd go about this. In particular, we're averse to adding more keyword arguments to the request methods. I don't think this approach is going to be the one we take: we should just be setting this appropriately in the headers and allowing users to override it.


Ok. I have no global view of the project.
I just tryed to match the verify=False, but it may just been added here ( https://github.com/kennethreitz/requests/blob/master/requests/utils.py#L553) once you reach 3.0.0.
socket.error raised from r.iter_content

My code

```
                r = requests.get(url, stream=True)
                for chunk in r.iter_content(chunk_size=16*1024):
                    if chunk: # filter out keep-alive new chunks
                        f.write(chunk)
                        f.flush()
```

raises an exception socket.error; I would expect it to be mapped to requests.exceptions instead.

```
    for chunk in r.iter_content(chunk_size=16*1024):
  File "/usr/lib/python2.7/dist-packages/requests/models.py", line 616, in generate
    decode_content=True):
  File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 225, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/lib/python2.7/dist-packages/urllib3/response.py", line 174, in read
    data = self._fp.read(amt)
  File "/usr/lib/python2.7/httplib.py", line 567, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/ssl.py", line 341, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 260, in read
    return self._sslobj.read(len)
error: [Errno 104] Connection reset by peer
```

@Lukasa, since you fixed another exception handling issue already, would you care to take a look at some point? Easy enough to work around but should be fixed to maintain a human API.


Agreed. Either we should catch it ourselves, or urllib3 should catch this stuff from read or stream. @shazow?


Isn't there already an open issue for this?


Ugh, we actually already fixed it. =( I just totally missed it in my code read. Should be fixed in the next release! =)


with what exception I can solve this issue?
Fix #2180.

@kennethreitz?


:+1: 


Any reason why only the first reference to `timeline.json` is fixed? Are the [other](https://github.com/kennethreitz/requests/blame/7e24176ca137cfc1c134343d991163d5f8779333/docs/user/quickstart.rst#L84) references to it valid?


@myint No, I just missed them. This should be fixed in master now. Thanks for reporting it!
Github timeline no longer available - makes poor example

The quickstart tutorial recommends fetching https://github.com/timeline.json, which does technically work, and it does return a block of JSON text. But it's an unideal example, because all it tells you is "this service isn't available, go use the Events API". Could do with replacement, I think.


Good catch! @sigmavirus24, you know the GitHub API best, what endpoint should we hit?


It wasn't actually me who caught it, but a student I'm mentoring. He sent me his non-working code, I tried it on my system (it worked - he had a completely unrelated issue), and that's where I saw this. :)


So that used to be the public timeline of events on GitHub apparently. If we want to just replace the URL, we could use `https://api.github.com/events`. That'll return the latest 25 events.


Let's do that.


Looks good to me!


we could always do httpbin too


We do httpbin elsewhere in that example, but I liked that it was useful information.
Add Session method to merge environment settings with per-request settings

See also #2129 


One minor note, otherwise LGTM.


Tests fail. 


Tests were passing locally so I rebased and pushed hoping that solves the issue.


Looks like httpbin is having intermittent failures? The same tests aren't failing on the same Python versions, and there is a lot of this

ProtocolError: ('Connection aborted.', error(110, 'Connection timed out'))


Hmm, just changed the deployment configuration.
Update PyPy version on ci.kennethreitz.org

The latest version of PyPy is 2.3.1 (Python 2.7.6), but the Jenkins boxes are running pypy 1.8

The pypy docs mention something about a sandboxed version shipping with 1.8, is that why it's so old on those machines?


@kennethreitz, we're going to need your input on this one.


Relatedly, we should be adding Python 3.4 to the test matrix.


I'll have to do the dirty work soon :)


Python 3.4 added. PyPy isn't a big deal.
Add support for connect timeouts

Modifies the timeout interface to also accept a tuple (connect, read) which
would be used to set individual connect and read timeouts for Requests. Adds
Advanced documentation explaining the interface and providing guidance for
timeout values.


For perspective, this is basically a rebase/rewrite of #1801.

I have always been in favour of this change, at least in principle, and I'm happy with the implementation as well. =)


okay I killed the extra format try/catch & rebased


So there is a latent issue here.

In the event that Requests supports the `urllib3.util.Retry` class at some future point, more things may be raised as MaxRetryError, including read timeouts. 

I don't know that there's anything to do right now since requests controls how retry is used, but worth mentioning.


Let's deal with one problem at a time. That's not a bug _right now_, it's just a trap for anyone who wants to implement that support (inevitably it would be you, Kevin, so it's all good. ;) )


I think the way we currently use the `Retry` class would allow users to import and use the `Retry` class with requests. I might be remembering incorrectly though.


:sparkles: :cake: :sparkles:


Time for a new release, once #2178 is ready. 


I would petition that we wait until @willingc can get her `json` parameter pull request in good shape for us to review.


@sigmavirus24 that can be 2.4.1 :)


@kevinburke want to add some docs? :)


@kennethreitz I added this section on timeouts:
http://docs.python-requests.org/en/latest/user/advanced/#timeouts

Just noticed I missed some method signatures, will add the docs for those
now.

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Fri, Aug 29, 2014 at 7:28 AM, Kenneth Reitz notifications@github.com
wrote:

> @kevinburke https://github.com/kevinburke want to add some docs? :)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/2176#issuecomment-53882868
> .
Clarify which tests to run in the Makefile

If you put your Virtualenv in the same folder as Requests (as I
usually do), py.test will pick up any test_\* files inside the
virtualenv and attempt to run them too, leading to the following error:
https://gist.github.com/kevinburke/1aa2b07e01de3a7daa15

Instead specify the test file we would like to run in the Makefile.


:+1: :cake:

This is a PITA. Let's do it.
https requests error

Just see my example: it works with curl 
[root@localhost ~]# curl -X GET https://openapi/face/ --cacert /export/server.pem 
{"datetime": "2014-08-21 08:30:30.896918", "face": "a smiling face"}[root@localhost ~]# 

But requests did not work, see

> > > r = requests.get('https://openapi/face/', cert='/export/server.pem')
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "/usr/lib/python2.6/site-packages/requests/api.py", line 55, in get
> > >     return request('get', url, *_kwargs)
> > >   File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
> > >     return session.request(method=method, url=url, *_kwargs)
> > >   File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 456, in request
> > >     resp = self.send(prep, *_send_kwargs)
> > >   File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 559, in send
> > >     r = adapter.send(request, *_kwargs)
> > >   File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 382, in send
> > >     raise SSLError(e, request=request)
> > > requests.exceptions.SSLError: [Errno 336265225] _ssl.c:339: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib

Can anybody help me


You're using the wrong argument. The `cert` command line argument provides a _client_ certificate, you want to provide the server certificate. Try this:

``` python
r = requests.get('https://openapi/face/', verify='/export/server.pem')
```
SSL verification failing for a valid GeoTrust Global CA certificate

I'm having trouble with SSL cert verification for a valid CA-signed cert ... and I can't seem to add the CA cert that I need to the cacert.pem file.

Computer: Raspberry Pi connecting to a website on IIS8/WS2012 (TLS 1.0/1.1/1.2; Note: SSL3 is not supported by this server)
OS: Arch Linux (3.12.26-1-ARCH)
Language: Python 3.4.1
Certificate Chain: my_site_cert > RapidSSL CA (Serial #: 0236d1) > GeoTrust Global CA (Serial #: 023456)

Trying to get an secure session open to some valid CA-signed endpionts (e.g,. Microsoft Azure endpoints with certificates signed by Baltimore CyberTrust Root) work fine with requests. However, attempting to open a secure connection to a website with an equally valid certificate signed by a GeoTrust Global CA fails with the invalid certificate error.

It looks like requests is using the CA bundle at:

```
/usr/lib/python3.4/site-packages/requests/cacert.pem
```

... and that bundle does contain a GeoTrust Global CA. Looking at the cert entry for the GeoTrust Global CA entry, I can see that it does not match the base64-encoded cert that I pulled down from my certificate chain. The serial numbers don't match either.

I took the base64-encoded certificate block for the one I presumably need

```
-----BEGIN CERTIFICATE-----
xxxxxxx
-----END CERTIFICATE-----
```

... and pasted it into the cacert.pem file that requests uses. I rebooted. This did not work. I'm still getting the exception:

```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)
```

What do I need to do to get my CA cert installed in such a way that requests will pick it up and grant my secure connection? The docs are too vague about the specific path to use for other ca bundles in Arch Linux (e.g., ca-certificates or openssl ca bundles), and there is no information about getting other CA certs in the right spot for requests to pick them up. I certainly don't want to run a production system with `verify=False`.


Is your system time correct?


Good grief! It's amazing how simple it was to fix this. Thank you so much for that. It turns out that I had the wrong command in my service shell script that runs on boot to move the hardware clock time to the system time (I see now that the correct command is `hwclock --hctosys`). DOH! Bart! I'm a .NET guy, so what do I know about Linux ... still way way back on the learning curve. Thanks again. I recommend this tiny little tip be put into the docs in the SSL Verification section.
Tracking Previous Requests

Addresses the issue brought up here:
https://github.com/kennethreitz/requests/issues/1929


This looks good to me! I've made a few notes inline. Do you mind fixing them up?


@Lukasa I will try to come up with a better way to test.


Shouldn't be too hard. The final response's history (`r.history`) contains the list of all responses in order. Each of those responses should have a slice of that history in its own history in the region `[0, i)`, where `i` is the index of the response in the original response's history.


@Lukasa I believe I have made the appropriate changes.