Raibek/requests-github-issues huggingface dataset:
Test failure for 2.8.0

I see this test failure:

```
y.test test_requests.py 
============================================================================================================================= test session starts ==============================================================================================================================
platform linux2 -- Python 2.7.10, pytest-2.8.2, py-1.4.30, pluggy-0.3.1
rootdir: /home/davidc/src/python/requests, inifile: 
collected 165 items 

test_requests.py ......................................................................................x...........................................................F..................

=================================================================================================================================== FAILURES ===================================================================================================================================
_______________________________________________________________________________________________________________ TestMorselToCookieExpires.test_expires_valid_str _______________________________________________________________________________________________________________

self = <test_requests.TestMorselToCookieExpires testMethod=test_expires_valid_str>

    def test_expires_valid_str(self):
        """Test case where we convert expires from string time."""

        morsel = Morsel()
        morsel['expires'] = 'Thu, 01-Jan-1970 00:00:01 GMT'
        cookie = morsel_to_cookie(morsel)
>       assert cookie.expires == 1
E       AssertionError: assert -3599 == 1
E        +  where -3599 = Cookie(version=0, name=None, value=None, port=None, port_specified=False, domain='', domain_specified=False, domain_in...False, secure=False, expires=-3599, discard=False, comment='', comment_url=False, rest={'HttpOnly': ''}, rfc2109=False).expires

test_requests.py:1455: AssertionError
=============================================================================================================== 1 failed, 163 passed, 1 xfailed in 33.37 seconds ===============================================================================================================
```

Is this expected ?


Sadly, this is a known problem. See #1859 for the initial report and #1860 for a proposed fix that never landed. Currently we don't have a good proposal for how to fix this permanently, but the test failure you have right there is benign.


understood. Thanks for letting me know
Please create v2.8.0 tag

Tag v2.8.0 seems missing: e499b354c4215b158db6d4210136e0b9adbe8d14


Sorry, pushed it up now. =)


Thanks
Le 12 oct. 2015 19:09, "Cory Benfield" notifications@github.com a écrit :

> Sorry, pushed it up now. =)
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2823#issuecomment-147463056
> .
Allow non-native-string method names.

Resolves #2818.


I'm starting with just the failing test to confirm that our environment can actually reproduce it.


@sigmavirus24 I'd like to slip this into 2.8.1, which I plan to release tomorrow. Mind doing a quick review before then?


I feel certain that we've fixed this already, perhaps in 3.0.0?

And a quick look shows we fixed it on July 18th: https://github.com/kennethreitz/requests/commit/ea1cbb3a1d9c6fb0edebb1bf0ac222c50054a367

I don't see discussion in the [bug](https://github.com/kennethreitz/requests/issues/2613) about why we didn't propose that against master though (although parts of it are backwards incompatible).

I'm +1 on fixing this and this looks like a good, simple, probably backwards compatible change.


Yeah, I don't really see why we didn't either. I think this fits here, and that past me was an idiot. =)
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

I've been getting this error randomly while using [RoboBrowser](https://github.com/jmcarp/robobrowser). RoboBrowser installs with requests 2.7.0 but I checked line 166 of current version (2.8) of retry.py and it's the same.

I should add that I made a small modification to a file in RoboBrowser as described here: https://github.com/jmcarp/robobrowser/issues/52#issuecomment-144929550

```
Traceback (most recent call last):
  File "fetch.py", line 139, in <module>
    go()
  File "fetch.py", line 120, in go
    result = fetch(url)
  File "fetch.py", line 29, in fetch
    browser.open(url)
  File "C:\Users\Vidpe\Desktop\mpb\venv\lib\site-packages\robobrowser\browser.py", line 205, in open
    response = self.session.request(method, url, **self._build_send_args(**kwargs))
  File "C:\Users\Vidpe\Desktop\mpb\venv\lib\site-packages\requests\sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Vidpe\Desktop\mpb\venv\lib\site-packages\requests\sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\Vidpe\Desktop\mpb\venv\lib\site-packages\requests\adapters.py", line 370, in send
    timeout=timeout
  File "C:\Users\Vidpe\Desktop\mpb\venv\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 617, in urlopen
    release_conn=release_conn, **response_kw)
  File "C:\Users\Vidpe\Desktop\mpb\venv\lib\site-packages\requests\packages\urllib3\connectionpool.py", line 598, in urlopen
    retries.sleep()
  File "C:\Users\Vidpe\Desktop\mpb\venv\lib\site-packages\requests\packages\urllib3\util\retry.py", line 175, in sleep
    backoff = self.get_backoff_time()
  File "C:\Users\Vidpe\Desktop\mpb\venv\lib\site-packages\requests\packages\urllib3\util\retry.py", line 166, in get_backoff_time
    backoff_value = self.backoff_factor * (2 ** (self._observed_errors - 1))
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
```


So a quick code read suggests that this is _impossible_ using standard requests. The `backoff_factor` attribute is defaulted to `0` and never assigned to `None` by any code in requests, as far as I can see.

However, a quick search of RoboBrowser reveals that _it_ sets the `backoff_factor`, [here](https://github.com/jmcarp/robobrowser/blob/a243352a25f093d8107067cc63f5784b00e6f39c/robobrowser/browser.py#L109), and in the default case will set it to `None`. `None` is not an allowed value for the backoff factor: this is a bug in RoboBrowser.
Problem with `json` parameter in POST for requests 2.8.0

The following code works in requests 2.7.0 but raises an assertion error in 2.8.0:

```
import requests

session = requests.session()
session.headers.update({'x-api-key': 'blah', "Accept-Encoding": "gzip, deflate",
                              "User-Agent": "magnet-sdk-python"})
session.verify = True
req = requests.Request(method='POST', url='https://portal.niddel.com/api/v1-beta/alerts', params=None, json={'a': 1, 'b': 2})
req = session.prepare_request(req)
assert req.body
```

Since I provided neither `files` nor `body`, I would have expected `json` to work.

In case you need this info to reproduce the issue, all tests were performed on a Mac OS X with Python 2.7.10 (installed using homebrew).


@asieira Thanks for this! This is a duplicate of #2813, and should be resolved in 2.8.1.


:+1:


Bom dia Pessoal, estou com o um projeto na empresa onde devo se comunicar com o serviço do governo e estou patinando para se autenticar no mesmo onde pede para enviar o certificado encapsulado em uma keystore como demostra o link abaixo:

https://val.portalunico.siscomex.gov.br/docs/api/

Já tentei de diversas formas, mas sempre me retorna o erro:

HTTPSConnectionPool(host='val.portalunico.siscomex.gov.br', port=443): Max retries exceeded with url: /portal/api/autenticar (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

Fiz o teste pelo Postman tambem e esta se comunicando corretamente, ate copiei o código gerado por ele, só que o mesmo não gera o parte do codigo do handshake:

O certificado esta VALIDO, e é utilizado no ambiente de produção, ou seja não tem nenhum problema com o mesmo.

import http.client 

conn = http.client.HTTPSConnection("portalunico.siscomex.gov.br") 

headers = {'role-type': "IMPEXP"} 

try:

    conn.request("POST", "/portal/api/autenticar", headers=headers) 

    res = conn.getresponse()

    data = res.read()

    print(data.decode("utf-8"))

except Exception as erro:

    print('Erro:'+str(erro))  

 

Segue abaixo o codigo que tenho:

# coding:utf-8

#!/usr/bin/env python 

from OpenSSL import crypto

from requests_pkcs12 import post

import requests_pkcs12 

url = "https://val.portalunico.siscomex.gov.br/portal/api/autenticar"

certificado = "certificado.p12"

 

#certif = pkcs12_filename = certificado

#key    =  pkcs12_password = '123456'

 

#certificado = (pkcs12_filename=certificado,pkcs12_password='123456') 

header = {"Content-Type": "application/json",                          

          "Role-Type": "IMPEXP"} 

try:

    post(url,headers=header,proxies =proxy, pkcs12_filename=certificado,pkcs12_password='123456')   

except Exception as e:   

    print('Erro:'+str(e))

 

Se alguém souber de como eu realizar essa autenticação ficaria muito grato. 

Obrigado ....
Python 3.5 install requests failed

```
C:\Users\gengjiawen>pip install --no-cache requests
Collecting requests
  Downloading requests-2.8.0-py2.py3-none-any.whl (476kB)
    100% |████████████████████████████████| 479kB 122kB/s
Installing collected packages: requests
Exception:
Traceback (most recent call last):
  File "c:\program files\python\lib\site-packages\pip\basecommand.py", line 211, in main
    status = self.run(options, args)
  File "c:\program files\python\lib\site-packages\pip\commands\install.py", line 311, in run
    root=options.root_path,
  File "c:\program files\python\lib\site-packages\pip\req\req_set.py", line 646, in install
    **kwargs
  File "c:\program files\python\lib\site-packages\pip\req\req_install.py", line 803, in install
    self.move_wheel_files(self.source_dir, root=root)
  File "c:\program files\python\lib\site-packages\pip\req\req_install.py", line 998, in move_wheel_files
    isolated=self.isolated,
  File "c:\program files\python\lib\site-packages\pip\wheel.py", line 339, in move_wheel_files
    clobber(source, lib_dir, True)
  File "c:\program files\python\lib\site-packages\pip\wheel.py", line 310, in clobber
    ensure_dir(destdir)
  File "c:\program files\python\lib\site-packages\pip\utils\__init__.py", line 71, in ensure_dir
    os.makedirs(path)
  File "c:\program files\python\lib\os.py", line 241, in makedirs
    mkdir(name, mode)
PermissionError: [WinError 5] Access is denied: 'c:\\program files\\python\\Lib\\site-packages\\requests'
```

I am on win10(64bit), python 3.5.0.
Python 3.4 has not this problem.
Run the terminal as Admin will fix this.


I believe this is a problem with your installation, not with requests. Do you encounter this problem with other packages? E.g., try running `pip install --no-cache hyperframe`.


@Lukasa yes, you are right,this is may be a issue of python 3.5,I will close this issue:)
Requests json argument is ignored in 2.8.0

In version `2.7.0` I was using `Requests` constructor with `json` parameter specified, like:

```
json_data = [ ... ];
r = requests.Request('POST', url, json=json_data)
```

And in the `2.8.0` version this code does not work, because I do not specify `data` parameter. Inside constructor we get `self.data` assigned as `[] if data is None else data`, so we finally get `self.data` equal to `[]`.

And because of this commit: 7a0cd16c1a5be16f3df395ee12ed11d0ddde7837 we get `json` parameter ignored, because now we have to explicitly specify `data={}` when calling constructor, like:

```
json_data = [ ... ];
r = requests.Request('POST', url, json=json_data, data={})
```

I think that this issue definitely breaks documented API.


Ah, yeah, that definitely happened. Sorry. =(

This was introduced in #2763. Looks like we need a slightly more subtle check than the one we introduced.


See #2814 for a proposed fix.
Don't treat NewConnectionErrors as timeouts.

Resolves #2811.

Fundamentally I view this mostly as a compatibility change, to get us back into line with how we've processed this kind of error in the 2.X series. I think I'd like to back it out in 3.0.0, which is why I've left a note in the code to that effect.


@sigmavirus24 How do you feel about removing this in 3.0.0?


+0


@Lukasa I've found a TODO made by you and associated with this issue https://github.com/requests/requests/blob/3.0/requests/adapters.py#L531
2.8.0 release raise ConnectTimeout on not known url

2.7.0 and 2.8.0 raises diff exception on not known url:

import requests
requests.get('http://fake')

2.8.0:
requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='fake', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7fac9c476110>: Failed to establish a new connection: [Errno -2] Name or service not known',))

2.7.0
requests.exceptions.ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))


Hmm, looks like urllib3 changed how it wrapped this exception. Let me see if I can find where this problem got introduced.


I suspect shazow/urllib3#703 may have introduced a new exception type that urllib3 treats differently to `socket.error`.


@Lukasa if the OpenStack gate is indicative of anything, it's between 1.10.4 and 1.11


Looks to me like it was actually done in 1.12: `NewConnectionError` is a subclass of `ConnectTimeoutError`, whereas previously `socket.error` wouldn't have been.


@Lukasa : yes looks like new NewConnectionError raised here: https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connection.py#L142


Hmm, that's not right though: `socket.error` should have entered the retry block.


Regardless, the reason we think it's a `ConnectTimeout` is because the new exception is a child of `ConnectTimeoutError`. This is somewhat tricky to deal with: `NewConnectionError` is _sometimes_ a timeout, but sometimes not. 


I think from requests' perspective we just want to treat `NewConnectionError` as _not_ a timeout, even though under the hood urllib3 will attempt to retry when it hits it.


Also cc'ing @shazow 


Ok, a proposed bugfix is in #2812. There's some discussion in there about whether this fixes a permanent bug, or whether it's strictly a compatibility fix.


I suspect some of this might be platform/configuration-specific. I can't remember the exact details but I think there was a distinction between how OSX vs Linux (maybe Posix vs GNU?) systems behave when you make a new socket to a bad host. It might also have to do with the DNS settings. I vaguely remember in one case it would try to resolve it and hang around until timeout, in the other case it would quickly eliminate it as not resolvable and return a lower-level error.

I tried to normalize them under the same exception but not sure if I succeeded.
Cert present in certifi but not in requests

https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem#L5026

Can the Entrust G2 certificate be added to the requests certificate store?


I think the core issue here is that requests' certificate store is somewhat out of date. I propose we ship a 2.8.1 containing a cert store update, specifically the `weak.pem` bundle that ships with certifi.

@hombremuchacho Someone I presume was your colleague or another team member (@aehlke) suggested that certifi was missing a certificate you needed, so you couldn't use it as a drop-in replacement. Did you try using `certifi.old_where()`?


(To be clear, the bundle pointed at by `certifi.old_where()` is exactly the same as the bundle I'm proposing in #2810, so if that works for you I'll know it's a good choice to move forward with.)


@Lukasa Thanks, I'll test this now.
Adding Entrust G2 CA Cert

Adding Entrust G2 CA Cert to cacerts.pem


I could be mistaken but I'm rather certain that we don't accept additions to the certificate bundle directly. They have to go through [certifi](/certifi/python-certifi) first. @Lukasa am I right?


We do not accept changes to the cert bundle in requests in this manner: certainly not one certificate at a time. 

Would you like to open an issue that discusses the actual problem you have encountered?


@Lukasa @sigmavirus24 Thanks for clarifying how this works. I'll open an issue. We use the requests library as part of our project and are missing a certain certificate -- this certificate is present in certifi; however, it hasn't been added to requests. https://github.com/certifi/python-certifi/blob/master/certifi/cacert.pem#L5026


To clarify, Certifi was not a drop-in solution for us as it was missing at least one cert that's bundled with Requests.
HTTP Proxy with prepared request (honouring env. var.)

Running the following it looks like the proxy env. var. is not honoured:

``` python
>>> # requests version 2.2.1 / python 3.4.0
>>> import requests
>>> import os
>>> os.environ['HTTP_PROXY'] = 'http://some.wrong.dns.example.org'
>>> req = requests.Request('GET', 'http://httpbin.org/get')
>>> r = req.prepare()
>>> s = requests.Session()
>>> s.send(r)
<Response [200]>
```

Is it the expected behaviour with the lower level interface ?
I was actually expecting request to transparently use HTTP_PROXY as it does with the main interface requests.get() or requests.request('GET',...).

Cheers


It's certainly the _coded_ behaviour: if you don't use something that goes through `Session.request` then you never get the environment variables.

However, I think this is unexpected. It doesn't seem like there's any reason we couldn't move the call to `Session.merge_environment` into `Session.send`, so I'm inclined to say we should do that. Unfortunately, it's a bit backward-incompatible IMO, so I'd want to sit on this until 2.9.0 or, more likely, 3.0.0. Thoughts @sigmavirus24?


OTOH such isolation from the environment could be desirable and the reason someone uses prepared requests in the first place.


So we explicitly made [`merge_environment_settings`](https://github.com/kennethreitz/requests/blob/e499b354c4215b158db6d4210136e0b9adbe8d14/requests/sessions.py#L615) a public method which could be used by people taking advantage of the PreparedRequest flow. Perhaps we never quite documented it very well when we added it, but I'm not really for moving most of the logic into send because then someone's going to want something that does far less than send does and we will be wanting to move things back into `Session.request`. I think the existing behaviour is fine. We probably just need to document `merge_environment_settings` as a _potential_ part of the Prepared Request flow.


Sounds reasonable to me.


Alright, thanks for the clarification.

Then how should I use merge_environment_setting in order to merge only HTTP proxies var ?

Is this correct ?

``` python
>>> import requests
>>> import os
>>> os.environ['HTTP_PROXY'] = 'http://some.wrong.dns.example.org'
>>> req = requests.Request('GET', 'http://httpbin.org/get')
>>> r = req.prepare()
>>> s = requests.Session()
>>> settings = s.merge_environment_settings(req.url, {}, None, False, None)
>>> s.send(r, **settings)
# => should raise a gaierror because of wrong proxy in os.environ
```

@sigmavirus24 : I agree, the existing behavior is fine, It just lacks some precision in the documentation though.


@mxjeff right, that's why I said we probably need to document it better.

Also your code sample is correct.
allows user paths for ca_cert verify path

currently you get the following error:

`SSLError("bad ca_certs: '~/example.crt'", Error([('system library', 'fopen', 'No such file or directory'), ('BIO routines', 'BIO_new_file', 'no such file'), ('x509 certificate routines', 'X509_load_cert_crl_file', 'system lib')],))`


Thanks for this! However, this is a change in the [urllib3](https://github.com/shazow/urllib3) library, which we vendor in. Can you please open this pull request over there? Thanks!
Update default user-agent (#2785)

- Updated default user agent so that it includes only the requests
  version;
- Added PyCharm / IDEA project directory to `.gitignore`.


If we can clean this up relatively swiftly it'll make it into 2.8.0: otherwise, it'll fall out until 2.8.1.


I wouldn't be comfortable including this in 2.8.1. 2.9.0 definitely, but not 2.8.1


@asieira No, I think I recommend adding it to your local `.git/info/exclude` file. Text-editor specific config really shouldn't go into the project repository, the world just gets really complex really fast that way.


Ok, will do that. 


LGTM. @sigmavirus24?


@Lukasa will you handle the release notes for this or should I do it later tonight?


@sigmavirus24 I'll do it.

@asieira Thanks so much for this change! :sparkles: :cake: :sparkles:


:bow: my pleasure, thank you all again for the awesome package. :)
IP Address of target host in Response

Hello,

Is there any way to get IP address of target host of a request? Looks like it's possible to get it from urllib3's HTTPResponse object with <Object>._fp.fp._sock.getpeername() but I couldn't find how to get it working with requests.

Is there any easy way or work around for it? Or can you please add it to feature requests and to Response object as an attribute? It would be good to use it like
r = requests.get("http://blabla.com")
print r.ip_addr

Best Regards,
Sezgin


Hi @sezginriggs thanks for taking the time to open this. We've discussed this previously on https://github.com/kennethreitz/requests/issues/2158 and decided against this. We're currently in a feature freeze (and will be for the rest of the project's life time in all likelihood) so we're not adding new features.

In the future, please search all issues using GitHub before opening new feature requests.

Cheers,
Ian


@sezginriggs use `stream=True`, grab the IP before you read the response body.

More here https://github.com/kennethreitz/requests/issues/2158
async and await

Python 3.5 [introduced](https://docs.python.org/3/whatsnew/3.5.html#new-features) coroutines with async and await syntax. Specification about that can be read in [PEP 492](https://www.python.org/dev/peps/pep-0492/).

Are there any plans to support this?  Because of network lags or slow connections it would make sense to make asynchronous http requests.


@screendriver At the current time there are no plans to support async and await. This is not because they aren't a good idea: they are. It's because to use them requires quite substantial code changes.

Right now requests is a purely synchronous library that, at the bottom of its stack, uses `httplib` to send and receive data. We cannot move to an async model unless we replace httplib. The best we could do is provide a shorthand to run a request in a thread, but asyncio already has just such a shorthand, so I don't believe it would be valuable.

Right now I am quietly looking at whether we can rewrite requests to work just as well in a synchronous environment as in an async one. However, the reality is that doing so will be a lot of work, involving rewriting a lot of our stack, and may not happen for many years, if ever.
Update Quickstart documentation of Response.json()

Fixes #2799 


This is really good! One note: can we remove the reference to `response.ok`? It's at risk of deprecation, so I'd rather not add new references to it.


Thanks. I beg to differ on `response.ok`. I find it useful and advise people to use it, although I'd prefer to change its implementation to avoid raising and catching exceptions. It could be as simple as `return self.status_code < 400`. I could update the pull request if you wish.


The problem with `ok` is that it's not entirely clear what exactly it means: it's ambiguous and weird. Changing it would be better, yeah. =)


I've updated the implementation of `ok` and added a docstring to it.


Sorry, that was me being really unclear. I didn't want you to change `ok` itself, I wanted you to change the reference to it. =( Sorry.


Feel free to edit the description to fit your needs. I still think using `ok` is clearer for new users (to whom the Quickstart guide is targeted), IMHO.


To be clear, I'm not going to accept a change to the implementation of `ok`, especially one that changes the contract. If you'd rather not make further changes to this, that's fine, I can merge it and make the changes myself.


My change to the implementation of `ok` certainly did not change the contract; it still behaves exactly the same. And certainly feel free to accept it or not; it isn't really important.

I'm just not comfortable changing the reference to `ok` in the Quickstart guide, so I'd rather you do it yourself if you really think it's better without it.


@Lukasa would you like to revert these changes, or should I?


Thanks so much @hosamaly!


You're welcome @Lukasa . Thanks for accepting it.
Quickstart note about json rendering is misleading

In the Quickstart document, the [JSON Response Content](http://docs.python-requests.org/en/v2.0-0/user/quickstart/#json-response-content) section says:

> In case the JSON decoding fails, `r.json` raises an exception. For example, if the response gets a 401 (Unauthorized), attempting `r.json` raises `ValueError: No JSON object could be decoded`

This is a bit misleading, especially that the [`json()`](https://github.com/kennethreitz/requests/blob/v2.7.0/requests/models.py#L798) method makes no attempt to validate that. It is accurate to say that failure to decode JSON will result in an error being raised. However, receiving 401 does not necessarily mean that JSON decoding will fail. Some APIs might return a JSON object in the response body (e.g. `{ 'error': 'Unauthorized' }`), in which case `r.json()` will work fine.


That's a fair observation. Got a suggestion for clearer language?
ImportError: cannot import name certs

opensuse 13.1 / python-requests = 2.7.0-63.1 / 
## In [8]: import requests;

ImportError                               Traceback (most recent call last)
<ipython-input-8-eedc5c747136> in <module>()
----> 1 import requests;

/usr/lib/python2.7/site-packages/requests/**init**.py in <module>()
     56     pass
     57 
---> 58 from . import utils
     59 from .models import Request, Response, PreparedRequest
     60 from .api import request, get, head, post, patch, put, delete, options

/usr/lib/python2.7/site-packages/requests/utils.py in <module>()
     23 
     24 from . import **version**
---> 25 from . import certs
     26 from .compat import parse_http_list as _parse_list_header
     27 from .compat import (quote, urlparse, bytes, str, OrderedDict, unquote, is_py2,

ImportError: cannot import name certs


this issue is fixed with version -requests = opensuse 13.1 / 2.7.0-66.1 /
Failure in tests on specfic host

Hi I am facing a weird issue both with python2 and python3 which seems to be limited to a specific host (my laptop). A few of the test cases are failing and I do not know why.

```
[rvdzwet@rvdzwet-asus tmp]$ git clone https://github.com/kennethreitz/requests.git
Cloning into 'requests'...
remote: Counting objects: 15705, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 15705 (delta 0), reused 0 (delta 0), pack-reused 15701
Receiving objects: 100% (15705/15705), 6.52 MiB | 147.00 KiB/s, done.
Resolving deltas: 100% (8874/8874), done.
Checking connectivity... done.

[rvdzwet@rvdzwet-asus tmp]$ cd requests/

[rvdzwet@rvdzwet-asus requests]$ ls
AUTHORS.rst  docs  ext  HISTORY.rst  LICENSE  Makefile  MANIFEST.in  NOTICE  README.rst  requests  requirements.txt  setup.cfg  setup.py  test_requests.py
[rvdzwet@rvdzwet-asus requests]$ ./test_requests.py 
................FF.................................................................................................................................
======================================================================
FAIL: test_auth_is_retained_for_redirect_on_host (__main__.RequestsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_requests.py", line 1002, in test_auth_is_retained_for_redirect_on_host
    assert h1 == h2
AssertionError

======================================================================
FAIL: test_auth_is_stripped_on_redirect_off_host (__main__.RequestsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_requests.py", line 995, in test_auth_is_stripped_on_redirect_off_host
    assert not r.request.headers.get('Authorization', '')
AssertionError

----------------------------------------------------------------------
Ran 147 tests in 33.126s

FAILED (failures=2)
```

Some details of the host:

```
[rvdzwet@rvdzwet-asus requests]$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: Fedora
Description:    Fedora release 21 (Twenty One)
Release:    21
Codename:   TwentyOne

[rvdzwet@rvdzwet-asus requests]$ python -V
Python 2.7.8

[rvdzwet@rvdzwet-asus requests]$ python3 -V
Python 3.4.1
```


Found the culprit (after looking looking at the streams with wireshark). It was caused by a erroneous `~/.netrc` entry.

```
[rvdzwet@rvdzwet-asus requests]$ cat ~/.netrc 
default login anonymous password user@site
```

Removing this entry fixed my issue. Thanks to you folks for providing the extended test suite, allowing quick debugging!
adapters: Fix bug causing ingored read_timeout for chunked requests

Whenever chunked request was made (using generators) read timeout value was ignored because socket timeout was not set.

This can be a problem when system default timeout is very long because blocks process waiting for response for a period of time that cannot be controlled.

Changes:
- `HTTPAdapter.send()` now sets read timeout on `low_conn.sock` socket when request body is chunked.
- socket.error, socket.timeout are explicitely translated to urllib3 exceptions using `conn._raise_timeout`
  as expected with non-chunked request bodies to be later translated to `requests` exceptions with existing code.
- test added for timeouts on chunked-message body requests

Additional notes:
- due to convention to use httbin in requests tests and no posibility
  to tests POST timeouts with `delay/:n` endpoint (method not allowed)
  test for this bug uses GET with chunked message-body. This is OK
  because RFC-2616 does not explicitely say that message-body is
  forbidden in GET requests. And also it works


Paging @shazow to make sure we don't step on any toes here with urllib3 by accident.


Looks painful but don't think urllib3 would break this.


This went how I expected. It was closed but never fixed. If it won't be fixed this way but is planned to be done in urllib3 can we just reopen it as an issue so it will be left as a reminder that such issue exists?


Also I found another reason why it may be worth fixing. Some servers can send a response before whole request is transmitted. I know HTTP was designed as a synchronous protocol but it looks like a lot of software implements it asynchronously.

One of notable examples is Jetty. In case of issues like unhandled read timeouts on its side it sends response before even consuming a request. I noticed that various services using this server may behave in unexpected ways if they send error response but client is still writing its response (this is another issue by the way). Currently I experience an issue where requests lock indefinitely on `sendto` sys call and never quit due to lack of timeout when talking to Jetty server using chunked transfer encoding.

I think that instead of saying that there is better way to do it we could merge this as a working fix to a critical issue (never-ending blocking calls) even if not elegant and only then think about rectifying this in urllib3.


@swistakm We have merged chunked upload support into urllib3. We have not updated to the version of urllib3 that contains that fix, but when we do we'll also remove our own support for timeouts.

However, requests should _never_ lock on sending a socket: that suggests the connection is still up but Jetty is not consuming any data from the socket, leading to the TCP stream being forced open.
pooled connections are not returned to the pool

Hi

When pool_block=True on an HTTPAdapter is used and the connection fails 
this one seems not return to the pool.

To reproduce:

```
# python
>>> import requests
>>> a = requests.adapters.HTTPAdapter(pool_maxsize=1, pool_block=True)
>>> s = requests.Session()
>>> s.mount("http://", a)
>>> s.get("http://localhost:12345")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 413, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(111, 'Connection refused'))
>>> s.get("http://localhost:12345")

# here the session objects is stuck.
```

Cheers,


After deeper investigation it seems to be that one: https://github.com/shazow/urllib3/issues/659


Duplicate of kennethreitz/requests#2687
Does't escaped ASCII characters when post json data by default


@tao12345666333 could you explain this? Why is this necessary? Did you encounter a bug? We don't just accept code thrown over the wall without an explanation.


@sigmavirus24 Thanks for your reply.

When I use requests to post json data, the json data will be escaped by default. 
My code is here:

```
#!/usr/bin/env python
# coding=utf-8
import requests


def push_message():
    body = {
        'touser': '@all',
        'msgtype': 'text',
        'agentid': 0,
        'text': {
            'content': '测试'
        },
        'safe': 0
    }

    res = requests.post(
        url='http://mytest_url',
        json=body,
        headers={
            'Content-Type': 'application/json',
            'encoding': 'utf-8'
        }
    )

    print res.content

if __name__ == '__main__':
    push_message()
```

The api server can't parse data correctly, `if include \\uxxxx will create fail!` (The api server is a third party)

Because of requests's default json dumps, all Chinese character will be escaped `\\uxxx`, so I want to change default json.dumps() by it's `ensure_ascii` options.


Hi， @sigmavirus24  I have modified the code before, does not change the original behavior, but to provide a keyword argument. Using this because escape does not an action of `prepare_body()` function.
Therefore, don't use explicit arguments.


@tao12345666333 in this case, you should simply encode the json data yourself and pass it through the `data` parameter. Thanks for sharing, though! I see where you're coming from. 


@kennethreitz Thanks! I will try again. :-)
requests.get(verify='path/to/self-signed-certificate') cannot verify SAN

Verifying a certificate which have Subject Alternative Names with IP addresses (or DNS names), Common Name should not be checked and those addresses should be checked instead.
[RFC6125 section 6.4.4 describes this.](http://tools.ietf.org/html/rfc6125#section-6.4.4)


@kissgyorgy Thanks for this!

We already do this for DNS names, as you can see in [this code](https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/packages/ssl_match_hostname/_implementation.py#L67-L105). The reason we don't do it for IP addresses in SANs is because we currently don't support authenticating IP addresses in SANs.

This is a bit of a flaw in urllib3, and we have an open issue for it in shazow/urllib3#258. Fixing that issue would automatically fix this problem, so I recommend focusing on that.
Withholding a post body in AuthBase until a security context has been established

**Background**
I'm working with a server which uses SPNEGO/GSS authentication, to keep it simple lets say the GSS mechanism is NTLM. As usual, the client and server exchange tokens to establish the context. What's different here is that once the context is established I want wrap (encrypt) the request body using the negotiated GSS context (in this case it would use NTLMv2 Session Security) and send it.

The authentication and encryption is not a problem. I'm using AuthBase and a response hook to exchange the tokens in the authenticate header. 

**Problem**
My AuthBase derived class is sending the plaintext request body every time during the token exchange. I only want to send it in its encrypted form on the final exchange.

Looking at the PizzaAuth example on the Requests page:

``` python
from requests.auth import AuthBase

class PizzaAuth(AuthBase):
    """Attaches HTTP Pizza Authentication to the given Request object."""
    def __init__(self, username):
        # setup any auth-related data here
        self.username = username

    def __call__(self, r):
        # !! = !! = !! = !! = !! = !! = !! = !! = !! = !! = !! = !! = 
        # I want to manipulate the request body here and ensure the request
        # that is returned is empty, or is otherwise encrypted
        # !! = !! = !! = !! = !! = !! = !! = !! = !! = !! = !! = !! = 
        r.headers['X-Pizza'] = self.username
        return r
```

If I change the request body to an empty string
`r.body = ""`

I find that the remote server does not respond. This is because I have changed the body, but the Content-Length header is not re-calculated, and so my empty request is sent with a non-zero content-length - leaving the server waiting for more bytes. Now, I can also set the Body and the Content-Length header to zero - and that seems to work.

**Question**
Is is safe to manipulate the request body in AuthBase, will this cause a memory leak or any quality problems? Is it safe to simply set the body to an empty string and set the Content-Length to 0 until my final response, and then simply set the body and set the right content length?


Yes, it is safe to set the request body. This will not cause memory leaks because we do not have any reference deletion anywhere in requests' codebase. Leaking the reference to the body will be absolutely fine. Yes, it should be safe to simply set the body to an empty string and set the content-length appropriately, then set them back on the final response.

Note that this is one possible approach. Another approach would be to issue new requests in `__call__`.

In future, these questions are better asked on StackOverflow: the users there have much higher bandwidth to provide quality answers to questions like this one.


Thanks Lukasa!
SSLError when use Pyinstaller

I try to mix my python desktop app to exe file. When execute without compiler is OK but EXE it show error as below

```
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\try\build\main\out00-PYZ.pyz\Tkinter", line 1536, in __call__
  File "<string>", line 154, in on_btn_download
  File "C:\try\build\main\out00-PYZ.pyz\usgsDownloader", line 57, in search
  File "C:\try\build\main\out00-PYZ.pyz\usgs.api", line 302, in search
  File "C:\try\build\main\out00-PYZ.pyz\requests.api", line 109, in post
  File "C:\try\build\main\out00-PYZ.pyz\requests.api", line 50, in request
  File "C:\try\build\main\out00-PYZ.pyz\requests.sessions", line 465, in request
  File "C:\try\build\main\out00-PYZ.pyz\requests.sessions", line 573, in send
  File "C:\try\build\main\out00-PYZ.pyz\requests.adapters", line 431, in send
SSLError: [Errno 2] No such file or directory
```

Compile command

```
C:\try>pyinstaller -F --hidden-import=requests main.py
```

What should I fix it? thank


This is not a bug in requests. This is a bug in PyInstaller since it doesn't know how to detect static package data (e.g., the certificate bundle that requests is packaged with).


Thanks sigmavirus24 for giving a hint.

Fix this by http://stackoverflow.com/questions/21201238/twilio-python-module-errors-after-compiling/21206079#21206079


I too have the trouble with Pyinstaller and Requests. Thanks folks for posting the question and for pointing us in the right direction, much appreciated!
Move CA Certificates documentation next to SSL Cert Verification

When reading the SSL Cert Verification documentation and working on directing requests to use a system's local CA store I never noticed the CA Certificates documentation as it is at the tail end of the Advanced page.  I suggest it would be better to have the CA Certificates documentation immediately follow the documentation regarding SSL Cert Verification.

No content change, just a relocation of existing verbiage.


Thanks @jimbrowne! :cake: :sparkles: :cake:
Use context manager for Session

Since a context manager is defined for session, i thought it would be better (and more pythonic) to use it. All the tests passed. Let me know what you guys think


I'm afraid we actually don't want this.

The use-case is extremely uncommon, but it's possible that if an exception is hit during request processing that a user may want to keep hold of state from the connection pool or urllib3 layer. Using the context manager potentially prevents that by forcing the Session closed.

Basically, remember that a context manager is not the same as this:

``` python
obj = something()
obj.action()
obj.close()
```

It's the same as this:

``` python
obj = something()

try:
    obj.action()
finally:
    obj.close()
```
Replace hardcoded class with dunder class attribute

Small suggestion as this was something I heard in Raymond Hettinger's talk at PyCon 2015


Btw if this is accepted, i can do it for the other modules/python files...in that way i can also read some of the requests code :stuck_out_tongue: 


I certainly want the change in `super` removed: as pointed out in the diff comment by @jmoldow, that isn't valid. Otherwise, I suspect it makes the class somewhat more subclass-friendly, which I don't object to. 


i am not sure why this is considered invalid. can you elaborate?


There is a reason that `super()` requires two arguments. It's because the first argument can't be deduced from `self`, because the first argument isn't (in general) `self.__class__`.

If you have a method-resolution order C -> B -> A -> object, `C.__init__` is going to call `super(C, self).__init__`, which will call `B.__init__`, which will call `super(B, self).__init__`, which will call `A.__init__`. The only call where the first argument is `self.__class__` is the first one, but you don't know in advance what will even be the first one.

The first argument needs to be the hard-coded class name, in order for super to figure out what the super object should refer to.


Actually `__class__` and `self.__class__` are a whole different level of magic that will thoroughly break this for any subclass. Sorry @ueg1990 this is not something I will allow in requests.


lol i do not have any objection to that. learnt something new in python :)
onto the next pull request
Redundant creation of list?

Hello, I came across this code under status_codes.py:

for (code, titles) in list(_codes.items()):
    for title in titles:
        setattr(codes, title, code)
        if not title.startswith('\'):
            setattr(codes, title.upper(), code)

in the first line (the outermost loop), i think the list(_codes.items()) step is redundant. it should just be iterating over _codes.items(). what i am not sure is if the items method here returns a list (python 2) or an iterator/generator (python 3). if its the former, maybe its better to loop over _codes.iteritems(). In either case, you do not really need to convert it to a list.

Though this is a very small issue, if its worth doing a PR, I can do it. (in that way, i can contribute my first PR to the requests library :) )


Thanks @ueg1990! I think you're right: feel free to provide a pull request. =)


## To be clear: please do not use iteritems. Just remove the use of list()

Sent from my Android device with K-9 Mail. Please excuse my brevity.


I will send the pull request today. But i am curious, why should we not use iteritems?


@ueg1990 It doesn't exist in Python 3. =) Given that this is a one-time operation, we don't care too much about the one-off memory allocation for the temporary list on Python 2.


oh yeah make sense.
minor refactoring to function

I was just looking through the code base and noticed this small "improvement" I could make. Is the focus more on readability or is this kind of refactoring acceptable as well? I understand that this format does decrease readability a little bit, so I didn't look into it too much, thought I'd get feedback first.   


Dictionary comprehensions were introduced in Python 2.7. Prior to that they are invalid syntax, e.g., for Python 2.5 and 2.6, both of which are supported versions for ~~pep8~~ requests. 

Thanks for contributing though @suhussai!

Perhaps you could look through the bugs and find one that doesn't have a linked PR to try to work on.

Cheers,
Ian!

**Edit** I said pep8 but I meant requests. I clearly can't keep my projects straight anymore. :(


Your right, I noticed that in the tests. Even though dictionary comprehension isn't supported directly in python2.6, this format works according to http://stackoverflow.com/questions/1747817/python-create-a-dictionary-with-list-comprehension:

```
dict((cookie.name, cookie.value) for cookie in cj)
```

I installed python2.6/2.5 and ran an example program using that format, and it worked as expected. Do we want to include this other format, which is supported by at least python 2.5 and greater?


So `dict()` is actually bizarrely slow. That said, being a style issue I don't think we're concerned enough to change it.


Ah, I didn't know that. Thanks for the information.
Fixed issue #2756

Now 'json' parameter will be used to prepare body only if the 'data'
parameter is not present


One minor comment, otherwise I'm fine. =)


With `if data is None` I get the following test failure. Please advice.

```
|======================================================================
ERROR: test_json_param_post_content_type_works (__main__.RequestsTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "test_requests.py", line 1062, in 
test_json_param_post_content_type_works
     assert 'application/json' in r.request.headers['Content-Type']
   File "/home/ravi/oss/requests-temp/requests/structures.py", line 54, 
in __getitem__
     return self._store[key.lower()][1]
KeyError: 'content-type'

----------------------------------------------------------------------
Ran 148 tests in 85.057s

FAILED (errors=1)
```

On Tuesday 08 September 2015 08:24 PM, Cory Benfield wrote:

> One minor comment, otherwise I'm fine. =)
> 
> —
> Reply to this email directly or view it on GitHub 
> https://github.com/kennethreitz/requests/pull/2763#issuecomment-138591292.


Ugh, you're right, our code sets data to the empty dict. Nevermind, I rescind that feedback. This LGTM, @sigmavirus24?


:+1:


The documentation also needs to be updated.
The `files` also supersedes `json` as already pointed out by @taion in comments of #2756 


@sumitbinnani I'd happily accept a pull request that adds only that documentation change.


This looks okay to me. How about you @Lukasa? We should probably not have this pending after having merged the documentation for next week.


I'm happy for us to merge this in for 2.8.0.