Raibek/requests-github-issues huggingface dataset:
URL with port causing socket.gaierror

The following is not working:

```
import requests
requests.post('http://localhost:8080/', data={"hello": "world"})
```

This works:

``````
import requests
requests.post('http://localhost/', data={"hello": "world"})```
``````

This works too:

``````
$ export HTTP_PROXY="http://localhost:8080"
$ python 
import requests
requests.post('http://localhost/', data={"hello": "world"})```
``````

Requests 2.1.0
Python 2.6.1
Mac OS X 10.6.8

httpie debug:

```
requests.request({'allow_redirects': False,
 'auth': None,
 'data': {},
 'files': {},
 'headers': CaseInsensitiveDict({'User-Agent': 'HTTPie/0.8.0-dev'}),
 'method': 'post',
 'params': {},
 'proxies': {},
 'stream': True,
 'timeout': 30,
 'url': 'http://localhost:8080/xxx/rest/product/',
 'verify': True})

Traceback (most recent call last):
  File "/Users/andi/.virtualenvs/xxxx/bin/http", line 8, in <module>
    load_entry_point('httpie==0.8.0-dev', 'console_scripts', 'http')()
  File "/Users/andi/.virtualenvs/xxxx/lib/python2.6/site-packages/httpie/core.py", line 95, in main
    response = get_response(args, config_dir=env.config.directory)
  File "/Users/andi/.virtualenvs/xxxx/lib/python2.6/site-packages/httpie/client.py", line 27, in get_response
    response = requests.request(**requests_kwargs)
  File "/Users/andi/.virtualenvs/xxxx/lib/python2.6/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/andi/.virtualenvs/xxxx/lib/python2.6/site-packages/requests/sessions.py", line 355, in request
    env_proxies = get_environ_proxies(url) or {}
  File "/Users/andi/.virtualenvs/xxxx/lib/python2.6/site-packages/requests/utils.py", line 490, in get_environ_proxies
    if proxy_bypass(netloc):
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 1567, in proxy_bypass
    return proxy_bypass_macosx_sysconf(host)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/urllib.py", line 1433, in proxy_bypass_macosx_sysconf
    hostIP = socket.gethostbyname(host)
socket.gaierror: [Errno 8] nodename nor servname provided, or not known
```


Hmm, that stack trace is interesting. I don't have a copy of Python 2.6.1 sitting around, but in 2.7 that code is subtly different, and explicitly splits the host and port. I wonder if early versions of 2.6 had a bug in their `proxy_bypass` code.


2.6.1 _is_ ancient. I think the latest release is 2.6.9 or so and that's the last security release it'll see.

The reality though is that this seems to be in the standard lib and I'm not sure there's very much we can do about it.


Sorry, did not see the forrest from the trees. Of course i should have been using 2.7, I forgot to pass that option to mkvirtualenv. Damn open space office ;) Sorry for the noise and thanks for pointing it out.



  File "/usr/lib/python3.5/socket.py", line 732, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):

gaierror: [Errno -2] Name or service not known
Session Uses Wrong Method When Making Requests

When making a request with a session, such as:

```
response = session.post("some_url", form_data)
```

the value of `response.request.method` can sometimes be a `GET` with a `response.request.body` that's empty.

This happens with surprisingly high frequency (appox. 1 in 10 requests).


You are not checking `response.history` which will undoubtedly show that the original response was a redirect. On redirect, we must change the verb and clear the body per the RFC. I'll wait for you to confirm that you see redirects in your history before closing this.


I am getting a 301 in `response.history`. Thanks.
failed to fetch URL which can be accessed by browser

I failed to fetch a web page from the following URL, but this URL is fine in my browser.
http://hkblog.xanga.com/2009/08/17/%E3%80%8A%E6%84%9B%E6%83%85%EF%BC%8C%E6%B2%92%E9%82%A3%E9%BA%BC%E7%BE%8E%E5%A5%BD%E3%80%8B/

The error messages are as follows:

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    cur_r = requests.get('http://hkblog.xanga.com/2009/08/17/%E3%80%8A%E6%84%9B%E6%83%85%EF%BC%8C%E6%B2%92%E9%82%A3%E9%BA%BC%E7%BE%8E%E5%A5%BD%E3%80%8B/')
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, *_send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 372, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='hkblog.xanga.com', port=80): Max retries exceeded with url: /2009/08/17/%E3%80%8A%E6%84%9B%E6%83%85%EF%BC%8C%E6%B2%92%E9%82%A3%E9%BA%BC%E7%BE%8E%E5%A5%BD%E3%80%8B/ (Caused by <class 'httplib.BadStatusLine'>: '')


I think the host 'http://hkblog.xanga.com' does not accept 'User-Agent' containing 'CPython'.
When i send same request with `header={'User-Agent' : 'python-requests/2.1.0 CPython/2.7.5 Darwin/13.0.0'}` (exactly same user-agent with `requests`), that error also occurs in urllib2.
But when i remove 'CPython/2.7.5' in user-agent, it works well both requests and urllib2.
So, it's not a bug of `requests`.


Looks like @daftshady nailed this one down
Bump the year ftw

s/2013/2014


May as well update LICENSE. =D


Hurrah! =D :+1:


Done. And to quote my commit message:

> No one checks docs or a license

:wink: 


Ooh, ooh, I do!


You would


> Bump the year ftw

The usual practice is to update the list of years each time the file is modified (but never remove a year). 
http://stackoverflow.com/questions/3487007/when-to-update-the-year-in-open-source-copyright-notice


@zedxxx I'm following the convention of the project. requests has been around (much) longer than a year, yes, but for as long as I've been around it has never had the year range in the license.


> has never had the year range in the license

Perhaps, it is time to correct this error.
Set trust_env as an option in request method

Hidden set trust_env in class Session aimed to trust the http_proxy env, can we have this as an option in Session.request or request api, for only in request method, this option are read to decide to trust http_proxy env or not.


There's nothing hidden about `trust_env`, it's [documented](http://docs.python-requests.org/en/latest/api/#requests.Session.trust_env). We're not going to change where `trust_env` lives, we're happy with the API as it stands.

Thanks for asking though!


My mistake about unclear expression.
Sometimes we'd like to just use requests.get() as simple, not to declare Session() then get().
Noticed that 'stream', 'verify', 'cert' all can be set either by requests.get() or change it in Session(), why not just let it append to the parameter list. http://docs.python-requests.org/en/latest/api/#requests.Session.request

Thanks for reply.


@V-E-O the parameters passed to `get`, `post`, `put`, and other VERB methods are parameters that pertain to the request itself, not to the environment and not to the session. `trust_env` pertains to the environment and how the session considers the environment when performing requests. In short, no we will not add `trust_env` to the parameter list of any of those methods.


@sigmavirus24 I understand your point, also fine with disabling proxy by setting `trust_env`. Just feeling odd first time I tried to use requests.get(..., proxies=None/{}) to bypass env proxies.
Anyway, it's OK with us, thank you all for reply!
Provide classifiers for PyPI that specify which Python versions are supported


We do: https://github.com/kennethreitz/requests/blob/master/setup.py#L56..L60


Then why do these don't appear on the PyPI page?

https://pypi.python.org/pypi/requests

On Mon, Dec 30, 2013 at 7:32 PM, Ian Cordasco notifications@github.comwrote:

> We do:
> https://github.com/kennethreitz/requests/blob/master/setup.py#L56..L60
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1834#issuecomment-31357160
> .


PyPI is a strange beast that is luckily being slowly replaced by [pypa/warehouse](https://github.com/pypa/warehouse). In the past I had issues where I created a package name manually and then metadata from `setup.py` didn't properly appear. I suspect this might be something similar. Regardless only @kennethreitz can change this and it isn't an actual bug in requests.
Add more interlinks between the documentation

Also fixes up tense in a few cases, and adds the `intersphinx` extension so we
can link to the urllib3 documentation when it is called out.

I should probably write documentation for how to do this somewhere as well...


This in principle looks fine, though I don't know about about intersphinx to be sure. =) Let's see what @sigmavirus24 thinks.


FWIW I wrote up docs on how to do this here:

http://kev.inburke.com/kevin/sphinx-interlinks/


Nice dude! Hook up all the intersphinx you want!

One thing to note — it should stay out of the Quickstart. Let's keep it to the Advanced, API Docs, Developer Docs, etc. :)
Requests must be installed as root when using pip to avoid errors


I wouldn't recommend doing this at all. If at all possible always install your packages to virtualenvs and avoid using sudo as much as you can.


@NickGeek This is simply not true. _Sometimes_ Requests must be installed using `sudo`: if you install directly into the site packages and you don't have the correct permissions for them. But if you're using a virtualenv or a copy of Python where you have permission to the site packages, you do not need `sudo`.

For that reason, we assume that people who see errors out of `pip` that relate to lack of permissions will just elevate, running `sudo !!`. Better to leave the `sudo` out of the docs.

Thanks for this though!


Okay, no problem.
r.text exception in Python 3.x

I use `r.context`, it is `bytes` type, then I use `r = requests.get('http://github.com'); print(r.text)`, but an error occurs as below:

```
UnicodeEncodeError: 'ascii' codec can't encode character '\xb7' in position 235: ordinal not in range(128)
```

is it a bug in python 3 ?  I use requests 2.1.0 version in openSUSE 13.1 x86_64.

Thanks


This is not the place to ask questions. Please ask your questions on [StackOverflow](http://stackoverflow.com/questions/tagged/python-requests).


@sigmavirus24 Ok, maybe my title confused you, now it is ok.


I was not confused and it's not a bug in `requests`. Your standard out encoding is `ascii`. StackOverflow is the place to seek help about that.
Incorrect Timezone

Recently, I tried to crawl some data from www.soccerway.com using 'requests' + 'beautifulsoup'.

Crawling went very smoothly until I found out that all of the time has been offset by 7 hours. (see below comparison)
![selection_004](https://f.cloud.github.com/assets/2215059/1807903/6b6b5802-6d22-11e3-92dc-1a920b98a257.png)

So I was thinking maybe 'requests' didn't have the right time zone settings. Because the website said that match time will be converted to visitor's local time.
![selection_005](https://f.cloud.github.com/assets/2215059/1807902/6b67fb6c-6d22-11e3-89ee-d1c034c4d288.png)

Any idea on how I can change the time-zone settings in 'requests' would be much appreciated.


Sorry, I just found out this issue has nothing to do with 'requests'.The same thing happened when I tried to crawl page with 'urllib2'.

To anyone who know how to solve the time-zone difference when crawling a webpage, please help.

Thanks.


The best place to ask that question is probably StackOverflow. I'm not sure what the answer is, but usually websites do this based on source IP address geolocation. Requests shouldn't be doing anything different here, which makes that a bit confusing. My recommendation would be to examine the headers your browser is sending to see if there's any information in there that the site could be using.


Thanks!
Use adapter pool size for proxies

I was doing hundreds of parallel requests, and kept seeing warnings about dropped connections because the connection pool was full.

So I increased the connection pool on the adapter, and increased it some more. And then even more. It did not help.

Then I noticed that only my proxy connections where dropped.
Some investigation revealed that proxy connections use their own pool subclass.
However, this pool is created without respecting the pool size setting.

This PR simply propagates the pool kwargs to the proxy pool.
I contemplated adding additional kwargs for the proxy pool size, but this seems uneccecery complexity.

If there is a use case for using one Session for proxied and direct requests, and the proxies needs a different pool size than the direct connections, I could add that.


Thanks for this!

I like this solution, I think it's the correct approach. We can quite safely overload these kwargs, and I think doing so provides the nicer interface.

Once again, it's annoying that we can't easily test this, but I'm happy with this solution as-is. @sigmavirus24?


Yea, I looked at writing a test case, but instead just hamered it with proxy requests and looked for warning -- there where none.


If you want to write a test (that would just the slightest bit brittle) we could introduce mock as a testing dependency. We could then do something like:

``` python
def test_proxy_receives_pool_settings(self):
    with mock.patch('requests.packages.urllib3.poolmanager.proxy_from_url') as proxy_from_url_mock:
        http = HTTPAdapter()
        proxy_from_url_mock.assert_called_once_with(
            # ...
        )
```

It's quite simple and it tests the intended behaviour. I'm just a bit wary of introducing mock without other tests relying more strongly on it. And I still need to re-write the tests so that they're fast, offline and thorough enough.


> If you want to write a test

Not terribly excited. But if it's the right thing to do, I'll do it. I'd really like this merged.


> Not terribly excited

Yeah testing is boring. I guess I'll cover this when I pull an all-nighter on the tests in the near future.

:+1:


Meh proxies suck.
SSLError exception leak

My program caught an ssl.SSLError which is not RequestException and I think it may be an uncaught exception? Requests version is 2.0.0.

Traceback:

```
......
  r = requests.post(self._token_url, data=data, timeout=self._timeout)
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/api.py", line 88, in post
  return request('post', url, data=data, **kwargs)
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/api.py", line 44, in request
  return session.request(method=method, url=url, **kwargs)
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/sessions.py", line 357, in request
  resp = self.send(prep, **send_kwargs)
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
  r = adapter.send(request, **kwargs)
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/adapters.py", line 367, in send
  r.content
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/models.py", line 633, in content
  self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/models.py", line 572, in generate
  decode_content=True):
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 225, in stream
  data = self.read(amt=amt, decode_content=decode_content)
File "/data/poi-op/env/python/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 174, in read
  data = self._fp.read(amt)
File "/usr/local/python27/lib/python2.7/httplib.py", line 543, in read
  return self._read_chunked(amt)
File "/usr/local/python27/lib/python2.7/httplib.py", line 585, in _read_chunked
  line = self.fp.readline(_MAXLINE + 1)
File "/usr/local/python27/lib/python2.7/socket.py", line 476, in readline
  data = self._sock.recv(self._rbufsize)
File "/usr/local/python27/lib/python2.7/ssl.py", line 241, in recv
  return self.read(buflen)
File "/usr/local/python27/lib/python2.7/ssl.py", line 160, in read
  return self._sslobj.read(len)
SSLError: The read operation timed out
```


Thanks for raising this issue!

This was originally spotted in #1787, and should be fixed in `master`. Try using the GitHub version of Requests to see if you can still hit it.


Closed due to inactivity.
Session hook is ignored

Hello.

My English is not so good, sorry.

Problem example:

```
def fail_on_error(r, *args, **kwargs):
    r.raise_for_status()

s = requests.session()
s.hooks.update(dict(response=fail_on_error))
r = s.get('http://httpbin.org/status/404')
```

This code doesn't raise an Exception. Is wasn't too obvious for me. Is it ok?

It is caused by `merge_setting` if this a problem, which overrides `Session` hooks with default `Request` hooks.

Thanks.


Thanks for raising this issue!

I can't reproduce this on my machine. Can you tell me what version of Requests you're using? You can find this out by getting `requests.__version__` in a Python shell.


This was fixed in version 2.1.0 of requests. 

I, like @Lukasa, see this work properly when tried interactively:

``` pycon
>>> import requests
>>> def fail_on_error(r, *args, **kwargs):
...     r.raise_for_status()
... 
>>> s = requests.session()
>>> s.hooks.update(dict(response=fail_on_error))
>>> r = s.get('http://httpbin.org/status/404')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/sessions.py", line 395, in get
    return self.request('GET', url, **kwargs)
  File "requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "requests/sessions.py", line 491, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "requests/hooks.py", line 41, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "<stdin>", line 2, in fail_on_error
  File "requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: NOT FOUND
```


Excellent, OK. This can be fixed by updating your version of Requests.


Thx.Sorry for outdated bug.
My version was 2.0.1.
SSL verify failed though cert appears to be OK

I'm trying to call a simple URL from a Debian stable server and I'm getting a certificate error:

> > > import requests 
> > > r = requests.get('https://mobile.amber.io')
> > > [..]
> > > requests.exceptions.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

The certificate should be valid. I've tried all SSL online checks, it works on all platforms. 

On my mac though it might be unrelated I'm getting SSL hosts don't match (there are a couple of hosts on that server with different certs). 


My psychic powers tell me that you're using Python 2.7, which doesn't support SNI out of the box. See [this StackOverflow answer](http://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484) for instructions on how to add SNI support.

Am I right?


That fixed it for me (installing urllib3 and those three packages). Thanks!
How to disable (not send it at all) a request header?

I'm using requests writing a program and it added the following header to my custom header:

```
Accept-Encoding: gzip, deflate, compress
```

My custom header is:

```
headers = {
    "User-Agent": "Shockwave Flash",
    "Accept": "text/*",
    "Connection": "Keep-Alive",
    "Pragma": "no-cache",
    "Host": "upload.youku.com"
}
```

Is it possible to disable the added header at all?


Yup. Like this:

``` python
r = requests.get(url, headers={'Accept-Encoding': None})
```


@ogrishman, @Lukasa explained on #1820 that this is a bug tracker and not a question and answer forum. Further questions you ask here will be closed without answer.
Removal of TLSv1 from the webserver gives socket.error: [Errno 54] Connection reset by peer error to python requests

I discovered that if I force nginx to use only TLSv1.2 and TLSv1.1 protocols, it will prevent the requests library from working. 

Other browsers are able to deal with this, including CURL but request doesn't, at least when is used on OS X with Python 2.7 

Reproduce, reconfigure webserver to use only recent SSL protocols: 

ssl_protocols TLSv1.2 TLSv1.1;

```
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /some/url (Caused by <class 'socket.error'>: [Errno 54] Connection reset by peer)
```


This sounds like a bug in urllib3. @shazow opinions?


Actually following the conversation on #171 this is not a bug in requests but in Python as @Lukasa explained.


My favourite.
Losing cookies in redirects after HTTPS POST

Hi,

in a session, I am sending HTTPS GET request, followed by a HTTPS POST request which results in a series of 302-redirects. I monitor traffic via mitmproxy, but the results are the same whether I use the proxy or not.

With allow_redirect=True, most of the cookies get lost between the first and second redirect (4 out of 6 got lost in my case).

With allow_redirect=False for the POST, I initiate the GET with allow_redirect=True and everything works just fine.

Here's the boiled-down code:

```
self.session = requests.Session()
self.session.get(get_url, headers=self.headers, proxies=self.proxies, verify=False)
self.session.post(post_url, data=payload, headers=self.headers, proxies=self.proxies, verify=False)
```

I cannot give you all details about the URL or the parameters since it is a login procedure, but I can certainly answer questions or experiment if you tell me what to do.

Edit: This happens in 2.1.0. But it works fine in 2.0.1.

Regards,

Roland


Thanks for raising this issue! In future, however, please take a quick look through both the open and the closed issues before opening yours. This issue has been reported twice already (in #1805 and #1809) and is fixed in the current master branch by #1808. You can either wait until the next release for the fix, or you can use the current GitHub code if you need the fix immediately.


Thanks for the quick answer. I did not make the connection, when I read the issue titles.
urllib3.exceptions.DecodeError being raised

I have been getting a `requests.packages.urllib3.exceptions.DecodeError` exception raised. From my understanding of the documentation, shouldn't this be translated to a Requests exception which inherits from `requests.exceptions.RequestException`?

This occurred using Requests 2.1.0 when making a GET request through a misconfigured HTTP proxy.

Here is the stack trace:

```
Traceback (most recent call last):
  File "proxy-manager.py", line 94, in <module>
    good, comment = is_good(proxy, base_headers, base_latency * 10)
  File "proxy-manager.py", line 42, in is_good
    headers = get_header_data(proxy)
  File "proxy-manager.py", line 14, in get_header_data
    r = requests.get('http://proxyjudge.us/azenv.php', proxies=proxies, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 485, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 388, in send
    r.content
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 676, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 615, in generate
    decode_content=True):
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/response.py", line 236, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/response.py", line 204, in read
    e)
requests.packages.urllib3.exceptions.DecodeError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))
```


Thanks for raising this! Yeah, it probably should.


Fixed by #1816.
make requests compatible with SAE

make requests compatible with SAE

Sina App Engine conn has a None valued sock


Hey @tg123 this PR belongs on [shazow/urllib3](/shazow/urllib3) as was noted in the [README in the packages directory](https://github.com/tg123/requests/tree/fa34ab816aa03a009ceb86db7e55f16ca6687ba4/requests/packages). Please be more careful next time.


sorry about that


No worries! Thanks for sending the PR though and fixing up urllib3! :cake:


in SAE, try:

```
import os
os.environ['disable_fetchurl'] = True
```

http://sae.sina.com.cn/?m=devcenter&catId=291#anchor_24be84a1512409fbeb5363421a1a19fe
Stop bundling urllib3, just add it to install_requires


We have discussed this issue with users several times in the past. A search of the issues that are closed would show that. Our opinion has not changed, nor has our reasoning. By vendoring urllib3 we have a very specific version that we have tested against that may include unreleased bug fixes and does not put pressure on @shazow. It also allows us to checkout a specific version of the repository and just work on it with what existed at the time. It's reliable and it will be how we work for the foreseeable future. I'll leave this open until @Lukasa can make his way around to it, but I'm strongly in favor of closing this.


Fixed by https://github.com/kennethreitz/requests/pull/1812


@sigmavirus24 The big problem is that distros remove the bundled version anyways, making conflicts when you want to handle exceptions / use urllib3 directly, since requests is using a different version than everything else.


For example, if you look at the patch Fedora keeps around for requests:

http://pkgs.fedoraproject.org/cgit/python-requests.git/commit/?id=2f898f274c560a0fb5ac48719a9529f68688fb7a

I think doing a requires>=<ver1>  <=<ver2>  would also be acceptable practice, allowing you to pin a range of versions you know are working with requests.  


Since #1812 has been closed, I feel it's fair to close this and end the discussion. We've discussed this publicly before and we have discussed it with many of the distro package maintainers who package requests for you and everyone else. Our stance has also always been to use PyPI and not the distro packages because they remove the vendored dependencies.
Fixes #1805

Sure cookies are persisted to the session, but those new cookies are not added
to the next prepared request. We need to update that new request's CookieJar
with the new cookies.

/cc @teleyinex
This allows me to fetch the URL you specified quite quickly.

TODO:
- [ ] Find a way to reliably test this. Perhaps introduce [Betamax](https://github.com/sigmavirus24/betamax) to make this test reproducible.


Looks reasonable to me. Testing is going to be a pain though.


Looks good too! Thanks for fixing it so quickly @sigmavirus24
Need a way to disable SSL verification for all requests calls

If you are using a 3rd party library or just have lots of requests calls inside your code it's almost impossible to to use `verify=False`.

We need a way to disable this for all requests calls, probbly an environment variable.


We used to have a way to globally configure the library like this in v0.x. We intentionally and viciously removed it in v1.x. We've not added it back and we will not add anything similar back, especially not to disable SSL verification and especially not in an environment variable.


I'm waiting for @Lukasa to weigh in before closing this.


Agreed. At this time we have no plans to make it possible to disable SSL verification on a global scope. SSL verification is and should be on by default.
Getting a public Google Spreadsheet doc gives you the too many redirects error

In version 2.0.0 when you try to get a document like this:

``` python
import requests
r = requests.get('https://docs.google.com/spreadsheet/ccc?key=0AsNlt0WgPAHwdEczcWduOXRUb1JUc1VGMmJtc2xXaXc&usp=sharing&output=csv')

print r.text
```

You get the CSV content, however with 2.1.0 you get the TooManyRedirects error (it tries 30 times), so it is not possible to download the document. Any idea where could it broken?


Ooh, this is weird: I don't even see that request get off my system.


Which is clearly a problem with my wireshark, since it definitely does. We seem to be spinning in a redirect loop, which it's hard to imagine is the fault of a Requests version.


But it is, because all is well in 2.0.1.


Then, why it is working with 2.0.0 on my machine after creating a new virtualenv just for testing this? It works also with version 1.2, so I don't know what could be wrong. I tried this code from [stackoverflow](http://stackoverflow.com/a/16948852/1960596):

``` python
>>> session = requests.session()
>>> r = session.get(url, allow_redirects=False)
>>> r.headers.get('location')
>>> for redirect in session.resolve_redirects(r, r.request):
...     print redirect.headers.get('location')
```

And you can see, how requests tries all the time to reload almost the same page. Could be something related that Google sets in the first call a cookie, and that cookie is not sent in the redirected request?


Urgh, this is a cookies problem. Observe:

``` python
>>> s = requests.Session()
>>> r = s.get('https://docs.google.com/spreadsheet/ccc?key=0AsNlt0WgPAHwdEczcWduOXRUb1JUc1VGMmJtc2xXaXc&usp=sharing&output=csv')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests\sessions.py", line 394, in get
    return self.request('GET', url, **kwargs)
  File "requests\sessions.py", line 382, in request
    resp = self.send(prep, **send_kwargs)
  File "requests\sessions.py", line 505, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "requests\sessions.py", line 99, in resolve_redirects
    raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects)
requests.exceptions.TooManyRedirects: Exceeded 30 redirects.
>>> r = s.get('https://docs.google.com/spreadsheet/ccc?key=0AsNlt0WgPAHwdEczcOXRUb1JUc1VGMmJtc2xXaXc&usp=sharing&output=csv')
>>> r.status_code
200
```

@gazpachoking, @sigmavirus24, I choose you!


Wow, that was fast ;-) Thanks for finding the bug!


It would seem like cookies extracted during redirects aren't being stored on the session (I haven't dug into the code yet).


Possible solution in #1808.
Move off of charade to erikrose/chardet

@dan-blanchard has taken over maintenance of [chardet](erikrose/chardet). We've merged charade into chardet so that chardet will now work on python2 and python3 without needing separate versions. I'd like to get rid of charade since it's been a responsibility that I had neglected and has been more trouble than it has been worth.

We should start using chardet proper again when they have it ready. (In all candor it should already be ready)


A new release of chardet is available with all the features we ever needed from charade. I'll throw together a PR to close this issue in a few minutes.
Issue a warning when using apparent_encoding.


I'm something in the region of -0 on this. @sigmavirus24?


I'm moreso in the half-open interval [-1, -0) on this if only because the warning will likely only be displayed once per detected encoding and not once per response if I remember the way they work correctly.

An example is like so:

``` python
def foo(word):
    warnings.warn(word)

foo('biz')
foo('biz')
foo('biz')
foo('bar')
foo('bar')
foo('baz')
foo('baz')
foo('baz')
```

Will only produce 3 warnings and will not achieve what you want @martinblech 


You're right @sigmavirus24, that's not what I want. I didn't know warnings worked like that, thanks for teaching me something new!

How about doing `log.warning()` instead of `warnings.warn()`? Here's how it would look on the REPL:

``` python
>>> import logging
>>> logging.basicConfig(level=logging.INFO)
>>> import requests
>>> print requests.get('http://lyrics.wikia.com/api.php', params={'artist': 'Javiera Mena', 'song': 'Al Siguiente Nivel', 'fmt': 'xml'}).text
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): lyrics.wikia.com
WARNING:requests.models:No encoding found in HTTP headers, using apparent "TIS-620".
<?xml version="1.0" encoding="UTF-8"?>
<LyricsResult>
    <artist>Javiera Mena</artist>
    <song>Al Siguiente Nivel</song>
    <lyrics>Esa visiรณn estรก
Yo la veo venir
Se me aparece
No tengo la nocion
Si en mi generacion
Se sentirรก

Si te vas no quiebro fuerza[...]</lyrics>
    <url>http://lyrics.wikia.com/Javiera_Mena:Al_Siguiente_Nivel</url>
    <page_namespace>0</page_namespace>
    <page_id>1626596</page_id>
    <isOnTakedownList>0</isOnTakedownList>
</LyricsResult>
```


First, calling it a warning is far more alarmist than making it an INFO level. Second, requests removed all of its logging statements in v1.0 and has intentionally kept them out since. 


I see. Then both `warnings.warn` and `logger.warning` are out of question.
Add params in custom authentication

Modifying the request params in a custom authentication implementation raises a `AttributeError: 'PreparedRequest' object has no attribute 'params'`.

Example:

``` python
class CustomAuth(requests.auth.AuthBase):
    def __call__(self, r):
        r.params['key'] = 'my_api_key'
        return r
```


Thanks for raising this issue!

This fails because you aren't modifying a `Request` object, you're modifying a `PreparedRequest` object. That only has a `url` property. You'll have to edit that manually. The API documentation for `PreparedRequest` objects is [here](http://docs.python-requests.org/en/latest/api/#requests.PreparedRequest).


A quick fix,

``` python
class CustomAuth(requests.auth.AuthBase):
    def __call__(self, r):
        r.prepare_url(r.url, {'key': 'my_api_key'})
        return r
```

`PreparedRequest` should have a params property imho. 


`PreparedRequest` should not have a params property. The `PreparedRequest` object is intended to be directly serialised into HTTP. That means no assembling or encoding URLs.


Everything @Lukasa said has been correct and as far as I'm concerned this is not a valid issue. Your concerns are duly noted and appreciated. Don't hesitate to open further issues if you think you've found other bugs.


@Lukasa @sigmavirus24 the method `prepare_url` from  `PreparedRequest` [already checks](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L375) if the given url contains previous params. I don't understand why `PreparedRequest` shouldn't have a params property as they are serializable into a HTTP req when the method `prepare` is called.


The `prepare` method is called on the `PreparedRequest` instance long before the auth handler receives it. We're not going to call `prepare` obsessively because we do not have to. That's why there's no `params` attribute.
Socket timeout exception thrown instead of RequestException

When running the following code....

``` python
response = requests.get(result, timeout=10)
```

Where `result` was http://63.224.204.109:8085/browse/category/allbooks, the following exception was thrown after hanging for a few minutes (and apparently disregarding the `timeout` argument?):

```
Traceback (most recent call last):
  File "find-calibre.py", line 34, in <module>
    response = requests.get(result, timeout=10)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 357, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 367, in send
    r.content
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 633, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 572, in generate
    decode_content=True):
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 225, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/usr/lib/python2.7/site-packages/requests/packages/urllib3/response.py", line 174, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python2.7/httplib.py", line 561, in read
    s = self.fp.read(amt)
  File "/usr/lib64/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
socket.timeout: timed out
```

Shouldn't this be caught by Requests, and passed on as a RequestException instead?


Same issue is already opened.
See https://github.com/kennethreitz/requests/issues/1787


As @daftshady points out, please check the issues tracker before opening a new issue.


Thanks for opening this @joepie91 and thank you @daftshady for taking the time to find the already opened issue.
Expose apparent_encoding_confidence.

When there's no encoding in the headers and it has to be autodetected from the
contents using charade, the Response object should expose the confidence value
so the user can decide whether the value of Response.text should be trusted or
not.


Thanks for this @martinblech!

Unfortunately, I'm opposed to this change. Not anything to do with the code itself, but I don't think adding this parameter to the `Response` object adds sufficient value to justify including it. If people _really_ want this information they can get it (by running `charade`/`chardet` themselves), and for a significant majority of people this information is simply unnecessary. Requests got where it was today by hiding complexity from users, and I'm in favour of that trend continuing. @sigmavirus24?


@Lukasa I understand and I love Requests' simplicity. Still, I think it's not safe for Requests to rely so heavily on charade as default and hide it completely from the user.

Here's some examples of what happens to common German, Spanish, French and Portuguese words after being "charadized":

``` python
>>> def charadize(s):
...     encoded = s.encode('utf8')
...     encoding = charade.detect(encoded)['encoding']
...     return encoded.decode(encoding)
...
>>> print charadize(u'Sonatine für Gitarre')
Sonatine fĂźr Gitarre
>>> print charadize(u'Canción de cuna')
Canciรณn de cuna
>>> print charadize(u'Martín Blech')
MartĂ­n Blech
>>> print charadize(u'Chatêau')
ChatĂŞau
>>> print charadize(u'Licença')
Licenรงa
```


That's not really a very fair test of Charade. Charade works by analyzing byte frequency and byte sequences, to work out what an encoding is likely to be. Using a single word puts the frequency of certain byte sequences way off, which will almost always cause charade to make an error. Consider putting a decent quantity of text through it in different encodings, or in the same encoding in different languages, to get a better idea of its effectiveness.


It might have looked like I was, and I apologize for that, but I'm not judging the quality of the Charade library itself. If I was, I wouldn't use my previous example as a benchmark, by no means.

I'm just trying to illustrate that charade (or any other charset detection library, for that matter) is perhaps more likely to fail than initially assumed when integrating it to Requests, and that applying it without any warning at all might not be a safe default after all.

At this moment I am looking at a database with ~100k song lyrics snippets retrieved from the LyricWiki API, and most non-English examples look like they've been affected by this glitch. It's no big deal, we can retrieve them again over the next couple of days, but it certainly would have helped to see a warning in the console that read `requests: no encoding found in HTTP headers, using autodetected <encoding name>`.


Maybe something like https://github.com/kennethreitz/requests/pull/1799 will do?
Fixed pickle support for requests.adapters.HTTPAdapter

Fixed issue https://github.com/kennethreitz/requests/issues/1777 by adding unpickle support to the HTTPAdapter.  

While unpickleing, the class will now set `proxy_manager` to an empty dictionary.


This fix is a subset of the fix in #1793. You and @sigmavirus24 should work out which fix we apply (maybe a merged set of the fixes).


My fix initializes those attributes ahead of time in the event that they ever get added to the `__attrs__` list. It seems the most fool-proof way at the moment and far less complex than first checking if it has an attribute. (I don't quite like `hasattr` or how frequently we use it.)

The only issue with both of our pull requests is that the original issue could still come back (sort of) if the for-loop ever initializes `proxy_manager` to `None`. I'm going to pull @erikcw's commit into my branch and work from there.


In retrospect, I tend to agree with @sigmavirus24 approach of initializing the attributes instead of using `hasattr` in this case.  Since it's before the loop, there are no concerns about clobbering pickled values with an empty dictionary -- while future-proofing the ability to move it into `__attrs__`.

My first approach to solving this was to simply add `proxy_manager` to `__attrs__` -- but that won't work because of the lambdas in `proxy_manager`.  My recommendation would be to initialize the attributes before the loop as suggested by @sigmavirus24 , and to incorporate my code comments to save future contributors from stumbling down the `__attrs__` approach.


I cherry-picked your commits into my PR. Thanks for your work here @erikcw ! Keep the PRs coming!
Requests 2.1.0 regression - cookies set on session.get are not remembered anymore for subsequent requests

The following line that made it happen has disappeared from Session.request:

```
    # Add param cookies to session cookies
    self.cookies = cookiejar_from_dict(cookies, cookiejar=self.cookies, overwrite=False)
```

If this was on purpose then some documentation of breaking changes such as this one would be nice to have.


That line is replaced with `self.cookies = merge_cookies(self.cookies, cookies)` in 2.1.0.
Is your problem happens when you provides cookies with `cookieJar`? 


Apologies, this was on purpose. See Issue #1728, which was fixed in #1729 (though actually merged as part of #1776). I'm very sorry that it got missed out of the changelog: that's my fault. I'll update it now.


Noted in 456c42b00d15c1ffb7fe610e4dad9164ab3eb6f0. Once again, I'm very sorry for that.


Ah, i did't notice that change. I also apologies about my misinformation


Thanks Cory for your prompt support and updating the changelog so promptly
so other people are not caught by it.

On Thu, Dec 12, 2013 at 5:23 PM, Cory Benfield notifications@github.comwrote:

> Noted in 456c42bhttps://github.com/kennethreitz/requests/commit/456c42b00d15c1ffb7fe610e4dad9164ab3eb6f0.
> Once again, I'm very sorry for that.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/1791#issuecomment-30437004
> .
Requests on PYPY 2.21 is over twice as slow as on CPython

A get request between two local servers in same data center that is typically 15ms on CPython is more like 33ms on the same box with latest PYPY.

Why should this be?


Before I go digging into this, what's your sample? I would want a fairly conclusive demonstration that this is actually a problem: that means something like doing 1000 requests to something on a local network (or even the same box) in CPython and the same again in PyPy, then checking the _average_ execution time for each request.

What exactly did you do to conclude this?


Ok
This hits couchbase on a server for a small slab of json in same data centre from same server running two virtualenvs - 1 cpython 2.74 and one pypy2.21. 

``` python
import requests
from utils.timefunctions import etime
from profilehooks import profile

@profile
def req():
    for i in xrange(2000):
        r = requests.get("http://data.atomised.io:8091/couchBase/default/1738568:1739813:29610")

def main():
    req()

if __name__ == "__main__":
    main()
```

![cpython](https://f.cloud.github.com/assets/456254/1725165/10453746-627c-11e3-9d30-9f8a6d3ef858.png)
![pypy](https://f.cloud.github.com/assets/456254/1725166/1048b3d0-627c-11e3-9e19-a0c3a88a52d7.png)


It seems to spend most of its time in `_socket.getaddrinfo()` (if I am reading the traces correctly).
So it shouldn't be the fault of requests but pypy. Maybe you want to profile that and report it to the pypy people.


will do


Yeah, @t-8ch seems to have the right analysis. In CPython, it takes 29milliseconds per call to `requests.get()`, of which 22ms is `getaddrinfo()`. In PyPy, it takes 47 ms per call to `requests.get()`, of which 35ms is `getaddrinfo()` That accounts for 13 ms of the 18ms difference. I imagine the remaining portion of that time difference is probably because the profiler doesn't play well with PyPy (I seem to recall that being a problem, though @alex will surely be able to correct me if it isn't).


Do you have a benchmark of just `getaddrinfo()`? /cc @fijal


Hmm, I wanted to test this. Running on Windows, I get garbage output when profiling on PyPy, but I wrote a quick test of `getaddrinfo()`:

``` python
import socket
import time

def req():
    for i in xrange(2000):
      res = socket.getaddrinfo('lukasa.co.uk', 'http')

if __name__ == '__main__':
    start = time.time()
    req()
    end = time.time()

    print "Elapsed time: %f" % (end - start)
```

Output:

```
PyPy:
Elapsed time: 1.203000

CPython:
Elapsed time: 1.011000
```

This _is_ slower, but only marginally. I don't think this is PyPy's fault, unless @handloomweaver can demonstrate the problem on their hardware/network.


Closed due to inactivity.
fixed https connection to be able to handle SSL_v3 when using it with Op...

...enSSL


This is `urllib3` change, not requests.
You should open PR in `urllib3` not here.


Good point.
On Dec 10, 2013 10:26 PM, "Park Ilsu" notifications@github.com wrote:

> This is urllib3 change, not requests.
> You should open PR in urllib3 not here.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/pull/1788#issuecomment-30295114
> .


Thanks for this, good luck at urllib3!


@anantasty next time, please check the README in the `requests/packages` directory which GitHub kindly displays when you visit [that directory](https://github.com/kennethreitz/requests/tree/master/requests/packages).
Made default_user_agent reusable

make the version reusable too.


I'm now -1 on this. I was neutral on the previous change, but each additional changeable value takes us one step closer to the logical endpoint of this trend, which is every parameter is editable. I think we should stop changing this, and just let people reimplement the totally trivial logic of this function if they feel they need it.


The slippery slope was my own concern about the last pull request. Furthermore, there aren't even any tests for this, not that their addition would make this anymore palatable to me.


It's not a big deal. I just think if you just change the name, without also changing the version, it will cause confusion. For example: 'other-package-name/2.1.0', where the name has changed, but the version is still requests version. Either all allowed to change, either all changes are not allowed.
Clean up #1779.

The unicode literals are a time bomb, let's just be rid of them.


Probably I would have fixed it if a message was left like "Clean up the unicode, then we are good to merge". 


@kracekumar I know. =) The only reason I did this was because Kenneth got eager to push out a release last night, and I wanted to make sure we didn't slow him down. =) Your PR was a good one, thanks for providing it!


@kracekumar it's no big deal at all :)


@kennethreitz @Lukasa That is great.