Raibek/requests-github-issues huggingface dataset:
Post requests to more complex HTML elements

Does the requests module support post requests to more complex HTML elements where values cannot be accessed just by dictionary lookup? For example, say we have the following HTML code, which is present on most Wikipedia pages to support the text of the article:

</div><textarea tabindex="1" accesskey="," id="wpTextbox1" cols="80" rows="25" style="" lang="en" dir="ltr" name="wpTextbox1">
(body of the article)
</textarea><div class='editOptions'>

Is there a way to access the part that says (body of the article) using the requests API, and post to it? I tried 'wpTextbox1' : '' in the dictionary, but this did not work.


Sorry, the HTML headers got messed up by the Github reply form, and I'm not sure how to post them to this page.


Here's a link to my Stackoverflow post of the same question, which has gotten not substantial replies: http://stackoverflow.com/questions/32447809/use-python-requests-to-access-mediawiki-text


This is not a support forum. Please be patient with the stack overflow community


While I understand that this is not a support forum, my experience with Github issues forums is that they also discuss possible improvements to APIs as well as documentation pages. Given that changing the values of more complex HTML elements is a task that should come up fairly often, I would think that this might be something you would add to your current API, if not already, or if it is possible already, described to some detail in your documentation page. I am not trying to be rude, just pointing out some facts......


Except that requests is an HTTP library, not an HTML library. We don't parse HTML for the user, nor will we ever. This seems to have been [explained to you](http://stackoverflow.com/a/32448030/1953283) already but you ignored it. The question you should ask in response to that answer is "Then what do I use to scrape the HTML." That's not something I'll answer for you here because Google can give you many better options and I am not Google.
Log in to Wikipedia not working

I'm trying to log into Wikipedia using the following code:

import requests

s = requests.Session()
url = 'https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page'
values = {'wpName' : '.....',
        'wpPassword' : '.....'}
s.get(url)
req = s.post(url, values)
print(req.cookies)
print(req.url)

However, it does not work. The req.url is the same as the login page (https://en.wikipedia.org/w/index.php?title=Special:UserLogin&returnto=Main+Page), and the cookie jar is empty. Does the MediaWiki software block the requests module, or am I doing something wrong?


Your URL is wrong.

Please open the page in your browser and enable your browser developer tools, and then try the login yourself. You'll note that the POST target is actually `https://en.wikipedia.org/w/index.php?title=Special:UserLogin&action=submitlogin&type=login&returnto=Main+Page`, not the URL you used (which is missing the `action=submitlogin&type=login` portions.

Additionally, your form is missing some values. My test (username 'test', password 'testpass') ends up sending the following values:

```
wpName:test
wpPassword:testpass
wpLoginAttempt:Log in
wpLoginToken:3cc5dc5e021f5790ad020deced537681
wpForceHttps:1
wpFromhttp:
```

The loginToken, in particular, is something you'll need to scrape out of the page. I'm not going to provide instructions here because this is a bug tracker, not a generalised help forum: if you need more specific advice I'd recommend using [StackOverflow](https://stackoverflow.com/).
python3: does not un-gzip replies

It seems that (at least) a 403 response with gzipped body is expanded when running Python2, but not so with Py3.

```
desk ~/src/librssreader$ python run.py 
AppId required! Contact app developer. See http://www.inoreader.com/developers
[ traceback because of unrelated missing error check ]
desk ~/src/librssreader$ python3 run.py 
[[[ binary gibberish ]]]
[ same traceback ]
desk ~/src/librssreader$ 
```

run.py is simply:

```
#!/usr/bin/python3

from librssreader.inoreader import RssReader, ClientAuthMethod
auth = ClientAuthMethod('USERNAME','CENSORED')
reader = RssReader(auth)
print(reader.getUserInfo())
```

You need to register with inoreader.com (or find another web server) before testing this.


Requests has no logic relating to response code that should affect whether or not the body of the request is expanded. If you read the content, you should get the gzip expanded. My money is that the 403 response does not include a Content-Encoding header. Can you show me the request/response please (sanitising secure headers if needed)?


Found it. The reply header looks like this:

```
HTTP/1.1 403 Forbidden
Server: nginx/1.7.9
Date: Mon, 07 Sep 2015 17:20:42 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 90
Connection: keep-alive
X-Powered-By: PHP/5.4.41
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Reader-Google-Version: 555-000
X-XSS-Protection: 1; mode=block
AppId required! Contact app developer. See http: //www.inoreader.com/developers
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip

[gzipped content]
```

Obviously the problem here is that Requests, on Python2, manages to ignore the inserted line of nonsense (note how the sender's server cleverly inserted a space after the colon), while on Python3 it does not.
I'll send them a bug report.

Still, there should not be inconsistent behavior between Py2 and Py3 versions.


So the header parsing is done by httplib (on Python 2, renamed to http.client on Python 3). This is a bug in the standard library, not in requests.

We cannot control the dependencies that are far beneath us (like the standard library).


Yup, unfortunately there's nothing we can really do about this at the moment, until such time as we stop using `httplib` to do the low-lying HTTP stuff for us.

I am extremely sorry about this. In the short term, I recommend you contact the server provider if possible and tell them to stop sending that nonsense in the HTTP headers (or at least to give the HTTP header a proper key).
Add PreparedRequest pickling tests

Trying to address #1558 by first creating a failing test, but I'm not sure when `PreparedRequest`s fail to be pickleable. Pulling early to see if anyone has suggestions on how to re-create the issue, also maybe these tests are useful on their own as well.


Thanks @keyan! From the issue:

> On a prepared request, the only thing I think can hurt us is the case when the body attribute is a file object. Is there a specific hook that we're concerned about not being pickleable?

So the concerns seem to be body file objects and hooks.


@keyan do you have intentions to update this with the tests we were looking to add?


Hey @sigmavirus24, I could use some advice on how to add a file object in the test environment. I just added a commit with a failing test containing a hook. Not sure how to address the actual issue though.


@keyan It is often easiest to just create a file handle to the file under test: `open(__file__)`.


Closing due to inactivity, (now) merge conflict, and failing tests. These look very much needed though, so please open another PR when ready to proceed!
Json supersedes data in prepare_body

When not a stream, json supersedes data in prepare_body:
https://github.com/kennethreitz/requests/blob/f5dacf84468ab7e0631cc61a3f1431a32e3e143c/requests/models.py#L446

This conflicts with the docstring, which indicates that json is only used when data is not specified:
https://github.com/kennethreitz/requests/blob/f5dacf84468ab7e0631cc61a3f1431a32e3e143c/requests/models.py#L195


Thanks for this! Yes, that's well spotted: we're definitely in conflict with the documented behaviour.


I may try to find someone who wants to get started in requests to see if they want to fix this: it's a perfect fix for someone relatively new to OSS.


BTW, `files` also supersedes `json` - the docs could use a small update there.


This is resolved in 2.8.0. Thanks all!
allow custom JSONEncoder for the request's json param

Using the [json param](http://docs.python-requests.org/en/latest/api/?highlight=json#requests.request) when making a request is awesome: 

``` python
payload = {'blip': 'blaz'}
r = requests.get('http://foo.bar', json=payload)
```

Using the [response property](http://docs.python-requests.org/en/latest/api/#requests.Response.json) is great as well, especially since you can use a custom json decoder:

``` python
r.json(cls=MyCustomDecoder)
```

It would be nice to allow a param to be passed that could be applied to the `dumps` of the [json value](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L417-L419).

``` python
payload = {'blip': Decimal(69)}
json_params = {'cls': MyCustomEncoder}
r = requests.get('http://foo.bar', json=payload, json_params=json_params)
```


I think this is interesting, but generally requests is very averse to adding new keyword arguments to `requests.*`. I don't think it's valuable enough to justify adding the new argument, I'm afraid. Sorry!


Is there a better way to always use a custom json encoder? This way is brittle and gross and does not even work with stuff in master (fine for 2.7.0).

``` python
import json
import functools
from decimal import Decimal

import requests


class CustomJSONEncoder(json.JSONEncoder):
    def default(self, obj):
        if isinstance(obj, Decimal):
            return round(float(obj), 2)
        return json.JSONEncoder.default(self, obj)

# yuck
requests.models.json_dumps = functools.partial(json.dumps, cls=CustomJSONEncoder)

r = requests.post('http://foo.bar', json={'foo': Decimal(69)})
```


I think the better way is to manually use it yourself:

``` python
r = requests.post('http://foo.bar', data=json.dumps(some_data, cls=CustomJSONEncoder), headers={'Content-Type': 'application/json'})
```


Yep, I am already doing it that way. Was wanting a way to globally have it use a specific encoder/decoder when messing with json


@drmaples then make a function that handles that "globally" for yourself. We won't be introducing this.
Fix unicode redirects on Python 3.

Resolves #2653.

Follows on from #2655: please read the discussion there for more information.


:+1:


there's a lot of pending merges — trying hard not to merge them :)


@kennethreitz You're doing an admirable job. ;)


Soon™


Oh right, this is for 3.0.0... did we have a test for this? If not I'm :+1: 


I don't think we do, though I can see if I can create one.


Tests are nice :-P


Ok, got some extra tests for `unquote_unreserved`, which is a start.


And a failing test for the original issue, which now passes. It's only valid on Python 3, so I added a skip for Python 2.

Unfortunately, this now requires a more up-to-date py.test version.


Hurrah! All tests green. Over to you @sigmavirus24.


One tiny comment. I'd like your thoughts on it (not necessarily action unless you agree). Otherwise :+1: 


Rebase required. 


Rebased.
Upgrade urllib3 to 1.11

I'm running into [the following bug](https://github.com/pypa/pip/issues/2753) in `pip`:

```
  File "/usr/lib/python2.7/httplib.py", line 780, in close
    self.sock.close()   # close it manually... there may be other refs
  File "/…/pip/pip/_vendor/requests/packages/urllib3/contrib/pyopenssl.py", line 213, in close
    return self.connection.shutdown()
  File "/…/OpenSSL/SSL.py", line 1522, in shutdown
  File "/…/OpenSSL/SSL.py", line 1178, in _raise_ssl_error

SysCallError: (32, 'EPIPE')
```

This is caused by [a bug in urllib3](https://github.com/shazow/urllib3/issues/612); urllib3 has since [patched](https://github.com/shazow/urllib3/commit/90946207d2aedd41fcfcf8e6d68b07fa5a834ada) the bug, but pip can't get the patch until requests upgrades their copy of urllib3.


We'll hopefully be creating a release soon. When that happens we'll upgrade our version of urllib3.
form data encoding can't be modifid

these days, I 'm doing a poster to an old system,it only accepts gbk encoding,but form data is utf8, python 3 can pass a encoding parameter to urlencode then pass to post method,but python 2 urlencode does'nt support this parameter.

any opinion about this?


This isn't something requests would do for you anyway, so what you should do is urlencode the data, and then pass it to the `data=` parameter, e.g., 

``` py
postdata = urlencode(...)
postdata = postdata.encode(...)
requests.post(url, data=postdata, headers={'Content-Type': 'application/x-www-form-urlencoded'})
```

It isn't pretty but requests clearly meets the 95% use case here. The remaining 5% is something we won't add support for to the API.

Alternatively, you could use the PreparedRequest flow and encode the body before sending the request, but that (I think) might end up being more work than what I've sketched out above.


Also in the future, if you're looking for help, please ask questions on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).


@sigmavirus24 sorry.

urlencode in python 2 encode unicode to ?

this is a problem
during redirecting the host adress in header did not update correctly

Situation:POST or GET a adress which will redirect an adress which the host adress in header is different with the last request.

I write a small script to simulate login my school account.But the exception,

> requests.exceptions.ConnectionError: ('Connection aborted.', error(10054, ''))
> is always happening.

I try to print the information of every request.So in [session.py](https://github.com/kennethreitz/requests/blob/master/requests/sessions.py) class SessionRedirectMixin function resolve_redirects ,I print the header information before a real request.

I find during redirecting,the header information is just copied  from its last request.So if a request is redirected to an adress which the host adress in header is different with the last request,it will failed.

I try to solve this problem.
I add these lines in function resolve_redirects at 172th line in [session.py](https://github.com/kennethreitz/requests/blob/master/requests/sessions.py) before send the request.

```
if headers.get("Host"):
    headers["Host"] = urlparse(url).hostname
```

And it worked for me.

more information:

```
import requests
print request.__version__
'2.7.0'
```


@HeartUnchange are you specifying your own `Host` header? If so, why?


~(≧▽≦)/~I got it.I delete the Host ,and it works.I see browser send this field ,so I add this(I am a beginer).Thanks for your help.
Session connection with stream=True unusable after response.close()

Please consider the following code:

```
import requests
session = requests.Session()
resp = session.get('http://www.w3schools.com/html/demo_sse.php', stream=True)
resp.close()
session.get('http://www.w3schools.com')
```

It results in the following exception:

```
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File ".../venv/lib/python3.4/site-packages/requests/sessions.py", line 477, in get
    return self.request('GET', url, **kwargs)
  File ".../venv/lib/python3.4/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File ".../venv/lib/python3.4/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File ".../venv/lib/python3.4/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ResponseNotReady('Request-sent',))
```

Unless I am doing something wrong, It appears to me that `resp.close()` returns the connection to the pool in an inconsistent state. What's interesting is that the exception does not occur when the last line in the example above connects to a different server (eg. `session.get('http://www.google.com')` instead of `http://www.w3schools.com`). What's also interesting is that the problem only occurs when working with a `requests.Session`. That is, the following code does not raise an exception:

```
import requests
resp = requests.get('http://www.w3schools.com/html/demo_sse.php', stream=True)
resp.close()
requests.get('http://www.w3schools.com')
```

I am using requests version `2.7.0`. The problem occurs on both Windows 7 with Python 2.7.10 and on OS X with Python 3.4.3.


This is a known issue (#2593), and was fixed in #2610. =) The next release of requests should contain the fix.


Awesome, thanks :-) Is there an ETA for the next release?


Nothing beyond "soon" I'm afraid. Hopefully our release process will get a bit more concrete over the next few weeks.


Ok, thanks for letting me know!
Very slow read of large files compared to many other modules/binaries (e.g. urllib/curl/wget)

Using python 3.4 and requests 2.7.0 (on both Ubuntu 14.04 and Mac OS X), I've noticed that fetching raw content of large files is significantly slower (~5x) with requests than many other applications and python modules, performance as below is typical:

```
import requests                                                             
import urllib.request                                                       
import time                                                                 

# returns a 100Mb JSON file                                                                     
url = 'http://some.host.on.local.network/'

def with_urllib():                                                          
    with urllib.request.urlopen(url) as f:                                  
        return f.read()                                                     

def with_requests():                                                        
    return requests.get(url).content                                        

s = time.time()                                                             
a = with_urllib()                                                           
print('urllib:', time.time() - s, 'seconds')                                

s = time.time()                                                             
b = with_requests()                                                         
print('requests:', time.time() - s, 'seconds')                              

assert a == b
```

This outputs:

```
urllib: 1.113746166229248 seconds
requests: 5.458410978317261 seconds
```

curl, wget, pycurl and urllib all complete in similar times (~1.1s on average), requests consistently takes ~5.4s on average, even with various combinations of parameters I've tried (e.g. with streaming, with different streaming chunksizes, etc.).


I believe this is the same bug report as #2371. We'd like to keep investigating this performance problem to see if we can work out where we're spending the time, so if you feel like trying to track it down yourself please do. Otherwise, it'll go on my backlog of work to be investigated.


Thanks, looking at #2371, it sounds like it could be the same thing. I'll do some additional investigation, and report there if I find anything.
params of  proxy_headers are  inconsistent with its docstrings

requests/adapters.py

```
def proxy_headers(self, proxy):
        """Returns a dictionary of the headers to add to any request sent
        through a proxy. This works with urllib3 magic to ensure that they are
        correctly sent to the proxy, rather than in a tunnelled request if
        CONNECT is being used.  

        This should not be called from user code, and is only exposed for use
        when subclassing the
        :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.

        :param proxies: The url of the proxy being used for this request.
        :param kwargs: Optional additional keyword arguments.
        """
```


Yup, that does look wrong. Are you interested in providing a pull request to make this fix?


Yes, I'm grad to.
quick fix to make session cookies work with mocked HTTP responses

I ran into an issue working with Responses/HTTMock where session cookies were not being set. Tracked it down to here, looked like a quick (possibly dirty?) fix. Doesn't break any tests!

Throwing it up here for wider consideration. Comments welcome.


Generally speaking, we don't really believe it's the role of this library to fix abstraction errors in Responses or HTTMock. Our goal is to function with the standard library, which this code does. In rare cases we allow for really obvious differences (e.g. `StringIO`), but we are _extremely_ reluctant to add more special case code to account for libraries that _pretend_ to be other libraries. This is particularly true when it comes to attribute access, given that Responses or HTTMock could easily just provide that attribute in their mocked response.

I recommend opening this issue with the respective projects. =) Sorry!
PyOpenSSL support doesn't work with TLSv1.1 and TLSv1.2

`docker-py` tries to use latest protocol version in the `ssl` module, so use `ssl.PROTOCOL_TLSv1_2`. Unfortunately the version of urllib3 included with requests (and likely upstream?) only has older versions:

https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/contrib/pyopenssl.py#L71


So raise this issue against urllib3?
"Interrupted System Call" in select.select

Hi there,

The `pyopenssl.py` under `requests/packages/urllib3/contrib/pyopenssl.py` is using `select.select` without trapping the exception raised when the underlying _select(2)_ system call returns EINTR.  This results in an exception:

```
    response = session.request(method, full_url, params=params, timeout=TIMEOUT, headers=HTTP_HEADERS)
  File ".../.virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File ".../.virtualenv/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File ".../.virtualenv/local/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File ".../.virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File ".../.virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File ".../.virtualenv/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 184, in recv
    [self.socket], [], [], self.socket.gettimeout())
select.error: Interrupted system call
```

Current versions of pyopenssl don't make these calls; perhaps this can be fixed by updating the internal dependency?


Thanks for reporting this! Unfortunately, this is a urllib3 problem: can you raise this issue over there?


Thanks, logged as shazow/urllib3#700 - I'll follow up with a request to update the internal urllib3 dependency once they deal with it there.
file:// protocol not available

Hello all, and first of all thanks for this awesome library.

While working with requests I was surprised to find out that it doesn't handle the file:// protocol. Even though I understand it's maybe not a key feature of this library it would come in handy for compatibility reasons.

Concretely I have sort of a 'Fetcher' class that servers files from a 'Repository', but this repository can be local or remote. In case of being local I tried (unsuccessfully, of course) to use the file:// protocol. Finally it forced me to considerably increase the number of lines of code I needed to handle the issue.

For example, something like this would be fine:

```
response = requests.get('file:///Users/User/my_file.txt')
```

So far it raises this exception:

```
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 649, in get_adapter
    raise InvalidSchema("No connection adapters were found for '%s'" % url)
requests.exceptions.InvalidSchema: No connection adapters were found for 'file:///Users/User/my_file.txt'
```

My questions are: 
1) Would there be a reason to not to implement this feature?
2) In case there isn't, shall I submit a a pull request, or it is already planned to add this functionality soon?


Thanks for this issue!

The reason we don't support the `file://` protocol is because requests is a HTTP library. Our core purpose is to do HTTP: anything else is outside the scope of the core library.

I believe there is a pluggable Transport Adapter for the file protocol floating around. You can also consider adapting one from [this answer](http://stackoverflow.com/a/22989322).


Before opening issues, please search all issues (not just open ones) for similar issues. [This](https://github.com/kennethreitz/requests/issues/847) is just the earliest issue I could find and there are several others (including pull requests).
content-length header value not available after get

I'm unable to get content-length reliably with the requests library whereas urllib2 seems to retrieve the data consistently. Not sure if additional code is required or if the URL I'm hitting is problematic.

Run this script and compare the results. I consistently get **None** from requests. Using the other test_url I consistently get a result from both. What am I missing?

``` python
# Test which demonstrates issues with requests get of content-length header

# This URL seems problematic (it's a several minute download)
test_url = "http://standards-oui.ieee.org/oui.txt"
# test_url = 'https://github.com/kennethreitz/requests/tarball/master'

def requests_getlen(url):
    import requests
    r = requests.get(url, stream=True)
    return str(r.headers.get('content-length'))

def urllib_getlen(url):
    import urllib2
    req = urllib2.urlopen(url)
    return str(req.info().getheader('content-length'))

print 'requests content-length: ' + requests_getlen(test_url)
print 'urllib2  content-length: ' + urllib_getlen(test_url)
```


Questions belong on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).

That said, have you checked to see if you're getting a chunked transfer-encoding?


Sorry, my bad. I thought it might have been a legitimate issue since the results were inconsistent between runs even using the alternate test_url value pointing at the tarball file on GitHub (that one would **sometimes** have content-length included, but not always). I'll figure out the appropriate question(s) and post on StackOverflow.

In the meantime, after some poking around with Charles, I determined that the difference stems from the fact that the Requests library specifies `gzip, deflate` as acceptable encodings by default and urllib2 specifies `identity` by default.

Here is code which worked for this URL with requests to obtain content-length header:

```
r = requests.head(url, headers={'Accept-Encoding': 'identity'}, stream=True)
return str(r.headers.get('content-length'))
```


> the results were inconsistent between runs even using the alternate test_url value pointing at the tarball file on GitHub (that one would sometimes have content-length included, but not always)

If you look at the raw data that we're sending we're sending the exact same time (some of the header order may be different but that's to be expected). This would seem to be a problem with GitHub, but as long as they're also sending `Transfer-Encoding: chunked` then they aren't required to send a content-length. Not everything on the internet has a content-length header associated with it.
Fixed minor rendering issue in documentation code blocks

Some of the code blocks on the quickstart page had an issue where they were rendered as two separate `<pre>`, making them inconsitent with the others.

Before:
![quickstart_ _requests_2 7 0_documentation_-_2015-08-15_21 51 01](https://cloud.githubusercontent.com/assets/6345/9290559/22c82672-4398-11e5-8e94-b5cbf2205709.png)

After:
![quickstart_ _requests_2 7 0_documentation_-_2015-08-15_21 51 26](https://cloud.githubusercontent.com/assets/6345/9290558/22c50a3c-4398-11e5-8c6e-51d06aed2090.png)


:sparkles: :cake: :sparkles: Thanks @bmispelon!


:rabbit2: :racehorse: :tada:
Docs: Fix links to `timeouts` section by using :ref:

This fixes three links (from docstrings) to the [<`timeouts`>](http://docs.python-requests.org/en/latest/user/advanced/#timeouts) section by using `:ref:` instead of linking to `.html` files (which are only available when building docs locally, but not in the published docs on RTD).

Fixes #2698


LGTM, thanks! :cake: :sparkles: :cake:
Add bitcoin donation info to README.rst

I would like to donate on a monthly basis, and I'm sure others do too. I'm not a particular fan of one time donations which you've recently added to the docs landing page.

I would however love to make small, monthly, bitcoin donations.


Looks like Requests already accepts (micro)donations [through Flattr](https://flattr.com/thing/442264/Requests), so you could use that for monthly donations. (But I'm not sure if you can manually set the amount you donate _specifically to requests_...)


This is a structural problem we need to work out in a way that is going to be most fair. I think the team need to sit down and discuss this a little bit at some point.


`1Fov3RQFSDTxxGApR19g1GgQazzoutiGCC`
Connection reset by peer on get requests

In making a request to a third party resource, https://www.example.com, I am getting the following traceback:

``` python
import requests; requests.get('http://www.example.com')

Traceback (most recent call last):
 (......traceback.....)
  File "/Users/python3.4/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(54, 'Connection reset by peer'))
```

For reference, I am on python 3.4.3, using requests 2.7.0, and running on Mac 10.9. What is very strange about this issue in particular is that the same call works if i run 'curl -i https://www.example.com' or if I access the URL using a browser. I believe it has something to do with the SSL handshake


Yeah, connection reset by peer means the remote peer doesn't like the TLS handshake much. It's difficult to know without tcpdumping what the problem is here. I don't see the problem on my end though: do you have PyOpenSSL installed?


It was an issue with the SSL certificate on their site, which has since been resolved. Thanks!
Add documentation in "More complicated POST requests" (Quickstart) to resolve #2686

As suggested in #2686, I added a snippet about `requests.post(json=...)` to [Quickstart#More complicated POST requests](http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests).


\o/ Beautiful, thankyou so much! :cake: :sparkles: 


Feel free to open a PR that adds you to AUTHORS as well. =)
Passing a list with size one to request.get transforms list into single element

Hello,

I'm using Python 2.7.6 and requests (2.6.2).
When I try to send a GET request with this payload :

``` python
{
  'event_parameter_2': datetime(2014, 3, 9, 6, 11, 35), 
  'event_parameter_3': [u'eligendi'], 
  'event_parameter_4': u'sc', 
}
```

The `event_parameter_3` field isn't a List type anymore, but a String type when my API gets the request.
This is how I send the GET request :

``` python
r = requests.get('http://' + host + ':' + port + '/event/log', params=generate_event())
print r.url
```

Which gets me the following output :

```
http://127.0.0.1:3000?event_parameter_2=2014-03-09+06%3A11%3A35&event_parameter_3=eligendi&event_parameter_4=sc
```

However, when having a list with multiple elements, I have no problem and get the following output :

```
http://127.0.0.1:3000?event_parameter_2=2012-10-12+01%3A29%3A37&event_parameter_3=occaecati&event_parameter_3=possimus&event_parameter_4=gy
```

I think that maybe when sending lists in url requests, formatting the url as such might be better ?

```
http://127.0.0.1:3000?event_parameter_3[0]=occaecati&event_parameter_3[1]=possimus
```

Thank you :smile: 


Thanks for this report!

Your summary is not quite accurate. The output that you are unhappy with in the first case is strictly formatted exactly the same as the output you _are_ happy with in the second case. The second case has multiple entries with the same name: the same number of entries as elements in the list. The first case also does: it just happens that there's only one element in the list.

You can see that if I break the sections out. Here is the two element case:

```
event_parameter_3=occaecati
&
event_parameter_3=possimus
```

Here's the one element case:

```
event_parameter_3=eligendi
```

This is exactly the same representation. =) We're being very consistent.

When sending lists in the URL there is no standard representation. The _most common_ representation is to have empty square brackets (`[]`), which you can do by doing `'event_parameter_3[]': [u'eligendi']`, but the reality is that there is no one right way to do this. Therefore, in the face of this ambiguity, we refuse to guess. You'll need to make sure the data is correct for whatever format your web service would like to receive.

Sorry we can't be more helpful!


Thank you for your answer ! :)

What I actually meant was that when passing a single sized list:

```
event_parameter_3=eligendi
```

There is no way to know whether it was an list or any other type when receiving the request, a string being represented the same way:

```
event_parameter_4=sc
```


Indeed. This is the problem with query strings: they aren't really fully-structured data. They're just a list of key-value pairs. There are some _conventions_ around representing 'lists' and other kinds of data structures in query strings, but they are only conventions, there are no rules.

As I said, the closest there is to a 'standard' convention is to use square brackets after the key name: `event_parameter_3[]`.


Oh, I didn't understand it this way !
Thank you very much ! :blush:
Capitalize response reasons.

This is a somewhat trivial thing, I ran into a "bug" while grouping responses by reason instead of status codes. This is of course a very particular case but just to be consistent it may be a good idea to have response.reason always be "NOT FOUND" instead of "Not Found", "Not found", "not found" etc.


Generally speaking, reason phrases are not to be trusted when provided by the server. There is no requirement that they be the same as the status code: nothing's really wrong with sending `200 NO PROBLEM BOSS`, for example, aside from the fact that the spec says you shouldn't.

For that reason, you should not rely on the reason phrase when making decisions unless the exact binary form of that phrase is something you care about. I don't think requests should canonicalise in any way here.

Thanks for the suggestion though!


+100 to what @Lukasa said. If you want normalized responses, you can use a mapping of status codes to whatever you'd like them to represent.


Note that I did not mention let's change `200 NO PROBLEM BOSS` to `200 OK` or vice-versa, the only proposed change is to capitalize whatever the reason coming from the server is just to avoid having to do it in an extra map().
Docs: Clarify that method-level parameters are not persisted in sessions.

This adds a paragraph to the **session docs** that clarifies the fact that **method-level parameters are not persisted across requests**, even when a session is being used (fixes #2488).

As an example I used cookies, and included a pointer to the [Cookie utility functions](http://docs.python-requests.org/en/latest/api/#cookies). In order to be able to link to that section I added some section labels in `docs/api.rst` (prefixed with `api-`, because otherwise there would be label collisions).


Eh, I'd rather not add a bunch of labels to one file just to be able to use _1_ in the docs.


@sigmavirus24 got it, I'll remove all the ones except the one for the cookies section. Should I stick with the `api-` prefix or drop that as well?


The api- prefix makes sense. Thanks.


@sigmavirus24 updated


Thanks @lukasgraf! :sparkles: :cake: :sparkles:
Document use of sessions as context managers

This adds some documentation on how to use sessions as **context managers** (fixes #2580). I also added a brief mention of **connection pooling** and its performance benefits.

I still got one question though: Is instantiating [`Session`](https://github.com/kennethreitz/requests/blob/9b067db19e20226dcb3aa407605d30942d085050/requests/sessions.py#L267) directly the proper API to document? I noticed that there's also a [`session()` factory function](https://github.com/kennethreitz/requests/blob/9b067db19e20226dcb3aa407605d30942d085050/requests/sessions.py#L674-L677) that explicitly mentions context management in its docstring. But I then saw this in the [API changes section](http://www.python-requests.org/en/latest/api/?highlight=backwards%20compatibility#api-changes):

> The `Session` API has changed. Sessions objects no longer take parameters. `Session` is also now capitalized, but it can still be instantiated with a lowercase `session` for backwards compatibility.

So I therefore used `Session()` in the examples, and I'm assuming that factory function is only there for backwards compatibility. Should I update its docstring with something like `Alias for backwards compatibility`?

Let me know how you'd like it documented, and I'll update my PR accordingly.


This is great, thanks!

:sparkles: :cake: :sparkles:
FR: Ability to enable raise_for_status for all requests on a Session object

Hey there,

Firstly, I absolutely adore requests, it's the reference library for Python and would like to thank you greatly for creating it.

Today I found the little utility function raise_for_status which I think is very useful, but I think it's a little tedious having to call this for each request if this is the way you prefer to work.

Would you please consider a way to allow for this to be run automatically each time a request is made?  I was thinking you could have a kwarg to Session which enables it for all requests, and possibly a kwarg for requests.get and so forth?

Thanks again! :smile: 
Fotis


Sorry @fgimian, it looks like this got lost!

Generally I don't believe automating a single function call is a particularly good idea. The cost of doing that is that it makes it hard to tell from reading code exactly what exceptions a given call into Requests might raise, because they depend on the `Session` object elsewhere in the code.

Given that the only advantage is to save you a single function call, I think this is probably not worthwhile. Sorry!


Thanks for your reply @Lukasa, I understand where you're coming from.  Really appreciate you considering my idea :smile:
Allow to modify query parameters in authentication classes

Some APIs like https://www.virustotal.com/en/documentation/private-api require you to pass the API key in a query parameter which is not possible at the moment because the request is prepared before the authentication class is applied.
Using the following authentication class:

``` python
class VirusTotalAuth(auth.AuthBase):
    def __init__(self, api_key):
        self.api_key = api_key

    def __call__(self, r):
        r.params['apikey'] = self.api_key

        return r
```

Produces this error:

```
AttributeError: 'PreparedRequest' object has no attribute 'params'
```

Which makes sense because you can't change a request's parameters after it has been prepared.


> Which makes sense because you can't change a request's parameters after it has been prepared.

You can change a request's parameters. The current way to do so (since you're actually receiving a `PreparedRequest` instance) is to:
1. Parse the url
2. Change the parameters
3. Unparse the url
4. Set the `url` attribute

This is well documented in other similar issues.

That said, I would posit that an Authentication Handler for this purpose is not ideal. Once you've authenticated, it makes more sense to add the API key to a `Session`'s `params` dictionary so it is added to every request there-after automatically.
Option to disable percent encoding disabled?

Hi,

I know it's not good practice to disable percent encoding, but the server I'm connecting to doesn't support it. There seemed to be an option using requests.defaults in an earlier version, but it seems to be gone now. Is there any way to disable percent encoding? Thanks.


This has been discussed before in:
- https://github.com/kennethreitz/requests/pull/757
- https://github.com/kennethreitz/requests/issues/1454

And I'm sure other issues (but those are just the first two that a search of the issues found). Our current behaviour suits easily more than 95% of our users and thus it will stay. If you need to bypass (or change) how we encode the URL, the [PreparedRequest flow](http://docs.python-requests.org/en/latest/user/advanced/#prepared-requests) is what you need. For questions about that, please use [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).
404 for timeouts advanced documentation links

Three files [in this search](https://github.com/kennethreitz/requests/search?l=python&q=<user/advanced.html#timeouts>&type=Code&utf8=✓) has links to `<user/advanced.html#timeouts>` which works good for documentation generated on the local machine.
But on the http://docs.python-requests.org/ there is no ".html" in pages urls so 404 Page not found is returned.

I think docs.python-requests.org will be ok if you change links to `<user/advanced/#timeouts>` but local machine documentation will be broken.


We should be using `:ref:` in our docs, so sphinx will handle this for us. Thanks for catching this @nilya
requests.get is partially ignoring proxies parameter?

I have a problem with Requests 2.7.0 and proxies. If I use the following code without setting the *_PROXY environment variables the requests takes up to 2 minutes to get the information I'm requesting by it does work. If I then set the HTTP_PROXY and HTTPS_PROXY env variables it takes 2 seconds to complete. It should be noted that the browsers on the machine are setup to use the proxy and the requests code does indeed pick up these settings from the registry as well as me specifying the values via the proxies parameter.

proxies = {
  "http": "http://%s:3128/" % ipaddr,
  "https": "http://%s:3128/" % ipaddr
}
r = requests.get("http://nvd.nist.gov/download/nvd-rss.xml", proxies=proxies)

I have profiled the code in pyCharm without use of the env variables and it says the following:
Name                        Call Count  Time(ms)    OwnTime(ms)
built-in method gethostbyaddr   3           41998   41998
built-in method gethostbyname   3           38378   38378

Debugging into sessions.py it seems that line 614 which calls get_environ_proxies takes quite a bit of time because it is indirectly calling socket.gethostbyname and socket.getfqdn in proxy_bypass_registry, request.py:2516 and the former throws an OSError but getfqdn succeeds giving the same answer as the rawHost variable.

Running the same test but with the HTTP_PROXY and HTTPS_PROXY environmental variables set shows in the profiler that the gethostbyaddr and gethostbyname take no time at all and the whole call takes 2 seconds.

Am I doing something silly here or is this something that can be fixed in requests? The proxy settings in the registry and passed via the proxies attribute just seem to be being ignored in parts of the code it would appear.


This sounds like you're experiencing some kind of timeout in the standard library's `urllib.proxy_bypass()` method: specifically, you seem to be experiencing some _really_ long DNS timeouts.

We're not strictly ignoring the proxies parameter, but we're trying not to be too clever about it in the code: we basically allow the system to provide us with the list of proxies that it wants to use and then merge those with the list of specified proxies. It's the most generic approach to the problem.

If you want to speed this up you can use a `Session` and set `Session.trust_env = False`, which will avoid this call altogether. Otherwise, we need to investigate why `urllib.proxy_bypass` is so expensive on your system when it is usually so cheap.
Address family not supported by protocol

I am getting the error when making a request to my web api hosted on another machine on IIS express, this is in order for me to debug my application, however when I run my python code I get the error 

requests.exceptions.ConnectionError: ('Connection aborted.', error(97, 'Address family not supported by protocol'))

Here is the traceback:

Traceback (most recent call last):
  File "Allocator.py", line 201, in <module>
    Main()
  File "Allocator.py", line 154, in Main
    requestResult = MakeRequest("")
  File "Allocator.py", line 36, in MakeRequest
    result = requests.get(constURLString + "?lastUpdate=" + urlParameters)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 69, in get
    return request('get', url, params=params, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 465, in request
    resp = self.send(prep, *_send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 573, in send
    r = adapter.send(request, *_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests-2.7.0-py2.7.egg/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(97, 'Address family not supported by protocol'))


Closing to centralize discussion on [StackOverflow](https://stackoverflow.com/questions/31695430/address-family-not-supported-by-protocol-making-request-to-webapi) given there is more detail in the question than was provided in the bug report.
Dealing with removal of Authorization header on redirect and diff domain

for services that require Authorization in the header, all redirect calls to a different domain would result in a 403-Forbidden.  Authorization should be preserved just as Authentication is.


Sorry, we won't be accepting this. This is specifically written to fix a [vulnerability](http://www.cvedetails.com/cve/CVE-2015-2296/) that was reported to us recently. Without knowing your specific use-case I can't advise you on how to replicate the original behaviour, but if you post specifics to [our Q&A area](https://stackoverflow.com/questions/tagged/python-requests) you'll get an excellent answer.


Thanks for looking into this request as well as providing the reference as to why it's not a good idea.  Appreciate the time.