Raibek/requests-github-issues huggingface dataset:
Clean up cookie docstrings, document cookie jar.

Following on from [a Stack Overflow question](https://stackoverflow.com/questions/27586779/requests-inability-to-handle-two-cookies-with-same-name-different-domain), this PR extends our cookie documentation [here](http://docs.python-requests.org/en/latest/api/#cookies) to document the cookie jar itself. This should make it more likely that users are able to discover how to make more subtle queries about cookies.


A couple comments. I'm also wondering if there would be use in the toolbelt for a class that extends this so that you can do 

```
s.cookies['cookie-name', 'domain', 'path']
```

If you use the custom cookie-jar. I do and don't like it. Alternatively, we could just include the recipe for the user in the documentation. It's not hard

``` python
class SpecialJar(requests.cookies.RequestsCookieJar):
    def __getitem__(self, name):
        if isinstance(name, str):
            return self._find_no_duplicates(name)
        if not isinstance(name, tuple):
            raise ValueError("Expected a string or a tuple but received {0}".format(type(name)))
        if len(name) == 2:
            path = None
            cookie_name, domain = name
        elif len(name) == 3:
            cookie_name, domain, path = name
        # Raise another ValueError in an else?

        return self._find_no_duplicates(cookie_name, domain=domain, path=path)
```


Updates made.

As for the toolbelt extension, that might be worthwhile, though I'm not sure I see the advantage. `.get()` is surely more natural (or at least seems that way to me).


Yeah it seems that way to me too. My problem with it is that default comes before domain and path so I expect people will not realize they need to use the keywords for domain and path to skip using a default.


Yeah, that's true, but it means that those who think `Session.cookies` is a dict get the behaviour they expect. Lesser of two evils, IMO.
Fix bug in renegotiating a nonce with the server

If a session runs long enough (without constant activity) then the server can
expire the nonce the session has negotiated. If that happens the session will
get a new 401 response which we were immediately returning to the user. A user
would then have to essentially reinitialize session.auth each time they get an
unexpected 401.

Also, there's no need for setattr calls when we can simply assign the
attribute on the instance.


LGTM. :cake:


@Lukasa I'm just waiting for @Jaypipes to verify this fixes their issues.


@sigmavirus24 I have now verified that this fixes the issue I was seeing. Thanks very much! :)


Cool. @Lukasa I'm going to cut 2.5.1 with this because it's a rather serious bug. (I have to wonder if it could be considered a CVE because someone could perform a denial of service on a client if it knows the clients behaviour and when to expire the nonce.)


Fine by me.
Request https url via http proxy

I recently tried to make a connection to a https url via a http proxy which doesn't support the 'connect' method (Crawlera http://scrapinghub.com/faq#https).
I'm using the following code:

import requests
proxies = {"http": "http://USER:PASS@paygo.crawlera.com:8010/",
               {"https:": "http://USER:PASS@paygo.crawlera.com:8010/"}

requests.get("https://wikipedia.org", proxies=proxies)

will result in 

> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/usr/lib/python2.7/site-packages/requests/api.py", line 55, in get
>    return request('get', url, *_kwargs)
>  File "/usr/lib/python2.7/site-packages/requests/api.py", line 44, in request
>    return session.request(method=method, url=url, *_kwargs)
>  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
>    resp = self.send(prep, *_send_kwargs)
>  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
>    r = adapter.send(request, *_kwargs)
>  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 378, in send
>    raise ProxyError(e)

Is there anything to do about that?


A HTTP proxy that does not support the CONNECT verb cannot transport a HTTPS request.


According to that proxy:
"... configure your HTTP client to use a HTTP proxy even for HTTPS URLs. However, not many clients support this for privacy reasons that don't apply to web crawling. cURL doesn't support it, but lwp-request does."
If "this is not possible" how does lwp-request support it?


Let's be clear about what you're asking requests to do.

A HTTPS request establishes a secure connection between your machine and the origin server. This means you encrypt the connection using TLS, which means you authenticate the origin server. This authentication is done using TLS certificates.

Combining this with proxies is tricky. The proxy can't be the other end of the TLS connection because it doesn't have the correct certificate (nor should it!). This means all you can possibly do is establish a TCP tunnel through the proxy: you connect to the proxy, the proxy connects to the remote end, and then all packets just get forwarded through. This allows you to still perform the TLS handshake. That's exactly what the CONNECT verb is for: it establishes that TCP tunnel.

To do this _without_ the CONNECT verb requires that you make the TLS connection with the proxy instead, then use it like a standard HTTP proxy. This is a bad idea: it represents a man-in-the-middle attack. I can't stress this enough: you _need_ to ensure that you trust Crawlera before doing this.

If you _really_ want to do it, requests should be able to: set the scheme of the proxy URL you pass us to `https` rather than `http`. That will cause us to establish the TLS connection with the proxy itself. 

That means your code changes to:

``` python
import requests
proxies = {"http": "http://USER:PASS@paygo.crawlera.com:8010/",
{"https:": "https://USER:PASS@paygo.crawlera.com:8010/"}

requests.get("https://wikipedia.org", proxies=proxies)
```
SSL verification fails with proxy

Using this code:

```
import requests
requests.get('https://github.com')
```

Throws the following exception:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 382, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: hostname '<Proxy>' doesn't match either of 'github.com', 'www.github.com'
```

where <Proxy> is the URL of the http(s)-proxy I'm using. I have set the environment-variables HTTP_PROXY and HTTPS_PROXY (lowercase, too) to that URL. Is it possible to avoid that error without appending verify=False so I don't have to mess with other projects?


Does the URL to your proxy begin with `https://`? That will cause us to try a TLS handshake with that proxy, which obviously fails as that proxy isn't authoritative for that domain. If this is the case, try setting the protocol to `http://`.


It does start with https://, but setting it to http:// doesn't help.


@looperhacks can you tell us what kind of proxy this is? Also, I doubt it will help, but have you attempted to use the `proxies` argument to `get`?


I don't know the software used, but using a squid-proxy causes the same problem. `proxies` doesn't help either.


When you say switching to `http://` doesn't help, do you mean you're getting the exact same error?


Yes. 


`curl` doesn't seem to accept `http_proxy=https://...`. I can't make Chrome use an HTTPS proxy either. Is this something people use?


It is extremely uncommon, but it does exist and we do technically support it.


Duplicate: https://github.com/shazow/urllib3/issues/531

BTW, there is no support for HTTPS proxies in requests/urllib3, see https://github.com/kennethreitz/requests/issues/1622


Ah, thanks for the reminder @schlamar, I had forgotten that we regressed/removed this.


Not quite, the current behavior is that HTTPS via HTTPS proxy silently degrades to HTTPS via HTTP proxy... ;)
POST to url with unicode character

Hello,

I have this code for making POST request to url with unicode character. I tried to replace `í` to `%C3%AD` but python is still telling me: `UnicodeEncodeError: 'ascii' codec can't encode character '\xf3' in position 1771: ordinal not in range(128)`

```
# coding=UTF-8
import requests

url = u'http://www.madridmovilidad.es/localizador-vehículo.aspx?idioma=es'
payload = {'veh_matricula': '5555HWS'}
url = url.encode('UTF-8')
r = requests.post(url, data=payload)

print(r.text)
```

I'm new to Python, but I have tried all I could find to solve python unicode "problems" but nothing helps. So maybe this is requests problem?

Can anyone help me, please?

Thank you!


I did the following on python 2.7 and 3.4 and had no problems

``` python
url = u'http://www.madridmovilidad.es/localizador-vehículo.aspx?idioma=es'
payload = {'veh_matricula': '5555HWS'}
requests.get(url)
requests.post(url, data=payload)
```

Could you provide the entire stacktrace? I suspect this is actually coming from your calling `print(r.text)`.


@sigmavirus24 You are right. That was because of `print(r.text)` call. Thank you!


@burdiyan the next time you have _questions_ and need help, please use our official Q&A section on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests). You'll get an answer much faster.
set charset in content type if UTF-8 data was encoded

Content type `'application/x-www-form-urlencoded'`, that is automatically provided by requests sometimes, should actually be `'application/x-www-form-urlencoded; charset=UTF-8'` if provided data is UTF-8.


This is incorrect behaviour, I'm afraid.

The `charset` parameter is only defined for documents of type `text/*`. That does not apply to `application/x-www-form-urlencoded`. Therefore, sending `charset` would be a specification violation.

Our behaviour is consistent with the W3C HTML5 specification, which says the following:

> If the form element has an accept-charset attribute, let the selected character encoding be the result of picking an encoding for the form.
> 
> Otherwise, if the form element has no accept-charset attribute, but the document's character encoding is an ASCII-compatible character encoding, then that is the selected character encoding.
> 
> Otherwise, let the selected character encoding be UTF-8.

We can't do either of the first two things because we don't know what the document or form looks like, so we can only do the third. If the user wants a different behaviour (per the first two points) they are required to do it themselves.

I'm sorry we can't accept this, but this is one of those things that seems like it should be simple but is actually quite complex.


OK, lesson learnt.
Added a save method to Response objects

Added an easy way to save content on the File System.

Should I write a test for this? I didn't find a test for `iter_content` so that's why I skipped testing `save`.


Thanks for this @syndbg! Unfortunately, we're in feature freeze, so we won't be accepting this patch. Sorry, and thanks for the work!


No problems, @Lukasa !

Can I help you somehow? I'm assuming a feature freeze is applied to fix bugs. 


If you see any open bugs you'd like to work on, feel free, we are happy to merge those. =)


And in the interest of full disclosure a lot of bugs that are categorized are probably waiting for us to start developing 3.0 (which is not a feature release but something we're doing to break some existing features in a backwards incompatible way). Other bugs may not be entirely decided upon so resurrecting discussions would be a good idea. Please do this to make sure you're working on a bug and not just something we forgot to close.
utils.guess_filename fails if the given parameter looks like a file object but has a non-string name attribute

A cherrypy uploaded file behave like a regular file, except that its name attribute is an int and passing it directly to requests fails because of that

(not sure if i should add myself to AUTHORS for this?)


I'm +0 on this if only because no one else has found this before and it would seem to be a bigger problem otherwise. It seems entirely harmless, but CherryPy seems to be an outlier in how it behaves here. Perhaps this is more appropriately a bug in CherryPy


Yeah, that's weird. Why would its name attribute be an int? I could understand a string representation of a number, but an int? Very weird. 


@arthurdarcet have you reported this as a bug to CherryPy?


Also this causes something of a regression. `name` could still be an instance of `builtin_str` but also be `''` which doesn't cause an exception but introduces backwards incompatible behaviour. Where we previously returned `None`, we will now return `''` which might create issues with what we end up generating with this function.


Poc, which will output `<class 'int'> 11`

``` python
#!/usr/bin/env python
import cherrypy, requests

class Root(object):
    @cherrypy.expose
    def index(self, infile):
        print(type(infile.file.name), repr(infile.file.name))
cherrypy.tree.mount(Root(), '/', {})

cherrypy.engine.start()
requests.post('http://localhost:8080/', files={'infile': open(__file__)})
cherrypy.engine.exit()
```


I've updated the commit to make sure no backward incompatible changes are introduced.

I haven't reported this to cherrypy because they are only specifying that they give a file-like object with a read method, nothing more, and since name isn't part of their API it's not really a bug on their side.

I agree that an int as a file name is weird to say the least, but imho requests should check that file.name indeed quack like a duck before using it like a duck.


@arthurdarcet I respectfully disagree that this isn't a bug on their side.


See also: https://bitbucket.org/cherrypy/cherrypy/issue/1346/uploaded-files-name-is-an-integer


Hi Ian,

Thank you for reporting this to cherrypy, I was planning to but I haven't
found the time to look into it.
I agree that integer file names should be considered a bug on cherrypy side
but I think this patch would only make requests a bit more resilient and is
relevant regardless of cherrypy issue

On Saturday, 13 December 2014, Ian Cordasco notifications@github.com
wrote:

> See also:
> https://bitbucket.org/cherrypy/cherrypy/issue/1346/uploaded-files-name-is-an-integer
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/pull/2379#issuecomment-66860416
> .


I agree @arthurdarcet. I just did not want the cherrypy issue to go unreported.


This makes requests no longer recognize unicode filename as valid under Python 2. See #2411.

EDIT: Actual English sentence!


@sjagoe come again?


Edited my hastily written comment. Opening an actual issue now.
readlines (without generators)

Hello,

I'm moving some Python code from `urlopen` to `requests` (because it will be easier for me to add a db cache (SQLite) mechanism using requests-cache)

The code I want to modify looks like

```
    with urlopen(url) as resp:
        lines = resp.readlines()
```

I don't find any equivalent to `readlines()`

Any idea ?

I have found a generator solution like

```
import requests
import requests_cache

requests_cache.install_cache('cache', backend='sqlite', expire_after=60*5)

response = requests.get('http://finance.yahoo.com/d/quotes.csv?s=AAPL+F&f=l1srs7t1p2', stream=True)

for line in response.iter_lines():
    print(line)
```

but requests_cache doesn't seems to like this stream parameter
see https://github.com/reclosedev/requests-cache/issues/33

So there is 2 solutions (and both can be valuable).
1. fix requests-cache : that's what I'm asking to @reclosedev
2. provide a request readlines() method (or give a tip to have something like that which will return a list and not a generator)

That will be very nice.

Thanks for requests... that's a great lib for humans


We will not be providing a `readlines` method. This is most certainly a bug in requests-cache and one I haven't seen or heard of in [CacheControl](https://github.com/ionrock/cachecontrol) which is our recommended caching library.

For what it's worth you can do:

``` python
import requests

with requests.get(url, stream=True) as resp:
    lines = list(resp.iter_lines())
```

That will give you a list of lines, but it will not fix the requests-cache bug, because that's a bug in that caching library.
AttributeError: 'module' object has no attribute 'python_implementation'

```
Python 2.7.3 (default, Feb 27 2014, 19:58:35) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> req = requests.request('GET', 'http://httpbin.org/get')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/symbol/python2.7.3/local/lib/python2.7/site-packages/requests/api.py", line 48, in request
    session = sessions.Session()
  File "/home/symbol/python2.7.3/local/lib/python2.7/site-packages/requests/sessions.py", line 284, in __init__
    self.headers = default_headers()
  File "/home/symbol/python2.7.3/local/lib/python2.7/site-packages/requests/utils.py", line 552, in default_headers
    'User-Agent': default_user_agent(),
  File "/home/symbol/python2.7.3/local/lib/python2.7/site-packages/requests/utils.py", line 521, in default_user_agent
    _implementation = platform.python_implementation()
AttributeError: 'module' object has no attribute 'python_implementation'
```


@reddypdl can you do `python -c 'import platform; print(platform.python_implementation())'`, if not can you tell us how you built this version of Python? The `platform` module is a standard library module and `platform.python_implementation()` has been around since Python 2.3 so if this is a real version of Python 2.7, then that function should be present. Also is it possible you have a `platform.py` file in the same directory? If so, please rename it.


Thanks  sigmavirus24 , i had a python package called platform in the same folder , so it was picking-up automatically.
"import requests" is very slow on python2 when ndg-httpsclient and pyasn1 are available

Juste after upgrading my laptop to jessie, I discovered a problem with python requests on py2. For the record, I've created a Debian bug report https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=772506

But to summerize the problem (in a fresh jessie docker container):

In a fresh virtualenv (on a jessie box):

```
(tst)root@ead8881e8ed5:/# pip install requests
Downloading/unpacking requests
  Downloading requests-2.5.0-py2.py3-none-any.whl (464kB): 464kB downloaded
Installing collected packages: requests
Successfully installed requests
Cleaning up...
(tst)root@ead8881e8ed5:/# time python -c "import requests" 

real    0m0.087s
user    0m0.044s
sys     0m0.040s

(tst)root@ead8881e8ed5:/# pip install ndg-httpsclient
[...]
(req)root@ead8881e8ed5:/req# time python -c "import requests" 
/req/local/lib/python2.7/site-packages/ndg/httpsclient/subj_alt_name.py:22: UserWarning: Error importing pyasn1, subjectAltName check for SSL peer verification will be disabled.  Import error is: No module named pyasn1.type
  warnings.warn(import_error_msg)
/req/local/lib/python2.7/site-packages/ndg/httpsclient/ssl_peer_verification.py:24: UserWarning: SubjectAltName support is disabled - check pyasn1 package installation to enable
  warnings.warn(SUBJ_ALT_NAME_SUPPORT_MSG)
/req/local/lib/python2.7/site-packages/ndg/httpsclient/subj_alt_name.py:22: UserWarning: Error importing pyasn1, subjectAltName check for SSL peer verification will be disabled.  Import error is: No module named pyasn1.type
  warnings.warn(import_error_msg)

real    0m0.100s
user    0m0.080s
sys     0m0.012s

(req)root@ead8881e8ed5:/req# pip install pyasn1
[...]
(req)root@ead8881e8ed5:/req# time python -c "import requests" 

real    0m0.769s
user    0m0.712s
sys     0m0.052s
```

Notice the last time measurement (almost 800ms), which is very painfull since I use requests in cmdline utilities and mercurial extensions...

David


Your debian bug report blames OpenSSL, and shows that the time is lost in yacc. This suggests to me that it should only happen the first time: parsing shouldn't be required on initialization after the first time. Is that right, or does it happen consistently?


It happens consistently (otherwise it would not be an issue).


Interesting. The Debian bug shows the time is in `cryptography`. Ping @alex?


Is this is reproducible just by doing `time python -c "import OpenSSL"`? If so, that would indicate that this is a problem with that package, not requests. (And we have no alternatives to be able to provide SNI for old versions of Python.)


Yes, python -c "import OpenSSL" is the slow fellow. And it's been reported already https://github.com/pyca/pyopenssl/issues/137

So I guess web can close this issue.
Set connection timeouts for HTTPS proxy attempts.

 Fixes #2336


@kevinburke Wrong repo. ;) Merge to urllib3 first, then we'll take the fix.


Sorry, that wasn't clear, it's been a long day (just got off a flight, LHR -> SFO).

What I meant was: we'll bring in a whole urllib3 when we move to the next release, so we shouldn't merge an individual fix.
api.dribbble.com SSL certificate does not validate

Built something with dribbble's API recently and noticed this.
This very may well be the fault of dribbble, but I don't know enough to diagnose. It works in browsers and curl, so it seems something's up with requests.

ssllabs shows the cert presented as a wildcard certificate https://www.ssllabs.com/ssltest/analyze.html?d=api.dribbble.com

However when connecting with requests 2.5.0 on python 2.7.8 an `SSLError` is raised

`SSLError: hostname 'api.dribbble.com' doesn't match either of 'dribbble.com', 'www.dribbble.com'`


(Pdb) self.sock.getpeercert()
{'notAfter': 'Mar 26 12:00:00 2015 GMT', 'subjectAltName': (('DNS', 'dribbble.com'), ('DNS', 'www.dribbble.com')), 'subject': ((('businessCategory', u'Private Organization'),), (('1.3.6.1.4.1.311.60.2.1.3', u'US'),), (('1.3.6.1.4.1.311.60.2.1.2', u'Massachusetts'),), (('serialNumber', u'001031096'),), (('streetAddress', u'Ste. 202'),), (('streetAddress', u'16 Front St.'),), (('postalCode', u'01970'),), (('countryName', u'US'),), (('stateOrProvinceName', u'Massachusetts'),), (('localityName', u'Salem'),), (('organizationName', u'Dribbble LLC'),), (('commonName', u'dribbble.com'),))}

So, the ssl package is telling us it does not have a wildcard. This is really strange because both ssllabs and symantec(https://ssltools.websecurity.symantec.com/checker/views/certCheck.jsp) are saying it presents as a wildcard.

I think there's something up with dribbble because I can send requests to soundcloud over ssl just fine. 


I'm travelling, so I can't debug this properly, but my first instinct is that this is an SNI issue. Does it work fine using requests + Python 3?


So that's also my first instinct @Lukasa but I don't think `getpeercert` would remove a wildcard `subjectAltName` entry randomly. Still, I'd like to know from @EthanBlackburn or @buttscicles what happens on Python 3 (preferably 3.3 or 3.4).


```
Python 3.3.5 (default, Sep 29 2014, 15:50:30)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__version__
'2.5.0'
>>> requests.get('https://api.dribbble.com')
<Response [200]>
>>>
```


Yeah... so it would seem our intuition was correct. Can you try again on python 2.x after doing `pip install -U requests[security]`?


We have a winner! SNI is the problem here.

You can fix this problem on Python 2 by following the [SNI instructions here](https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484).


I beat @Lukasa to commenting on an issue! Holy smokes!


The `-U` flag didn't seem to work for the bundle, I had to uninstall and then reinstall, but that did work, thank you!


In the future there's also `--force-reinstall` :)
requests.post() issues a GET with verify=False on insecure ssl environment

I did a search to see if I could find any similar issues prior to filing the bug. It seems like something within the way urllib3 is behaving, in particular, but I'm not sure.

I have an API endpoint where GET is disabled so only POST and OPTIONS are valid methods. It works fine when I interact with it when it's hosted on HTTP but when I host it over HTTPS, the response is a 405. It behaves the same if I'm using either `requests.post()` or `requests.Session().post()`. 

Specifically, here's an example of the request / response cycle:

```
>>> resp = requests.Session().post(url, headers=self._headers, verify=False)
>>> resp.content
'{"code": 405, "total": null, "version": 1, "errors": [{"message": "Method \'GET\' not allowed."}]'
```

On an HTTP environment, with the same call (including `verify=False`), I get the expected response. I've also been able to successfully hit the same URL with the same headers using Postman and cURL without issue on the HTTPS server.

Is this a bug with the library or am I doing something wrong?


What is the output of `resp.history`. It's plausible you're being redirected and the status code we're getting is causing us to switch the request to a `GET` (as we should be).


Ah, okay. Thank you for the pointer. In history was a Response [301] and I traced it to the fact that nginx was redirecting me from http:// to https:// with the response. I should have used HTTPS to begin with. Thanks for the help!
Release v2.5.0

TODO:
- [x] Tag release
- [x] Upload to PyPI
- [x] Create release on GitHub


:cake: LGTM.

Ready when you are @kennethreitz.


:shipit: 


@kennethreitz wanted to get this out today and added both of us to PyPI as maintainers so I'm going to wrap this up right now.


- [Tag](https://github.com/kennethreitz/requests/releases/tag/v2.5.0)
- [Release](https://github.com/kennethreitz/requests/releases/tag/v2.5.0)
- [PyPI](https://pypi.python.org/pypi/requests/2.5.0)


omg you guys are the best


:sparkles: :cake: :sparkles:
Update tests to work offline

There are still some that we cannot force to work offline but this improves a lot of things.


Some of the tests you port to `httpbin()` use `http` and others use `https`, is that significant?


@alex those tests that were using https didn't seem to have any necessity to rely on https. They were likely written by me and I just write `'https://httpbin.org'` everytime I want to talk to httpbin by force of habit.
Reduce bus factor

Keys to the kingdom for @Lukasa and @sigmavirus24.

This includes....
- [ ] DNSimple collab access]
- [x] PyPi collab access

I think that's it...


What about ci.kennethreitz.org


@kevinburke hmm, i suppose so . 


Op on the IRC channel as well


@kevinburke if someone tells me what command to type in, i'll do it lol


I'm not sure we need op on the channel. I've never seen any problems arise in there. 

```
/msg ChanServ FLAGS #python-requests lukasa +voAti
/msg ChanServ FLAGS #python-requests sigmavirus24 +voAti
```

Should allow @Lukasa and I to OP ourselves with ChanServ when necessary. Ideally not having OP on by default will be a positive impact on the channel's atmosphere 


@sigmavirus24 now has access to dnsimple. @Lukasa, can you tell me your dnsimple email address? 


Just sent you an invite anyway — hopefully it works :)


"lukasa is not registered."

sigma is good. 


@kennethreitz Sorry, used a different one: lukasaoz@gmail.com.

Always best to have 8 million email addresses.


cool, i'll say this is done now. 


CI is still something to think about...


Yeah. I'm curious why that's no longer building PRs or anything. If you want, I can set up ci.sigmavir.us to do CI here for now, but I'll need access to settings (or will need to sync up with @Lukasa about adding it)
Combining pool_block == True with head blocks

I am using requests to read redirection targets from 302 responses for a list of urls.

The core function is:

```
def getRedirect(url):
    print('requesting: %s' % url)
    request = reqsSess.head(url)
    print('done: %s' % url)
    request.raise_for_status()
    location = request.headers['location']
    request.close()
    return location
```

To avoid I/O lock, I am using gevent to spawn a separate `getRedirect` for all urls in my list (about 1000) and wanted to use requests' adapter rate limiting. Here I limited to one connection for testing purposes, but obviously the idea was to raise this once confirmed the it works. (Note, raising to 2 makes no difference to the issue.)

```
import requests
connection_limit = 1
adapter = requests.adapters.HTTPAdapter(pool_connections=connection_limit,
                                        max_retries=3,
                                        pool_block=True)
reqsSess = requests.session()
reqsSess.mount('http://www.mydomain.com', adapter)
```

The problem is, that I only see "requesting: url1", etc only eight times. Then the script blocks. Am I doing something wrong? I thought this might be related to #1967 but adding `request.close()` did not help.


Additional observations:

With `connection_limit = 1` and `pool_block=True`, about 6-8 (changing) `requesting: url...` statements are printed. Wireshark shows only a single DNS request and response. **No http requests of any kind are transmitted.**

With `connection_limit = 2` and `pool_block=True`, exactly 11 `requesting: url...` statements are printed. Wireshark shows only 10 DNS requests and responses. **No http requests of any kind are transmitted.**

With `connection_limit = 1` and `pool_block=False`, `requesting: url...` statements are printed interspersed with the warning that the connection pool is full. **Wireshark shows http HEAD requests are transmitted and responses received.**
about GET request

In requests, GET request need "params = request_body", but POST (or ohter) request  need "data=request_body",
so 
i think about is let the request data together.
i know HTTP BODY and URL params need a different handler, but we can let this handler in send_request function (means use URL or BODY).
and
it's there any "design pattern"  about using "params" and "data" ? 
thx.

file: api.py line: 23-24


Nothing prevents you from using both `params` and `data` simultaneously. I'm not sure I understand what you're asking. Regardless, it seems as though you're asking a question and this is not a forum it is an issue tracker. Questions belong on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).


sorry for my english. but i gotcha.
I just forget POST can upload file (bytes) also.
 thx.
work around for dealing with a multipart/related multipart upload instea...

...d of a multipart/form-data. Was trying to integrate file upload support for the HipChat v2 API with will and discovered that requests can't deal with the type of multipart request their upload api requires.

https://www.hipchat.com/docs/apiv2/method/share_file_with_room

I think this approach is a complete mess... but not sure about the best way to go about dealing with it.


Hey @netjunki,

We're not expanding the API any longer. Requests is under a feature freeze unless Kenneth blesses it for inclusion (which is rare and highly unlikely). You could probably subclass the `MultipartEncoder` from the requests-toolbelt and achieve what you're looking for here pretty easily.

Thanks for the pull request
is there any way to make requests waiting for ajax to respond?

a site (http://echo.msk.ru/) is responding to the request in a second. However, with a simple request.get I cant put a hand on the site content. For sure, I can use selenium for that, but is there any simple way to make requests wait for answer?

thanks in advance


requests does not wait for AJAX because it doesn't read HTML. You need something that will read the HTML of the site to learn where the Javascript is, then obtain it and execute it. requests is not built to do that, though it could be a _component_ of doing it.

My recommendation would be to use the browser developer tools to see if you can make the AJAX request yourself in a programmatic way.


thanks, I got it. Yeah, I usually use Selenium lib for python to solve this
Added ability to use list of proxies for each scheme. Proxies are randomly selected.

Basic Usage:

``` python
import requests
r = requests.Session()
p =  {'http':['proxy1.com:81', 'proxy2:82', 'proxy2:82'], 'https': 'proxy4.com'}
r.proxies = p

print r.get('http://ya.ru')
print r.get('http://google.com')

print requests.get('http://ya.ru', proxies=p)

print r.get('https://www.google.ru')
print requests.get('https://www.google.ru')
```


I'm not sure I understand how this is supposed to work.


Currently, we are can use only one proxy for scheme, but some sites have limits on the number of requests from 1 IP (per second, minutes, etc)
If we make a spider for the site - we will have big problems (
This function allows you to immediately set large list of proxies for the scheme(protocol) and use random for each request (without code changes).
Example without it this changes:

``` python
import requests
import random
r = requests.Session()
p =  {'http':random.choice(['proxy1.com:81', 'proxy2:82', 'proxy2:82']), 'https': 'proxy4.com'}
r.proxies = p
print r.get('http://ya.ru')
# we are must change proxy for next request
p['http'] = random.choice(['proxy1.com:81', 'proxy2:82', 'proxy2:82'])
r.proxies = p
print r.get('http://ya.ru')
```


So no one has ever requested this before and I haven't found anyone else searching for this on stackoverflow. Given that this is such a niché feature, we won't be accepting this. The good thing is that you already have the code to make this work (for the most part). What you want is a class that's either subclasses `dict` or `collections.MutableMapping`. It will contain allow us to do `.get('http://')` for example and the class will give us one value that you choose to give us.


Thanks for the answer. I just proposed a variant)
Properly serialize RecentlyUsedContainers for cache

RecentlyUsedContainers are threadsafe so they require a lock and as such
cannot be serialized with pickle directly. To handle it, we need to
convert it to a dictionary first and then back when deserializing.

Fixes #2345


looks good! any way to add a test exposing this issue?


> any way to add a test exposing this issue?

Besides the test that failed in the first place? Jenkins isn't running for a reason only @kennethreitz can determine. Had @Lukasa or I been running the tests locally before approving PRs, it probably wouldn't have reached this point.


+1 LGTM.
feature request: add a timeout to DNS lookups

I don't think it's possible. But it would be nice.

Currently if:
1. you set your DNS resolvers to something that's not listening on port 53, eg 123.123.123.123
2. you try to make a request with a timeout

The request will use the system timeout, usually upwards of 2 minutes.

It would be nice to be able to attach a timeout to DNS lookups. Go attaches one for example by resolving DNS in a goroutine with a thread timeout.


in theory you could run the socket.connect() in a thread and time it out


Given that requests doesn't handle sockets itself, I think this is a feature you should be requesting in urllib3. Before you open an issue there, I wonder if @shazow would care to express their opinion here first.


I would love to have the ability to include a DNS timeout, but spawning a separate thread is not a viable option (for urllib3, at least).


So something that should probably be pushed upstream to bugs.python.org as a feature request and will have to be a 3.5+ only feature. 


okay i submitted http://bugs.python.org/issue22889

## 

Kevin Burke
phone: 925.271.7005 | twentymilliseconds.com

On Sun, Nov 16, 2014 at 9:19 PM, Ian Cordasco notifications@github.com
wrote:

> So something that should probably be pushed upstream to bugs.python.org
> as a feature request and will have to be a 3.5+ only feature.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2347#issuecomment-63261047
> .
Uploading Binary Files Without Reading Into RAM

hi there. 

i try to upload a file via this:
http://docs.python-requests.org/en/latest/user/advanced/#post-multiple-multipart-encoded-files

but it does seem to read it into ram first. because i get memory errors on my low ram vps.

it would be good if it would have worked like this:
http://docs.python-requests.org/en/latest/user/advanced/#streaming-uploads

my example code:
import requests
with open('example.rar', 'rb') as f:
    r = requests.post('https://anonfiles.com/api/v1/upload', files={'file' : f})

print r.text


If you look at http://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file you'll see that we mention this limitation and recommend [requests-toolbelt](/sigmavirus24/requests-toolbelt). There's documentation on that project and it should be simple to use. Any problems encountered while using it should be filed against that project.


oh sorry i'm blind. thnx for the support
have a nice day


@daft117 Happy to help.
Using RecentlyUsedContainer for session cache breaks pickling

The changes in 387c8f8 have broken the ability to pickle a Session object.

Possible solutions:
- Re-implement `RecentlyUsedContainer` to not use threading locks
- Do not attempt to pickle the session redirect cache (personally, I prefer this one)

Reasons this wasn't caught earlier:
- Jenkins has magically stopped working
- No one has been running the suite locally before merging/approving changes


Note to @kennethreitz, we **can not** do a release until we fix this.


So we can't just remove `redirect_cache` from the list of `__attrs__`, it breaks the tests. So ... I'm thinking we have to special case how we serialize a `RecentlyUsedContainer`. We can call `dict` on the cache, but to repopulate it, we need to look over the items and then set them individually on the container which seems suboptimal. For now, we can do that until we maybe get something merged upstream in urllib3. The difficult thing is that the container is threadsafe, so I imagine an `update` method wouldn't be fit it well.


oops. :(


@mattrobenolt Not your fault, our CI was busted so you had no way to know. =)


Also, we told you to take this approach. Not your fault in the slightest.
Partially addresses Issue #1572

Addresses the LocationParseError but not the DecodeError from kennethreitz#1572. When running
test_requests.py, I got an error in test_session_pickling which resulted in a TypeError. I'm not sure of the reason for the TypeError but I have commented out that test.


Why have we commented out a test here?


@Lukasa it's a test failure on master apparently. Besides that, I'm -1 on this. A `LocationParseError` happens long before a connection is even attempted. This should be an `InvalidURL` error


Also the test failure was introduced sometime since 122c92e59002cad20c29be4e9ec9d4f6d50a1ef6


Since it's complaining about Locks not being picklable, I suspect this has something to do with 387c8f8 since the `RecentlyUsedContainer` uses threading's locks. One solution is to not try to pickle the redirect cache. That said, this discussion belongs in its own issue.


@sigmavirus24 If I've understood you correctly, I have changed the ConnectionError to an InvalidURL error.


@ContinuousFunction could you amend your commit messages? Two commits in a row with the same message is incredibly unhelpful when looking at the log.


@sigmavirus24 sorry about that. I've changed the message.


@ContinuousFunction can you rebase this, uncomment the test that you commented, and ensure the tests pass. Then force-push to this same branch?


@sigmavirus24 I believe I have made the appropriate changes.


Thanks @ContinuousFunction I'll run these tests now and make sure they pass before merging.


There was a test failure on Python 3 because `urllib3.exceptions.LocationParseError` doesn't have a `message` attribute. That's fixed in bd3cf95e34aa49c8d764c899672048df107e0d70.

Thanks @ContinuousFunction !


@sigmavirus24 My pleasure!
chardet ImportError: cannot import name universaldetector

Hello,

When I try to retrieve the text attribute from a request, i get the following error:
Traceback (most recent call last):
  File ".\sublime_plugin.py", line 175, in on_load
  File ".\sublime_plugin.py", line 154, in run_timed_function
  File ".\sublime_plugin.py", line 174, in <lambda>
  File ".\SublimeYouCompleteMe.py", line 244, in on_load
  File ".\plugin\ycmd_request.py", line 136, in send
  File ".\plugin\ycmd_request.py", line 43, in post_data_to_handler
  File ".\plugin\ycmd_request.py", line 109, in json_from_response
  File ".\requests\models.py", line 756, in text
    encoding = self.apparent_encoding
  File ".\requests\models.py", line 637, in apparent_encoding
    return chardet.detect(self.content)['encoding']
  File ".\requests\packages\chardet\detect.py", line 11, in detect
ImportError: cannot import name universaldetector

On python 2.7.8

I think this is an error in chardet's detect function in **init**.py, which does an import on function level instead of module level.
The import is "from . import universaldetector" and since it is called from requests/models.py it searches in the requests folder for universaldetector, which obviously fails.
I confirmed this by changing the line to "from packages.chardet import universaldetector" which hilariously solves the problem.

But I wonder if this bug is down to my specific configuration, because it seems like a bug that would affect a lot of people?


This is simply because you're using requests inside a sublime package and sublime has it's own weird Python that it ships with (if I remember correctly).


Ah thanks for the insight.
I expected something like that, because it would affect too many people otherwise.
I do know the encoding beforehand so I've used that to avoid importing the chardet module.
Thanks again!


No worries @ivankoster
"keep-alive" over a HTTP/1.0 request issue

Debugging a nasty "300: Bad request" error I noticed an invalid behaviour when requests-2.4 is used in combination with HTTP/1.0.

Tring to reproduce the issue this is what I found...

``` python
import httplib
import requests

# Workaround for the IncompleteRead issue.
# See http://bugs.python.org/issue14044
httplib.HTTPConnection._http_vsn = 10
httplib.HTTPConnection._http_vsn_str = 'HTTP/1.0'

r = requests.get('https://www.google.com')
r.request.headers
```

On requests-2.4.3 (from PyPi) the request headers are

``` python
{'Connection': 'keep-alive',
'Accept-Encoding': 'gzip, deflate',
'Accept': '*/*',
'User-Agent': 'python-requests/2.4.3 CPython/2.7.8 Linux/3.16.0-24-generic'}
```

On requests-2.3.0 (Ubuntu 14.10 official repos) the request headers are (notice no keep-alive here)

``` python
{'Accept-Encoding': 'gzip, deflate',
'Accept': '*/*',
'User-Agent': 'python-requests/2.3.0 CPython/2.7.8 Linux/3.16.0-24-generic'}
```

Reading the documentation I guess HTTP/1.0 is not supported by the requests library but I'd like to suggest at least a nice WARN log message to inform the result can be weird.

Some setups don't seem to like the _invalid_ combination "keep-alive" in the headers and HTTP/1.0 and even if in most cases everything is working fine at least once I received a "300: Bad request" error.


Sorry, this is a bit unclear to me. Are you explicitly setting the version string on all `HTTPConnection`s? You can override the `Connection: keep-alive` header, like so

``` pycon
Python 2.7.8 (default, Oct 19 2014, 16:03:53)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> requests.__version__
'2.4.1'
>>> r = requests.get('https://httpbin.org/get', headers={'Connection': None})
>>> r.request.headers
{'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.4.1 CPython/2.7.8 Darwin/14.0.0'}
```

If you use a session you can do:

``` pycon
>>> s = requests.Session()
>>> s
<requests.sessions.Session object at 0x10e569dd0>
>>> s.headers
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.4.1 CPython/2.7.8 Darwin/14.0.0'}
>>> del s.headers['Connection']
>>> r2 = s.get('https://httpbin.org/get')
>>> r2.request.headers
{'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.4.1 CPython/2.7.8 Darwin/14.0.0'}
```

Technically we were supposed to be sending this header since 1.0 (and we were at some point) so this isn't something we should need to warn most users about. We also don't introspect httplib at any point to determine if you've overridden any of the values because 99% of our users don't do that. And if they do, they know that if they get unexpected results it isn't our fault for expecting something different.
Create a response object from archived string or file HTTP responses.

So I'm working with warc files. Warc files are a storage medium for saving and archiving websites, so they contain plenty (possibly thousands) of text HTTP request / response pairs. I need a way to parse these responses into usable objects.

Extracting the text of the HTTP response from the warc file is easy enough, but I can't find a decent python library that will convert that response text into a response object. About the best I can find is this:

http://stackoverflow.com/questions/24728088/python-parse-http-response-string

which requires me to set up fake server connections using a really ugly library. As a work around it kinda works, but it would be nice to work with something a bit more human. 

In my field the internet archive is growing in importance and I can see this issue coming up for more people than just me. It would be a nice feature to have.


This feature really belongs in another library altogether. The problem is that the HTTP parsing logic in requests is in `httplib`, which is well down our stack and outside our control. This means we can't pull it out into common code.

_However_, @dstufft has been working on a formal HTTP parsing library which might be usable as a base for this use-case. What do you think, @dstufft?


Uh, yea you can parse that with my http11 lib, but it has no docs currently and would require some assembly right now I think. In particular I don't think it actually parses bodies currently.


I agree. This belongs in another library. If that library chooses to interact with requests that'd be great and they could probably take a similar approach to [betamax](/sigmavirus24/betamax). I'm happy to advise, but this isn't a bug and it's at this point a rejected feature.

Thanks @recrm for starting this discussion.