Raibek/requests-github-issues huggingface dataset:
Content-Encoding: Gzip not automatically decoded in 2.6.1

When using requests on python 2.7.8 ubuntu 14.10 from pypi, the current version 2.6.1 exhibits different behaviour from 2.6.0. Response.content is a gzipped binary string, instead of a decoded unicode string in 2.6.0. Is this change in behaviour intended? Running this script in both 2.6.0 and 2.6.1 shows the difference.

```
import requests
import gzip
import json
from StringIO import StringIO

resp = requests.get('https://liquidstate.atlassian.net/rest/api/2/serverInfo')
try:
    print 'Reading as JSON'
    print resp.json()
except Exception:
    print 'Failed to load JSON, reading as Gzip'
    print 'Reading as Gzip'
    content = StringIO()
    content.write(resp.content)
    content.seek(0)
    gz = gzip.GzipFile(fileobj=content)
    print json.loads(gz.read())
```

Under version 2.6.0:
Reading as JSON
{u'versionNumbers': [6, 5, 0], u'serverTitle': u'JIRA', u'buildDate': u'2015-04-08T00:00:00.000+1000',        u'scmInfo': u'0dc1796b325ae7e85b1dca228885a3417f39097a', u'baseUrl': u'https://liquidstate.atlassian.net', u'buildNumber': 65000, u'version': u'6.5-OD-01-120'}

And under version 2.6.1:
Reading as JSON
Failed to load JSON, reading as Gzip
Reading as Gzip
{u'versionNumbers': [6, 5, 0], u'serverTitle': u'JIRA', u'buildDate': u'2015-04-08T00:00:00.000+1000', u'scmInfo': u'0dc1796b325ae7e85b1dca228885a3417f39097a', u'baseUrl': u'https://liquidstate.atlassian.net', u'buildNumber': 65000, u'version': u'6.5-OD-01-120'}


Thanks for this!

This is actually a regression in urllib3, so I've raised an issue there (shazow/urllib3#593). We'll fix it up and release a fix shortly. =)


This is likely to be an issue for many people. Thanks for fixing!


Was broken for us as well this morning (used by our product's CI tests), workaround was to revert to 2.6.0. Thanks for prompt fixing!


:+1: GitHub really needs a silent +1 button, so people can acknowledge that we've seen a bug, without polluting the comment space. 


@JonathonReinhart no joke. Feel free to [Contact](https://github.com/contact) them though and tell them that. I know I have in the past and the more people who ask for it, the better our chances of getting it (I would hope).


I just released 2.6.2 to PyPI with the fix for this issue. Please upgrade as appropriate.


Thanks for the quick fix!
Add contribution guidelines

https://github.com/benoitc/gunicorn/blob/master/CONTRIBUTING.md


Presumably this document will be pretty similar to the content of [these](http://docs.python-requests.org/en/latest/dev/philosophy/) [two](http://docs.python-requests.org/en/latest/dev/todo/) pages.


Thoughts on adding something akin to [this](https://github.com/sigmavirus24/requests-toolbelt/blob/master/CODE_OF_CONDUCT.rst) as well?


Also the bit in the [Standard Library](http://docs.python-requests.org/en/latest/dev/philosophy/#standard-library) section about having just reached v1.0.0 is a bit ... dated.


Oh also our [Runtime Environments](http://docs.python-requests.org/en/latest/dev/todo/#runtime-environments) list is very wrong. PyPy 1.9 is very old and dated. We don't actually test against it either and I don't think we support 3.1 or 3.2 either. =(


simpler than pledging, i think

"be cordial, or be on your way", "adhere to our philosophy", etc.


@kennethreitz In that case, what do you think of [this doc](https://hyper.readthedocs.org/en/latest/contributing.html).


@Lukasa wonderful!


Alright, basic proposal is in #2572.


With #2572 merged, I'm closing this. :cake:
Make an official logo and stickers for it

Proposed logo:

:sparkles: :cake: :sparkles:


If we could get a designer to render it in a way that wasn't just emoji, this'd be perfect.


/cc @kennethreitz 


This would be the second emoji logo stick I have!

:muscle: :computer: :us: 


YES!


`>>> import requests`
:sparkles: :cake: :sparkles:


the github3.py stickers are some of my favorites. 


Yeah the Digital Services stickers are the reason why I thought of :sparkles: :cake: :sparkles: stickers as a real good idea. I still know the designer who did the github3.py stickers. I can see if she's available for hire to do this. I'll happily handle paying for the logo.


When are you looking for this to be done?


`¯\_(ツ)_/¯`


@missingdink whenevs :)


:+1: :hankey: :muscle: 


@missingdink :cake: 


Does James Halliday take commissions? I adore his readmes http://substack.net/art


@missingdink actually came through @hickford. We just need to revamp our stuff with the new logo.
requests does not respect the timeout set by socket.setdefaultimeout

Steps to reproduce:

``` py
import socket
import requests

socket.setdefaultimeout(0.0001)
# Expected this to raise exception, but it does not.
requests.get('http://google.com')

# These 2 raise an exception
requests.get('http://google.com', timeout=socket.gettimeout())
requests.get('http://google.com', timeout=socket._GLOBAL_DEFAULT_TIMEOUT)
```

Looks like the default "None" value is propagated to urllib3, which has `socket._GLOBAL_DEFAULT_TIMEOUT` set as the default timeout.

Is this by-design or a bug?


Duplicate of #2228. Please check open issues in the future for possible duplicates.
Why does the PreparedRequest.prepare method provide None as the default parameter value for those who cannot be None? 

Hi,
I just inspect some test cases of your project, and find some test cases like

``` python
def test_prepare_unicode_url():
    p = PreparedRequest()
    p.prepare(
        method='GET',
        url=u('http://www.example.com/üniçø∂é'),
        hooks=[]
    )
    assert_copy(p, p.copy())
```

When I remove some parameters for calling the prepare method, it will raise exceptions(with url being missed) or traceback(with hooks being missed). 
Is it unreasonable to provide None default parameter for some arguments cannot be None? 
(A related PR is #2301).

Thanks. 


I see no good reason to leave those as `None`-able defaults. I think we could get away with patching this in a minor release, because while it's _technically_ backward-incompatible, in practice I don't think anyone can be correctly relying on this behaviour. @sigmavirus24?


Personally, I like the idea of more or less _suggesting_ that users specify the arguments by keyword. If anything we should be documenting the fact that some of these parameters need to be provided. That said, I don't like that we have to pass hooks as an empty list so I'm sending a patch with a fix for that.
EditorConfig

Previously proposed and declined in #893.

I am opening this because I think it is time to reassess whether EditorConfig is appropriate for this project.

I did some GitHub diving and found the following Python projects which use EditorConfig:
- [flask-wtf](https://github.com/lepture/flask-wtf)
- [cookiecutter-django](https://github.com/pydanny/cookiecutter-django)

I also found the following Python-powered websites that use EditorConfig:
- [MDN](https://github.com/mozilla/kuma)
- [gratipay](https://github.com/gratipay/gratipay.com)
- [mozilla.org](https://github.com/mozilla/bedrock)

This file is a human-readable auto-enforced whitespace style guide for all files (_not just Python_).

For:
- **users of editors with built-in support** (e.g. PyCharm, Coda 2, SourceLair) this file will ensure the code style is enforced automatically.
- **users with a plugin installed** (available for most major editors) this file will also auto-enforce the code style
- **all other users** this simply serves as a white space style guide.


Thanks for this!

I do not believe that EditorConfig is appropriate at this time. In general, I resent the relentless expansion of dotfiles that various tools require people to add to the project root. I appreciate that these tools are close to many people's hearts, and that they provide value for plenty of people, but it's extremely difficult to justify any tool that is not actually considered 'enforced' by some form of automated tooling. The risk is that if we accept this it becomes difficult to reject it for other tools, and we rapidly end up with an expansion of dotfiles that have no love.

For this reason, I'm afraid I don't think I can accept this. =( Sorry!
Recommended packages.

Beginnings of work to have a list of recommended third-party (fourth-party?) packages.

@sigmavirus24 @kennethreitz feel free to push on top of this branch to add other things if you want.


@kennethreitz added the requests-toolbelt to the sidebar a while back. Would it be best to move it here?


Maybe have it here as well?


Let's replace the toolbar link to a link to this page. 


`==kennethreitz`. I'll push an update for that tonight I think


@sigmavirus24 send another pr with that :)
Cookies issue causing segfault still in 2.6?

FYI I think there might still be a cookies related issue... I was having issues two weeks ago with cookies.py in 2.5.3 i think, but I have downloaded the new requests 2.6 running on raspberry pi for a datalogger every minute, and it's much better but now it segfaults every 24 hours or so in a slightly different place.  (FYI I'm not very competent on python debugging yet)  

Traceback show's it segfaulting on line 241 in python2.7/rfc822.py (it shows as line 243 in my screenshot but I had already added a couple of lines above for debugging - it is whatever line "for line in self.headers" is on in getAllMatchingHeaders), invoked ultimately from requests/cookies.py.

The process is just repeatedly loading a webserver.html page on a PLC we have on our intranet, to read it's variables, so it's not going over the internet.

When running, normally the 'lst' variable is always returned as [] so I dont even need this procedure and for now I've bypassed the "for line in self.headers" bit to keep it from segfaulting once a day.

If you want more info do please let me know what you need.

And of course, thanks for the great work.

![screen shot 2015-04-12 at 08 17 02](https://cloud.githubusercontent.com/assets/10351369/7104822/fce30c78-e0f1-11e4-8d40-ea12fb032df0.png)


Or is this just a python2.7 bug?


@11phc it is just a python2.7 bug. I would check bugs.python.org to figure out which version number has the fix. If you can, I would update your version of python to 2.7.9 on your Pi.

Cheers,
Ian


ok thanks very much Ian.  I was on version 2.7.3, but unfortunately going to Python 2.7.9 still doesn’t seem to fix it.  I might have to go to Python 3.x I fear.

piers

On 12 Apr 2015, at 12:21, Ian Cordasco notifications@github.com wrote:

> @11phc it is just a python2.7 bug. I would check bugs.python.org to figure out which version number has the fix. If you can, I would update your version of python to 2.7.9 on your Pi.
> 
> Cheers,
> Ian
> 
> —
> Reply to this email directly or view it on GitHub.


Sorry…. I’ve got another segfault now in requests/api.py which I think probably is in requests, right? 

python 2.7.9
requests 2.6.0

its a data logger program, so it just happens once every 24 hours or so, after doing a requests call every minute.

anything useful I can do to isolate it?
Shall I open another issue in github?

thanks

On 12 Apr 2015, at 12:21, Ian Cordasco notifications@github.com wrote:

> @11phc it is just a python2.7 bug. I would check bugs.python.org to figure out which version number has the fix. If you can, I would update your version of python to 2.7.9 on your Pi.
> 
> Cheers,
> Ian
> 
> —
> Reply to this email directly or view it on GitHub.


Can you provide any extra information here first? A traceback? More details about how you're using requests? Anything that we could possibly use to help you.
Add a timeout to select(), otherwise we can block forever on a bad socket

All the other select.select() calls in this module use a timeout. Without this bugfix, a server that fails to complete the handshake will block forever. We've seen this in production.


Can you open this pull request on urllib3? All your code changes are there.


Please re-open this issue. I just checked `urllib3` and it has this same fix in it so `requests` needs to update your code from urllib3.


It appears great minds think alike https://github.com/shazow/urllib3/commit/14d09fc8fd9df697e534e8e10cbe6f913289c748


Indeed. However, there's no need to re-open the issue, we'll bring a new urllib3 version before our next release.
Fix MisssingSchema error message does not handle unicode input correctly #2540


I think this won't work on Python 3 with a binary URL, though I need to test that.


You rigth. This add 'b' to url. 

```
requests.exceptions.MissingSchema: Invalid URL b'example.com': 
No schema supplied. Perhaps you meant http://b'example.com'?
```


Whether it is necessary to display  URL completely in error message?
Will be enough only hostname?
For example, instead of the:

```
requests.exceptions.MissingSchema: Invalid URL 'example.com/page?query=blabla&foo=23': 
No schema supplied. Perhaps you meant 'http://example.com/page?query=blabla&foo=23'?
```

Output this:

```
requests.exceptions.MissingSchema: Invalid URL 'example.com/...': 
No schema supplied. Perhaps you meant 'http://example.com/...'?
```

Then we can solve this bug with applying IDNA to the hostname.


The better thing to do might be to call `to_native_str`, from our utility package.


This work in both Python's versions.


Can you wrap the line at 79 characters?


Thanks @sh1buy! :sparkles: :cake: :sparkles:
Document filtering of 'None' values for url-encoded parameters

Currently, when passing a dictionary/list of pairs to the request constructor's `params` parameter, any 'None' values are filtered when encoding the request url. If this is intended behavior or not something that will change, then this should be documented (actually, even if it is subject to change, it should probably by documented anyway so that people don't rely on it)


It already is documented. =) See [this block](http://docs.python-requests.org/en/latest/user/quickstart/#passing-parameters-in-urls).
Fully disable cookie handling in Sessions

This is a feature request, not a bug.

When creating a new `Session` in order to issue multiple requests, one can specify a `CookiePolicy` to reject all cookies (e.g. [this way](http://stackoverflow.com/a/21714597/2250036)). But for each new request, a new `RequestsCookieJar` is created, with a `DefaultCookiePolicy`, where previous cookies are merged.

Since there are no previous cookies, the behaviour is correct (no cookies are accepted), but the creation of those objects is costly and leads to a decreased performance. Working around this issue, with a subclass of Session that does not perform any cookie handling, gives a speedup of 1.32x or more, in our measurements.

Furthermore, it is a bit surprising that `Session` imposes this object creation to users, even if the user has specified that no cookie handling is needed (by setting the custom `CookiePolicy` after initialization), and even for a single request that does not create a `Session` (since a `Session` is created under the hood by all convenience methods like `requests.get`, `requests.post`).

Why is this behaviour so?  Are there plans to create a variant of `Session` where cookies can be fully disabled?  May we contribute our own workaround, if there is interest from other users?


Thanks for this feature request!

The reality is that you haven't told the `Session` no cookie handling is needed, you've only told the cookiejar it owns. The `Session` has no way to be told that you don't want cookie handling, and it's simply not a feature requests possesses.

The reality is, vastly more users want cookie handling on and functioning than want to turn it off. For that reason we really have no plans to make it possible to fully disable cookie handling. If you're sufficiently worried about performance that you want to turn off one of requests' most important high-level features, I recommend investigating whether you can use `urllib3` just as easily. =)


Thanks to you for your swift answer!  We will look into using directly `urllib3`…

> El 10/4/2015, a las 13:04, Cory Benfield notifications@github.com escribió:
> 
> Thanks for this feature request!
> 
> The reality is that you haven't told the Session no cookie handling is needed, you've only told the cookiejar it owns. The Session has no way to be told that you don't want cookie handling, and it's simply not a feature requests possesses.
> 
> The reality is, vastly more users want cookie handling on and functioning than want to turn it off. For that reason we really have no plans to make it possible to fully disable cookie handling. If you're sufficiently worried about performance that you want to turn off one of requests' most important high-level features, I recommend investigating whether you can use urllib3 just as easily. =)
> 
> —
> Reply to this email directly or view it on GitHub https://github.com/kennethreitz/requests/issues/2544#issuecomment-91518206.
Simple get request never returns

The following request never returns:

```
import requests
r = requests.get('http://en.wikipedia.org/w/api.php?rcprop=ids&format=json&action=query&rclimit=10&rctype=edit&list=recentchanges&rcnamespace=0', headers={'user-agent': 'api test'})
```

I've tried under both Python 2.7 and Python 3.

Adding a timeout (eg 5 seconds) causes the request to complete successfully - but adds 5s latency to every request.  And, of course, makes the request fail if the API call takes longer than 5s to process.  So I guess what is happening is that the server has finished sending the response but Requests is sat waiting for ... something ... to happen.  What?  Is it possible that the server is sending a wrong content-length?


Sorry, this turned out to be because IPv6 claims to work on my network but doesn't work very well.  Setting a timeout would then cause the IPv6 request to timeout and the request would fall back to IPv4, which succeeds.
MisssingSchema error message does not handle unicode input correctly

using requests 2.6.0, Python 2.7.6

```
import requests
requests.get(u"hällo world")
```

```
---------------------------------------------------------------------------
UnicodeEncodeError                        Traceback (most recent call last)
<ipython-input-3-bf1281984b3a> in <module>()
----> 1 requests.get(url)

/Library/Python/2.7/site-packages/requests-2.6.0-py2.7.egg/requests/api.pyc in get(url, **kwargs)
     66 
     67     kwargs.setdefault('allow_redirects', True)
---> 68     return request('get', url, **kwargs)
     69 
     70 

/Library/Python/2.7/site-packages/requests-2.6.0-py2.7.egg/requests/api.pyc in request(method, url, **kwargs)
     48 
     49     session = sessions.Session()
---> 50     response = session.request(method=method, url=url, **kwargs)
     51     # By explicitly closing the session, we avoid leaving sockets open which
     52     # can trigger a ResourceWarning in some cases, and look like a memory leak

/Library/Python/2.7/site-packages/requests-2.6.0-py2.7.egg/requests/sessions.pyc in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    448             hooks = hooks,
    449         )
--> 450         prep = self.prepare_request(req)
    451 
    452         proxies = proxies or {}

/Library/Python/2.7/site-packages/requests-2.6.0-py2.7.egg/requests/sessions.pyc in prepare_request(self, request)
    379             auth=merge_setting(auth, self.auth),
    380             cookies=merged_cookies,
--> 381             hooks=merge_hooks(request.hooks, self.hooks),
    382         )
    383         return p

/Library/Python/2.7/site-packages/requests-2.6.0-py2.7.egg/requests/models.pyc in prepare(self, method, url, headers, files, data, params, auth, cookies, hooks, json)
    302 
    303         self.prepare_method(method)
--> 304         self.prepare_url(url, params)
    305         self.prepare_headers(headers)
    306         self.prepare_cookies(cookies)

/Library/Python/2.7/site-packages/requests-2.6.0-py2.7.egg/requests/models.pyc in prepare_url(self, url, params)
    359         if not scheme:
    360             raise MissingSchema("Invalid URL {0!r}: No schema supplied. "
--> 361                                 "Perhaps you meant http://{0}?".format(url))
    362 
    363         if not host:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 1: ordinal not in range(128)
```


Ugh, Python 2, why do you suck so much?

I don't consider this a highly urgent bug because to hit it requires that you be using not just a unicode URL but an invalid one. It's also impossible to hit in Python 3, which further limits its scope. That said, it's definitely a real bug and if anyone wants to fix it they should consider taking a shot.


Thanks for your quick reply.


This is now fixed. =)
Noisy SSL warnings

Is there a way to suppress these noisy SSL warnings? I got them by specifying requests.get(..., verify=False) when I was trying to use some self-signed web services.

/usr/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)


`requests.packages.urllib3.disable_warnings()`.


Thank you!
Document that "PreparedRequest.prepare_cookies" can only be called once

I've done the documentation changes suggested in #2532.

I also documented a way to work around this in the docstring, but I'm not sure if it'd be good to do so. (As it may lead to hacked-together solutions) Then again, other solutions like creating a brand new `Request` might lead to worse problems. (e.g.: loss of data)

@sigmavirus24, what do you think?
Copy a PreparedRequest's CookieJar reliably

When a PreparedRequests's cookie jar is not a RequestsCookieJar instance, it
will not have a "copy" method. By adding _copy_cookie_jar we can reliably copy
cookie jars so that we have an actual copy instead of the same instance on
different prepared requests.

This also updates the RequestsCookieJar.update logic to create copies of
cookies from the other jar.

Closes #2527


LGTM. :cake:
Place VendorAlias first in meta_path

When other libraries or tools add items to the meta_path, we need to preempt
some of their import hooks to be sure modules can be properly found. This also
prevents problems importing built-in modules on Python 2 where it will first
attempt to import something like:

```
requests.packages.chardet.sys
```

By placing our VendorAlias first, the above will fail and then it will fall
back to trying to import sys directly instead.

Closes #2465
__build__ out of sync? Still at 0x020503, looks like 2.5.3 rather than 2.6.0

Could it be that

```
__build__ = 0x020503
```

at https://github.com/kennethreitz/requests/blob/master/requests/__init__.py#L46
went out of sync? It looks like 2.5.3 rather than 2.6.0.

Best, Sebastian


Yup, this looks wrong.
"import requests" takes a full second?

Hi!

I noticed that the `--help` output of a tool of mine was noticably slower than that of my other tools. It took a full second to show up. I traced the lagging down to the call to `import requests` (and [worked around it](https://github.com/hartwork/backup-my-hub/commit/509b30e53cccefd39c68311e456bba813e0be455) by importing requests only after processing command line options).

`import requests` takes at least a second on both the SSDed Debian and HDDed Gentoo, with version 2.5.3 as well as 2.6.0. Compared to the 30 milliseconds that other modules take to import, that's a long time:

```
# time python -c 'import requests; print requests.__file__'
/usr/lib64/python2.7/site-packages/requests/__init__.pyc

real    0m1.099s
user    0m1.055s
sys     0m0.039s

# time python -c 'import argparse; print argparse.__file__'
/usr/lib64/python2.7/argparse.pyc

real    0m0.030s
user    0m0.018s
sys     0m0.010s
```

It would be great if you could work on reducing import time. Many thanks!


I believe the slow part of importing requests is importing pyopenssl. In theory you could delay the import until the first time a request is made, though I'm pretty sure the team is not in favor of that approach.


> I believe the slow part of importing requests is importing pyopenssl. 

That makes perfect sense:

```
# time python -c 'import OpenSSL.SSL; print OpenSSL.SSL.__file__'
/usr/lib64/python2.7/site-packages/OpenSSL/SSL.pyc

real    0m1.014s
user    0m0.991s
sys     0m0.018s
```


@hartwork Are you using a new virtualenv for each test? Mostly importing pyopenssl takes a while because cffi does an implicit C compile at import time in some cases, but that should only happen the first time.


No virtualenv over here, no.


```
# for i in {1..10} ; do sh -c "time python -c 'import OpenSSL.SSL'" |& sed -n '2,2p' ; done
real    0m1.004s
real    0m0.993s
real    0m0.996s
real    0m0.994s
real    0m0.997s
real    0m0.999s
real    0m0.998s
real    0m0.991s
real    0m0.999s
real    0m0.997s
```


Interesting.

Fundamentally, this is a `cryptography` bug, probably pyca/cryptography#1764. I'm afraid until that's resolved we cannot do anything about this.


On my little ARM system from NextThing this takes SEVEN SECONDS.

@afcady Please report the issue to the appropriate upstream.
2.5.1 labeled latest release at GitHub (rather than 2.6.0)?

Hi!

Looking at
https://github.com/kennethreitz/requests/releases
I need to click on "Show 3 newer tags" to see 2.6.0 uncovered.
2.5.1 is marked the latest release. Is that intended?
It's 2.6.0 on PyPI https://pypi.python.org/pypi/requests/ .

Best, Sebastian


We don't really have a consistent policy of updating GitHub's releases. For our release notes we strongly recommend looking at PyPI or our documentation.


So... no fix? 2.5.1 remains the latest on GitHub?


@hartwork as @Lukasa has already kindly explained, PyPI is the source of truth. Not GitHub. You might notice that there are _several_ releases missing notes on GitHub.


My point are not the releases notes. My point is the fix potential user confusion. The release page up here raises more questions than it answers. If it's trouble to keep in sync, okay. If not, my vote to keep it synced, even just versions without release notes. That's all.


@hartwork it is not trivial.


Okay :)
AttributeError- LWPCookieJar instance has no attribute 'copy'

Hi,
In version 2.x:
<b>[test-case]</b>

``` python
import requests
from cookielib import LWPCookieJar
cookiejar = LWPCookieJar()
req = requests.Request('GET', 'http://httpbin.org/get', cookies=cookiejar)
r = req.prepare()
r.copy()
```

<b>[traceback]</b>
...
AttributeError: LWPCookieJar instance has no attribute 'copy'

<b>[comments]</b>
It seems that not all instances of cookiejar have the copy method, but in the copy method of PreparedRequest, it call the copy method of cookies without any checks.


What is your proposed solution?


Hi @sigmavirus24,
Thanks for quick response.  This situation is a little tricky that the base CookieJar class does not supply the 'copy' method but the RequestsCookieJar does. However, in the method 'prepare_cookies' of PreparedRequests, you permit all the instances of the subclass of CookieJar preserved as its original form(which may not contain 'copy' method). As I suppose, one uniformed way is to remove the copy method of  RequestsCookieJar and use the python copy function when copy is needed(This may need changes of other places). The other is more conservative, that is, restrict the self._cookies of PreparedRequest to be only the instance of RequestsCookieJar or else check whether having copy method before calling this method of a CookieJar instance.

``` python
def prepare_cookies(self, cookies):
    """Prepares the given HTTP cookie data."""

    if isinstance(cookies, cookielib.CookieJar):   #<=====
        self._cookies = cookies
    else:
        self._cookies = cookiejar_from_dict(cookies)
    ...
```


@zhaoguixu check out #2534 


As mentioned in #2077, self._cookies may be None

``` python
p._cookies = _copy_cookie_jar(self._cookies)   # need check whether it is None
```


Thanks for catching that @zhaoguixu. I updated the PR to account for that.
Add support for digest authentication with an HTTP proxy

Currently, requests only supports HTTP basic authentication to a proxy. It would be very useful to support digest authentication with a proxy as well.

Additionally, there should be some way to signal that requests should not attempt to pass proxy credentials in plaintext before receiving the digest nonce from the proxy.


@justintime32 Thanks for the feature request!

It should be entirely possible to write a short authentication handler that does exactly what you need: a quick Google showed [this](http://stackoverflow.com/questions/13506455/how-to-pass-proxy-authentication-requires-digest-auth-by-using-python-requests) as the top result. Because of the ease of adding such a thing yourself, and because it's relatively infrequently used, we don't believe there's much advantage in bringing Proxy Digest Auth into the core library.


Hi @Lukasa, I actually started by writing an auth handler. The main issue I ran into was that it only works for non-SSL requests. SSL requests through a proxy are made through a CONNECT tunnel, and the CONNECT request must be authenticated. The auth handler appears to be unable to hook into the proxy tunnel creation step, and therefore SSL requests will always fail.

Are there other ways to add this authentication without modifying the core? Maybe some more hooks around the proxy tunnel creation?


Unfortunately, what you need is not possible with `httplib`. To do the CONNECT tunnel in httplib, you end up in the `_tunnel` method, which has the following code:

``` python
    def _tunnel(self):
        connect_str = "CONNECT %s:%d HTTP/1.0\r\n" % (self._tunnel_host,
            self._tunnel_port)
        connect_bytes = connect_str.encode("ascii")
        self.send(connect_bytes)
        for header, value in self._tunnel_headers.items():
            header_str = "%s: %s\r\n" % (header, value)
            header_bytes = header_str.encode("latin-1")
            self.send(header_bytes)
        self.send(b'\r\n')

        response = self.response_class(self.sock, method=self._method)
        (version, code, message) = response._read_status()

        if code != 200:
            self.close()
            raise OSError("Tunnel connection failed: %d %s" % (code,
                                                               message.strip()))
```

As you can see, there is no way to hook into a 407 response here. We can only do this by overriding the way the HTTP connection functions, which is something we do in [urllib3](https://github.com/shazow/urllib3): I recommend opening a feature request there.
TypeError: __str__ returned non-string (type Error)

OS X 10.9
Python 2.7.9

```
$ pip freeze
cffi==0.9.2
cryptography==0.8.1
enum34==1.0.4
gitdb==0.6.4
GitPython==0.3.6
ndg-httpsclient==0.3.3
pyasn1==0.1.7
pycparser==2.10
pyOpenSSL==0.14
requests==2.6.0
requests-toolbelt==0.3.1
six==1.9.0
smmap==0.9.0
```

On request to website with TLS 1.0 `https://codeforge.lbl.gov/frs/download.php/409/fastbit-ibis1.3.8.tar.gz` I'm getting:

``` python
Traceback (most recent call last):
  File "./main.py", line 118, in <module>
    resp = downloader.run()
  File "/Users/mktums/Projects/brew404/downloaders.py", line 64, in run
    resp = self.fetch()
  File "/Users/mktums/Projects/brew404/downloaders.py", line 59, in fetch
    resp = s.send(req, verify=verify_ssl, allow_redirects=True, timeout=20)
  File "/Users/mktums/.virtualenvs/brew/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/Users/mktums/.virtualenvs/brew/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/Users/mktums/.virtualenvs/brew/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 544, in urlopen
    body=body, headers=headers)
  File "/Users/mktums/.virtualenvs/brew/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 344, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
  File "/Users/mktums/.virtualenvs/brew/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 314, in _raise_timeout
    if 'timed out' in str(err) or 'did not complete (read)' in str(err):  # Python 2.6
TypeError: __str__ returned non-string (type Error)
```


Thanks for reporting this! This is currently tracked under shazow/urllib3#556, so I'm going to close this issue to centralise discussion there.


@Lukasa Thanks for fast reply!


how to fix this bug? any ideas?


@gdonzy Please read the above issue, which links to a related tracking issue.


Thanks a lot. When I use "requests.get("xxxx", verify=False)", the bug is fixed.


Can I re open this? I am getting this even for a signed certificate.  I cannot use verify = False


No, this was fixed a long time ago. You should update instead.
Save and load session

I'd like to be able to save and load a session to disk. I think this means just saving and loading the cookies, but perhaps there are other nuances that would need to be included.

I'm interested in using Robobrowser (https://github.com/jmcarp/robobrowser) which wraps requests. I'd like to be able to persist the browser between running my application, which means being able to save and load the session to disk.

There is a SO post regarding how to do this. I think tellingly, the most popular answer is wrong.
http://stackoverflow.com/questions/13030095/how-to-save-requests-python-cookies-to-a-file

Thanks,
Jim


The easiest thing to do is just to pickle the whole `session` object:

``` python
import requests, requests.utils, pickle
session = requests.session()
# Make some calls
with open('somefile', 'w') as f:
    pickle.dump(session, f)
```

``` python
with open('somefile') as f:
    session = pickle.load(f)
```


As an addendum: looks like Python3 expects the 'b' flag during write and read:

```python
with open('somefile', 'wb') as f:
    pickle.dump(session, f)
```
```python
with open('somefile', 'rb') as f:
    session = pickle.load(f)
```


I tried using this approach, however I could not get TLS session resumption to work after reloading the pickled session. Does anyone know how to achieve this behavior?
ConnectionError and socket.gaierror occurred on IPv6-enabled system

I tried to make a request with a tool that uses python-requests on a Ubuntu 14.04 server with IPv6 enabled (with inet6 interface configured at `/etc/network/interfaces`) and encountered the following errors:

``` bash
$ 115down 
Traceback (most recent call last):
  File "/usr/local/bin/115down", line 130, in <module>
    main()
  File "/usr/local/bin/115down", line 126, in main
    args.sub_num, args.count, args.sub_count, args.tasks)
  File "/usr/local/bin/115down", line 64, in get_entries
    api.login(username, password, section)
  File "/usr/local/lib/python2.7/dist-packages/u115/api.py", line 248, in login
    if self.has_logged_in:
  File "/usr/local/lib/python2.7/dist-packages/u115/api.py", line 297, in has_logged_in
    r = self.http.get(CHECKPOINT_URL)
  File "/usr/local/lib/python2.7/dist-packages/u115/api.py", line 56, in get
    r = self.session.get(url, params=params)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 467, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 455, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 558, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='passport.115.com', port=80): Max retries exceeded with url: /?ct=ajax&ac=ajax_check_point (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)
Exception requests.exceptions.ConnectionError: ConnectionError(MaxRetryError("HTTPConnectionPool(host='passport.115.com', port=80): Max retries exceeded with url: /?ct=ajax&ac=ajax_check_point (Caused by <class 'socket.gaierror'>: [Errno -2] Name or service not known)",),) in <bound method API.__del__ of <u115.api.API object at 0x7fe060092210>> ignored
```

When disabled IPv6 by commenting relevant lines in `/etc/network/interfaces` and reboot, everything works. 

Anyone have any ideas?


`socket.gaierror` seems to suggest that you're failing a DNS lookup. Try this:

``` python
import socket
socket.getaddrinfo('passport.115.com', 80)
```

in both IPv6 and IPv4 mode, and see if the problem reproduces.


To reproduce,

```
>>> import socket
>>> socket.getaddrinfo('passport.115.com', 80)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.gaierror: [Errno -2] Name or service not known
```

But I think I got the cause, when I tried to disable IPv6 editing the default `/etc/network/interfaces`, I found DNS is set only for IPv6 of the eth0 interface but not for IPv4:

```
dns-nameservers 2001:4860:4860::8844 2001:4860:4860::8888 8.8.8.8
```

So I manually added 8.8.8.8 and 8.8.4.4 to IPv4 and it worked.

It seems the default system is not "properly" configured for this scenario


I'm glad you were able to diagnose the problem!
chardet.detect() call in models.py fails when using IronPython.

In models.py, chardet.detect(self.content)['encoding'] fails to work with IronPython with an exception: 'Expected a bytes object, not a unicode object'. It is due to a check in chardet's __init__ ( isinstance(aBuf, unicode) ) and IronPython's way of handling strings as unicode type.

Excerpt from chardet's **init**:

```
def detect(aBuf):
        if ((version_info < (3, 0) and isinstance(aBuf, unicode)) or
                (version_info >= (3, 0) and not isinstance(aBuf, bytes))):
            raise ValueError('Expected a bytes object, not a unicode object')
```

I tried passing bytes(self.content) to chardet instead but that solution fails short because in compat.py, we are redefining bytes to str, which under IronPython is
unicode.

Can we perhaps add some way of handling IronPython properly or is it something that should be fixed in chardet instead?


I think this needs to be fixed in chardet itself, sadly. Happily, it's now under active development, over at [chardet/chardet](https://github.com/chardet/chardet), so opening the issue over there should get it promptly looked at.


Actually this looks like a problem with IronPython that voids requests expectations. `Response.content` should always be a `bytes` object. That's what we expect to get back from the socket via httplib. If we aren't getting bytes back, that sounds like a bug in the way IronPython has copied in the standard library. We have no way of encoding it back to bytes intelligently and IronPython should really really really not be doing that.
shorter and faster version extraction from __init__.py

@sigmavirus24 I though it could be improved a bit. Tell me what you think.


if you think readability has been tampered too much, I'll make a pull request with this less-pure-but-softer version:

```
-    for line in fd:
-        m = reg.match(line)
-        if m:
-            version = m.group(1)
-            break
+    version = reg.search(fd.read())
```


No objections from me. :+1:


Do you have benchmarks? Also, why are we optimizing something that is run at most once by a user (during pip install)?


It's not optimizing. It's making it simpler. And thus that it's simpler also makes faster.
What you did was opening the file, buffering it, crawling it line by line (triggering a search for line terminators whereas we don't care about) and applying a regex on each line.
What I do is opening and reading the whole file and applying the regex on the content.
This uninteresting [microbenchmark](https://gist.github.com/deronnax/984499ee7ef62b9b81f4) shows us it's 3 time faster, but that's not my point. It's simpler, that's what matter :)


Either way, this change is incomplete in its current form. I've left feedback on what is necessary to fix it.


Yeah, sorry for the cardet, I thought about it and forgot.
I think using the dollar and thus requiring the "version" line ends with a newline is a bad idea : it might end with a whitespace or a comment, it's valid.
I think the cardet suffices. And it thus behaves the same as your version did ;)


>  I think using the dollar and thus requiring the "version" line ends with a newline is a bad idea : it might end with a whitespace or a comment, it's valid.

No one asked for you to add anything other than the `^`.

Thanks for fixing this up. Any further feedback @Lukasa?


Nope. Fine by me. =)


The

> Please fix the expression to check for line beginnings and **endings**

 misled me :)


I misspoke twice then! Sorry for the confusion @deronnax. Thanks for the contribution! :cake: 


You're welcome :)
Can't copy-paste international URL

Observe this link to an article: http://alaxon.co.il/article/%D7%9C%D7%95-%D7%99%D7%9B%D7%9C%D7%95-%D7%97%D7%AA%D7%95%D7%9C%D7%99%D7%9D-%D7%9C%D7%93%D7%91%D7%A8/

If you click it, you'll see Hebrew letters in your address bar. (At least in Chrome.) The article works.

Now, try accessing it in requests:

```
>>> import requests
>>> requests.get('http://alaxon.co.il/article/%D7%9C%D7%95-%D7%99%D7%9B%D7%9C%D7%95-%D7%97%D7%AA%D7%95%D7%9C%D7%99%D7%9D-%D7%9C%D7%93%D7%91%D7%A8/')
Starting new HTTP connection (1): alaxon.co.il
<Response [403]>
```

Why is that? 


Note that I get the same 403 error response when I put the Hebrew URL in `requests.get` without the percentage escape sequences. If I put this Hebrew URL in the browser, it works fine.


My mistake, this was a completely unrelated problem with the site.


Heh, I was just diving in. Thanks for solving it yourself!
Minor Patch TypeError thrown

Python2.7, str(err) was throwing a TypeError as it doesn't override the 'toString' method **str**.

Wrapped offending block in a quick try block, catching only TypeError exceptions.


Thanks! :cake:


@Montycarlo please see #2514. This should have been opened at [urllib3](/shazow/urllib3) and the change needs to be much better since this will introduce a breaking change to urllib3.


Sorry @Montycarlo! I should have caught that earlier. =(


Sorry for wasting your time! Keep up the awesome work :)
Request works from command line, but not from within script.

If I run the following from the command line, it works fine.  If I try to run it from a script I get the AttributeError.  Why is those things I said?

import requests
r = requests.head('https://api.github.com/user')

AttributeError: 'module' object has no attribute 'head'


Is there a file named 'requests.py'? Python would try to import that file, not the module, in that case.


Nailed it!  thanks Kevin!
Expand on chunked handling.

Once we merge in the new urllib3, this will update the docs to reflect the new best way to handle chunked transfer encoding.


looks fine to me

how about specifying the size of a chunk? will it be possible?


Yup, passing the chunk size parameter to `iter_content` will do it. What you _can't_ do is change that size from chunk to chunk.


You are right! Should have checked the code.

The reason I asked because I was slightly confused by this sentence. I would probably update it to:

``` diff
-iterate chunk-by-chunk by calling ``iter_content`` with a chunk
-size parameter of ``None``.
+iterate chunk-by-chunk by calling ``iter_content`` (it is ``None``
+by default, but you can set chunk size yourself by providing an ``int``).
```

But you're the native speaker here. So I'm assuming that's just me not getting it.


Agree that it needs rewording, though that wording isn't quite right (it's not `None` by default, it's `1`).


pong


Always be updating PRs.
request.files is empty after POSTing a file

Hey guys, I don't know if this is the right place to make this bug report, but I could sure use some help, I have been banging my head against a wall for a bit. See this repo where I'm reproducing my error: https://github.com/bepetersn/special-repo. 

Using Flask, I'm seeing weird behavior around its `request` object. After uploading a file with the requests library, e.g. `requests.post(uri, files=<my_files>)`, by the time the request propagated to my view function, `request.files` was empty. Oddly, I observed that the contents of the file itself was available under `request.form[None]`. 

After quite a lot of debugging, I saw that Werkzeug received roughly the following as the form/multipart-encoded data, and proceeded to try to parse it:

> Content-Disposition: form-data; name_=utf-8\'\'Spirit%20Airlines%20-%20cheap%20tickets%2C%0A  %20cheap%20flights%2C%20discount%20airfare%2C%20cheap%20hotels  %2C%20cheap%20car%20rentals%2C%20cheap%20travel.pdf; filename_=utf-  8\'\'Spirit%20Airlines%20-%20cheap%20tickets%2C%0A%20cheap%20flights  %2C%20discount%20airfare%2C%20cheap%20hotels%2C%20cheap%20car%20rentals  %2C%20cheap%20travel.pdf\r\n  

Do you notice the little "*" characters just after the "name" and "filename" header definitions? From my sense of things, Werkzeug looks for exactly "name" and "filename", doesn't know how to parse these in their place, fails to find an attachment name, and thus doesn't create a `werkzeug.datastructures.FileStorage`. For this reason, `request.files` is empty.

What does anyone think? Is requests responsible for adding these extra characters in?


I see the line responsible here: https://github.com/kennethreitz/requests/blob/35d083e1665beff39aabe47a79cd1f867b897b0c/requests/packages/urllib3/fields.py#L45. However, it's pretty clear that it's intentional. I'm curious if anyone can point me to what purpose it has (maybe something to do with RFC 2231?), -- and whether it's really correct, if I'm somehow giving requests bad data, or even if werkzeug isn't handling this stuff correctly..


This problem is almost certainly because you're passing a unicode string into requests 'files' parameter. Make sure your strings are all bytestrings first. =)


I will try that. :) Beats reading about 4 RFCs to check if I should actually be wanting to submit a pull request to werkzeug.


Well, I don't fully get the logic of it, but what I actually needed to do was get rid of the newline that was in the filename I was passing to the `files` parameter of `requests.post`.


Ah, yes, that'll do it to.

The problem here is that the multipart/form-encoded spec doesn't ordinarily allow newlines in the form fields, so you need to do some awkward work to get around them. That involves the encoding we've used, which is sadly relatively poorly supported by many servers. =(


Thanks for your help, Corey.
On Mar 18, 2015 4:54 PM, "Cory Benfield" notifications@github.com wrote:

> Closed #2505 https://github.com/kennethreitz/requests/issues/2505.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2505#event-258690327.