Raibek/requests-github-issues huggingface dataset:
Broken commit 5e6ecdad9f69b1ff789a17733b8edc6fd7091bd8

"git fsck" thinks [this commit](https://github.com/kennethreitz/requests/commit/5e6ecdad9f69b1ff789a17733b8edc6fd7091bd8) is broken:

```
error in commit 5e6ecdad9f69b1ff789a17733b8edc6fd7091bd8: invalid author/committer line - bad time zone
```

```
commit 5e6ecdad9f69b1ff789a17733b8edc6fd7091bd8
tree 0d1971c86d047f9530aee85a50b8fc1ca12723ba
parent 9471b0ab889a4684f87952cb951f7f4dc3f59dda
author Shrikant Sharat Kandula <...> 1313584730 +051800
committer Shrikant Sharat Kandula <...> 1313584730 +051800
```

Would it be possible to repair the history?


No it isn't possible to repair the history. That will change the SHA value of every commit after it and update the timestamps on ALL of them. It will invalidate every tag since that commit and it will cause far too many headaches.


To be clear, it is _technically_ possible to repair this, also if repairing it didn't affect the repository's history from that point on, I would be very much in favor of it, but it irrevocably breaks expectations around our history and will cause more trouble than not. You appear to be the first person to have checked this and found it to be an issue.

Even if `git fsck` reports the commit as being damaged, `git` continues to work just fine. This doesn't realistically affect the use of the repository or any person's ability to contribute to it. I appreciate you reporting this, but it simply isn't something we're going to fix.


> You appear to be the first person to have checked this and found it to be an issue.

Oh, really? I thought I'm just too lazy to find an old and closed issue.

I just happen to have "transfer.fsckObjects = true" in my ~/.gitignore...

@sigmavirus24, yep, this looks like a very popular repo. Forking just for one commit would be a solution, but... Ok, I'll just disable the checks in .git/config after the checkout.


@anatolyborodin Recent versions of git (`master`, but not yet in any released version) have support for configurable fsck checks. So you will be able to set `receive.fsck.badtimezone = warn`, for example, if you would like the usual checks, but would prefer to ignore this particular (minor) problem.
convert requests command

I'm using `requests` module in my app but when i run it in google app engine, raise **SSL Connection** exception:

```
File "/base/data/home/apps/s~myapp/1.385939809966824449/lib/requests/adapters.py", line 429, in send
    raise SSLError(e, request=request)
SSLError: Can't connect to HTTPS URL because the SSL module is not available.
```

I google this problem and read some answers that said SSL module is not in _GAE while list_.
I can send a simple GET request with `urllib`. It was ok.
What is equivalent of `requests.request('GET', request_url, params=params, files=files)` in `urllib`?


Requests does not consider Google App Engine a supported platform at this time, so I'm afraid we can't help you here. I recommend asking your question on StackOverflow. =) Sorry!
Upgrade to urllib3 1.11 to fix connection pool exhaustion

An important fix has been made to urllib3 which fixes http connection pool exhaustion when connections fail:

https://github.com/shazow/urllib3/issues/644

Can you release a new version which uses urllib3 1.11 ?


We plan to: see #2678.

For future reference, two of the three requests core developers are also urllib3 core developers. urllib3 does not sneakily release software without us noticing. ;)


## I'm not a core developer on urllib3. :-P... Or is Kenneth?

Sent from my Android device with K-9 Mail. Please excuse my brevity.


Pfft, sure you are.


PR #2678 is currently using a checkout of urrlib3 from some commit that is slightly earlier than 1.11. I understand 1.11 wasn't released yet when the PR was first created. But since I don't see any comments on that PR indicating either way, I was wondering: will you be updating the PR to use the released version, or is there a reason the PR is still using the checkout that it's using?


We will update the PR. =)
Misleading POST documentation

The GitHub example at [#more-complicated-post-requests](http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests) is very misleading. It would be great to mention the `json` parameter. If you want to know what it does, you'll have to navigate all the way through until the `PreparedRequest` class.


The documentation should be something like:
`:param json: (optional) json data, the passed in object gets serialized to json (json.dumps()) and the content type of the request is set to  'application/json'.`


@melbic Yeah, sorry, that documentation was written before the `json` parameter was added.

Would you like to propose a pull request with some new wording?


I would rather that pull request be non-destructive. In other words, please do not remove what's already there, merely add. Many people use the "latest" documentation against older versions of requests. We don't really maintain old versions of the documentation so we need the documentation to be somewhat backwards compatible. The section you mention is exactly that. **Adding** information about the `json` parameter is fine, but please note the version for that section so people know "I'm on 2.2.1 installed on Ubuntu 14.04, because the package archives are so wildly out of date, I can't use that feature because it was introduced in 2.4.0 (or whenever we actually introduced it)."
[Request] Simple argument to set an overall timeout for the request

The `timeout` parameter is useful but does not provide a way to set an overall timeout for the request.

But in keeping with _simple is beautiful_, it would be terrific if we could do:
`requests.get(url, overall_timeout=10)`

Thanks in advance for considering it. :)


As a workaround, the `@stopit.threading_timeoutable` seems to work.
implement delayed retry support to failed requests

Currently requests library supports a retry mechanism where you can specify how many retries to make untill it will fail, which is great unless you have a downtime of the service of only few seconds.

In order to make usage of requests more resilient to temporary errors we need an option to delay further retries, preferably using a fibonacci-like increase delay on sequential  retries.

Example, `start_delay=5` seconds and `max_retries=10`, would use these delays: 5, 10, 15, 25, 40, 65, 105, ...

I observed that lots of people are encountering the same issue and I do think that is much better to have this as part of requests and not having to hack our own httpadaptor to implement it. 

I would go so far to say that I would even argue for having it enabled by default, when max_retries is specified.


Thanks for the suggestion!

The supported way to do this in requests is to use a HTTP adapter. I notice you've spotted this but consider it to be a 'hack'. It's not: the requests project considers the HTTP adapter a first-class part of our interface.

The change is very simple:

``` python
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

class RetryAdapter(HTTPAdapter):
    def __init__(self, *args, **kwargs):
        super(RetryAdapter, self).__init__(*args, **kwargs)
        self.max_retries = Retry(total=self.max_retries, backoff_factor=5)

s = requests.Session()
s.mount('http://', RetryAdapter())
s.mount('https://', RetryAdapter())
```

This is a very simple adapter and could easily be dropped in. We may even want to add it to the requests-toolbelt: a pull request there would be very welcome. (/cc @sigmavirus24)

Thanks for the suggestion!
Fix docs for passing a list of values for a query string

The special `[]` notation at the end of the field name is not necessary to get a field to appear with multiple values in the query string.


Yeah, this is _mostly_ true. I'm going to merge this as-is @peplin, but I'd really love it if you'd follow up with another pull request that adds a paragraph about _why_ you might want to add the `[]` (namely, because many servers expect that notation).

Thanks so much for the contribution! :cake: :sparkles: :cake:
Proposed 2.8.0

Includes
- #2674 
- #2567 
- #2523 
- #2706
- Bump for urllib3

Reminder to look at http://ci.kennethreitz.org/job/requests-pr/ for build status


So http://ci.kennethreitz.org/job/requests-pr/714/PYTHON=3.3/console is an interesting error. I'll work on tracking it down


Here's hoping http://ci.sigmavir.us/job/requests-real/ picks up on this. @kennethreitz's CI is not picking up on the new commits.


Ah, here we go: http://ci.kennethreitz.org/job/requests-pr/715/


Let's be extra cautious of #2567


We are. That's why we're shipping it in 2.8.0: it should avoid us immediately breaking a ton of production software. =)


@kennethreitz #2567 is already running on Debian (and maybe ubuntu?) against Requests 2.7.0 and has yet to produce any bug reports there. It's been running there for _at least_ a month. Also iirc, Cramer signed off on it and pip has been using something similar to #2567 for a while now.


what does @mitsuhiko think?


Replied to #2567.


Any traction on this release? Would love to see https://github.com/shazow/urllib3/issues/556 fixed :)


> Any traction on this release? Would love to see shazow/urllib3#556 fixed :)

+1


## Sorry Laurent. What are you trying to add 1 to? This isn't JavaScript so we don't do auto type coercion and that comment doesn't parse.

Sent from my Android device with K-9 Mail. Please excuse my brevity.


Heh, sorry, was just trying to add my vote as we just hit the problem, in case this PR needs some love ;)


Comments aren't for voting though. We have this planned we just don't have it entirely ready. This was also just to have a branch to have all the open PRs merged into one place so we can test all of them together. This isn't necessarily the PR that will actually merge to create 2.8.0


@sigmavirus24 IIRC Ubuntu either pulls from Debian testing or unstable during a new OS release cycle, so if #2567 was put in to one of those branches it will be in Ubuntu by now. Also you would be able to check the patches that are in the source for packaging as well. just my 2 cents


Ok, in preparation for releasing 2.8.0 I am going to merge this now. I'm aiming to get 2.8.0 out the door by the middle of this week, and merging this will help force me to do that.
can't use proxy-authorization in header 

hi, i try to use Proxy-Authorization header with my request, but it doesn't appear in request headers. If i change header name to 'Proxy-Authorization1', for example, everything is ok, i can see it in output. 

``` python
proxy = {'http': 'http://ip:port/'}
token = base64.encodestring('user:pass').strip()
headers = {'Proxy-Authorization': 'Basic ' + token, 'Proxy-Connection': 'Keep-Alive', 'Accept': '*/*'}

r = requests.get('http://url', headers=headers, proxies=proxy) 
```

the reason why i'm trying to do this - with authentication like 'http://user:pass@ip.port' or via HTTPBasicAuth request fails with 407 Authentication Required

so, is there a way to explicitly define Proxy-Authorization header ?


Well, in the sample code above you aren't actually telling requests which proxy to talk to. =)

This is easiest to do by using the username and password in the proxy URL, and then not providing the header at all:

``` python
proxy = {'http': 'http://user:pass@ip:port/'}
r = requests.get('http://url', proxies=proxy) 
```


yep, tried this way, but proxies were still returning 407. however, now everything is ok again, i begin to suspect this is proxy's issue.
Convert non-string header values to their string representation.

Fixes https://github.com/kennethreitz/requests/issues/2675


The change you've made is in urllib3. @shazow may want this change, in which case we should accept it over at the main urllib3 repository. If he does not want it, we should instead move it into the requests code.


-1 on having this in urllib3's urlopen, that's far too low-level to mangle user input. Requests is probably the more appropriate layer for this.

Alternatively, it could live in `HTTPHeaderDict` once we get it usable as input as well as output, but that's blocked on https://github.com/shazow/urllib3/pull/633.


That's pretty much what I thought.

In that case, I think the right place for this is `requests.models.PreparedRequest.prepare_headers()`.


Thanks for the review @Lukasa and @shazow :) I've updated the PR now. 
https://github.com/kennethreitz/requests/blob/master/test_requests.py#L1275 fails for me on Python 3 because the keys are not in order, but looking at the implementation the keys are not expected to be in order? 


Yup, it's a known issue. =)


@Lukasa I've updated the PR.


See discussion on #2675 before continuing work here. I'm not sure we want to encourage this.


Closing based on discussion on #2675 
Alas, it couldn't be my first contribution to requests. May be some time in the future :)
Catch and wrap ClosedPoolError

Partially resolves #1572: "urllib3 exceptions passing through requests
API". #1572 

Inspired from @Lukasa's previous 2605be11d82d42438ac7c3993810c955bde74cef.

This is my first PR to requests library; feel free to give me feedback. I generally have a fast response time. Also, available on IRC and Twitter (@ArcTanSusan).


Aw no, we lost my awesome work?

![image](https://cloud.githubusercontent.com/assets/1382556/8743585/e94a1910-2c67-11e5-9387-f09a00a9b26c.png)

All joking aside, this looks good to me, I'd be happy to merge it. However, it's an API change, so it _at least_ needs to go into 2.8.0 and may need to go into 3.0.0. @sigmavirus24?


2.8.0 seems reasonable to me. I don't think we need to wait for 3.0.0 to wrap a urllib3 exception
Low-level exceptions leaking from Response.iter_content()

Up through v2.6.0, `Response.iter_content()`would catch all expected exceptions coming from `self.raw.stream()`, and wrap them in instances of various subclasses of `RequestException`.

In v2.6.1, urllib3 was updated to 1.10.3, which has issue https://github.com/shazow/urllib3/issues/673 that causes `self.raw.stream()` to raise low-level `httplib.HTTPException`s, particularly `httplib.IncompleteRead`. Because of this, as of v2.6.1 of requests, `Response.iter_content()` leaks these low-level of exceptions, instead of catching them and wrapping them in `RequestException` instances.

Issue https://github.com/shazow/urllib3/issues/673 has been fixed in master, but hasn't been released yet.

To fix this, requests could catch `httplib.HTTPException` along with `urllib3.exceptions.ProtocolError`, converting both into `ChunkedEncodingError`. Or requests can upgrade to the newest version of urllib3 when it is released.


Requests will upgrade to the newest version of urllib3 when it is released. =)

Thanks for the report!
Fix quickstart "Custom Headers" example intro

Previously this section prefaced an example with:

```
For example, we didn't specify our content-type
```

But, the actual example set a custom user-agent header on the request. This
changes it to say "user-agent" instead which matches the given example.


Alternatively the example could be changed to actually set the content-type header instead of the user-agent header - either way I think they should be consistent :)


Thanks @petedmarsh, I missed this when I changed the example from content-type to user-agent!
Re-enable Travis-CI builds


Ideally this isnt merged until #2668 is fixed.


We will not be turning on Travis CI. ci.kennethreitz.org just needs a proper kick. /cc @kennethreitz 


The main repo doesnt need to enable Travis. This helps other developers testing on their forks.


The main repo should be testing pull requests, which will help other developers on their forks. =)


That only helps developers when they submit a pull request.  Some developers like to check their work against all configurations in the build matrix before doing a pull request.
If travis was enabled for pull requests, https://github.com/kennethreitz/requests/pull/2666 would not have been merged as travis would have commented on the pull request that it was broken.
If http://ci.kennethreitz.org/ is the preferred approach, it should be submitting comments to pull requests, or people with merge rights should be checking it manually before merging.


> If http://ci.kennethreitz.org/ is the preferred approach, it should be submitting comments to pull requests, or people with merge rights should be checking it manually before merging.

No-one is denying that. ci.kennethreitz.org is currently broken. It should be functioning correctly. Currently it's the preferred approach, but obviously we need to get it working again.

As to developers on their forks, we encourage developers to open pull requests early if they want to do testing. Again, all of this assumes our process is functioning correctly: right now it is not. This is an organisational issue within the project, and re-adding travis will not fix it, especially because Travis is _very_ flaky when running our tests and regularly fails when it shouldn't.
insecure platform warning and failure

This might have something to do with the 2.4.0 release and using certi but I have no idea.  Similar to #2255 I am seeing an issue with executing the samples on the readme.

Why this happens with a non ssl request, I have no idea.

```
import requests
 r = requests.get('http://en.wikipedia.org/wiki/Monty_Python')
/usr/lib/python2.6/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
>>> r = requests.get('http://en.wikipedia.org/wiki/Monty_Python', verify=False)
/usr/lib/python2.6/site-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
/usr/lib/python2.6/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
```


Wikipedia redirects you to a HTTPS endpoint. If you check `r.url` you'll see you ended up at a HTTPS version of that page, which is how the insecure request was made. =)
Adds extra tests for CaseInsensitiveDict

This adds extra tstes for `requests.structures.CaseInsensitiveDict`
- Test for NotImplemented in `__eq__`
- Adds test for `copy()`
- Adds test for `__repr__()`


@Lukasa how does this look to you? Looks fine to me. I don't mind a couple extra tests personally.


I have one note, but I'm happy to add in extra tests.


@Lukasa I believe I've addressed your concern for that :)


Yeah, that looks good to me. \o/
cookiejar_from_dict support SimpleCookie

Hi there! :)

I have a suggestion about support `http.cookies.SimpleCookie` by `requests.cookies.cookiejar_from_dict`.

Example:

``` python
import http.cookies

import requests
import requests.cookies

cookie_str = 'Key1=Value1;Key2=Value2'  # e.g can be obtain from cfg
cookie = http.cookies.SimpleCookie(cookie_str)

sess = requests.session()
sess.cookies = requests.cookies.cookiejar_from_dict(cookie)

```

`SimpleCookie` have mapping interface and `RequestsCookieJar` can assign cookie with `Morsel` value, but `cookiejar_from_dict` don't do this right now.

I thinking about that fixup, because `RequestsCookieJar` just extension of common `CookieJar`:

``` python
cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
->
RequestsCookieJar.set(cookiejar, name, cookie_dict[name])
```

But method name is strongly say us `from_dict`, SimpleCookie not a dict, just mapping. May be we need new method. I don't know. What do you think?


Have you tested your example code? It works fine for me...


Yes, of course! Code works fine before you send first request. See the picture.
If you need I can send full report (.html doc) (rendered by mako lib).

On the screen you see Python\lib\http\cookiejar.py : 1289
![report_part](https://cloud.githubusercontent.com/assets/735150/8637499/2700ca06-28ad-11e5-8317-812696ffcfdd.PNG)


In which case, it's not clear to me what the bug is. Given that `cookiejar_from_dict` works fine with `SimpleCookie`, what problem do we have?


`SimpleCookie` handle thier values as `Morsel` object, and when we pass `SimpleCookie` object to cookiejar_from_dict, value write without "unpacking".


I think this is not a bug, i just want to do a enchancment request. I suggest my solution, but i think this is not clear enough.


Aha, ok, good. =) Now we have a problem.

However, I think this is a sufficiently simple problem that it doesn't warrant a helper method in requests. It's two lines of code to get the behaviour you want:

``` python
for item in cookie.items():
    sess.cookies.set(*item)
```


Ou! You are right :+1: Thanks


Glad I could help! :cake: :sparkles:
fix some small spell problem


Thanks for this @mudongliang!

Unfortunately, neither of those is actually incorrect, they're just informal English.

> (or branch off of it)

To "branch off" of something is to create a new branch from it. Thus, to "branch off of it [master]" is to create a new branch from the master branch.

> Send a pull request and bug the maintainer until it gets merged or published.

In this case, "bug" does not mean the noun "bug" (a defect or imperfection), it means the verb "to bug" (to bother, to annoy, to pester). The sentence therefore means "Send a pull request and annoy the maintainer until it gets merged or published".

Sorry we can't merge this, but thanks for opening it! :cake:
Only pass useful timeouts to _get_conn

So `_get_conn` doesn't use a urllib3 `Timeout` object, it only takes a float. It also only does stuff if `block` is `True`. Given that by default we use `block=False`, it won't do anything, so let's not use it. However, if someone changes that with a global constant they may want to be able to change this too, so let's make it a named constant.


/cc @shazow 


:custard:


LGTM
HTTPAdapter timeout type error

At https://github.com/kennethreitz/requests/blob/master/requests/adapters.py#L378 you call

conn._get_conn(timeout=timeout)

The timeout object you pass is a Timeout but that internal method in urllib3 expects a float. This results in a type error about float + Timeout. 

A float works there but I wasn't sure which to use.


Yeah, I think I spotted this before. I have no idea why we never fixed it, but it's definitely a bug.


I found it while chasing the Connection pool is full warning on chunked
posts. No parameter settings to HTTPAdaptor seem to help.

On Fri, Jul 3, 2015 at 11:55 AM, Cory Benfield notifications@github.com
wrote:

> Yeah, I think I spotted this before. I have no idea why we never fixed it,
> but it's definitely a bug.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2660#issuecomment-118380232
> .


See #2661 for what I think is a fix for this.


Yes, that is equivalent to my hacked test.

On Fri, Jul 3, 2015 at 12:05 PM, Cory Benfield notifications@github.com
wrote:

> See #2661 https://github.com/kennethreitz/requests/pull/2661 for what I
> think is a fix for this.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2660#issuecomment-118382859
> .


Fixed. =) Thanks @gbishop!
Extend "verify" to accept directory argument

On my systems I exclusively work with collections of CA certificates stored as individual files in a single directory (with appropriate c_rehash symlinks) rather than the large monolithic "CA BUNDLE" file with every cert concatenated inside.  Most *nix distributions provide both.  A directory of certs at /etc/ssl/certs (or similar) as well as the single-large-file.

Monolithic files are probably easiest for most turnkey users who are only going to be connecting to a public server that uses a popular CA.  They can simply download a single file from the internet and point the API at it to get up and running.  However, the directory-based approach is much more manageable for users who need or want to customize their CA selection.  Remove some old untrusted CAs or include a personal or corporate CA.

The openssl and pyopenssl APIs provide for both.  load_verify_locations() accepts both a cafile and capath argument.  You may specify one or the other or both.  However, requests always passes "verify" as the cafile argument, and correctly documents that the input must be a file.  If you try to pass a directory anyway, openssl gives you "SSLError: [Errno 21] Is a directory".

Please expand the definition of the "verify" parameter to also accept directory arguments.  If the argument is a directory, pass it as the capath argument to load_verify_locations(), else continue passing it as cafile.


I like this.


Alright, cool, let's do it.


I think this is an issue for `urllib3` -- if I understand correctly (and it's very possible I don't!), when `requests` wants to send a request, it [asks for an `adapter`](https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L567) -- that adapter wraps a `urllib3` connection via `get_connection()`. When that `urllib3` connection has [its socket wrapped](https://github.com/shazow/urllib3/blob/master/urllib3/connection.py#L237), the call to `load_verify_locations` [is made](https://github.com/shazow/urllib3/blob/master/urllib3/util/ssl_.py#L264), but it simply passes the first parameter (which in this case is just a file location). So in order to pass a `capath` argument through, `urllib3`'s `ssl_wrap_socket`'s function would need to change.


That's entirely correct. However, this is an issue that we want in requests, so for the moment we drive it here. =) An _enhancement_ is required in urllib3, but the exact form that will take is not yet known.


Ah gotcha, thanks for the clarification!


Ok, let's get started on this.


The urllib3 part of this work is in shazow/urllib3#701.


We can't progress this until urllib3 next pushes a release that contains the appropriate change, so that we can update our bundled dependency. This is on hold until that time.


See #2858.
Allow get_netrc_auth to raise parse/permission errors to caller

If the netrc file exists but cannot be parsed or read, get_netrc_auth
silently fails.

Add a new argument `ignore_errors` which when set to False will cause
any parse/permission errors to be raised to the caller.  The default
value is True, which means the default behavior is unchanged.

Fixes #2654

Change-Id: I7436aaaf593178673ab84fd9e7ab4bcb0e3fe75e


With a badly formed netrc file, `netrc.netrc` fails:

```
>>> from netrc import netrc
>>> netrc()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/netrc.py", line 32, in __init__
    self._parse(file, fp)
  File "/usr/lib/python2.7/netrc.py", line 94, in _parse
    file, lexer.lineno)
netrc.NetrcParseError: bad follower token 'fssdfs' (/home/user/.netrc, line 23)
```

Default behavior of `requests.utils.get_netrc_auth` is to silently fail:

```
>>> from requests.utils import get_netrc_auth
>>> get_netrc_auth("www.example.com")
```

With this patch, passing `ignore_errors=False` allows the parse error to be raised:

```
>>> get_netrc_auth("www.example.com", ignore_errors=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "requests/utils.py", line 101, in get_netrc_auth
    _netrc = netrc(netrc_path).authenticators(host)
  File "/usr/lib/python2.7/netrc.py", line 32, in __init__
    self._parse(file, fp)
  File "/usr/lib/python2.7/netrc.py", line 94, in _parse
    file, lexer.lineno)
netrc.NetrcParseError: bad follower token 'fssdfs' (/home/user/.netrc, line 23)
>>> 
```


This is simple enough. Personally I'd rather have something like

``` py
from requests import utils

utils.get_netrc_auth('www.example.com', raise_errors=True)
```

I've found people tend to become more confused when passing `False` as an override instead of `True`.


@sigmavirus24 thanks for the feedback.  Added a new commit to change the logic as suggested.


Cool. LGTM. I'll let @Lukasa weigh in.

Thanks @dpursehouse 


This LGTM.
Handle complex redirect URIs on Python 3

Resolves #2653.

This is one of those annoying changes that's almost impossible to test because of the sheer complexity of our redirect handling code. This also doesn't make it any simpler, sadly.

As to what version we merge this into, I proposed it against the master branch. I don't think it belongs in 3.0.0 (`resolve_redirects` isn't really part of our public API), but it could definitely break people's stuff. Next minor release feels appropriate, but I'd like to hear what you think @sigmavirus24.


The tests fail miserably on this :(

One way to fix the failing tests is to either change `unquote_unreserved` to call split with a `b'%'`, or to immediately decode the URL to `utf-8`, e.g.,

``` py
if is_py3 and isinstance(url, str):
    url = url.encode('latin1').decode('utf8')
```

Personally, I'd like those to be on separate lines to make it easier to find errors in the future, but that's just me =P.

Personally, I'm a bit ... skiddish to include this in 2.8. I think 3.0 is the _safer_ option because we don't regularly test against internationalized domains. Then again, I have to wonder how many people are using requests on Python 3 with redirects that behave like this since this is the first report we've had like this in the 2+ years that we've had this code in place. Perhaps that makes it safe, but I'm not a fan of that logic. =/


Wouldn't it be better to file a bug report on Python 3? I always thought that Python 3 had some kind of "UTF-8 everywhere where possible" policy, so I wonder why they decode with ISO-8859-1.


> Wouldn't it be better to file a bug report on Python 3? I always thought that Python 3 had some kind of "UTF-8 everywhere where possible" policy, so I wonder why they decode with ISO-8859-1.

No that's not the policy. Even so, it doesn't make sense in the context of HTTP/1.1 where headers were (until the latest revision of the specification late last year) always supposed to be Latin-1 (a.k.a, ISO-8859-1). http.client is doing the right thing on Python 3 by returning text as was described in the original version of the HTTP/1.1 specification


Yeah, this definitely isn't _wrong_ for Python 3, it's trying to do the best it can, it just happens to be inconsistent. I'll think some more, I think I can restructure the code to get it to work properly.


Ok, I've fixed the broken tests. I agree with you though @sigmavirus24, I think this is scary enough to want to go into 3.0.0, so I'm closing this PR and opening a new one with the correct target.


Please see #2754 for the new version of this change.
utils.get_netrc_auth silently fails when netrc exists but fails to parse

My .netrc contains a line for the github auth, [like this](https://gist.github.com/wikimatze/9790374).

It turns out that `netrc.netrc()` doesn't like that:

```
>>> from netrc import netrc
>>> netrc()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/netrc.py", line 35, in __init__
    self._parse(file, fp, default_netrc)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/netrc.py", line 117, in _parse
    file, lexer.lineno)
netrc.NetrcParseError: bad follower token 'protocol' (/Users/david/.netrc, line 9)
```

`get_netrc_auth` catches the `NetrcParseError` [but just ignores it](https://github.com/kennethreitz/requests/blob/master/requests/utils.py#L106).

At least having it emit a warning would have saved some hair-pulling.


Yeah, this is tricky. You can see the reason we added this logic in #449.

Given that you're using the function explicitly, maybe the best option is to add a kwarg to `get_netrc_auth` that allows it to be used in an explodey way (when users like yourself want to be explicit), but that allows us to continue to use it in a not-explodey way.


Because this can be done in a backwards compatible way, I've tagged this optimistically for 2.8.0
Documentation of verify parameter inconsistent with examples on stackoverflow

requests is a very nice replacement for ugly code you would write using urllib and urllib2.  After converting my code to requests and testing extensively I found a glitch when packaging my code with pyinstaller (I guess the same thing happens with py2exe).  When connecting to https sites requests apparently depends upon a certificate file which is hard to find in the release tree.  (When you package python code with pyinstaller you must also package any required data files.)

Posted solutions to this problem show setting the 'verify' parameter for the post and get methods to the path to a cacert.pem file grabbed from the certifi python package.  I tried this solution and it works.  (I also tried the documented cert parameter pointing to a cert and key file pair used by my django server but that did not work.)  One such answer is at http://stackoverflow.com/questions/10667960/python-requests-throwing-up-sslerror

This demonstrates that the verify parameter has undocumented functionality:  it accepts more than the True and False values shown in the documentation.

Could you correct the documentation to document this functionality.  And hopefully you could explain the difference between setting a pem file with the verity parameter vs the cert parameter.


@nmgeek You may want to take a look at http://docs.python-requests.org/en/latest/user/advanced/#ssl-cert-verification where it is documented: 'You can pass verify the path to a CA_BUNDLE file with certificates of trusted CAs.'

This really could use an example though.


@t-8ch is right, this _is_ documented, just not very well. As to your point of `verify` vs `cert`, that's covered in the same section of the prose documentation that @t-8ch linked to.
Display content as part of HTTP error messages


Unlike #2648 I think this is a not a good idea.

The content of a message is potentially extremely large or unbounded. It should not appear in tracebacks. Additionally, if you've set `stream=True` because you're expecting a large response in any form then the body may not have been downloaded yet, in which case we will actually block behind obtaining the request body before we throw the exception. Generally I'd say an exception we _know_ we want to throw should not block behind I/O before being thrown.

If you want to log this information out yourself you can do: catch the exception from `raise_for_status` and then consume the content yourself. =)


Ok Thanks for the feedback :)
Display URL as part of HTTP error messages

It seems convenient to include the URL in the error message in case you get an unexpected error.

E.g.:

```
In [1]: import requests

In [2]: resp = requests.get('http://www.google.com/eofdfdfdfdfd')

In [3]: resp
Out[3]: <Response [404]>

In [4]: resp.raise_for_status()
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-4-00e7077cfb5b> in <module>()
----> 1 resp.raise_for_status()

/Users/marca/dev/git-repos/requests/requests/models.py in raise_for_status(self)
    835
    836         if http_error_msg:
--> 837             raise HTTPError(http_error_msg, response=self)
    838
    839     def close(self):

HTTPError: 404 Client Error: Not Found for url: http://www.google.com/eofdfdfdfdfd
```


Seems reasonable enough to me. I'm happy to take this. @sigmavirus24?


Seems like a good UX improvement to me. :+1:
incorrect status_code

import requests
import time
import random

url = 'http://myserver.com'

n = 0

while(True):
    resp = requests.get(url)
    now=time.time()
    n += 1
    print(n, resp, resp.status_code, time.ctime(now))
    print(resp.headers)
    r=random.randint(1,5)
    time.sleep(r)
    print('\t\t\t slept',r, 'seconds')
# 

Output:

1 <Response [200]> 200 Thu Jun 18 23:15:57 2015
{'pragma': 'no-cache', 'set-cookie': 'OutlookSession=d2b7de620d414931b8100369275493d4; path=/; secure; HttpOnly', 'expires': '-1', 'x-owa-version': '14.3.181.6', 'x-powered-by': 'ASP.NET', 'date': 'Fri, 19 Jun 2015 06:15:56 GMT', 'content-type': 'text/html; charset=utf-8', 'vary': 'Accept-Encoding', 'content-encoding': 'gzip', 'content-length': '3251', 'cache-control': 'no-cache, no-store'}
             slept 5 seconds
2 <Response [200]> 200 Thu Jun 18 23:16:02 2015
{'pragma': 'no-cache', 'set-cookie': 'OutlookSession=9093d72589364d9894173a5ed0475a45; path=/; secure; HttpOnly', 'expires': '-1', 'x-owa-version': '14.3.181.6', 'x-powered-by': 'ASP.NET', 'date': 'Fri, 19 Jun 2015 06:16:01 GMT', 'content-type': 'text/html; charset=utf-8', 'vary': 'Accept-Encoding', 'content-encoding': 'gzip', 'content-length': '3251', 'cache-control': 'no-cache, no-store'}
.....
# 

While actual response code is 300 as seen in wireshark capture:
HTTP/1.1 302 Object Moved\r\n


￼
![screen shot 2015-06-18 at 11 21 48 pm](https://cloud.githubusercontent.com/assets/10746779/8248193/d4ae4228-1610-11e5-9b63-f84c66b703bb.png)


This is not a bug.

Requests will follow all redirects until the chain is complete. What has happened here is that you've been redirected from HTTP to HTTPS by the second 302, which means your wireshark filter does not show the next transaction. =)

If you print `resp.history` you'll see the full redirect chain we followed.
deflate failure on google appengine above requests 2.3.0

if response was given with deflate,gzip or just deflate (Content-Encoding: "gzip, deflate") then requests on google appengine would failure this line
packages/urllib3/response.py:440

``` python
if self._original_response and self._original_response._method.upper() == 'HEAD':
```

It happens because self._original_response is replaced by google sdk and has different format from inner Response class objects.
gae:

``` python
<google.appengine.dist27.gae_override.httplib.HTTPResponse instance at 0x7f66f51cbf80>
```

origin:

``` python
<httplib.HTTPResponse instance at 0x7f59742b0638>
```

Though original_response._method is not str (it's actually int).

This breaks content deflating e.g. requests will process request normally but response.content would contain binary (compressed) data.

Tested on 2.7.0 branch. 2.3.0 works well.


Sorry, this is a bug in [urllib3](https://github.com/shazow/urllib3), do you mind opening an issue for it on that repository?


This should be fixed when https://github.com/shazow/urllib3/pull/620 is merged and we pull it in for a new release.


np, thanks for advance :)
make api.py support keepalive

Use an object pool, making api.py support for keepalive


Thanks for this @duanhongyi!

However, I'm :-1: on this idea. My primary objection is that this adds implicit global state. Generally speaking I don't think libraries should maintain any internal state at all except what is truly necessary for their function. Wherever possible we should provide 'state objects' to users that the user has control over, ensuring that users own the lifetimes of objects: we already do this.

I think having a pool of sessions in the background is a bad idea. It'll lead to bug reports from users who aren't expecting it and it'll lead to complaints from users who don't like the memory profile it causes. Most importantly, it makes it _very very difficult_ to obtain reproducible behaviour out of the top-level API, because what exactly happens on a given request is actually dependent on the entire lifetime of the program up until that point. `requests.get()` may or may not work depending on whether there are cookies present in the particular `Session` you're using. It may or may not work depending on whether the remote server supports keepalive connections (a surprising number don't handle it well).

I'll let @sigmavirus24 express an opinion as well, but I'm sorry, I doubt we'll merge this.


@Lukasa 

I agree with you, but the global shared socket connection, which is common in programming languages such as Java, is also practical. and perhaps adapters.py is more appropriate for doing this.


Any user who wants it can easily achieve a very similar effect:

```
from requests import session as requests
```

This gives the implicit shared pool that everyone wants. Bear in mind the session API is a strict superset of the `api.py` API, so it should be a simple replacement.


I agree with @Lukasa. This will also likely negatively affect things like grequests which build on the api module's design. I also don't think we want this in `requests/adapters.py`.
Requests does not support encoding headers to ascii?

A simple test case:

``` python
request = requests.Request(method='GET', url='https://httpbin.org/get')
request.headers = {u'referer': u'http://xx.com?x=føø'}
prepared = request.prepare()
requests.Session().send(prepared)
```

```
Traceback (most recent call last):
  ...
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 566, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 331, in send
    timeout=timeout
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 558, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python2.7/httplib.py", line 975, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1008, in _send_request
    self.putheader(hdr, value)
  File "/usr/lib/python2.7/httplib.py", line 955, in putheader
    hdr = '%s: %s' % (header, '\r\n\t'.join([str(v) for v in values]))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 17-18: ordinal not in range(128)
```


Requests does not encode header values for you because we don't know what encoding you want to send. You'll need to pick an encoding for the header value yourself and encode it.
Same HTTPConnection object returned to pool twice

Not totally sure how to reproduce this yet but I am definitely getting a case where I am getting the "Connection pool is full" message because the same connection object is being returned to the pool twice  (and not surprisingly the second time the pool says that it is full).

I am using gevent (yes I realize that is probably crucial), but even so ...

The first time the connection gets returned is in requests.adapters:

410                 else:
411                     # All is well, return the connection to the pool.
412                     conn._put_conn(low_conn)

The second time the connection gets returned is in requests.packages.urllib3.response:

285         finally:
286             if self._original_response and self._original_response.isclosed():
287                 self.release_conn()

I'd love to  be able to supply a simple example, but the code path is a little to convoluted for me to work out quickly so I'm looking for hints/tips here if you have any.

Thanks,
Giles


I've seen this and wasn't able to come up with a simple case to reproduce it. /cc @shazow since we discussed it


Maybe we need a mutex around our release_conn() innards.


Actually it turns out that the bug is super easy to reproduce.

https://gist.github.com/gilesbrown/3832fc29100213171968

It doesn't require gevent, all it requires is doing a chunked PUT (or POST etc).


Interesting. This line of code looks fishy to me:

``` python
# All is well, return the connection to the pool.
conn._put_conn(low_conn)
```

We do that internally, despite having set `preload_content=False`. We may well then consume the content elsewhere.

I think that's just wrong, really, and it opens us up to bugs.


And, quite expectedly, commenting out that block causes everything to function properly. I just think that's wrong.


It should be noted that [I was the idiot](https://github.com/kennethreitz/requests/commit/31c0962e83c8f3b043b3faceff880f69a9a4dbdc) who put that release in the code in the first place, nearly two years ago. Past me is a jerk. =P


@Lukasa I did always like future-you better than past-you.


Well it's good that you're able to fix your fishy past so easily.
Handle upgrade to WebSocket (Feature Request)

it would be awesome if there was a way to handle the response for a request that upgrades to a websocket connection, for example:

``` python
url = "https://host.domain.com/mypath"
headers = {
    "Authorization": "myrealm <my-auth-token>",
    "Connection": "upgrade",
    "Upgrade": "websocket",
    "Sec-Websocket-version": "13",
    "Sec-Websocket-key": "<some-challenge-key>",
}
payload = {
    "some_key": ['some', 'values']
}
def handler(message):
    print(message)
response = requests.post(url, headers=headers, json=payload, websocket_callback=handler)
```


I'm afraid that we're extremely unlikely to add support for this at the requests level in this form. What you'd have to do is use `stream=True` and then grab the socket out of the response object at a very low level.


I have to agree with @Lukasa. This is not on the roadmap for future versions of requests at this point.


All right, fair enough.
Does requests support url anchor

Can the url  containing both parameters and anchor, like http://xx.com?a=1&&b=2#here


It absolutely can, but you have to build the URL yourself. If you ask requests to encode the parameters for you (using params) then we'll throw the anchor away.

It seems like a reasonable request to keep hold of that anchor: @sigmavirus24?


I'm confused about what's being discussed:

``` py
>>> req = requests.Request(method='GET', url='http://xx.com?a=1&&b=2#here')
>>> p = req.prepare()
>>> p.url
'http://xx.com/?a=1&&b=2#here'
```

Indicates that the URL is not affected. Are we talking about something like:

``` py
requests.get('http://xx.com', params='a=1&&b=2#here')
```

That shouldn't work because we expect params to be a dictionary

``` py
requests.get('http://xx.com', params={'a': '1', 'b': '2#here'})
```

In this case, we'd have to do a lot of pre-processing on the params to make sure the value with `#here` is at the end and not encoded... which may not be what people want.

But I'm probably completely missing the point.


The thing you _might_ expect to work is this:

``` python
requests.get('http://hostname/path#fragment', params={'param': 'value'})
```

That does not work.

All the other things function correctly.


Yeah, I'm confused:

``` py
>>> import requests
>>> r = requests.Request(method='GET', url='http://example.com/foo#bar', params={'a': 'b'})
>>> p = r.prepare()
>>> p
<PreparedRequest [GET]>
>>> p.url
'http://example.com/foo?a=b#bar'
>>> r = requests.get('https://httpbin.org/get#fragment', params={'a': 'b'})
>>> r.json()
{u'origin': u'127.0.0.1', u'headers': {u'Host': u'httpbin.org', u'Accept-Encoding': u'gzip, deflate', u'Accept': u'*/*', u'User-Agent': u'python-requests/2.7.0 CPython/2.7.9 Darwin/14.1.0'}, u'args': {u'a': u'b'}, u'url': u'https://httpbin.org/get?a=b'}
>>> r.url
u'https://httpbin.org/get?a=b#fragment'
```


Ah crap, I'm an idiot. I looked at the JSON response, forgetting that the fragment doesn't get transmitted to the server. Duh.

This all works totally fine!


@kaizengliu Please do not use the _bug_ tracker to ask questions. Our question forum is located [here](https://stackoverflow.com/questions/tagged/python-requests).


I got it.  It can work in the way like this

```
requests.Request(method='GET', url='http://example.com/foo#bar', params={'a': 'b'})
```

Thank you very much. And I am sorry,  I will ask question in [here](https://stackoverflow.com/questions/tagged/python-requests)