Raibek/requests-github-issues huggingface dataset:
Add ability to load ca bundle from data

SSLContext.load_verify_locations allows to specify cadata with 2.7.9+, 3.4+

Thanks for this! Some notes:

- Firstly, this patch contains changes to the urllib3 project. This is a sub-project vendored in its entirety unchanged into Requests.  That means any changes to urllib3 need to be made on that project, and they will then make their way into Requests over time.
- Secondly, I am not sure how important this patch is. In the current Requests release it is possible to pass a SSLContext in to a transport adapter, as discussed in a few other issues (e.g. [here](https://github.com/kennethreitz/requests/issues/3774#issuecomment-267871876)). This means it's already possible to use cadata, which drastically reduces the urgency of this patch.
- Thirdly, I suspect in v3 we'll want to refactor this so that verify can accept an `SSLContext` directly, rather than continuing to overload it with more stringly-typed inputs.

Because of the first reason I cannot merge this patch directly: you'd need to open a PR on urllib3 first. However, you may want to consider whether it's worth pursuing the "pass an SSLContext" approach instead.

Agree, passing custom SSLContext solves the issue.

Could you update documentation to better reflect that, with an example of passing custom SSLContext that loads its certificates from memory?

@Lukasa One question: won't implementation of `ssl_wrap_socket` override custom CA certificates by calling load_verify_locations inside?

@Lukasa That's how I did it so far:

```python
from requests.adapters import HTTPAdapter


class CustomSSLContextAdapter(HTTPAdapter):
    def __init__(self, *args, ssl_context=None, **kwargs):
        super().__init__(*args, **kwargs)
        self._ssl_context = ssl_context

    def init_poolmanager(self, *args, **kwargs):
        kwargs['ssl_context'] = self._ssl_context
        return super().init_poolmanager(*args, **kwargs)
```

It's certainly possible. You may need to make further adapter overrides so you can set `ca_certs` and `ca_cert_dir` to `None`.

@Lukasa Something like this:

```python
class CustomSSLContextAdapter(HTTPAdapter):
    """
    HTTP adapter with custom SSLContext.

    >>> from requests.sessions import Session
    >>>
    >>> with Session() as session:
    >>>     session.mount('https://', CustomSSLContextAdapter(ssl_context=...))
    >>>     session(method=..., url=..., ...)
    """
    def __init__(self, *args, ssl_context=None, **kwargs):
        """
        @param ssl_context: Custom SSL context.
        @type ssl_context: ssl.SSLContext
        """
        self._ssl_context = ssl_context
        super().__init__(*args, **kwargs)

    def init_poolmanager(self, *args, **kwargs):
        kwargs['ssl_context'] = self._ssl_context
        return super().init_poolmanager(*args, **kwargs)

    def cert_verify(self, conn, *args, **kwargs):
        super().cert_verify(conn, *args, **kwargs)
        conn.ca_certs = None
        conn.ca_cert_dir = None
```
?

I think that should be fine.
Timeout in stream request closes connection from client side

I've hit a problematic behaviour with python3 and requests, stemming from urllib3. Consider the following example usage with tight read timeout.

After getting the first exception, the connection is cleanly shut down from clientside. Serverside gets Broken pipe. I'd expected to be able to continue reading from the stream, without needing to reconnect.

Reading the urllib3 explanation clarifies the issue somewhat, but is this wanted behaviour?
```
#!/usr/bin/env python3
import requests
res = requests.get("http://localhost/stream", stream=True, timeout=(10.0, 1.0))
reason = None
while not reason:
    try:
        line = res.raw.readline()
        print(line)
    except Exception as e: print (e)
```

Full traceback is
```
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 228, in _error_catcher
    yield
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 310, in read
    data = self._fp.read(amt)
  File "/usr/lib64/python3.5/http/client.py", line 448, in read
    n = self.readinto(b)
  File "/usr/lib64/python3.5/http/client.py", line 488, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib64/python3.5/socket.py", line 575, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "example.py", line 8, in <module>
    except Exception as e: raise (e)
  File "example.py", line 6, in <module>
    line = res.raw.readline()
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 320, in read
    flush_decoder = True
  File "/usr/lib64/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3.5/site-packages/requests/packages/urllib3/response.py", line 233, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost'): Read timed out.
```

Timeouts are considered error conditions: that is, we assume that if the server takes longer than a timeout to generate a value, you want us to treat that as an error. That means we tear the connection down, because it's now in an indeterminite state: the server may have gone away, or any number of other bad things have happened.

This is entirely expected behaviour. Requests does not support a form of non-blocking operation, so if you want to do stuff while you're waiting for downloads you should set longer timeouts and do your HTTP requests in a separate thread, allowing your main thread to execute while the background thread does the HTTP.
Cannot clone repo due to bad timezone error

```
nyuszika7h@cadoth ~/src > git clone https://github.com/kennethreitz/requests.git
Cloning into 'requests'...
remote: Counting objects: 18867, done.
remote: Compressing objects: 100% (11/11), done.
error: object 5e6ecdad9f69b1ff789a17733b8edc6fd7091bd8: badTimezone: invalid author/committer line - bad time zone
fatal: Error in object
fatal: index-pack failed
```

Thanks for raising this issue! In future, please search the issue history before opening new issues on projects, as you'll often find that your issue has been reported before. In this case, this is a duplicate of #3088, #3008, and #2690.

Sorry. I did search, but didn't get the right terms.

If you have an error message, it's usually sufficient just to try to search for that. In this case, I searched for "fatal: index-pack failed" and #3088 popped right up.

Yeah, well, I searched for "timezone" because I didn't realize there's a space in it in the error message. (Also, it's strange, I don't remember enabling fsck, but I found it in my git config.)

fsck is on-by-default on new installs of git, so unless you're persisting a gitconfig from an older version of git you'll always find it on. =)

I do have a custom gitconfig, so that's strange. This kind of issue could have been avoided if they enabled fsck by default in the first place, but oh well. You can't rewrite 6 years of history now, that would just mess things up even more. I do wonder how that weird timezone ended up in a commit though.

Yeah, it's really annoying because it causes legitimate problems for lots of people, especially those who use git in automated scripts, but we really can't do anything about it now. =(

Even though it hurts to do this, you may want to consider rewriting the history to fix this problem once and for all. For Breathe I ended up doing this and I added a notice to the top of the README, in the biggest heading possible, for a while.

Projects using this as a submodule will have to update the SHA once. But currently many setups encounter a clone failure meaning existing submodules will not work at all and cloning this repo directly won't either.

Ref: https://github.com/michaeljones/breathe/issues/340#issuecomment-394396673
Improve discoverability of OAuth 2 support

The previous summary gave the impression that requests-oauthlib only supports
OAuth 1.

This updates makes it clear that it also supports OAuth 2, and links directly to the
use case specific authentication flow guides.

I'm currently working on adding OpenID Connect support to a pre-existing web application (https://github.com/fedora-infra/anitya/issues/329) and found it very hard to find good documentation on writing a full semi-automated integration test where the only manual step is the user authorising a temporary access token.

Part of the problem was hitting this page in the docs and thinking that requests didn't have native OAuth 2 support - it was only after clicking through to the full requests-oauthlib documentation that I realised that that initial impression was wrong.

Thanks for having the code available to find!

You may be amused by the horrible hack I wrote to automate obtaining a refresh token for Anitya's Fedora Account System integration tests: https://github.com/ncoghlan/anitya/blob/authenticated-api/request_oidc_credentials.py :)

permalink for above broken link:
https://github.com/release-monitoring/anitya/blob/c369715f5ae7873066ca92b7ae3c412f01711e16/request_oidc_credentials.py
Add **kwargs to cookiejar_from_dict

It's can be useful for set not default cookies attributes values, for example "domain" or "discard".

So, while we can do this, are you not likely to be better served by creating a CookieJar yourself at this point? `cookiejar_from_dict` as a utility function is mostly intended for internal use to support the usage of dict in our APIs: we kind of expect that users who want to get more granular control over cookie settings will simply directly use a CookieJar.

I agree with @Lukasa. While I know a significant number of people use this function, it's not a public API that we support and even so, this is the one of the least optimal ways of accomplishing what you want.
Use proxy to hide IP

Hi,
I'm trying to use a proxy to hide my IP with my script.

I'm just trying 
`requests.get("https://api.ipify.org?format=json", proxies={"http":proxies[0][:-1]}).json()`

But I get my real IP address, no the proxy's one. It that normal ? How is it possible to hide my IP with requests ?

Thanks

The proxy dictionary is keyed on the scheme of the URL you are *requesting*. In your example you are requesting a *https* URL with a proxy dictionary that only has a *http* key. This means Requests thinks you don't want to proxy https requests, and so sends them directly. Add *https* to your proxy dictionary as well, using the same proxy, and you should be fine.
Why LifoQueue and not simply Queue (or custom LifoQueue)?

During some profiling, I found out one slow operation is `self.pool.get()` in `_get_conn()`.

In urllib3, in `connectionpool.ConnectionPool`, requests uses `LifoQueue` (which uses a list instead of a much faster deque):
https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/connectionpool.py#L64

My suggestion is one could use a custom LifoQueue as:
```python
class LifoQueue(Queue):
    def _get(self):
        return self.queue.pop()
```
or simply use a Queue, if Lifo is not really needed.


The LIFO property of the LifoQueue is very much needed: this is to ensure the increased likelihood of using a warm TCP connection, which provides substantial efficiency bonuses. The reason it's a queue, instead of just a deque, is to get the thread-safety guarantees that the queue module provides.

You are welcome to attempt to optimise this, but those two semantics need to be preserved. I suspect you'll find that the major performance cost is actually the locking, and by the time you reintroduce that the perf gains are minimal.

Further the code you're linking to is actually in urllib3, thus making this not only an invalid defect report but also one on the wrong project.
Customized proxies seems not work, how can we know the proxies's IP address send to server?

Take this as an example: 

```
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import requests 

proxies = {"http":"http://111.222.333.444:80"}
r = requests.get("https://www.google.com", proxies=proxies, timeout=5)
r.status_code
r.content 
```

Obviously, the proxies "111.222.333.444" is not correct, but why I still got status_code 200??

How can we check the proxies we used to send to server?  


The `proxies` dictionary is defined as having the following form: the keys are the URL schemes for the URLs you are requesting, and the values are the URLs for reaching the proxy.

That means when we make a web request, we take the scheme of the URL you're requesting and look for a proxy for it. In your case, the scheme is `https` (because you're looking for `https://www.google.com`). Your proxy dictionary has no `https` key, so that means "no proxy", so we just go directly to Google.

If you change your `requests.get` to be `http://www.google.com`, then you'll find that we attempt to reach the proxy and error out.

Thanks, so rapid reply!!
Can you also reply the second question: can we retrieve the proxies send to server? some time, I got error, and I am pretty sure the schemes are same but still got an error, like use free proxies, some of them may out of date. 

@withr Unfortunately, not easily. Generally speaking you should look at the proxies dictionary you provided. Alternatively, you can write a custom Transport Adapter that will provide the information you need, though that's quite a lot of work.
Help: how to upgrade urllib3 for solve "filedescriptor out of range in select()" issue

requests version:  2.12.4
python version:  2.7.9
platform: linux(CentOS)

issue occured:
```shell
ValueError: filedescriptor out of range in select()
```
detail description : https://github.com/shazow/urllib3/commit/a49bec58fa5f7aca76d1d0b2f1975eb094648eab


and this issue have been fixed :
https://github.com/shazow/urllib3/issues/589
https://github.com/sigmavirus24/urllib3/commit/eb8264f55be9cd6b2d0be2833c5ba48058ce999d


I wonder why requests don't upgrade urllib3? and **how can I upgrade urllib3 for requests?**



Requests uses a vendored copy of urllib3, so we ship our own updates to it. That means you cannot upgrade urllib3 separately to requests.

While I am here, I should note that there is no released version of urllib3 that contains that fix, you'd have to use urllib3 from master. If that's the case, you are best served by cloning the Requests repository and updating the copy of urllib3 inside it and installing that.

@Lukasa Sorry to bother on closed issue but I'm wondering why should this fix not being updated in Request?

@RobGThai We pushed requests v2.13 today which contains that fix.
Require pytest-mock for the tests

test_requests.py `test_session_close_proxy_clear` uses the
`mocker` fixture, which is provided by pytest-mock.

Now I look at it, the pytest-httpbin required version also differs between setup.py and requirements.txt (as a downstream distributor I also much prefer `>=` to `==`, but...:>)

Perhaps it would be worth reducing test_requirements to a non-versioned list like [urllib3 does](https://github.com/shazow/urllib3/blob/master/setup.py#L48-L54)? That allows a minimum starting point for someone running tests through setup.py, while avoiding having to remember updating dependencies in multiple places. It looks like the chosen versions in test_requirements were simply the latest ones available when test_requirements was added (6c2942b).

FWIW, we distributors like `install_requires` and `test_requires` over the 'everything together' format of `requirements.txt` because we have the same split at the distro level: we need to have all the test requirements present when *building* the package (assuming we're going to run the test suite) - so as `BuildRequires`, in RPM terms - but we only need the generated packages to require the stuff from `install_requires` at install time (so as `Requires` in RPM terms). Since `requirements.txt` draws no distinction it's hard for us to use.

We have no `requirements.txt` requirement: all the setup.py stuff should work. It's just that requirements files can be handy for things like tox, which can try to be clever about tracking changes on it. 

Regardless, we should being setup.py into line with requirements.txt for testing purposes. I see no reason to remove the version pins at this time.
Pyinstaller error with requests version > 2.11.1

Hi
I am developing an application with python, using _requests_ library in it.
I make executable file with _pyinstaller_ library.

My application works with newest version of _requests_ when I run it through python commands or inside IDLE. But when I get EXE output with _pyinstaller_, It doesn't run correctly. Here is the message:

"
... (trace files)
ImportError: No module named 'queue'
During handling of the above exception, another exception occurred:
... (trace files)
ImportError: No module named 'urllib3'
Failed to execute script.
"
So I figured out that there is something wrong with _**urllib3**_ library. Then I downgraded _requests_ to "2.11.1" version (One before updating bundled urllib3 to 1.19) and created another EXE file. Rerun it, and worked!

Now I know that there is something wrong with newer versions of _requests_ (> 2.11.1) and _pyinstaller_. But don't know how to fix it. Could you plz help me?

Thanks.

This is strictly a PyInstaller issue. Requests works fine, and vendors all of its dependencies. Clearly PyInstaller is having trouble locating the dependent module, even though it is entirely within the Requests package. That problem is PyInstaller's, and you'll need to take it up with them.

This is caused by requests' bundling it's own copy of `six`.
Invalid detection of utf-32-be

```python
import requests

requests.get('http://localhost/example.json').json()
```

```
Traceback (most recent call last):
  File "example.py", line 3, in <module>
    requests.get('http://localhost/example.json').json()
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 850, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.6/json/__init__.py", line 344, in loads
    s, 0)
json.decoder.JSONDecodeError: Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)
```

```
# wget --server-response http://localhost/example.json
--2016-12-29 10:49:23--  http://localhost/example.json
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Thu, 29 Dec 2016 03:51:03 GMT
  Server: Apache/2.4.10 (Debian)
  Last-Modified: Thu, 29 Dec 2016 03:42:32 GMT
  ETag: "c-544c3e019ba33"
  Accept-Ranges: bytes
  Content-Length: 12
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: application/json
Length: 12 [application/json]

```

http://localhost/example.json is file encoded with utf-32-be.
```json
{}
```

Fix #3791
3780: Lazy load idna library

Changes based on comment from previous Pull Request: https://github.com/kennethreitz/requests/pull/3787

Fix for issue: https://github.com/kennethreitz/requests/issues/3780


@Lukasa Please let me know your thought on this. I think it should address your concern in my previous Pull Request.

@sigmavirus24  Not sure why it is showing the status as "Changes requested" even though they are fixed in commit: https://github.com/kennethreitz/requests/pull/3789/commits/965eb3d1525e3d384623c0267654c2f07acc4aea

May be it due to additional import of `sys` (needed but was missing in previous PR)

@moin18 that will be in place until sigmavirus24 has a chance to review your changes again. Today is still a "holiday" for a lot of people working in the US, so sigmavirus24 likely won't respond immediately. Things look pretty cleaned up now, so I think you can leave these changes and he'll respond when he has a spare moment :)

@sigmavirus24  I have fixed the comments you mentioned. Please can you review them once more? :)

@moin18 would you be willing to rebase this rather than merging master into this branch? Alternatively, @Lukasa how do you feel about squash merging this?

I have no objection to squash merging.

@sigmavirus24 @Lukasa cleaned up the commit history. Merged all commits into single commit.
Lazy load idna to free memory

Fix for issue: https://github.com/kennethreitz/requests/issues/3780

Lazily loading `idna`. This CR includes utilities to:
* lazily load idna package
* And, delete module from sys.modules after use

Changes looks to work fine for me. Now, None of the reference to `idna` is getting used directly. GC will release the memory when required.

I think it will be a better idea to create a separate package to lazy load module and add that package as the part of both `urllib3` and `requests` package

urllib3 does not vendor idna, so such a package cannot really be easily shared. 

Additionally, Requests does not carry patches to urllib3: it vendors an unedited copy. Any urllib3 patch you need to make must be proposed to that project. 

@Lukasa : In that case I will be creating a separate patch for urllib3 discarding the urllib3 related changes from this PR. Apart from that, does the `requests` library related changes looks good to you?

I think it looks really over complex. Repeatedly loading and unloading the module is solving a non-existent problem. All we should do is defer loading as late in the process as possible.
fix urllib3 documentation link

The link to urllib3 documentation in the appengine warning message shows " SORRY This page does  not exist yet."

This PR fixes the documentation link.

Thanks for the patch!

@Lukasa it looks like this is a modification in the appengine module in urllib3. The warning update may need to go over there, otherwise we'll overwrite it the next time we update urllib3.

Agh good spot, I missed that. Can we open a new patch on urllib3 that DTRT?

@Adusei would you be interested in submitting this patch to [urllib3](https://github.com/shazow/urllib3) so this can be fixed at the source?

urllib3 has the proper documentation link.. checking it now, and seems that they fixed the issue with these two commits last month:

- https://github.com/shazow/urllib3/commit/097b935bea9fa575c59391e8c30a3ab00c974fae
- https://github.com/shazow/urllib3/commit/9688f0311839ce79c59b5254a8c68c148e73528c


Great, things should be set for future releases then. Thanks for taking a look @Adusei!
SSL context

In some instances, requests is used to access services via SSL that disable TLS v1.0 for security reasons but there is no way to configure what SSL/TLS versions are acceptable nor pass in a SSL context. This is very disappointing.

For example, I *need* to be able to create this context and use it, but I can't...
```
import ssl
context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
context.options |= ssl.OP_NO_TLSv1
```

Thanks for this report!

Firstly, services that disable TLSv1 should still be compatible with Requests. Requests advertises support for all TLS versions higher than SSLv3 in the default configuration, so services should simply choose TLSv1.2. Requests will happily do that. 

Secondly, you absolutely can pass an SSL context as of v2.12. See [this comment](https://github.com/kennethreitz/requests/issues/3777#issuecomment-267998174) for an example of how to do it.

I hope that calms your disappointment. 

I should note: *please* do not use the context in your original comment. It's severely insecure. Please always use `create_urllib3_context` and then edit it accordingly: you'll get much higher security that way, and we'll keep moving forward with more secure configurations. 

> Firstly, services that disable TLSv1 should still be compatible with Requests. 

In an ideal world, yes. This is the real world which is far from ideal.

> Requests advertises support for all TLS versions higher than SSLv3 in the default 
> configuration, so services should simply choose TLSv1.2. Requests will happily do 
> that.

Obviously not.

> Secondly, you absolutely can pass an SSL context as of v2.12. See this comment 
> for an example of how to do it.

Also not true. I have to create an entire adapter to pass into it instead of simply passing in a context. However, it does work! I was able to successfully modify Zeep to nicely handle that new adapter for the session and it worked like a champ but I still had to construct my own context that was only slightly different than the one urllib3 made. 

> I should note: please do not use the context in your original comment. It's severely 
> insecure. Please always use create_urllib3_context and then edit it accordingly: 
> you'll get much higher security that way, and we'll keep moving forward with more 
> secure configurations.

So far from the truth. What does `create_urllib3_context` do? Literally the same thing I did.

In the end, thank you for the suggestion because that adapter did exactly what I needed it to. Unfortunately Zeep is such a mess that it may be a moot point although I should be able to modify Suds to use requests. I am still a fan, and advocate, of the `requests` module! 

Cheers Lukasa!
Closes #3780 High memory usage due to idma import


Because of how Python imports work, this also will not have any effect. Specifically, `from x import y` is approximately the same as `import x; y = x.y`. I'm afraid we need a better solution than even this. 

Sorry!

By better solution do you mean the changes in: `idna/core.py` that is imported by `__init__.py`?

Ideally we'd only bring in IDNA when it was actually needed: that is, when we had a domain we had to internationalise.

What are your thoughts on this?

```
@staticmethod:
def _get_idna_encoded_host(host):
      from . package import idna
      try:
         host = idna.encode(host, uts46=True).decode('utf-8')
     except idna.IDNAError:
         raise UnicodeError

.....

if not unicode_is_ascii(host):
    try:
        host =self._get_idna_encoded_host(host)
    except UnicodeError:
         raise InvalidURL('URL has an invalid label.')
elif host.startswith(u'*'):
    raise InvalidURL('URL has an invalid label.')
```
                

@moin18 The problem I ran up against was that `idna` is part of `packages` so whenever `packages` is loaded (e.g. in `models.py`) `idna` will be loaded at that point. So the fix would be for `idna` not to be loaded at that point but somehow still be loaded when needed :/
Closes ##3780 High memory usage due to idma import

Fix related to bug: https://github.com/kennethreitz/requests/issues/3780

`idma` package will now be imported within the function

Thanks for this!

However, this patch does not meaningfully reduce the memory usage. Any use of Requests will encounter this branch immediately and unconditionally, meaning that in practice all that we have done is delay the loading of the module. I think we need a better solution than this one.

How about doing selective import of the needed objects?

Raised another PR: https://github.com/kennethreitz/requests/pull/3783


On Wed, Dec 21, 2016 at 7:11 PM, Cory Benfield <notifications@github.com>
wrote:

> Closed #3782 <https://github.com/kennethreitz/requests/pull/3782>.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/kennethreitz/requests/pull/3782#event-901602069>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AE_RIPmien3rnNTVgjj6Y4O2yNmqOqr9ks5rKSyVgaJpZM4LS6ge>
> .
>
updating idna.uts46data from upstream project


This is meant to include https://github.com/kjd/idna/pull/33 which will fix https://github.com/kennethreitz/requests/issues/3711.

The patch was done by running `make idna`. Also patching Makefile itself, removing `sed` invocation, which isn't necessary anymore.

Thanks for this patch @mplonka. However, we would generally prefer to update the bundled packages ourselves at the time of release: that way, we have some confidence that we have done the sensible thing. Can you pare this patch down to just the makefile change, please?

Thanks for reviewing this PR, @Lukasa .

As per your request, here's the revert commit.

Thanks! For the sake of clarity, do you mind squashing these into one commit? 

I actually wanted to suggest squashing.

@Lukasa it seems that idna hasn't been updated yet, but a new version has been released in the mean time. Will idna be updated to fix the Jython issue?

@LordGaav Per [our documentation](http://docs.python-requests.org/en/master/community/release-process/#hotfix-releases), we do not update vendored dependencies in patch releases. Please wait for v2.13, which will contain an update.

That was quick. I'll use `requests!=2.12.*` in my requirements.txt in the mean time then. Thanks.
idna package increases memory usage by ~20MB

I am running an Ubuntu 16.04.1 LTS machine with Python 3.5.2 which has about 40 Python scripts loaded, each one of which uses Requests. I recently updated Requests and suddenly found I was running out of memory.

Upon further investigation I discovered that as of version 2.12.0, the `idna` package has been included in requests and this has caused the memory consumption for each script to increase by about 20MB.

This can be observed with the following steps:

1. Install Requests version 2.11.1
2. Run `python3 -c "import requests; import time; time.sleep(60)" &`.
3. While the previous command is running, run `top` or `htop` and observe the resident memory usage of the command.
4. Install Requests version 2.12.0.
5. Repeat steps 2 and 3.

Is there any way that Requests could lazily load the idna library so that this extra 20MB of memory is only used when actually working with IDNA domains?

Yeah, requests could absolutely lazily import idna. Anyone who would like to can work on a patch to defer this import. =)

Just had a look at where `idna` is used (https://github.com/kennethreitz/requests/blob/7d2dfa86841fae49fa82f4f59098d5be862d1ba0/requests/models.py#L371) and it seems it's being used for every request; is there any check we can do to see whether `idna` needs to be loaded here?

@DanielGibbsNZ Not enormously effectively, no. In principle the check in the `except` block could be used, I suppose.

Something like this perhaps?

```
if not unicode_is_ascii(host):
    from .packages import idna
    try:
        host = idna.encode(host, uts46=True).decode('utf-8')
    except (UnicodeError, idna.IDNAError):
        if not unicode_is_ascii(host) or host.startswith(u'*'):
            raise InvalidURL('URL has an invalid label.')
```

That's sensible enough, though we would want to avoid doing the `unicode_is_ascii` check twice.

Hmm, looks slightly more complicated than I thought to actually get this working as I'm not sure how to get `packages/__init__.py` to lazily load `idna`. Additionally, `urllib3`'s `pyopenssl` uses `idna` as well. I might leave this for someone more experienced than I.

@Lukasa: `idna` is also imported in `urllib3/contrib/pyopenssl.py`. Here: https://github.com/kennethreitz/requests/blob/master/requests/packages/urllib3/contrib/pyopenssl.py#L46

In order to remove the reference of `idna` via garbage collector, corresponding changes in urllib3 are also required. The idea is to do lazy loading of idna package only on demand basis. 

Let me know if I am on correct path. If yes, I will raise the bug and pull request to change this behavior in urllib3 as well.

Sample diff is available here: https://github.com/kennethreitz/requests/pull/3787

For the urllib3 case, we could also lazily load idna, but let's fix it one step at a time.

Looks like it's already fixed there.

Looks like this is now resolved with #3789.
Handle Content in new tab

I tried to scrap report from https://www.theice.com/marketdata/reports/datawarehouse/ConsolidatedDailyVolumeOIReport.shtml?selectionForm=

We can find Report Date, Asset Class and a Submit button
When we hit submit it actually opens in a new Tab as 

https://www.theice.com/marketdata/reports/datawarehouse/ConsolidatedDailyVolumeOIReport.shtm

I tried simulating POST, but it always says Session Time Out.

On Contrast if you could see
https://www.theice.com/marketdata/reports/175.
When you enter dates and hit submit, it opens but not in a new tab, in this case requests is able to get the data.

Is there any solution for this? Please suggest.

Questions belong on [StackOverflow](https://stackoverflow.com) using the [python-requests tag](http://stackoverflow.com/questions/tagged/python-requests) not in the defect/bug tracker.

But it's not working?

@arunchandramouli taking a quick look at those websites, it looks as if they require JavaScript in order for you to successfully scrape their data. Requests does not (and has never) executed JavaScript. You need a different tool. Requests is working exactly as expected. There is no bug here.
Proxy socks windows

`$ pip install requests[socks]`

`>>>import requests`
`>>>proxies = {'http': 'socks5://127.0.0.1:9150','https': 'socks5://127.0.0.1:9150'}`
`>>>r = requests.get('https://api.github.com', proxies=proxies)`
`Traceback (most recent call last):`
`...`
`   raise InvalidSchema("Missing dependencies for SOCKS support.")`
`requests.exceptions.InvalidSchema: Missing dependencies for SOCKS support.`

Package is installed win_inet_pton with pysocks automatically.
`$ pip install win_inet_pton`

Can you provide the output of `pip freeze`, and confirm that when you type `import socks` that that works?

I confirm for him ;)

```
>> pip freeze
future==0.16.0
psutil==5.0.0
PySocks==1.6.5
requests==2.12.4
six==1.10.0
win-inet-pton==1.0.1
```


```
>> python
Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import socks
>>>
```


@yoyoprs I believe @Lukasa wanted you two to `import socks` and then check that the version was the same as what `pip` reports.

Workaround:

```
pip install win-inet-pton
```
testing HTTPDigestAuth hooks

#1979 is still opened as "unresolved" but it looks like a patch (#2253) was merged to address it in 2014. There wasn't a testing framework at the time, so no tests were included to verify things actually worked. I haven't been able to concoct a end-to-end test with httpbin for this, but the individual pieces work as intended. I can work on devising a socket test if needed.

I think I'd be much happier seeing a socket test than the httpbin ones: they just end up testing implementation details more than actual behaviour.

I don't think either of these are testing anything specific to httpbin's implementation, they should work the same regardless of which server they hit as long as the status codes are correct.

I added a somewhat monstrous test to simulate the full lifecycle of 401->302-401->200. I'm cutting a few corners with the actual digest piece because of the thread state, but the scenario we're trying to prove is correct. I've tested this both against 2.2.1 and 2.12.3 without the `handle_redirect` hook mounted. It's failing as expected in those scenarios.

> I remain unconvinced about whether checking num_401_calls is a good idea or whether we should consider it an implementation detail.

Sorry, could you clarify this a little? Are you referring to this in relation to the tests, or the actual implementation? I'm not sure how else we'd test that this is appropriately reset since that's the single flag that determines whether the 401 challenge is responded to or not.

I mean it about the tests. Generally speaking if we can we should restrict ourselves to testing observable behaviour, by which I mean bytes output or public return values. 

Ok, I'll remove those assert statements and rely on having expected header and status_code values.

Alright, assertions about the state of `num_401_calls` should be gone now and we're only relying upon observable behaviour.

Alright, we're on only socket tests now testing behaviour rather than implementation details.

@Lukasa I rescinded my last comment on `num_401_calls` because while what I was testing worked, it's not reproducible in the wild. I still think there may be value in testing the hooks individually but I don't think it's a blocker here. These tests cover the use cases we're concerned about in #1979.

Cool, tests are good. Thanks for the work @nateprewitt!
Make code match behavior as defined in RFC 2818 and RFC 6125 and documented behavior

The CN should be checked if no dNSName exist in subjectAltNames. The code treated a IP address as DNS name by appending it to dnsnames. This way CN was not checked in a certificate which contained the hostname as CN but only IP addresses as SAN.

See also http://stackoverflow.com/questions/41089539/authentication-issue-with-ssl-certificate-using-python-requests-lib

Thanks for this! For the moment, I'm not going to merge this, for the following reasons.

1. This patch affects urllib3, which we vendor without changes. It would need to be proposed to that project. 
2. The inclusion of match_hostname is currently intended to behave the same way on all platforms. Thus, if we edit it, we'd also have to stop preferring the stdlib until they matched our behaviour. This patch is therefore incomplete.
3. Your assertion about the incorrectness of our logic is not as clear-cut as it seems. Our behaviour currently matches curl, Firefox, and Chrome, which is some good company to be in (see curl/curl#1065).
4. It seems the Python crypto experts [aren't convinced of this either, and ordinarily I'd default to respecting their logic](http://bugs.python.org/issue28938).
5. There is basically no good reason to have a cert that fails in this manner. CN has been deprecated for years, SAN has been around forever. Just put the hostname in the SAN field as well.
Revert "use enforce_content_length in Requests"

This is more of a TODO reminder. Given the upcoming work for urllib3 2.0, which I'm assuming will be used in a Requests 3.0.0, this won't work. We don't need to remove this immediately, but I'll leave this here for when we're sure 2.0 is happening.

Heh, we won't forget: we'll run immediately into errors because I removed the kwarg entirely. Let's not queue up a whole bunch of PRs: let's focus on doing 2.0 first. :wink:
Add deprecation warnings for 3.0

Add extra test parameter for basic auth encoding

Don't merge yet, I think I swapped the warning message and class

r4r @Lukasa @nateprewitt 

@Lukasa when you get another second to merge master into Proposed/3.0.0, I'll throw up the other patch.

Already done. ;)

hell yeah!

Looks like we might be hitting a problem in 2.6 with zero length values in the string formatter (None specifically).
adding string casting for non-bytes values

This addresses #3753, I'll open a PR for 3.0.0 removing this when we merge master.

i support this

it's stupid, but no reason not to 

LGTM. Thanks for this @nateprewitt! :sparkles:

And I've just caught the 3.0.0 branch up, so you can go ahead and open the second PR now.

@sigmavirus24 you're suggesting using `warnings.warn` to actively notify the user about the deprecation, correct? I can add that and the test here in a moment if you're not already working on it.
301/302 redirects for non-POST with body

This patch is an attempt to address #2590 by relaxing the way Requests handles redirects. 

Taking a look at how we’re handling stripping bodies, I modified the conditional to avoid an overly complicated check. The only requests we want to strip bodies off of are ones that have been modified from the redirect. That means the method will have already been changed to GET in `rebuild_method` and we can perform a uniform stripping for only GET requests. Personally, I'd like to have `rebuild_method` return a value, either a boolean or the new method, and use that as the parameter for the check. Then move the code inside the conditional into it's own function. This will help a bit with the overall size of `resolve_redirects`.

I also removed the 302 catch-all which was put in place in #1704 when RFC 2616 was still the standard. The previous mandates for 302 aren’t included in [RFC 7231 6.4.3](https://tools.ietf.org/html/rfc7231#section-6.4.3) and don’t appear to be reproducible in Chrome or Firefox with any tools supporting PUT, PATCH, or DELETE. (Note: if there’s a repro for this that I’m missing please let me know and I’ll gladly revert this. I’m just going off of the information available.)

The "test all methods" tests may be a bit overboard, I added them primarily for my own assurance to make sure I wasn't missing edge cases. I can remove those if they're deemed unnecessary.

Alright, comments updated.

> Personally, I'd like to have rebuild_method return a value, either a boolean or the new method, and use that as the parameter for the check. Then move the code inside the conditional into it's own function. This will help a bit with the overall size of resolve_redirects.

Thoughts on mucking with this? If not, I think we're good to go.

@Lukasa, I think this is ready for a peek whenever you've got a moment :)

Ok, 301-303 individual tests removed in favor of the big ones.
extensions issue with requests-2.12.3-py3.5.egg

after upgarde to 2.12.3 under win10, below issue is encountered, any suggestion?
  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\sessions.py", line 535, in post
    return self.request('POST', url, data=data, json=json, **kwargs)

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\adapters.py", line 423, in send
    timeout=timeout

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\packages\urllib3\connectionpool.py", line 594, in urlopen
    chunked=chunked)

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\packages\urllib3\connectionpool.py", line 350, in _make_request
    self._validate_conn(conn)

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\packages\urllib3\connectionpool.py", line 835, in _validate_conn
    conn.connect()

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\packages\urllib3\connection.py", line 330, in connect
    cert = self.sock.getpeercert()

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\packages\urllib3\contrib\pyopenssl.py", line 324, in getpeercert
    'subjectAltName': get_subj_alt_name(x509)

  File "d:\Anaconda3\lib\site-packages\requests-2.12.3-py3.5.egg\requests\packages\urllib3\contrib\pyopenssl.py", line 171, in get_subj_alt_name
    ext = cert.extensions.get_extension_for_class(

AttributeError: 'Extensions' object has no attribute 'get_extension_for_class'

Hey @luwm001, this appears to be a duplicate of #3710. I believe you'll be able to solve this issue by updating your `cryptography` module version.

Thanks.
test auth doesn't generate incorrect http headers

These are a few tests from #3338 which is slated for Proposed/3.0.0. #3066 is the issue #3338 was originally opened to fix, but has since been resolved on master with #3082 and #3535. Adding these tests will allow us to confirm #3066 is indeed resolved and help prevent regressions until 3.0.0 is released.

Is the plan for this to land in 3.0.0 or master? Your summary isn't entirely clear.

Based on some of the other issues you linked I *think* you did want these tests in master.

Yep, master was the goal here. Thanks!
Error Auth Basic (since 2.12.2)

This error appears since last version (my code works with 2.11, 2.9 etc...)

```python
 File "C:\Users\vqmg0203\Desktop\app\core\models\database.py", line 211, in update_infos
   r = self.request_get(self.user_url)
 File "C:\Users\vqmg0203\Desktop\app\core\handlers\http.py", line 30, in wrapped
   return f(self, *args, **kwargs)
 File "C:\Users\vqmg0203\Desktop\app\core\handlers\http.py", line 130, in request_get
   allow_redirects=False)
 File "C:\Python27\lib\site-packages\requests\sessions.py", line 501, in get
   return self.request('GET', url, **kwargs)
 File "C:\Python27\lib\site-packages\requests\sessions.py", line 474, in request
   prep = self.prepare_request(req)
 File "C:\Python27\lib\site-packages\requests\sessions.py", line 407, in prepare_request
   hooks=merge_hooks(request.hooks, self.hooks),
 File "C:\Python27\lib\site-packages\requests\models.py", line 306, in prepare
   self.prepare_auth(auth, url)
 File "C:\Python27\lib\site-packages\requests\models.py", line 527, in prepare_auth
   r = auth(self)
 File "C:\Python27\lib\site-packages\requests\auth.py", line 68, in __call__
   r.headers['Authorization'] = _basic_auth_str(self.username, self.password)
 File "C:\Python27\lib\site-packages\requests\auth.py", line 38, in _basic_auth_str
   b64encode(b':'.join((username, password))).strip()
TypeError: sequence item 1: expected string, NoneType found
```

You cannot pass `None` as an entry to basic auth. This never worked sensibly: we ended up just putting the literal string `"None"` into the basic auth header.

In the future, please search **closed and** open issues before creating new ones that are duplicates.

This is a duplicate of **two** other issues: #3750 and #3736

Sorry for the duplicate :( 
Thanks for the explanation
Basic Auth broken when username / password is None

It appears commit 329a5cfdd4500fb1ce405a3242ef95f9197d24ba has broken `_basic_auth_str` and it now throws an exception `TypeError` (see below) when the supplied username and password is `None`.

Befor this commit, in requests 2.1.1, it was possible to supply None values for username and password when authentication is optional - in this particular case Home Assistant writing to InfluxDB with default configuration.

Exception thrown:
```python
TypeError: sequence item 0: expected a bytes-like object, NoneType found
```

This is a duplicate of #3736

In the future, please search closed and open issues before creating new ones that are duplicates.

Sorry. I looked at the open tickets only!