Raibek/requests-github-issues huggingface dataset:
response.apparent encoding throws error after using iter_lines

When using iter_lines with stream=True, and later trying to get the response.apparent_encoding, I get an exception as follows:

`RuntimeError: The content for this response was already consumed`

So my question is: Is this the proper behavior? I mean, shouldn't I be able to get the apparent_encoding even after consuming the content using stream=True? 
I was thinking on running the chardet on the buffer result of consuming the iteration but I'm thinking that this beats the purpose since the result is already encoded.

Am I missing something?

Thanks.

Thanks for opening this issue!

> Is this the proper behavior?

Yes. :smile:

> I mean, shouldn't I be able to get the apparent_encoding even after consuming the content using stream=True?

No. :smile: When you set `stream=True`, you are telling Requests that you *do not* want us to buffer the content for any reason. That means we do not store it, which means we cannot hand it off to chardet to process. `apparent_encoding` cannot be used in conjunction with `stream=True`. If you are going to consume all the content into a single buffer, you can just safely use `stream=False`.

Thanks for your answer. Thought would be that.
Install in Docker does not work as expected

I am attempting to build a Docker image (base image ubuntu:14.04) on a Windows machine where one of the commands are:
```
RUN pip install git+https://github.com/kennethreitz/requests
```
The functionality of this is to use pip inside the container to install the latest version of requests. It has functioned correctly in the past, but it currently throws an error. 
```
Step X/Y : RUN pip install git+https://github.com/kennethreitz/requests
 ---> Running in 33fa1f8103a0
Downloading/unpacking git+https://github.com/kennethreitz/requests
  Cloning https://github.com/kennethreitz/requests to /tmp/pip-ZPkqbQ-build
  Running setup.py (path:/tmp/pip-ZPkqbQ-build/setup.py) egg_info for package from git+https://github.com/kennethreitz/requests
    error in requests setup command: Invalid environment marker: platform_system == "Windows" and python_version<"3.3"
    Complete output from command python setup.py egg_info:
    error in requests setup command: Invalid environment marker: platform_system == "Windows" and python_version<"3.3"
```
The error is being triggered by the following extra requirement introduced [here](
https://github.com/kennethreitz/requests/commit/d89f8c0d70195d1ee707426ac7f3bdab870e0a18#diff-2eeaed663bd0d25b7e608891384b7298R100). 

The correct thing to do in this case is of course to lock the version of requests in the `pip install`-command and I have done this. I do however suspect that this could be an issue in the future if this commit is released.

This error is caused because the version of `pip` and `setuptools` being used in Ubuntu 14.04 is extremely old (necessarily it's from before 2014), and so doesn't understand markers. Frankly, that doesn't bother me very much. Assuming you're using a virtualenv inside your Docker image, you can resolve this by running `pip install -U setuptools` and `pip install -U pip` and that will fix your problems. If you aren't, you shouldn't be installing from `pip` as you'll totally bust your OS.

Hi @PetaPetaPeta thanks for opening this, but as @Lukasa explained there is nothing we can do to fix this for ancient versions of pip and setuptools. As such, I'm closing this issue.
ModuleNotFoundError: No module named 'email.parser'; 'email' is not a package

when i run : import requests 
i get the error as following:

~~~shell
D:\python\Python36\python.exe "D:/Python Work Space/test003.py"
Traceback (most recent call last):
  File "D:/Python Work Space/test003.py", line 1, in <module>
    import requests
  File "D:\python\Python36\lib\site-packages\requests\__init__.py", line 60, in <module>
    from .packages.urllib3.exceptions import DependencyWarning
  File "D:\python\Python36\lib\site-packages\requests\packages\__init__.py", line 29, in <module>
    import urllib3
  File "D:\python\Python36\lib\site-packages\urllib3\__init__.py", line 8, in <module>
    from .connectionpool import (
  File "D:\python\Python36\lib\site-packages\urllib3\connectionpool.py", line 11, in <module>
    from .exceptions import (
  File "D:\python\Python36\lib\site-packages\urllib3\exceptions.py", line 2, in <module>
    from .packages.six.moves.http_client import (
  File "D:\python\Python36\lib\site-packages\urllib3\packages\six.py", line 203, in load_module
    mod = mod._resolve()
  File "D:\python\Python36\lib\site-packages\urllib3\packages\six.py", line 115, in _resolve
    return _import_module(self.mod)
  File "D:\python\Python36\lib\site-packages\urllib3\packages\six.py", line 82, in _import_module
    __import__(name)
  File "D:\python\Python36\lib\http\client.py", line 71, in <module>
    import email.parser
ModuleNotFoundError: No module named 'email.parser'; 'email' is not a package
~~~
but i have already installed it
see:

~~~shell
D:\Python Work Space>pip install email
Requirement already satisfied: email in d:\python\python36\lib\site-packages\email-6.0.0a1-py3.6.egg
~~~

Email is a standard library module that should have come with Python. I *suspect* that `pip install email` made that worse, not better. However, more likely you have a file in the same directory as your code called `email.py`, you need to change that filename.
Allowing users to override the default agent using env variable

We need to override the default user agent that requests uses. I think its ok to allow overriding the default user agent using environment variables.

I didn't find other options (except monkey patching default_user_agent function in utils), maybe this can help other users as well.

Thanks for this!

I don't really understand why Requests should merge this into the tree. If you want to have this behaviour in your application, you can just use `Session` objects and set `Session.headers['User-Agent']` to whatever you like.

In fact, I'd say that having this functionality in Requests as provided here is not good. Mostly this is because this ignores the `trust_env` variable that Requests normally uses, preventing applications from opting-out of the environment. I am also concerned about unintended side-effects: environment variables are a very sledgehammer approach to this problem, such that any application using Requests with the default user-agent variable will be affected, even if that's not the one the user was concerned about. It's almost always better for applications to work with environment variables, rather than have the libraries go looking around for environment variables to control their behaviour.

So I'm afraid I don't think we'll be merging this in its current form, sorry!
Add requests.Session(timeout) parameter

...as a parameter which gets inherited globally by `get()` and other HTTP methods.

```
session = requests.Session(timeout=30)
session.get(...)
session.get(...)
```

Would be equivalent to:

```
session = requests.Session()
session.get(..., timeout=30)
session.get(..., timeout=30)
```

Thoughts?

Thanks for opening this! This is a duplicate of #3341: please read the discussion there for more.

Oh sorry about that.
Deadlocks when retry with pool_block=True

I created a repo with a testcase for this issue:

https://github.com/mattbillenstein/requests-retry-pool-hang

I can repro using gevent and threads.  It seems there are a few other issues exhibiting deadlocks, so hopefully this testcase is helpful in getting to the bottom of this.

Ok, so I have something that fixes this for me -- retries are implemented using recursion and before recursing, the connection isn't released back to the pool, so for every retry, we would grab another connection while still holding onto the old one for no real reason afaict.  See:

https://github.com/mattbillenstein/requests/commit/524b43154553489bf60216acb9030ce5b36f7215

Connections should be released here: https://github.com/mattbillenstein/requests/commit/524b43154553489bf60216acb9030ce5b36f7215#diff-56e4c9a9f8955a91d87d7da5335a7dddR668.

Yeah, I traced through that, the connection is not put back to the pool there.  If I add a print in there:

        finally:
            if not clean_exit:
                # We hit some kind of exception, handled or otherwise. We need
                # to throw the connection away unless explicitly told not to.
                # Close the connection, set the variable to None, and make sure
                # we put the None back in the pool to avoid leaking it.
                conn = conn and conn.close()
                release_this_conn = True

            print clean_exit, release_this_conn
            if release_this_conn:
                # Put the connection back to be reused. If the connection is
                # expired then it will be None, which will get replaced with a
                # fresh connection during _get_conn.
                self._put_conn(conn)

And reduce my test to just one one request:

    $ time ./retry_pool_hang_gevent.py  --retry
    True False
    True False
    True False
    True False
    True False
    .
    real    0m0.898s
    user    0m0.206s
    sys     0m0.041s

So it seems like Requests is not asking for that connection to be returned to the pool. This feels like a urllib3 issue; it seems like urllib3 leaks connections with `preload_content=False` and retries set. Mind opening an issue there?
Remove some unused imports.


# [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=h1) Report
> Merging [#3991](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=desc) into [master](https://codecov.io/gh/kennethreitz/requests/commit/44106f18b75bfbafa63b1a1910366d57d554bc41?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/kennethreitz/requests/pull/3991/graphs/tree.svg?src=pr&width=650&token=mZGswIiXLa&height=150)](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3991      +/-   ##
==========================================
- Coverage   89.72%   89.72%   -0.01%     
==========================================
  Files          15       15              
  Lines        1947     1946       -1     
==========================================
- Hits         1747     1746       -1     
  Misses        200      200
```


| [Impacted Files](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/auth.py](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=tree#diff-cmVxdWVzdHMvYXV0aC5weQ==) | `86.41% <ø> (-0.09%)` | :arrow_down: |
| [requests/adapters.py](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=tree#diff-cmVxdWVzdHMvYWRhcHRlcnMucHk=) | `92.89% <ø> (ø)` | :arrow_up: |
| [requests/models.py](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=tree#diff-cmVxdWVzdHMvbW9kZWxzLnB5) | `93.42% <100%> (ø)` | :arrow_up: |
| [requests/utils.py](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=tree#diff-cmVxdWVzdHMvdXRpbHMucHk=) | `84.98% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=footer). Last update [44106f1...19ba9f1](https://codecov.io/gh/kennethreitz/requests/pull/3991?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Upload data progress

I saw examples of using `iter_content` to get progress of the download, but I do not see a corresponding API for upload a blob. I am requesting such feature. Thanks.

If you are uploading a blob of data you can provide either an iterable object (which will be uploaded using chunked encoding) or a file-like object that provides a `read` method (which, if we can determine a length, will be streamed to the server). In either case, we have to call into your code to retrieve the next chunk of data, so you will get notified whenever we've sent the last chunk you gave us. That's the expected way to be notified about uploading data.

Is that sufficient for your needs?

@gladiacxtylish you can look at an example in [requests-toolbelt](https://github.com/sigmavirus24/requests-toolbelt/blob/3ac79c3ca358a7d3152a3d06a4f33306f9b6772f/requests_toolbelt/multipart/encoder.py#L316) for this. Cheers!
Allow empty data values in multipart form upload

A multipart form upload with data field such as `{'resources': []}` is transformed as `[]`, leaving out the `resources` key.

Can you provide sample code that demonstrates the problem, as well as an example of what you believe the correct body should look like?

For example, 
```
data = {'resources': []}
files = {'file': open('file.zip', 'rb')}
requests.put('/upload', data=data, files=files)
```

And what is your expected outcome?

Thanks for the response.  I think `new_fields` can take up the value as it is without any transformation if the data field is in that structure. 

At line https://github.com/kennethreitz/requests/blob/master/requests/models.py#L139 , `new_fields` can contain the data value. At least, we should be able to maintain the `resources` key.

Sorry, I meant literally "what should the request body look like in this case?"

This should probably work
```
body --7c5c7d822e7646a5b3c6a1a6b9e3d72f
Content-Disposition: form-data; name="resources"

[]
--7c5c7d822e7646a5b3c6a1a6b9e3d72f
Content-Disposition: form-data; name="file"; filename="file.zip"
```

If `resources` is populated for example:
```
body --7c5c7d822e7646a5b3c6a1a6b9e3d72f
Content-Disposition: form-data; name="resources"

{'sha1': '4fc46b3bd86860a07f84bce98e1cd6fb0f7ef0c4', 'size': 9212}
--7c5c7d822e7646a5b3c6a1a6b9e3d72f
Content-Disposition: form-data; name="file"; filename="file.zip"
```

Here's what a sample request body looks like with curl:
```
curl -v -k -X PUT -F 'resources=[]' -F "application=@/tmp/application.zip" https://api.system.domain.net/v2/apps/130adaba-b017-4234-8d2c-8f2874334e3d/bits                          
* TCP_NODELAY set
> PUT /v2/apps/130adaba-b017-4234-8d2c-8f2874334e3d/bits HTTP/1.1
> User-Agent: curl/7.51.0
> Accept: */*
> Content-Length: 44852
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=------------------------854d952145f669ed
>
< HTTP/1.1 100 Continue
< HTTP/1.1 201 Created
< Content-Length: 2 
< Content-Type: application/json;charset=utf-8
< Date: Sat, 01 Apr 2017 22:05:26 GMT
< Server: nginx
< X-Content-Type-Options: nosniff
< X-Vcap-Request-Id: ffd5d2c7-f9f3-44da-7db4-e62e71fbadde
< X-Vcap-Request-Id: ffd5d2c7-f9f3-44da-7db4-e62e71fbadde::da9da2b6-8afc-48d5-af5a-aebbbd9a9a68
<
* Curl_http_done: called premature == 0
{}
```

The curl data you showed does not show the request body, only the request header. Based on your example, you can get the result you want by doing:

```python
data = {'resources': '[]'}
files = {'file': open('file.zip', 'rb')}
requests.put('/upload', data=data, files=files)
```

Ha. Hackity. Thanks
Add appveyor testing of Requests.

In general it'd be good to have more testing, but it's also possible that via PR #3979 we'll get unexpected different behaviour on Windows than on other platforms. For this reason, we should start also testing on Windows.

This PR takes the configuration used for urllib3, updates it for Requests, and then provides it. We'll still need to actually enable Appveyor so that it'll try to build, which I cannot do as I don't have admin on the repo. @kennethreitz, mind giving that a try?

# [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3983?src=pr&el=h1) Report
> Merging [#3983](https://codecov.io/gh/kennethreitz/requests/pull/3983?src=pr&el=desc) into [master](https://codecov.io/gh/kennethreitz/requests/commit/f6c58ec5820e29c1214c8e3ff887f571c32ffd48?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/kennethreitz/requests/pull/3983/graphs/tree.svg?token=mZGswIiXLa&src=pr&height=150&width=650)](https://codecov.io/gh/kennethreitz/requests/pull/3983?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3983   +/-   ##
=======================================
  Coverage   89.72%   89.72%           
=======================================
  Files          15       15           
  Lines        1947     1947           
=======================================
  Hits         1747     1747           
  Misses        200      200
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3983?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3983?src=pr&el=footer). Last update [f6c58ec...fae7530](https://codecov.io/gh/kennethreitz/requests/pull/3983?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


i'm very skeptical, but let's give it a shot

doesn't travis have a windows option?

build failed, looks like it's because of 2.6 maybe. https://ci.appveyor.com/project/kennethreitz/requests

Travis does not have a Windows option and does not include one in their roadmap. 

So the reason those builds are failing is because I have to work out where the hell GNU make is on those systems. Given that our CI builds use make, I need to find where it is. If you're happy to be a bit patient, I'll open a PR to fix it in the next few hours
configure sudo: false on travis


# [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3982?src=pr&el=h1) Report
> Merging [#3982](https://codecov.io/gh/kennethreitz/requests/pull/3982?src=pr&el=desc) into [master](https://codecov.io/gh/kennethreitz/requests/commit/52a3526a99e0129ef239ec811a125272398a9f7c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/kennethreitz/requests/pull/3982/graphs/tree.svg?width=650&height=150&src=pr&token=mZGswIiXLa)](https://codecov.io/gh/kennethreitz/requests/pull/3982?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3982   +/-   ##
=======================================
  Coverage   89.72%   89.72%           
=======================================
  Files          15       15           
  Lines        1947     1947           
=======================================
  Hits         1747     1747           
  Misses        200      200
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3982?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3982?src=pr&el=footer). Last update [52a3526...047056c](https://codecov.io/gh/kennethreitz/requests/pull/3982?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Cool, thanks @graingert!
enable travis pip cache


Thanks @graingert!

# [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3981?src=pr&el=h1) Report
> Merging [#3981](https://codecov.io/gh/kennethreitz/requests/pull/3981?src=pr&el=desc) into [master](https://codecov.io/gh/kennethreitz/requests/commit/524627ef21818e06a33bf4b54cd9c93a7dc35b21?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/kennethreitz/requests/pull/3981/graphs/tree.svg?src=pr&token=mZGswIiXLa&width=650&height=150)](https://codecov.io/gh/kennethreitz/requests/pull/3981?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3981   +/-   ##
=======================================
  Coverage   89.72%   89.72%           
=======================================
  Files          15       15           
  Lines        1947     1947           
=======================================
  Hits         1747     1747           
  Misses        200      200
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3981?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3981?src=pr&el=footer). Last update [524627e...af383b3](https://codecov.io/gh/kennethreitz/requests/pull/3981?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
No Content-Length header when POSTing or PUTing an empty Django InMemoryUploadedFile

I'm trying to send a PUT request passing as the data kwarg an instance of a InMemoryUploadedFile (which is the model Django uses for uploaded files, see for reference https://docs.djangoproject.com/en/1.10/ref/files/uploads/#django.core.files.uploadedfile.InMemoryUploadedFile), like this:

```
files = request.FILES.getlist('file')
for uploaded_file in files:
    [...]
    requests.put(upload_url, data=uploaded_file)
```
The problem is that the resulting request will not have the expected Content-Length header, but the Transfer-Encoding one. I think the problem is at https://github.com/kennethreitz/requests/blob/master/requests/models.py#L492, where requests sets one header or the other based on the `if length` check. In the case of a stream with 0 length, the check fails and the Transfer-Encoding header gets set.

The snippet above works as intended when the file is non-empty (because the `if length` check passes), and also `requests.put(upload_url, data=uploaded_file.read())` works (as this is not a stream anymore, but it could cause problems when reading a huge file in memory).

Why is the transfer encoding header a problem? HTTP/1.1 servers are required to understand it. 

The problem is not in the Transfer-Encoding header, but the lack of the Content-Length header: if I try to upload the file to Amazon S3 (for example), it will reply with a 411 Content Length Missing response.

Ok, so this boils down to an S3 behaviour, as S3 requires the content length. 

Requests does not in general consider this behaviour a bug: I consider S3 at fault here. However, it's understandably a problem. You should be able to get the behaviour you want by writing a small proxy object that sends all `read` calls to the Django object and that implements `__len__`, which will be enough for us to work out its length. 

Oh I apologise. You're posting an empty file. Requests really does not provide an easy way to override our logic here, which is very deliberate (plenty of users pass us file objects that return a length of zero but which nonetheless return data from `read()`). For this reason the easiest thing to do is likely to use the [PreparedRequest flow](http://docs.python-requests.org/en/master/user/advanced/#prepared-requests) to tweak the headers of the prepared request to suit this case. 

Thank you for opening this @CianciuStyles. I believe if you follow @Lukasa's advice you'll be able to solve your problem nicely. As a result, I'm closing this since it's not a bug and you now have a suitable workaround. Cheers!
Don't use `urllib.proxy_bypass` on Windows

The `urllib.proxy_bypass` implementation on Windows is using forward and reverse DNS requests. This can result in major performance issues (I guess this depends on the DNS server, right now I have seen delays up to 15 seconds!). Additionally, this behavior is contrary to all major browser implementations.

This could even be a security issue as people might rely on the proxy as a privacy feature and in this case the client shouldn't do any DNS requests as this could leak private information.

I would recommend using a custom implementation which isn't doing any DNS request.

References:

* https://github.com/kennethreitz/requests/issues/2988
* http://bugs.python.org/issue29533

If you are in favor of this proposal I might be able to do a PR.


In principle we are not attached to the specific implementation of `proxy_bypass`, only the net result. Any change away would be required to reproduce all relevant functionality and to provide complete test coverage. If you're open to doing that work we'll happily consider the PR. 

Basically, this would be the copy/pasted Windows versions of [ `urllib.proxy_bypass`](https://github.com/python/cpython/blob/98b1c82/Lib/urllib.py#L1633) and [`urllib.proxy_bypass_registry`](https://github.com/python/cpython/blob/98b1c82/Lib/urllib.py#L1581), the latter without the DNS lookups.

Not quite sure how to test that in a sensible way. Maybe mocking/monkeypatching registry access (`_winreg`)? 

Mainly note for myself: `urllib.proxy_bypass_registry` with host `172.16.1.12` and proxyOverride `172.16.1.1` evaluates to True. Not sure if this is a bug or implementation detail (matching start of string).

proxyOverride `192.168.` (without star) and host `192.168.0.1` also evaluates to True. 

In the config dialog it says "do not use proxy server for addresses **beginning with**" (screenshot [here](https://blogs.msdn.microsoft.com/askie/2015/10/12/how-to-configure-proxy-settings-for-ie10-and-ie11-as-iem-is-not-available/)). So I assume the urllib matching behavior is correct and matches the Window behavior.

#3979
'headers' field does not allow multiple headers with the same name

A response returns multiple `Set-Cookie` headers. When I wanted to see these using `response.headers`, only a single `Set-Cookie` header is shown. I have expected all `Set-Cookie` headers to be shown.

Is it intentional to show only one header key-value pair per header key?

Requests does show multiples, but it joins them into a single header field. This allows users to get a slightly more intuitive behaviour than returning lists, though it does reduce fidelity slightly.
ETA on 2.14? + docs request on release schedule

1. Is there an ETA on 2.14?  I couldn't find any milestone info and there seems to be a few dozen PRs merged since January.

2. There should probably be a line about the release schedule in the contributors guide or elsewhere, so I people like me don't ask annoying questions.

I don't mean to prod for a release, but just get a general idea of when one will likely be made.  I've got a couple of tiny open source projects that rely on a handful of the forthcoming changes and just want to know when I should schedule an afternoon for packaging and updates.

Heh, there is no schedule. The schedule is "whenever the core devs feel that a release is warranted". In this case, I have an OmniFocus task for this but am currently on holiday, so would expect that we'll release next week. 

Generally speaking I'd say that a schedule doesn't work for a project like this one: we're sufficiently small in scope that it's pretty easy to coordinate the work. However, if you disagree I'd be interested to hear it. 

Well, I'm not talking about an "official" release schedule... just that a lot of projects tend to have something general like "we aim for a release every X months or Y pull requests/core changes".

I had actually looked through the history to try and predict when a release happens, and it tends to be around 3 months (sometimes 2, sometimes 4).  Looking at the commit history though, it seems like there were a lot more PRs merged than usual since 2.13.0... so I expected 2.14 to be on the earlier side.  

2.4  - 2014/08/29
2.5 - 2014/12/01
2.6 - 2015/03/14
2.7 - 2015/05/03
2.8 - 2015/10/08
2.9 - 2015/12/15
2.10 - 2016/04/29
2.11 - 2016/08/06
2.12 - 2016/11/15
2.12 - 2017/01/24










Heh, frankly it's looking like another 3 month period basically on the nose. 

I have no issue with people doing averages of our release cycle, but I'm nervous about providing that analysis ourselves. It changes the questions from "when is the next release?" to "why is there no release yet?", which isn't really a net improvement. 

We have no release cadence other than coincidentally.
Custom Handler

How do you extend requests to use a custom handler for a security scheme?
I'm looking for documentation on how to extend the API basically.
Thanks

What specific kind of API extension do you have in mind? Generally speaking you'd do it by subclassing the Session object.

I have a site that is token based security, and I want to create a handler to work with this.  So I know the username, password, expiration, and token url, and I want the authentication handler to handle the creation and update of the token based on a given time frame.   

It's token based, but not oauth? You may find Requests-Oauthlib's design instructive of how to handle a complex token based auth system. 

Thanks,  I think I figured it out.  So you inherit from AuthBase, then you write your logic in ```__call__```

My issue now is how do I insert the token parameter into a multi-part request.  For regular POST and GET calls it's easy.  Just append the token to the body or parameters off of the object ```r```.

If I insert the form-data into the r.body of the form request, I a message from server saying it's not valid:

Real quick and dirty method:

```
token = "abc123
item = '--%s\r\nContent-Disposition: form-data; name="token"\r\n\r\n%s\r\n' % \
                    (r.headers['Content-type'].split(';')[1].strip().split('=')[1], token)
r.body = str.encode(item) + r.body
```

It looks right from Fiddler, but apparently it isn't

There is a body terminating boundary (`--uuid--`) that comes last, after which data will usually be ignored. You'd also need to update the content-length, which you may not have done.

Essentially, you'll want to insert your data *before* the terminating boundary, not after.

ahh thanks for that! that fixed it. 

Thank you for all the help
Requests.Get Response Includes Unexpected Values

Hi,

I encountered an issue when using the requests.get command with Python 3.5.1. I described the issue in [this Stack Overflow post](http://stackoverflow.com/questions/43218259/python-requests-get-response-includes-unexpected-values). The solution was to use Python 2.7.13, so I'm not 100% sure whether the issue is related to the base Python package or how the requests library works with the base packages, but I thought I would post the issue just in case. 

Hope that helps!

That looks very much like a bug in httplib. If it's possible for us to recreate the bad request from our own machines we can work out exactly what's going on: otherwise we'll need to wait for a good repro. 

LUCKIEST HUMAN BEING NOW!!!. After setting the HTTP on the same server, Requests was able to return results from the server instead waiting infinitely. Here is what response.content has:

`b'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">\r\n<html>\r\n<head>\r\n<title>504</title>\r\n<meta http-equiv="Cache-Control" content="no-cache"/>\r\n</head>\r\n<body>\r\n<p>\r\nGateway Timeout \r\n</p>\r\n</body>\r\n</html>\r\n'`

After five days, solution is near. Have to hunt down the timeout setting on the server that is doing this havoc
requests.exceptions.ConnectionError

`response = requests.post(url, data=payload ,verify=False, timeout=(3.05, 6.05))`

works fine in python shell but not in python manage.py shell (django)
I am getting below error 
'requests.exceptions.ConnectionError: HTTPConnectionPool(host='ggk-wrl-win1', port=9090): Max retries exceeded with url: /home/trusted/token (Caused by NewConnectionError('<requests.packages.urllib3.connec)'

You have truncated the exception message so I can't see what it is, do you mind providing the full exception string?

![screenshot from 2017-04-18 17 16 40](https://cloud.githubusercontent.com/assets/26059420/25129293/fcf012f2-245a-11e7-9918-2411e7178104.png)
![image](https://cloud.githubusercontent.com/assets/26059420/25129307/0e63289e-245b-11e7-9e28-43ae74e8cccf.png)


You're encountering a DNS error for the host name `ggk-wrl-win1`. Can you confirm your network is set up correctly?

yes, setup is good


From inside `manage.py`, can you run:

```python
import socket
s = socket.create_connection(('ggk-wrl-win1', 80))
```

and tell me what happens please?

![image](https://cloud.githubusercontent.com/assets/26059420/25132475/59037a96-2467-11e7-84ea-62b1e70b010f.png)


So, as you can see, this is a problem with your environment and not requests. Inside your environment that host name is not resolvable. We cannot help you work out why that is.

but, its works in python shell
![image](https://cloud.githubusercontent.com/assets/26059420/25133783/5b4bb3f0-246b-11e7-8de6-d4273d70ffa6.png)


Yes, which strongly suggests that your specific environment is at fault. You need to diagnose why the manage.py shell won't make outbound network connections. 

I think the issue with my docker setup, thank you

@srikanth-chandaluri  are you find solution ?

> I think the issue with my docker setup, thank you

are you find solution
fix handle of non-ascii location on redirects issue #3888

fix for #3888 

# [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3969?src=pr&el=h1) Report
> Merging [#3969](https://codecov.io/gh/kennethreitz/requests/pull/3969?src=pr&el=desc) into [master](https://codecov.io/gh/kennethreitz/requests/commit/fd7dcd278557c517a0df6f6a4415c4922e6bd0d5?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/kennethreitz/requests/pull/3969/graphs/tree.svg?height=150&width=650&token=mZGswIiXLa&src=pr)](https://codecov.io/gh/kennethreitz/requests/pull/3969?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3969      +/-   ##
==========================================
+ Coverage   89.72%   89.73%   +0.01%     
==========================================
  Files          15       15              
  Lines        1946     1949       +3     
==========================================
+ Hits         1746     1749       +3     
  Misses        200      200
```


| [Impacted Files](https://codecov.io/gh/kennethreitz/requests/pull/3969?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/sessions.py](https://codecov.io/gh/kennethreitz/requests/pull/3969?src=pr&el=tree#diff-cmVxdWVzdHMvc2Vzc2lvbnMucHk=) | `93.75% <100%> (+0.06%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3969?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3969?src=pr&el=footer). Last update [fd7dcd2...722b1da](https://codecov.io/gh/kennethreitz/requests/pull/3969?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


thanks for the quick reply, if you think we should make the test parameterized with some more location headers on it, let me know.

Yep, looks good to me too!

Alright let's do it!

great thanks for merging and the fast responses! 😃
URLs with Square Brackets []

We were using Requests with FileStack without issues until we ran into this URL format:

**https://process.filestackapi.com/store=l:S3,p:"/watermarked/"/watermark=f:wUaFDoPfRAiT50LQZWs9,p:[bottom,right],s:63/p2aaZ8pzS8md1vLzY7V2**

the **p:[bottom,right]** part seems to be causing issues (the request doesn't properly make it to their end).  FileStack's documentation suggests using curl with the -g option:

> -g, --globoff
> 
> This option switches off the "URL globbing parser". When you set this option, you can specify URLs that contain the letters {}[] without having them being interpreted by curl itself. Note that these letters are not normal legal URL contents but they should be encoded according to the URI standard.
> https://curl.haxx.se/docs/manpage.html

Is there a similar option for Requests?  I saw that it's not a valid URI, so I am also talking to FileStack to be able to resolve this but I am wondering if there is a temporary solution while I wait for them.

Looking at the response URL, it seems the quotes are encoded but not the brackets?

**https://process.filestackapi.com/store=l:S3,p:%22/watermarked/%22/watermark=f:9shDBUAMSMm58i5r3Ujr,p:[bottom,right],s:17/uj16GiSQa2bzDCVY34vw**

but the same happens in the p:center case which works just fine.


Are you sure you need it? Requests won't treat them as globs, so that's not a concern. We will encode them, but the snippet of documentation you provided suggested that that is fine. Have you actually tried the request at all?

Yes, I tried it without the square brackets, no luck.  And yes, I tried the request.  I get a 400 error back from FileStack only when the brackets are present.

The last URL I posted above is from the response.url property after the request is made.  Looks like the quotes are encoded but the brackets are not.  I'll see if FileStack has any news on this.  Not sure if this is a requests or FileStack issue at this point, will update when I know more.  Might be good leaving this issue just in case anyone runs into a similar problem.

Looks like the solution was simple, just manually encode the bracket characters like this:

**(https://process.filestackapi.com/store=l:S3,p:%22/watermarked/%22/watermark=f:wUaFDoPfRAiT60LQZWs9,p:%5Bbottom,right%5D,s:17/nPh71zWTTCmQXD9ho2ZF)**
waiting for a very long time for the reponse

python version:2.7
requests version:2.11.1

when the response expected to be redirect(301)，after I send out the request.get message, it will take a long time to get response.
however in the wireshark can see the response is immediately returned.it seems like somewhere in code after get the response takes the time out

Is the response body complete immediately? Can you provide a URL to reproduce this error?

This has been stale for nearly a month. We can reopen this if we get more information and can determine that this is actually a bug in Request.
when i import requests, UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 1: ordinal not in range(128),how to solve this?

>>> import requests
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python2713\lib\site-packages\requests\__init__.py", line 52, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "C:\Python2713\lib\site-packages\requests\packages\urllib3\contrib\pyopenssl.py", line 47, in <module>
    from cryptography import x509
  File "C:\Python2713\lib\site-packages\cryptography\x509\__init__.py", line 7, in <module>
    from cryptography.x509.base import (
  File "C:\Python2713\lib\site-packages\cryptography\x509\base.py", line 16, in <module>
    from cryptography.x509.extensions import Extension, ExtensionType
  File "C:\Python2713\lib\site-packages\cryptography\x509\extensions.py", line 14, in <module>
    from asn1crypto.keys import PublicKeyInfo
  File "C:\Python2713\lib\site-packages\asn1crypto\keys.py", line 22, in <module>
    from ._elliptic_curve import (
  File "C:\Python2713\lib\site-packages\asn1crypto\_elliptic_curve.py", line 51, in <module>
    from ._int import inverse_mod
  File "C:\Python2713\lib\site-packages\asn1crypto\_int.py", line 56, in <module>
    from ._perf._big_num_ctypes import libcrypto
  File "C:\Python2713\lib\site-packages\asn1crypto\_perf\_big_num_ctypes.py", line 31, in <module>
    libcrypto_path = find_library('crypto')
  File "C:\Python2713\lib\ctypes\util.py", line 53, in find_library
    fname = os.path.join(directory, name)
  File "C:\Python2713\lib\ntpath.py", line 85, in join
    result_path = result_path + p_path
UnicodeDecodeError: 'ascii' codec can't decode byte 0xcf in position 1: ordinal not in range(128)


Hey there. You have raised this issue before, [here](https://github.com/kennethreitz/requests/issues/1252#issuecomment-291438569), and so I will give the same advice again: this appears to be a problem with asn1crypto, not Requests: please open an issue on their repository.
fix unicode decode error on py2 when handling redirect without scheme

as discussed on #3959 , fix the UnicodeDecodeError happening because implicit decoding of str string into unicode while using python2.

This issue happens when handling redirection without scheme  (i.e. Location: //URL) on python2 and only if the location is unicode.

First time applying a PR on GitHub so let me know if there is something I can do to make it better :).

Thanks,
Shmulik

# [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3963?src=pr&el=h1) Report
> Merging [#3963](https://codecov.io/gh/kennethreitz/requests/pull/3963?src=pr&el=desc) into [master](https://codecov.io/gh/kennethreitz/requests/commit/84a60407628e83c004e3ea24a2d856a55d7a5c59?src=pr&el=desc) will **increase** coverage by `0.1%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/kennethreitz/requests/pull/3963/graphs/tree.svg?width=650&src=pr&token=mZGswIiXLa&height=150)](https://codecov.io/gh/kennethreitz/requests/pull/3963?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master    #3963     +/-   ##
=========================================
+ Coverage   89.61%   89.72%   +0.1%     
=========================================
  Files          15       15             
  Lines        1946     1946             
=========================================
+ Hits         1744     1746      +2     
+ Misses        202      200      -2
```


| [Impacted Files](https://codecov.io/gh/kennethreitz/requests/pull/3963?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/sessions.py](https://codecov.io/gh/kennethreitz/requests/pull/3963?src=pr&el=tree#diff-cmVxdWVzdHMvc2Vzc2lvbnMucHk=) | `93.68% <100%> (+0.74%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3963?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kennethreitz/requests/pull/3963?src=pr&el=footer). Last update [84a6040...a3e597c](https://codecov.io/gh/kennethreitz/requests/pull/3963?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

agree, its probably better - changed
Set-Cookie header didn't handled

Test code:
```
In [2]: rq = requests.get('http://httpbin.org/cookies/delete?k2=&k1=', cookies={'k2':'1'})

In [3]: rq.content
Out[3]: '{\n  "cookies": {\n    "k2": "1"\n  }\n}\n'
```
requests should delete `k2` and return `"cookies": {}`, but it didn't.
Should act like:
```
➜  ~ http http://httpbin.org/cookies/delete\?k1\=\&k1\=2 'Cookie: k1=1' -p Hb
GET /cookies/delete?k1=&k1=2 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie:  k1=1
Host: httpbin.org
User-Agent: HTTPie/0.9.4

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href="/cookies">/cookies</a>.  If not click the link.

➜  ~ http -F http://httpbin.org/cookies/delete\?k1\=\&k1\=2 'Cookie: k1=1' -p Hb
GET /cookies HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: httpbin.org
User-Agent: HTTPie/0.9.4

{
    "cookies": {}
}
```

Yes, this is somewhat intended behaviour. For setting cookies like this Requests isn't really sure what the scope of the cookie is: is it only the scope of the single request domain and path? The whole domain? The domain and subdomains? All hosts?

For that reason, given that the remote site didn't set it, we ignore the deletion. As you can see from this, Requests does handle cookies properly:

```python
>>> s = requests.Session()
>>> s.cookies
<RequestsCookieJar[]>
>>> r = s.get('http://httpbin.org/cookies')
>>> r.content
b'{\n  "cookies": {}\n}\n'
>>> r = s.get('http://httpbin.org/cookies/set', params={'k1': '1'})
>>> r.content
b'{\n  "cookies": {\n    "k1": "1"\n  }\n}\n'
>>> s.cookies
<RequestsCookieJar[Cookie(version=0, name='k1', value='1', port=None, port_specified=False, domain='httpbin.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={}, rfc2109=False)]>
>>> r = s.get('http://httpbin.org/cookies/delete', params={'k1': ''})
>>> r.content
b'{\n  "cookies": {}\n}\n'
```

The way to get cookies that behave the way you want is to use a `Session` and the cookiejar available at `Session.cookies`. Alternatively, you can do what httpie is doing, which is to not treat cookies specially at all and just set them as a regular header:

```python
>>> rq = requests.get('http://httpbin.org/cookies/delete?k2=&k1=', headers={'Cookie': 'k1=1'})
>>> rq.content
b'{\n  "cookies": {}\n}\n'
```
Use libproxy gain WPAD/PAC support for free

Currently requests does figuring out of whether a proxy should be used for a given URL. [WPAD/PAC](https://en.wikipedia.org/wiki/Web_Proxy_Auto-Discovery_Protocol) files describe those requirements, so if requests were to support WPAD proxy descriptions it would need to also parse it and figure what proxy to use for what URL.

On the other hand requests could delegate that task to libproxy and clean up a bunch of code (and issues) from the project.

By the way since I'm interested I could help implementing it.

So Requests cannot easily use libproxy because it is a binary dependency. We therefore cannot vendor it, and so cannot unconditionally rely on it being present. I would be open to a series of patches that add support, but libproxy is also not (as far as I can see) downloadable from PyPI either.

All of this together suggests that adding optional libproxy support would provide minimal benefit: an overwhelming majority of users do not have access to libproxy, and we have no way to distribute it to them.

My suggestion, then, is that someone (anyone who wants to be helpful) starts distributing binary wheels of libproxy on PyPI. At that point, once there is a reliable distribution method, we can discuss how reasonable it is to have Requests migrate to use it. However, until that time it's just not practical for Requests to add that dependency.

Sorry! 

I think pypac (https://pypac.readthedocs.io/en/latest/) is offering this functionality, correct ?
Should it be integrated / merged in some way to requests ?
The could allow a proxies="pac" kind of functionality out of the box.

Yeah seems like it should work. I'll see if I can get it to function at work.
HTTPS isn't working if proxy is down

I'm using requests with proxies and when proxy-server is down for some reason, requests make request without proxy(using the origin IP). Also, this issue appears when making https request with proxy which not support https. Could you please implement functionality, which could allow me to make proxy-only requests?

Requests should not be making requests without the proxy when the proxy is not working. Can you show us the code you're using please?

I've figured it out, i guess. This happened if making request in `while True` loop.