Raibek/requests-github-issues huggingface dataset:
When Location decoding fails, fall back to original

Issue #3888 correctly identified Location headers as *usually* containing UTF-8
codepoints (when not correctly URL encoded), but this is not always the case.
For example the URL
http://www.finanzen.net/suchergebnis.asp?strSuchString=DE0005933931 redirects
to `b'/etf/ishares_core_dax\xae_ucits_etf_de'`, containing the Latin-1 byte for
the ® character.

If UTF-8 decoding fails, it is better to fall back to the original.

This issue was found via https://stackoverflow.com/questions/47113376/python-3-x-requests-redirect-with-unicode-character

Crumbs, tests fail on 2.x because it encodes a bytestring (latin-1 encoded), while Python 3 handles a Unicode value. Returning a native `latin-1` string should work there.

Nope, `to_native_string()` returns a `str` on Python 2. Suggestions to produce consistent output on 2.x and 3.x appreciated; just returning `location.decode('latin1')` doesn't work either.

And another thought: Python 3 ends up with UTF8 bytes in the URL-encoded redirection URL regardless of what encoding the server used in the Location header. Surprisingly, this specific server doesn't appear to care (both variants end accepted and return the same response), but for other servers this may necessarily be the same. Most will expect the exact same byte sequence to be used for the next location. How should requests handle those?

All of this is distressingly difficult for us to handle appropriately. The biggest issue is that we do not control header decoding on Python 3 (as noted in the code comments above the change you made), so things get tricky fast.

The core issue though is that we cannot "retain the original": we need to transition the string to a native form. Have you tried using `to_native_string(resp.headers['location'], 'latin1')` to see if that resolves the test failure?

> Have you tried using `to_native_string(resp.headers['location'], 'latin1')` to see if that resolves the test failure?

I did, it doesn't, because in Python 2 you'd get a bytestring still. That is then urlencoded to a different representation from the Python 3 Unicode string path.

@mjpieters What is the different urlencoding output in each case?

For the Latin1 `å` character, Python 2 outputs `%E5`, Python 3 `%C3%A5`, so the Latin-1 and UTF-8 bytes URL-encoded.

You can reproduce these in Python 3 with:

```python
>>> from urllib.parse import quote
>>> quote('å', encoding='utf8')
'%C3%A5'
>>> quote('å', encoding='latin')
'%E5'
 ```

So, just to be clear: when given a byte string in Python 2, the quote library just quotes its bytes directly. When given a unicode string on Python 3, the quote library encodes it and then quotes the bytes?

Exactly. And you can tell `quote()` what encoding to use too; the default is UTF-8. So if we can store the encoding for the location header (UTF-8, or if that fails, the fallback to Latin-1) we could use that information to re-encode to the same.

That sounds like it'd be the best approach, if we can swing it.

Any update on this?

__Edit:__ I see there's https://github.com/psf/requests/pull/4933 as well.
support extraction of certificate bundle from a zip archive

Resolves #4369 by checking whether the default CA certificate bundle is being imported from a zip archive. If it is, extract the certificate bundle to a temp folder so it can be handled by the standard OpenSSL libraries.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=h1) Report
> Merging [#4371](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/81b63419cbd8a6f0c9c2b79083d25dc2260156e3?src=pr&el=desc) will **decrease** coverage by `0.28%`.
> The diff coverage is `60%`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4371/graphs/tree.svg?height=150&width=650&token=mZGswIiXLa&src=pr)](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #4371      +/-   ##
=========================================
- Coverage   88.28%     88%   -0.29%     
=========================================
  Files          18      18              
  Lines        2031    2050      +19     
=========================================
+ Hits         1793    1804      +11     
- Misses        238     246       +8
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/adapters.py](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=tree#diff-cmVxdWVzdHMvYWRhcHRlcnMucHk=) | `92.99% <100%> (ø)` | :arrow_up: |
| [requests/utils.py](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=tree#diff-cmVxdWVzdHMvdXRpbHMucHk=) | `81.75% <57.89%> (-1.13%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=footer). Last update [81b6341...b79df49](https://codecov.io/gh/requests/requests/pull/4371?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@Lukasa added tests. I also moved logic to `requests.utils` to avoid sullying the simple beauty of `requests.certs` and refactored to avoid my earlier use of module loading introspection in favor of a less magical approach.

done!

Thank you very much! :sparkles: :cake: :sparkles:
Actually gmo used

Not serious. Not cool.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4370?src=pr&el=h1) Report
> Merging [#4370](https://codecov.io/gh/requests/requests/pull/4370?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/81b63419cbd8a6f0c9c2b79083d25dc2260156e3?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4370/graphs/tree.svg?height=150&width=650&token=mZGswIiXLa&src=pr)](https://codecov.io/gh/requests/requests/pull/4370?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4370   +/-   ##
=======================================
  Coverage   88.28%   88.28%           
=======================================
  Files          18       18           
  Lines        2031     2031           
=======================================
  Hits         1793     1793           
  Misses        238      238
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4370?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4370?src=pr&el=footer). Last update [81b6341...2b09ab6](https://codecov.io/gh/requests/requests/pull/4370?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
rl print response headers

Summary.

## Expected Result

What you expected.

## Actual Result

What happened instead.

## Reproduction Steps

```python
import requests

```

## System Information

    $ python -m requests.help

```
<paste here>
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).
Bugfix issue #4325 by AzePUG team member Elshad Agayev

PoolManager object was initialized once and assigned to self.poolmanager property of HTTPAdapter class. Thats why updating settings (for example - verify=False to verify=True) does not affect. I reinitialized PoolManager on every connection and problem solved.

adapters.py:166
```
def init_poolmanager(self ...):
   ....
   self.poolmanager = PoolManager(num_pools=connections, maxsize=maxsize,
                                       block=block, strict=True, **pool_kwargs)
   return self.poolmanager
```

adapters.py:320
```
def get_connection(self...):
   ...
   conn = self.init_poolmanager(self._pool_connections, self._pool_maxsize, block=self._pool_block).connection_from_url(url)
```

I believe the issue you are attempting to fix was scheduled for Requests 3.0 and is already fixed on the branch of code dedicated to that release. As such, I'm closing this. Thanks for contributing!
bugfix issue #4353 by AzePUG team member Elshad Agayev

- Added new function validate_url into the file utils.py at the line 858
- Before to call prepend_scheme_if_needed function we are checking validity of proxy url in the file adapters.py at the line 307, if wrong then raising ProxyError exception with clear error message

by Azerbaijan Python User Group

Thanks for contributing to requests, @elshadaghazade! Unfortunately, this issue is already being addressed in #4356. I'll be closing this one in favor of the earlier PR.
Fix flake8 errors


# [Codecov](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=h1) Report
> Merging [#4359](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/81b63419cbd8a6f0c9c2b79083d25dc2260156e3?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `75%`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4359/graphs/tree.svg?height=150&width=650&token=mZGswIiXLa&src=pr)](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4359   +/-   ##
=======================================
  Coverage   88.28%   88.28%           
=======================================
  Files          18       18           
  Lines        2031     2031           
=======================================
  Hits         1793     1793           
  Misses        238      238
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/adapters.py](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=tree#diff-cmVxdWVzdHMvYWRhcHRlcnMucHk=) | `92.99% <0%> (ø)` | :arrow_up: |
| [requests/models.py](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=tree#diff-cmVxdWVzdHMvbW9kZWxzLnB5) | `93.45% <100%> (ø)` | :arrow_up: |
| [tests/testserver/\_\_init\_\_.py](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=tree#diff-dGVzdHMvdGVzdHNlcnZlci9fX2luaXRfXy5weQ==) | `73.33% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=footer). Last update [81b6341...1c9693d](https://codecov.io/gh/requests/requests/pull/4359?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


No thanks.
POST with files and timeout does both?

Hi, im making a requests.post call with sometimes large files object and a 5s timeout. Im observing that even if the call does indeed raise a timeout exception, the full file still ends up being sent.. Somehow.

Is this intentional or are there any nuances to using timeout with files?

A timeout is a complex thing. In this case, the timeout is applied *per socket operation*. That means each send, regardless of how much data, and each read. 

In this case you are likely timing out on the read. That will lead to the behaviour you’re describing. 

If you want a total timeout you can arrange it, but it’s definitely tricky. 

Hi, so just to be clear, if uploading the file actually takes minutes, but the request is constantly sending sone data, would the timeout trigger, or does it see that data is being sent, so all is good?

So long as at least one byte reaches the wire per timeout interval, it would think all is good.
Check if host is invalid for proxy

According to RFC3986, the authority section can be empty for a given URL,
however, for a proxy URL, it shouldn't be. This patch adds a check to verify
that the parsed URL will have a valid host before creating the proxy manager.

Fixes #4353

# [Codecov](https://codecov.io/gh/requests/requests/pull/4356?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@234f80a`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4356/graphs/tree.svg?width=650&height=150&src=pr&token=mZGswIiXLa)](https://codecov.io/gh/requests/requests/pull/4356?src=pr&el=tree)

```diff
@@           Coverage Diff            @@
##             master   #4356   +/-   ##
========================================
  Coverage          ?   87.2%           
========================================
  Files             ?      18           
  Lines             ?    2055           
  Branches          ?       0           
========================================
  Hits              ?    1792           
  Misses            ?     263           
  Partials          ?       0
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4356?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/adapters.py](https://codecov.io/gh/requests/requests/pull/4356?src=pr&el=tree#diff-cmVxdWVzdHMvYWRhcHRlcnMucHk=) | `91.74% <100%> (ø)` | |
| [requests/exceptions.py](https://codecov.io/gh/requests/requests/pull/4356?src=pr&el=tree#diff-cmVxdWVzdHMvZXhjZXB0aW9ucy5weQ==) | `100% <100%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4356?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4356?src=pr&el=footer). Last update [234f80a...9a8a826](https://codecov.io/gh/requests/requests/pull/4356?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@sigmavirus24 Is there anything else that I might have missed in the last update of the PR?

I think I saw feedback that we should use a different exception class and I kind of agree. I think it makes sense to subclass `ProxyError` to create `InvalidProxyUrlError` or something similar so people can catch that specifically

I have updated the PR with `InvalidProxyURL`

I think @nehaljwani is on vacation right now.  I'd rebase the PR for him if I could.

I have rebased the PR.

Friendly nudge :sweat_smile: 

Thanks for your contribution @nehaljwani!
Use another SSL engine than OpenSSL

Requests should let the ability to use other SSL backends than OpenSSL (which has numerous problems), for example NSS.

This is an enormously difficult request to achieve. Python is extremely tightly tied to OpenSSL, and so therefore Requests is too (because it ultimately uses httplib). Changing this would require a very substantial patch, and take an enormous amount of developer time. This is sadly just not a priority for the development team. While we'd welcome a patch to do this, it will need to be organised separately from this project.
Add something to the docs about hooks on Session()

I had to work this out by reading the source code for `Session` – better to put it front-and-centre in the docs.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4352?src=pr&el=h1) Report
> Merging [#4352](https://codecov.io/gh/requests/requests/pull/4352?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/c4c031fc5c62fac25c155e6870962736eb363dc9?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4352/graphs/tree.svg?token=mZGswIiXLa&src=pr&height=150&width=650)](https://codecov.io/gh/requests/requests/pull/4352?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4352   +/-   ##
=======================================
  Coverage   88.28%   88.28%           
=======================================
  Files          18       18           
  Lines        2031     2031           
=======================================
  Hits         1793     1793           
  Misses        238      238
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4352?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4352?src=pr&el=footer). Last update [c4c031f...c5ed41e](https://codecov.io/gh/requests/requests/pull/4352?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Is http://docs.python-requests.org/en/master/user/advanced/?highlight=hooks#event-hooks not helpful? It's in the same document.

Ah, I see, it doesn't explain you can have hooks on a session. (Sorry, still ingesting coffee)

(Review markups in progress.)

I’ve pushed the change to switch to dict literals, and clarified the use of multiple hooks on a session – I’ll do the same for the functional API later.

Does “later” mean “in this PR when I next have time” or “in a follow-up PR”? Just want to make sure we’re all on the same page. 

Later = “in this PR, when I am slightly less crushed under PyCon UK prep”

👍

Turns out the train is a convenient time for doing small changes. How’s this for an example of multiple hooks?
Postback woes

Summary.

When trying to simulate postBack messages in an ASP.NET system, I find that Requests quotes
spaces as '+', but the server seems to expect '%20'. I've ready quite a few posts on the 'net about this,
but no (more or less) elegant solution to solve this using Requests. I tried to convert quote the dict to a string 'manually' using `requests.utils.quote`, which works for the spaces, but this function does not quote '/', which seems to be expected by eg. `__VIEWSTATE`.

## Expected Result
A possibility to specify POST operations with %20 quoting

## Actual Result

POST is quoting with '+'

## Reproduction Steps

s = requests.session()
...
        testdata = {"name": "MARIA    "}

        self.s.post(LAST_MOV,
                    data = testdata)
...
Logged from wireshark:

(POST header mostly removed)
Content-Type: application/x-www-form-urlencoded
name=MARIA++++


## System Information

requests 2.13.0
python 3.6.0
Slackware 14 current


The [HTML RFC](https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1) says:

> Space characters are replaced by '+', and then reserved characters are escaped as described in [RFC1738], section 2.2: Non-alphanumeric characters are replaced by `%HH', a percent sign and two hexadecimal digits representing the ASCII code of the character.

So Requests is doing the right thing.

@lutzhorn I don't believe @JohnCC330 is requesting we change the behaviour here. I believe they're looking for help. That said, there's no explicit ask and this is not a Q&A forum.

A short answer, is to URL encode the data yourself using the stdlib.

```py
>>> urllib.parse.urlencode({b'name': b'MARIA    '}, quote_via=urllib.parse.quote)
'name=MARIA%20%20%20%20'
```

If you then pass that to us, e.g.,

```py
data = urllib.parse.urlencode({b'name': b'MARIA    '}, quote_via=urllib.parse.quote)
requests.post(LAST_MOV, data=data, headers={'Content-Type
: 'application/x-www-form-urlencoded'})
```

That should achieve what you want. Either way, the server you're talking to is not specification compliant (as @lutzhorn already mentioned) so your best bet is to handle the data yourself.

In the future, please direct all questions to [StackOverflow](https://stackoverflow.com)
Fix case-insensitive comparison in Session.get_adapter

While trying to get the adapter for an url, the url is lowered before comparing with the prefix, but the prefix is not, so if it contains non-lowercase characters (eg. `https://example.com/sOmE_WeiRD_pReFIX/`), it won't return the correct adapter.

The other solution is to lower the prefix while adding the adapter. I have opted for the first solution to isolate in one place the case-insensitivity feature.

Edit:
Another alternative could be to document that the prefix should be passed lowered to Session.mount() (this is what I am doing for now to work-around the issue).

Yes, I will update the pull request with them.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4349?src=pr&el=h1) Report
> Merging [#4349](https://codecov.io/gh/requests/requests/pull/4349?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/db9da3e0d24c9902e801c45031beb3588d98b3e5?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4349/graphs/tree.svg?width=650&height=150&src=pr&token=mZGswIiXLa)](https://codecov.io/gh/requests/requests/pull/4349?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4349   +/-   ##
=======================================
  Coverage   88.28%   88.28%           
=======================================
  Files          18       18           
  Lines        2031     2031           
=======================================
  Hits         1793     1793           
  Misses        238      238
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4349?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/sessions.py](https://codecov.io/gh/requests/requests/pull/4349?src=pr&el=tree#diff-cmVxdWVzdHMvc2Vzc2lvbnMucHk=) | `95.55% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4349?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4349?src=pr&el=footer). Last update [db9da3e...2afc1cb](https://codecov.io/gh/requests/requests/pull/4349?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@Lukasa I have added one test to check that it works with mixed case, another to check that the comparison is case-insensitive (this two are the ones that would have been failing before), and also another test to check prefix matching works as expected, ie. returning the adapter with the longest prefix that match (as I haven't found any test about it).

Mind bringing this branch up to date with master?

Done!

Thanks! 🎉
TypeError: __str__ returned non-string (type Error)

**Summary.**
I want to install arduino-sketch on my raspberry pi. I have python 3 and 2.7 already installed.
But while installing it with pip following error has been occurred...

## **Expected Result**
Successfully installed arduino-sketch
## **Actual Result**
/usr/local/bin/pip run on Mon Apr 10 02:00:33 2017
Downloading/unpacking arduino-sketch
  Getting page https://pypi.python.org/simple/arduino-sketch/
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/lib/python2.7/dist-packages/pip/commands/install.py", line 290, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundl$  File "/usr/lib/python2.7/dist-packages/pip/req.py", line 1178, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 194, in find_requirement
    page = self._get_page(main_index_url, req)
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 568, in _get_page
    session=self.session,
  File "/usr/lib/python2.7/dist-packages/pip/index.py", line 694, in get_page
    req, link, "connection error: %s" % exc, url,
TypeError: __str__ returned non-string (type Error)


Okay guys I resolved the error.
The error was due to inaccurate date and time of Raspberry PI. To resolve date and time you can do this:

    sudo raspi-config
    Internationalization options
    Change Time Zone
    Select geographical area
    Select city or region
    Reboot your pi
OR you can manually correct time by:
sudo date -s "Mon Aug  12 20:14:11 UTC 2014"

@PalashSharma15  Since you have resolved the error, you can close this issue.

Thanks @lutzhorn and @nateprewitt..
I am a beginner to github and didn't checked out the close feature.
HTTP_PROXY/HTTPS_PROXY not used with urllib3

It appears that environment variables `HTTP_PROXY` and `HTTPS_PROXY` as described here:

http://docs.python-requests.org/en/master/user/advanced/#proxies

are no longer effective, since requests uses urllib3 instead of urllib.

## Expected Result

```shell
$ export HTTP_PROXY="http://10.10.1.10:3128"
$ export HTTPS_PROXY="http://10.10.1.10:1080"

$ python
>>> import requests
>>> requests.get('http://example.org')
```

## Actual Result

`HTTP_PROXY` and `HTTPS_PROXY` not used.

## Reproduction Steps


```shell
$ export HTTP_PROXY="http://10.10.1.10:3128"
$ export HTTPS_PROXY="http://10.10.1.10:1080"

$ python
>>> import requests
>>> requests.get('http://example.org')
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": "2.1.1"
  }, 
  "idna": {
    "version": "2.6"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.14"
  }, 
  "platform": {
    "release": "16.7.0", 
    "system": "Darwin"
  }, 
  "pyOpenSSL": {
    "openssl_version": "1010006f", 
    "version": "17.3.0"
  }, 
  "requests": {
    "version": "2.18.4"
  }, 
  "system_ssl": {
    "version": "100020cf"
  }, 
  "urllib3": {
    "version": "1.22"
  }, 
  "using_pyopenssl": true
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Request hasn’t used urllib in more than 5 years, and those environment variables have worked fine that entire time. What makes you think they don’t work?
Downloading gzip files is broken

I'm trying to download a binary gzip file with a GET request.

## Expected Result

When I download a gzip binary file I would expect the file to be the same size and content as it is on the server.

## Actual Result

The file is uncompressed without any indication and written with a different size and contents.

## Reproduction Steps

```py
def download_file_chunked(url, filename, user, password):
  req = requests.get(url, auth=(user, password), stream=True)
  total_size = int(req.headers.get('content-length', 0)); 
  with open(filename, "wb") as f:
    for chunk in req.iter_content(1024): 
      if chunk:
        f.write(chunk)
```

Some headers in the response:

```
{
  'Date': 'Tue, 17 Oct 2017 10:07:37 GMT', 
  'Server': 'Apache/ProXad [Jul 22 2015 14:50:04]', 
  'Connection': 'close', 'Accept-Ranges': 'bytes', 
  'Content-Length': '17762', 
  'Content-Type': 'application/x-gzip', 
  'Content-Encoding': 'gzip'
}
```

It seems that requests is not only automatically decoding the response (which is gzip-encoded), but also jumps on the gzip data in the response and decodes that as well. This is very counter-intuitive and hard to debug. I found that the following works:

```py
def download_file_copyfileobj(url, filename, user, password):
  import shutil
  req = requests.get(url, auth=(user, password), stream=True)
  print(req.headers)
  with open(filename, "wb") as f:
    req.raw.decode_content = False
    shutil.copyfileobj(req.raw, f)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "1.9"
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.2"
  },
  "platform": {
    "release": "16.7.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "1010006f",
    "version": "17.3.0"
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": true
}
```

Please see that linked issue, I have a theory why this occurs. 

@sztomi is correct. If the server is sending Content-Type *and* Content-Encoding set to `gzip` for a file that was compressed only once then it is in error. Requests will still only decompress the data once though.

Thanks, this indeed was the problem.
headers shouldn't be a dictionary, it should be a list

## Summary.

Requests lib has implemented headers as a dictionary, yet the web supports a repeating header with the same key.  Subversion via WebDAV is tech that uses that.  For example, "svn log" sends up:

```
DAV: http://subversion.tigris.org/xmlns/dav/svn/depth
DAV: http://subversion.tigris.org/xmlns/dav/svn/mergeinfo
DAV: http://subversion.tigris.org/xmlns/dav/svn/log-revprops
```

## Expected Result

All three 'A' headers got sent to the server

## Actual Result

Only one did

## Reproduction Steps

```python
foo = requests.request(verb, url, headers={"A": "one", "A": "two", "A": "three"})
```

## System Information

All versions of requests



Ha!  You can fool it with trailing spaces on the key:

```
 headers={
     "DAV" : "http://subversion.tigris.org/xmlns/dav/svn/depth",
     "DAV " : "http://subversion.tigris.org/xmlns/dav/svn/mergeinfo",
     "DAV  " : "http://subversion.tigris.org/xmlns/dav/svn/log-revprops"
 }
```



Did you try doing `headers=[('A', 'one'), ('A', 'two'), ('A', 'three')]`? I believe we support this use case when sent that way.

`headers` must be a `dict`which is verified here:

https://github.com/requests/requests/blob/54ad3d6dcc173576d3f9784985e88d31a4997677/requests/utils.py#L858-L861

A `list` is not supported.

@paul-hammant What API would you propose to verify multiple-value headers? `headers={"A": "one", "A": "two", "A": "three"}` makes no sense because this is `{'A': 'three'}`. What about a list:

````python
headers = {
    "A": ["1st value of multi-value header", "2nd value of multi-value header],
    "B": "single value header"
}

@lutzhorn, I don't believe we'll allow the user to pass a list since it would go against our stance of not guessing on what the user wants in their header.

While we don't currently support a list of tuples, we use that in other places such as the `data` param and it seems like a possible option.

I'm not convinced we need to make a change here though. RFC 7230 states pretty clearly that:
```
3.2.2.  Field Order
   [...]
   A sender MUST NOT generate multiple header fields with the same field
   name in a message unless either the entire field value for that
   header field is defined as a comma-separated list [i.e., #(values)]
   or the header field is a well-known exception (as noted below).
```

So @paul-hammant you should be able to accomplish what you're looking for with something like:

```python
dav_vals = ["one", "two", "three"]
headers = { 'DAV': ','.join(dav_vals)}
foo = requests.request(verb, url, headers=headers)
```

I'm proposing we close this, because this is solvable with one line and probably doesn't warrant the weight of adding tuple support.



Yes, the RFC is strict on that. Using `join` is a good idea for how to follow the RFC and accomplish what @paul-hammant  has in mind. 

@paul-hammant I think is related to #4290

Apache2/Subversion are not as strict as the RFC. No matter - there are multiple ways through this. Thanks for not insta-closing :)
Include license file in the generated wheel package

The wheel package format supports including the license file. This is done using the `[metadata]` section in the `setup.cfg` file. For additional information on this feature, see:

https://wheel.readthedocs.io/en/stable/index.html#including-the-license-in-the-generated-wheel-file

@jdufresne can rebase this commit onto master when you have a moment. Thanks!

Rebased on latest master.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4342?src=pr&el=h1) Report
> Merging [#4342](https://codecov.io/gh/requests/requests/pull/4342?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/8982efa9e46172b42c8cf6cdcc1a3f4c75e670ce?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4342/graphs/tree.svg?token=mZGswIiXLa&width=650&height=150&src=pr)](https://codecov.io/gh/requests/requests/pull/4342?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4342   +/-   ##
=======================================
  Coverage   88.28%   88.28%           
=======================================
  Files          18       18           
  Lines        2031     2031           
=======================================
  Hits         1793     1793           
  Misses        238      238
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4342?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4342?src=pr&el=footer). Last update [8982efa...1595e43](https://codecov.io/gh/requests/requests/pull/4342?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @jdufresne! ✨
Python 3.7 not officially supported yet

We don't actually test on 3.7 and 3.7 isn't released for another eight months, so don't yet claim support.

Re: https://github.com/requests/requests/pull/4329#pullrequestreview-69421398 and closes https://github.com/requests/requests/pull/4329.

Also, Python 3.3 was also dropped in https://github.com/requests/requests/pull/4231 so remove mention of those.
Issue with certs.where()

I'm seeing a cascading failure from the certifi project with no particular way to override the behavior other than changing python code.

Would be nice to be able to at least supply an environment variable to override the CERTS path.

## Expected Result
A usable request object that validates over HTTPS.

## Actual Result

This is not a file, it's a location in a zip file
```
/tmp/spark-7b667869-6fa4-4a2d-af58-87f6218ce59f/userFiles-bc90eb3c-0796-4c83-844f-14333c8090b0/google-cloud-storage-py27-none-any.all.zip/certifi/cacert.pem
```
```
IOError: Could not find a suitable TLS CA certificate bundle, invalid path: /tmp/spark-7b667869-6fa4-4a2d-af58-87f6218ce59f/userFiles-bc90eb3c-0796-4c83-844f-14333c8090b0/google-cloud-storage-py27-none-any.all.zip/certifi/cacert.pem
```
## Reproduction Steps

Note, this is using PySpark

```python
sc.addPyFile("s3a://big-data-analytics-scratch-prod/project_mesoads/eggs/google-cloud-storage-py27-none-any.all.zip")

import requests
brokencrt = requests.utils.DEFAULT_CA_BUNDLE_PATH
print(brokencrt)
requests.get('https://google.com/')
```

## System Information
This is run via PySpark on Python 2.7 caused by certifi: https://github.com/certifi/certifi.io/issues/14

You can provide an environment variable: `REQUESTS_CA_BUNDLE`.
Respect the NETRC environment variable, if set.

If the NETRC environment variable is set, use it to
load the user's netrc file. If not set, default to
searching the user's home directory for  a `.netrc`
or `_netrc` file.

Issue #4318

Not too sure how to go about testing this, at least with the `os.environ` check done at module load time. I'm thinking about moving it inside `get_netrc_auth` to facilitate testing. With the check done there, a test could look something like:

1. Create an .netrc file in the temp directory
1. Use `utils.set_environ` context manager to temporarily set `NETRC`
1. Call `get_netrc_auth` and test that it is using the values from the temp file, in a similar manner as the `test_basicauth_with_netrc` test.

Does that sound like a good approach? I'm not sure that I like having the `os.environ` check inside the function, but I can't think of a good alternative.

Closing due to inactivity.
Requests hang multiprocessing

I'm trying to `get `4 Stack-overflow pages simultaneously with `Requests `and `multiprocessing`.

## Actual Result

Instead of the processes finishing, i get 4 zombie processes. I really tried 100 things, i read everything i could find(or understand) including the documentation, especially [here](http://docs.python-requests.org/en/latest/user/advanced/#keep-alive). I frankly don't know if this is a matter of requests or of multiprocessing but...

## Reproduction Steps

```python
import multiprocessing as mp
import requests
from bs4 import BeautifulSoup
from random import randint
import threading
import sys   
sys.setrecursionlimit(10000)




class Spider(object):
		
	# define a example function
	def rand_string(self,length, output):
		s = requests.session()
		#s.keep_alive=False
		random_post=randint(1000000,9999999)
		response=s.get('https://stackoverflow.com/questions/'+str(random_post))
		soup=BeautifulSoup(response.content,'lxml')
		try:
			title=soup.find('a',{'class':'question-hyperlink'}).string
		except:
			title="not found"

		output.put(title)
	
	# Setup a list of processes that we want to run
	def run(self,output):

		processes = [mp.Process(target=self.rand_string, args=(x, output)) for x in range(3)]

		for p in processes:
			p.start()
	
		# Exit the completed processes
		for p in processes:
			p.join()
			p.terminate()
	
		# Get process results from the output queue
		results = [output.get() for p in processes]
	
		print(results)
	
	# Run processes

if __name__ == '__main__':

	output = mp.Queue()

	spider=Spider()
	spider.run(output)


			

```


Try setting a timeout of 30 seconds and see if that resolves your issue. I suspect a connection is getting wedged, preventing your program from completing.

Hi, the problem got solved, it seems like it wasthe fact that `title` was type `bs.4.Tag` and the attempt to pickle it would hang the program. ( see the answer [here](https://stackoverflow.com/questions/46710707/requests-and-multiprocessing/46733628#46733628)).
Pylint can't import requests.packages.urllib3

When I import anything from `requests.packages.urllib3`, `pylint` marks such lines as errors: `Unable to import '...' (import-error)`

## Reproduction Steps

```bash
virtualenv -p python3 test
source test/bin/activate

# taken from http://docs.python-requests.org/en/master/user/advanced/#example-specific-ssl-version
echo "from requests.packages.urllib3.poolmanager import PoolManager" > test.py

pip install requests==2.18.4
pip install pylint
pylint --errors-only test.py

************* Module test
E:  1, 0: Unable to import 'requests.packages.urllib3.poolmanager' (import-error)
```
This is only happening with requests>=2.16. Previous versions are fine.

I can see the same behaviour in Python 2.7 (I know, I know... :) )

## Expected Result

Ideally I would like `pylint` to be able to perform the import so it can use the collected information for further variable analysis.

That doesn't necessarily mean changing `requests`, I'm happy to change my code to solve this problem. For example importing "base" `urllib3` instead of the packaged one, although I don't know whether that has any consequences and is safe to do. I saw the comment in `packages.py` that the code is there for compatibility reasons but http://docs.python-requests.org is using it so I assumed that's still the canonical way.

Any recommendations/fixes?

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.3"
  },
  "platform": {
    "release": "4.10.0-35-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```

If you know that you’re using the most recent requests you can safely just use the base urllib3. That is my recommendation in this case.

Thanks for the recommendation! If it's official it might be worth updating the docs so users don't wonder about the packaged `urllib3`.

@shiftysymbol, urllib3 is no longer packaged as of 2.16. We now have a file that points the namespace requests.packages.urllib3 to your currently installed version for backwards compatibility reasons.

Cheers @nateprewitt -- I simply wasn't sure how safe it was to use `urllib3` directly and whether it was the right thing to do since the docs don't mention it and examples still use the bundled `urllib3`. But then I saw comments in the source code so wanted to double check.
ssl client certificate requests reponse status code is 400

Summary.
The server USES nginx to implement HTTPS
## Expected Result
ssl client certificate requests reponse status code is 400
What you expected.
Certificate request
## Actual Result
the nginx error is  21:unable to verify the first certificate
What happened instead.

## Reproduction Steps

```python
import requests
r = requests.get(url,
                 cert='0ea.pem'
                 )
print(r)
```

## System Information

    $ python -m requests.help

```
<paste here>
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Can you show us the contents of the .pem file please? Can you also provide the output of `python -m requests.help` as the template asked for?

From my test, I found that in ubuntu 14.04.3, this won't happened, but in ubuntu 16.04.1, 400 status code is replied when requesting HTTPS.
It's a clue, I haven't dig any deeper yet, but this may give you some hints.
Request blob:domain.com

Summary.

## Expected Result

I want do download blob:domain.com files

## Actual Result

With Requests:
 `No connection adapters were found for 'blob:domain.com/aaaaa-bbbbbb-cccc'`
With python urllib2:
`<urlopen error unknown url type: blob>`

## Reproduction Steps

```python
import requests
requests.get('blob:domain.com/aaaaa-bbbbbb-cccc')
```

## System Information

    $ python -m requests.help

```
{                         
  "chardet": {            
    "version": "3.0.4"    
  },                      
  "cryptography": {       
    "version": ""         
  },                      
  "idna": {               
    "version": "2.6"      
  },                      
  "implementation": {     
    "name": "CPython",    
    "version": "2.7.14"   
  },                      
  "platform": {           
    "release": "10",      
    "system": "Windows"   
  },                      
  "pyOpenSSL": {          
    "openssl_version": "",
    "version": null       
  },                      
  "requests": {           
    "version": "2.18.4"   
  },                      
  "system_ssl": {         
    "version": "100020bf" 
  },                      
  "urllib3": {            
    "version": "1.22"     
  },                      
  "using_pyopenssl": false
}                         
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Requests is a HTTP library, and so only supports HTTP schemed URLs. There will be no support added for other kinds.
The latest version requests error（Value for header......must be of type str or bytes, not <class 'dict'>)，But use verion 2.3.0 is ok。

Hi,
   - I use the latest version requests 2.18.4（Python2 or Python3），An error occurred：
```
requests.exceptions.InvalidHeader: Value for header {clientInfo: {'longitude': '', 'traderName': 'Google Nexus 6 - 5.1.0 - API 22 - 1440x2560', 'partner': 'Tencent', 'nettype': 'wifi', 'clientAppVersion': '2.1.7', 'screenwidth': '1440', 'deviceCode': 'cbbf2edb-340f-3191-b804-46e6f9163fc1', 'clientVersion': '22', 'clientip': 'fe80::a00:27ff:fea4:efdc%eth1', 'screenheight': '2392', 'latitude': '', 'clientSystem': 'android', 'clientYunyuVersion': '7.6.3'}} must be of type str or bytes, not <class 'dict'>
```
  - But use version 2.3.0，http request is normal。
  - The script like this：
```
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import json
import requests

headers = {
    'timestamp': '1506578296',
    'signature': '57c2c971541ef71387742f480f4166e3',
    'platform': '3',
    'clientInfo': {"clientAppVersion":"2.1.7","clientYunyuVersion":"7.6.3","clientSystem":"android","clientVersion":"22","deviceCode":"cbbf2edb-340f-3191-b804-46e6f9163fc1","traderName":"Google Nexus 6 - 5.1.0 - API 22 - 1440x2560","partner":"Tencent","screenwidth":"1440","screenheight":"2392","nettype":"wifi","latitude":"","longitude":"","clientip":"fe80::a00:27ff:fea4:efdc%eth1"}}

product_nid = '32032002530101'
detail_url = 'http://m.meitun.com/newapi/item/getProdBasicInfo?cityid=&mt=1&promotionid=2552789&promotiontype=0&protocol=http&provinceid=11302&sku={}&topicid=145409'.format(product_nid)

detail_info = json.loads(requests.get(detail_url, headers=headers).text)
title = detail_info['data']['name']
price = detail_info['data']['price']
total_sale_vol = detail_info['data']['unk']
img_url = detail_info['data']['imageurl'][0]
spu_id = detail_info['data']['spu']
```
 - Please confirm。Tks！

Requests tightened its interface to only allow header values to be strings, and to not automatically coerce them to strings. You can get the old behaviour back by putting `str()` around your header value.
Warn about possible slowness on older versions of pyOpenSSL

Give a warning when `cryptography < 1.3.4` and `pyOpenSSL < 16.0.0` as discussed in #4309 

I think I squashed everything right this time. Please let me know if I missed something!

Great! Thanks for working with me and helping me get some experience, I really appreciate it!

I've updated the changelog, added myself to AUTHORS.rst, and fixed the commit message.