Raibek/requests-github-issues huggingface dataset:
Incorrect work of requests

<!-- Summary. -->
For some reason, making request via requests library forces a site to return page for enterring captcha.
The same request made with urllib gives expected result.

## Expected Result
The same response as made with urllib, i.e. get a full page response
<!-- What you expected. -->

## Actual Result
Request made with requests caused to get a response with captcha
<!-- What happened instead. -->

## Reproduction Steps

```python
import requests

url = "https://yandex.ru/pogoda/tbilisi"

headers = {"Host": "yandex.ru"}

resp = requests.get(url, headers=headers, verify=False)
print(res.text
```

```python
import urllib.request

req = urllib.request.Request("https://yandex.ru/pogoda/tbilisi")
res = urllib.request.urlopen(req)
print(res.read()))
```

## System Information

    $ python -m requests.help

```json
{

  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.1.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.9"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.28.1"
  },
  "system_ssl": {
    "version": "1010107f"
  },
  "urllib3": {
    "version": "1.26.9"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false

  "paste": "here"
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->


I suspect it is something related with ssl, because, when i decided to capture traffic from these two methods and compare, i was using httpdebugger and their certificate.
In this case everything worked as expected.

Hi @beqabeqa473, the service you're trying to connect to is performing user-agent fingerprinting and rejecting your requests. This is done by a number of services to prevent webscraping with common tools like Requests. We won't be able to provide help with that on this issue tracker though.

Hello.

Yes, of course i know about user-agent, and i passed correct user-agent from my browser as a header, but it redirects to captcha page.

I was seeing something similar earlier with requests, and in that case also, i was not able to reproduce issue with http debugger with enabled https encryption.
SSLError: [SSL: BAD_ECPOINT] bad ecpoint (_ssl.c:997)

- macos,python3.10,requests 2.28.1

- request
```
requests.get("https://kjt.hebei.gov.cn/www/index_ssl/index.html",verify=False,headers={"User-Agent":"firefox"})
```
- error info
```
SSLError: HTTPSConnectionPool(host='kjt.hebei.gov.cn', port=443): Max retries exceeded with url: /www/index_ssl/index.html (Caused by SSLError(SSLError(1, '[SSL: BAD_ECPOINT] bad ecpoint (_ssl.c:997)')))
```
- python3.10 -m requests.help
```
{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.8"
  },
  "cryptography": {
    "version": "36.0.0"
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.5"
  },
  "platform": {
    "release": "21.5.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "101010cf",
    "version": "22.0.0"
  },
  "requests": {
    "version": "2.28.1"
  },
  "system_ssl": {
    "version": "1010111f"
  },
  "urllib3": {
    "version": "1.26.7"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": true
}
```

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

may be? https://bugs.python.org/issue32882
how set_ecdh_curve("X25519") in requests?
[2.28.1] IncompleteRead errors on large file download using iter_content (working 2.28.0)

Receiving ChunkedEncodingErrors with IncompleteRead while receiving a reponse with a large binary using transfer-encoding: chunked.

## Expected Result

A response with a large binary body (500MB+) send using `transfer-encoding: chunked` with the appropriate header gets processed in steps using `inter_content` without error. 

Behavior as it was in 28.0.0.

## Actual Result

The processing fails with a random number of missing bytes raising `ChunkedEncodingError (Connection Broken: IncompleteRead(X bytes read, Y bytes more expected))`

## Reproduction Steps

Send a request to an endpoint returning a large binary (500MB+) in the reponse body using transfer encoding chunked.

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": "5.0.0"
  },
  "charset_normalizer": {
    "version": "2.1.0"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.10"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.28.1"
  },
  "system_ssl": {
    "version": "101010df"
  },
  "urllib3": {
    "version": "1.26.9"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": false
}

```


Hi @Moandco, could you provide some more information on the issue?

A few questions that are raised immediately are

a.) is this specific to a single server/service or broadly reproducible?
b.) is this error happening consistently with every large download?
c.) if it is consistent, downgrading immediately resolves the issue?


There was a minor change with iter_content in 2.28.1 but it should be largely cosmetic. The ChunkedEncodingError comes from a ProtocolError in urllib3 which has always been possible. We haven’t changed anything around that logic.

The issue is caused by the server-side. The package upgrade was a coincidence. Thanks for your time.
requests.cookies.RequestsCookieJar: popitem() does not work

`requests.cookies.RequestsCookieJar`'s `popitem()` method doesn't seem to work even if cookies is not empty.

## Expected Result

From [the doc](https://requests.readthedocs.io/en/latest/api/#requests.cookies.RequestsCookieJar.popitem):

>remove and return some (key, value) pair as a 2-tuple; but raise KeyError if D is empty.

(Also, I'm not sure what exactly is "D" here. I assume it means the cookies obj itself.)

## Actual Result

It always raises KeyError even when it's not empty.

## Reproduction Steps

```python
import requests

r = requests.get('https://google.com')
print(len(r.cookies)) # = 3

r.cookies.popitem()
```

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": "4.0.0"
  },
  "charset_normalizer": {
    "version": "2.0.10"
  },
  "cryptography": {
    "version": "3.4.8"
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.1"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "101010cf",
    "version": "20.0.1"
  },
  "requests": {
    "version": "2.28.1"
  },
  "system_ssl": {
    "version": "1010107f"
  },
  "urllib3": {
    "version": "1.26.3"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->


I found the issue and hopefully fixed it but I couldn't find the documentation entry, perhaps because `popitem` is not implemented explicitly in the code. Is there any way to improve the doc?

Correct @kianelbo support was not intended for that method but we can't use the mapping interface without it. To be fair, exposing cookies as if they're a simple mapping was a huge design mistake. Unfortunately we're here now so I think we have to fix this unless @nateprewitt or @sethmlarson agree that we should fix the cookies design in some other backwards compatible fashion

Yeah, I agree we should just fix it. Long term, I'm in favor of us reworking cookies entirely, as we've discussed before.

For the moment, this seems reasonably straight forward to get the inherited methods working. Can we get a quick manual pass to verify this fixes the others that come through mapping?

@nateprewitt do you mean just making the inherited methods work for now? They're currently ok other than `popitem` which the PR fixes.

Thanks for confirming, @kianelbo. I have your PR on the shortlist for review this week and we'll look at getting this change added.

Hi. Any plans for reviewing the PR? I was really excited to see it merged.
Werkzeug importing failed

Heya to all,

I already check Internet before botherin you all. Here's the deal: i installed all the requirements.txt and i wanted to launch the tests, but i raise an _**ImportError**_ with _**werkzeug.wrappers**_. Here is the complete error:

![Sans titre](https://user-images.githubusercontent.com/80721211/176997660-9ca8d709-4610-4c1f-93f4-948b428ac946.png)

Here is my Pip List:

![Sans titre](https://user-images.githubusercontent.com/80721211/176997804-cf619a8a-41f3-42ff-b6fe-4ccbc100fead.png)

So i upgraded Pip, i changed the version for werkzeug, i searched Google for answers but, the solutions that i found are not correct for my case. I'll be glad if someone could help me, i really want to:

- Run the tests

- Contribute to Requests

- Become the best programmer

But just not seeing that error will make me happy.

Thanks for reading and good day!



Hello there,
 I've read your issue and the werkzeug code,
 there is a command in their code that might help you
`    .. versionchanged:: 2.0
        Combine ``BaseResponse`` and mixins into a single ``Response``
        class. Using the old classes is deprecated and will be removed
        in Werkzeug 2.1.`
I think you should try :
`from Werkzeug.wrappers import Response`
Hope that helps you, 
GoodLuck 👍🏼 

Unfortunately we don't have time to help people debug their developer environments. You should ask on StackOverflow or do some additional learning about managing Python environments.
refactor: prefer dictionary comphrension to loop

A dictionary comprehension can create the dictionary on one line, cutting out the clutter of declaring an empty dict and then adding items.

Squeezing code onto one line can make it more difficult to read, but for comprehensions this isn't the case. All of the elements that you need are nicely presented, and once you are used to the syntax it is actually more readable than the for loop version. Given the attribute names, this is still readable due to lack of conditional behavior.

Another point is that the assignment is now more of an atomic operation - we're declaring what `cookie_dict` is rather than giving instructions on how to build it. This makes the code read like more of a narrative, since going forward we will care more about what `cookie_dict` is than the details of its construction.

Finally comprehensions will usually execute more quickly than building the collection in a loop, which is another factor if performance is a consideration. As cookies are variable in quantity, this may be of value in this location

Thanks for the PR @kkirsche.
refactor: prefer f-strings to format where fits within black lengths

Python added f-strings in version 3.6, with [PEP 498](https://www.python.org/dev/peps/pep-0498/). F-strings are a flexible and powerful way to format strings. They make the code shorter and more readable, since the code now looks more like the output.

F-Strings are also more performant than using `.format` (e.g. see [F-String Speed Considerations](https://realpython.com/lessons/f-strings-speed-considerations-performance/), [F-String Benchmarks](https://www.scivision.dev/python-f-string-speed/) for two quick examples) because the python runtime can insert the variables into the string while parsing rather than backtracking to insert them into the placeholders, reducing the number of times the lines need to be processed
refactor: use contextlib.suppress instead of empty except

Use [contextlib](https://docs.python.org/3/library/contextlib.html)'s suppress method to silence a specific error, instead of passing in an exception handler.

The context manager slightly shortens the code and significantly clarifies the author's intention to ignore the specific errors. The standard library feature was introduced following a [discussion](https://bugs.python.org/issue15806), where the consensus was that

> A key benefit here is in the priming effect for readers... The with statement form makes it clear before you start reading the code that certain exceptions won't propagate.
Replace bare exception handlers with contextlib.suppress

Use [contextlib](https://docs.python.org/3/library/contextlib.html)'s suppress method to silence a specific error, instead of passing in an exception handler.

The context manager slightly shortens the code and significantly clarifies the author's intention to ignore the specific errors. The standard library feature was introduced following a [discussion](https://bugs.python.org/issue15806), where the consensus was that

> A key benefit here is in the priming effect for readers... The with statement form makes it clear before you start reading the code that certain exceptions won't propagate.
adapt `requests`' version range check to release 5.0.0 of `chardet`

## Summary

`chardet` version 5.0.0 was released on pypi on 25th of June 2022. If this version is picked up by your dependencies, `requests` version 2.28.0 will issue the following warning:

```
.../lib/python3.9/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.8) or chardet (5.0.0)/charset_normalizer (2.0.4) doesn't match a supported version!"
```

I am not sure whether version 5.0.0 of `chardet` is compatible with `requests`. According to the documentation, `requests` supports `chardet` version 4. It would be good to verify the compatibility with `chardet` version 5 and adapt the version range check accordingly


## Reproduction Steps

Install the latest version of  `chardet`:

```shell
$ pip install --upgrade chardet
```

Import `requests` in python to trigger the warning:
 
```python
import requests
```

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": "5.0.0"
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.13"
  },
  "platform": {
    "release": "5.3.18-150300.59.76-default",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.27.1"
  },
  "system_ssl": {
    "version": "1010104f"
  },
  "urllib3": {
    "version": "1.26.9"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": false
}
```


Closing as a duplicate of https://github.com/psf/requests/issues/6177. Please search for open and closed issues before opening a new issue.

Thx, searched for an issue, but did not find it. Will search more intensely next time! ;-)
v2.28.1

2.28.1 (2022-06-29)
-------------------

**Improvements**

- Speed optimization in `iter_content` with transition to `yield from`. (#6170)

**Dependencies**

- Added support for chardet 5.0.0 (#6179)
- Added support for charset-normalizer 2.1.0 (#6169)

I'm deferring the decisions on #6159 and #6167 until we see larger data points. They have 5 and 2 user engagements respectively which seems very low for the current 2.28.x adoption rate. For the time being, there are straight forward workarounds for both.

> I'm deferring the decisions on #6159 and #6167 until we see larger data points. They have 5 and 2 user engagements respectively which seems very low for the current 2.28.x adoption rate. For the time being, there are straight forward workarounds for both.

Hi, what is the workaround for #6167? I am adding ":authority" and the request fails @nateprewitt 

@afriedma please keep conversations constrained to relevant issues. I believe this was already [answered](https://github.com/psf/requests/issues/6167#issuecomment-1159781034) in the other issue you commented on. If you're unfamiliar with PreparedRequests, you can find them in our [documentation](https://requests.readthedocs.io/en/latest/user/advanced/#prepared-requests).
Incorrect dependency warning about chardep v5.0.0

Warning about chardet version does not match requirements:
/usr/local/lib/python3.8/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version! warnings.warn(

If install chardet==4.0.0 this warning does not appears.

root@wret:/# pipdeptree -p requests
requests==2.28.0
  - certifi [required: >=2017.4.17, installed: 2022.6.15]
  - charset-normalizer [required: ~=2.0.0, installed: 2.0.12]
  - idna [required: >=2.5,<4, installed: 3.3]
  - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.9]

## Expected Result

No warning about dependency of chardep

## Actual Result

/usr/local/lib/python3.8/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version! warnings.warn(

According pipdeptree requirements are match

## Reproduction Steps

python -m requests.help

## System Information

/usr/local/lib/python3.8/site-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.9) or chardet (5.0.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn(
{
  "chardet": {
    "version": "5.0.0"
  },
  "charset_normalizer": {
    "version": "2.0.12"
  },
  "cryptography": {
    "version": "37.0.2"
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.13"
  },
  "platform": {
    "release": "5.10.104-linuxkit",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "30000030",
    "version": "22.0.0"
  },
  "requests": {
    "version": "2.28.0"
  },
  "system_ssl": {
    "version": "101010ef"
  },
  "urllib3": {
    "version": "1.26.9"
  },
  "using_charset_normalizer": false,
  "using_pyopenssl": true
}

Docker image:
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"


Hi @drfunjohn, please check open and closed issues before opening new ones. Dupe #6177.
(Github Actions [Lint, CodeQL Analysis]) Update action versions

Currently at the moment, Lint is behind by one version for this repository and CodeQL Analysis is behind by two versions. I've updated to the latest versions for them and have checked the repositories used in the files and have confirmed that the versions exists.

Hi @ArjunSharda,

Thanks for the PR. Typically, we don't accept misc infrastructure changes without notable benfits. This is to reduce PR noise and avoid incomplete updates. In the future, we'll likely constrain these files to dependabot/maintainers only.

It looks like none of the changes in setup-python or checkout are relevant to our workflows. There are some bug fixes in CodeQL that may be useful though. I've started CI and we can look at merging afterwards.

> Hi @ArjunSharda,
> 
> Thanks for the PR. Typically, we don't accept misc infrastructure changes without notable benfits. This is to reduce PR noise and avoid incomplete updates. In the future, we'll likely constrain these files to dependabot/maintainers only.
> 
> It looks like none of the changes in setup-python or checkout are relevant to our workflows. There are some bug fixes in CodeQL that may be useful though. I've started CI and we can look at merging afterwards.

I understand. Thank you
Update Dependency with `chardet>=3.0.2,<6`

https://github.com/chardet/chardet/releases/tag/5.0.0 was released on 2022-06-25 with remove of Python 3.6 support, which also align with our Python > 3.6 requirement.

Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

Thanks for the PR, @hswong3i. We’ll also need an update in the \_\_init\_\_.py for the version range. You can find an example [here](https://github.com/psf/requests/pull/5688/files).

We’ll also need to do some quick manual regression testing to verify there isn’t anything substantially breaking in the major version. 

> Thanks for the PR, @hswong3i. We’ll also need an update in the __init__.py for the version range. You can find an example [here](https://github.com/psf/requests/pull/5688/files).

@nateprewitt thank you for your prompt reply, and I had also discover this when running GitLab CLI.
 
> We’ll also need to do some quick manual regression testing to verify there isn’t anything substantially breaking in the major version.

As your reference, https://github.com/python-gitlab/python-gitlab/blob/main/setup.py using request, and I got following message before apply this PR:
```
/usr/lib/python3/dist-packages/requests/__init__.py:109: RequestsDependencyWarning: urllib3 (1.26.9) or c
hardet (5.0.0)/charset_normalizer (2.1.0) doesn't match a supported version!
  warnings.warn(
```

Now my re-packaged https://build.opensuse.org/package/show/home:alvistack/psf-requests-2.28.0 working fine with https://build.opensuse.org/package/show/home:alvistack/python-gitlab-python-gitlab-3.5.0 under Ubuntu 22.04 with Python 3.10.

Result after running `make` and `make ci`: [report.xml.txt](https://github.com/psf/requests/files/8986052/report.xml.txt)
```
================================================ short test summary info =================================================
FAILED tests/test_requests.py::TestTimeout::test_connect_timeout[timeout0] - Failed: The connect() request should time ...
FAILED tests/test_requests.py::TestTimeout::test_connect_timeout[timeout1] - Failed: The connect() request should time ...
FAILED tests/test_requests.py::TestTimeout::test_total_timeout_connect[timeout0] - requests.exceptions.ReadTimeout: HTT...
FAILED tests/test_requests.py::TestTimeout::test_total_timeout_connect[timeout1] - requests.exceptions.ReadTimeout: HTT...
===================== 4 failed, 575 passed, 13 skipped, 1 xfailed, 24 warnings in 172.28s (0:02:52) ======================
make: *** [Makefile:8: ci] Error 1
```
Update Dependency with `charset_normalizer>=2.0.0`

https://github.com/Ousret/charset_normalizer/releases/tag/2.1.0 was released on 2022-06-20, with our current dependency with `charset_normalizer~=2.0.0` we couldn't upgrade to 2.1.0 accordingly.

Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

Thanks for the PR, @hswong3i. Unfortunately this is a duplicate of #6169.

@nateprewitt thank you for clarify and sorry for my miss. Looking forward for upstream solution get merged.

In between I give a quick dirty workaround for my packaging scripts, and it is now working with CentOS 8 + molecule 4.0.0 + charset_normalizer 2.1.0 + requests 2.28.0 (patched):
* https://github.com/alvistack/psf-requests/commit/4910bb75c0122efd77e11274de2793c95e2f630b
* https://build.opensuse.org/package/show/home:alvistack/psf-requests-2.28.0
python -m pip install requests==2.27.1 instead installs everything in requirements.txt besides requests.

python -m pip install requests==2.27.1  maybe python is thinking its installing the requested stuff? please help.

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

We aren't responsible for the behavior of pip
Clarify setting proxies exceptions

This adds an example for the `NO_PROXY` environment variable and explains making exceptions to the proxy settings using per-host proxy entries.

@nateprewitt Can this be merged please? This still [confuses](https://github.com/psf/requests/pull/5596#issuecomment-1403476580) many people.

Bumping this.
I spent several days trying to sort out why a no_proxy setting wasn't working in a python system, and would have really loved to have this clarification in the docs.
Use yield from instead of looping yield

This merge request updates the stream yield functionality to use `yield from` instead of `yield` within a for loop. Because this is yielding from an iterable, we can use `yield from` which is not only slightly shorter but also on average 15% more performant than using `yield` inside of a loop. This is a result of some of the optimizations included as part of [PEP 380](https://peps.python.org/pep-0380/)

Yep, part of splitting up https://github.com/psf/requests/pull/6165 to get past that...feedback
Allow charset normalizer >=2 and <3

The current definition `charset_normalizer~=2.0.0` does not allow charset-normalizer 2.1.0 which has been released a couple of hours ago. The newer version seems to be compatible with requests and [the compatibility check](https://github.com/psf/requests/blob/da9996fe4dc63356e9467d0a5e10df3d89a8528e/requests/__init__.py#L80-L84) suggests that requests should allow versions up to 3.0.

All tox environments passed locally.

Similar and duplicated fix from https://github.com/psf/requests/pull/6175#issuecomment-1165150374:

> In between I give a quick dirty workaround for my packaging scripts, and it is now working with CentOS 8 + molecule 4.0.0 + charset_normalizer 2.1.0 + requests 2.28.0 (patched):
> * https://github.com/alvistack/psf-requests/commit/4910bb75c0122efd77e11274de2793c95e2f630b
> * https://build.opensuse.org/package/show/home:alvistack/psf-requests-2.28.0
redirect https://requests.readthedocs.io/en/master to https://requests.readthedocs.io/en/latest

the old https://requests.readthedocs.io/en/master page links to an old v2.26.0 with broken algolia search https://github.com/psf/requests/issues/5254 https://github.com/psf/requests/issues/6142 and is linked to by search engines and other documentation 


https://www.google.com/search?q=requests+readthedocs
https://developer-portal.driver-vehicle-licensing.api.gov.uk/apis/vehicle-enquiry-service/code-examples.html#python-using-requests
https://github.com/search?q=requests.readthedocs.io%2Fen%2Fmaster&type=code


Thanks for letting us know, @graingert. I've tried a few things with prefix redirects but they don't work as one might expect. I found a pattern for exact redirects that appears to have most URLs migrated to /en/latest. The base url without a trailing slash (https://requests.readthedocs.io/en/master) doesn't appear to work with any combination though. If anyone has ideas, I'm willing to try them.

Given almost all relevant URLs appear to be forwarding correctly now, I'll resolve this.

The first hit on Google redirects here https://readthedocs.org/projects/requests/ which doesn't seem right

@dvla can you update your links?

> The first hit on Google redirects here https://readthedocs.org/projects/requests/ which doesn't seem right

Hmm, this may be personalized based on geolocation or account. I'm seeing https://requests.readthedocs.io/en/latest/ as the first link on Google. I'm not sure what our options are to affect SEO on ranking though. If you have ideas, I'm happy to give them a try.

Edit: I'm also seeing the https://developer-portal.driver-vehicle-licensing.api.gov.uk/apis/vehicle-enquiry-service/code-examples.html#python-using-requests links working correctly. Is that not what you're seeing?

Yeah it's much much better than it was, thanks for fixing this!

Hopefully the SEO link juice will percolate through soon
Add union operator support to CaseInsensitiveDict (PEP 584)

Adds support for the `|` and `|=` operators for `CaseInsensitiveDict`.

This copies the functionality of `dict` from [PEP 584](https://peps.python.org/pep-0584). Even though this PEP was implemented in 3.9, the functionality implemented to `CaseInsensitiveDict` still works in all currently supported versions (3.7+).

This also brings a level of consistency with other Mapping types, i.e. `OrderedDict`, `MappingProxyType`, `ChainMap`, `WeakKeyDictionary`, and more have this as supported behavior.

The code itself is a slightly modified version of the [reference implementation](https://peps.python.org/pep-0584/#reference-implementation).

Example usage:
```python3
session = Session()
session.headers |= {'Accept', 'application/json'}
```
refactor: clean up various refactorings

1. Use `contextlib.suppress` rather than empty `except`
2. Prefer f-strings to `.format` and `+`-based concatenation for improved performance.
3. Remove empty `elif` via refactoring.
4. Use `yield from` instead of `yield` for 15–20% performance gain (on average)
5. Generate `list()` rather than iteration
6. Prefer generators to looped iteration

Oops, accidentally applied a named expression. I'll fix that

Thanks for the effort but we do not accept arbitrary refactorings of numerous disparate places in the code that aren't related. This tends to invite floods if other questionable large scale refactorings. The refactoring is usually largely subjective and unsolicited.
How to write python script setting system-wide http proxy using requests

Hello,

I've written a Python script exporting http proxy:

```
import os

proxy = "http://proxy:port"

os.environ['http_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy

```

But the thing is that I want to set the http proxy on system wide without exporting the environment variables using the requests library.

How to do that? I am Python newbie.. Please help.

I am using macOS by the way.

I don't have a mac to test this but I did find this answer from [stackoverflow](https://stackoverflow.com/a/39934154).
According to that I would set the proxy like this:
`os.system("networksetup -setwebproxy Ethernet {} {}".format(url, port))`
You can check the [networksetup documentation](https://keith.github.io/xcode-man-pages/networksetup.8.html) for more infos.

Thank you for your answer. 
I executed the script but it has returned the following error:

```
Traceback (most recent call last):
  File "networksetup on.py", line 9, in <module>
    Proxy_on()
  File "networksetup on.py", line 7, in Proxy_on
    os.system('networksetup -setwebproxy Wi-Fi '+proxy+' '+port)
TypeError: can only concatenate str (not "int") to str
```

This is the script:

```
import os

proxy = "http://127.0.0.1"
port = 8080

def Proxy_on():
    os.system('networksetup -setwebproxy Wi-Fi '+proxy+' '+port)

Proxy_on()
```

Hi @1nrho12, it appears this isn't related to Requests but rather a general Python question. We'd ask you move this to a platform like [StackOverflow](https://stackoverflow.com/). We try to reserve the issue tracker for Requests defects as responses to this issue emails 1400 people.
Fix HTTPS websites with system-wide HTTP proxy on Windows

Due to urllib bug, requests fails to open any HTTPS websites on Windows
if there's system-wide HTTP proxy configured. This is because urllib
incorrectly prepends the protocol to the proxy host and port, as in:

```
{'http':  'http://host:port',
 'https': 'https://host:port',
 'ftp':   'ftp://host:port'}
```

Such configuration forces urllib3 to use HTTPS proxy (Secure Web Proxy,
also known as TLS Proxy) for HTTPS URLs, which the configured proxy
most likely does not support.

Detect incorrect behavior and rewrite the protocol to http.

Issue #6139

Could anyone take a look at this merge request please?
> ![Screenshot_2020-06-15-21-55-17-097_com termux](https://user-images.githubusercontent.com/66963067/84682209-f15ffc00-af52-11ea-9d3c-78b7bded9f57.jpg)


Hi @7Kyra, this isn't an issue with Requests but a general Python question. You'll need to ensure the key you're retrieving exists. If you have further questions please open a question on [StackOverflow](https://stackoverflow.com).
Ability/option to automatically handle `Retry-After` in responses.

I am aware that "Requests is not accepting feature requests at this time." but am asking anyway.

Currently, there are many APIs that use the `Retry-After` header for throttling, but API client code generally does not take that into account. When thinking about how I would suggest adding throttling to atlassian-python-api, I realized that, just like following of redirects, this might make more sense to handle at a lower level in the Requests library.


Hi @stevecj, this is already possible in Requests. We don't implement any of the Retry behavior, that comes from urllib3.

You can see in our adapter that we take a [`max_retries`](https://github.com/psf/requests/blob/2a6f290bc09324406708a4d404a88a45d848ddf9/requests/adapters.py#L140) argument that take an int to instantiate a urllib3 [`Retry`](https://github.com/urllib3/urllib3/blob/23af17403c965a90d98cd13785ed385db6b022e6/src/urllib3/util/retry.py#L92) object.

By default, Requests does not retry. If you'd like to enable that functionality, create a Session, re-mount a new HTTPAdapter instance with your Retry and you're set. Note that Retry has an explicit [`respect_retry_after_header`](https://github.com/urllib3/urllib3/blob/23af17403c965a90d98cd13785ed385db6b022e6/src/urllib3/util/retry.py#L257) which is defaulted to True.

e.g.
```python
import requests

adapter = requests.adapters.HTTPAdapter(max_retries=3)
s = requests.Session()
s.mount('http://', adapter)
s.mount('https://', adapter)
```
v2.28.0

2.28.0 (2022-06-09)
-------------------

**Deprecations**

- ⚠️ Requests has officially dropped support for Python 2.7. ⚠️ (#6091)
- Requests has officially dropped support for Python 3.6 (including pypy3). (#6091)

**Improvements**

- Wrap JSON parsing issues in Request's JSONDecodeError for payloads without
  an encoding to make `json()` API consistent. (#6097)
- Parse header components consistently, raising an InvalidHeader error in
  all invalid cases. (#6154)
- Added provisional 3.11 support with current beta build. (#6155)
- Requests got a makeover and we decided to paint it black. (#6095)

**Bugfixes**

- Fixed bug where setting `CURL_CA_BUNDLE` to an empty string would disable
  cert verification. All Requests 2.x versions before 2.28.0 are affected. (#6074)
- Fixed urllib3 exception leak, wrapping `urllib3.exceptions.SSLError` with
  `requests.exceptions.SSLError` for `content` and `iter_content`. (#6057)
- Fixed issue where invalid Windows registry entires caused proxy resolution
  to raise an exception rather than ignoring the entry. (#6149)
- Fixed issue where entire payload could be included in the error message for
  JSONDecodeError. (#6079
Add validation for header name

Following up on #6083, this refactors our header validation function from 2.11.0 to consider the header name. We'd originally avoided adding header name validation because we wanted to limit the change scope to header splitting with new lines. Since then the standard library has made similar changes to ours and now raises a ValueError in `http.client`. This gives us inconsistent errors depending on which portion of the header you provide a bad value.

```python
>>> requests.get("https://httpbin.org/get", headers={":bad": "header"})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/nateprewitt/Work/OpenSource/requests/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/nateprewitt/Work/OpenSource/requests/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/nateprewitt/Work/OpenSource/requests/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/nateprewitt/Work/OpenSource/requests/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/Users/nateprewitt/Work/OpenSource/requests/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/Users/nateprewitt/Work/OpenSource/urllib3/src/urllib3/connectionpool.py", line 727, in urlopen
    httplib_response = self._make_request(
  File "/Users/nateprewitt/Work/OpenSource/urllib3/src/urllib3/connectionpool.py", line 433, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/nateprewitt/Work/OpenSource/urllib3/src/urllib3/connection.py", line 309, in request
    super().request(method, url, body=body, headers=headers)
  File "/Users/nateprewitt/.pyenv/versions/3.10.4/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/nateprewitt/.pyenv/versions/3.10.4/lib/python3.10/http/client.py", line 1323, in _send_request
    self.putheader(hdr, value)
  File "/Users/nateprewitt/Work/OpenSource/urllib3/src/urllib3/connection.py", line 274, in putheader
    super().putheader(header, *values)
  File "/Users/nateprewitt/.pyenv/versions/3.10.4/lib/python3.10/http/client.py", line 1250, in putheader
    raise ValueError('Invalid header name %r' % (header,))
ValueError: Invalid header name b':bad'
```

This PR ports the [validation scheme](https://github.com/python/cpython/blob/6b9122483f1f26afb0c41bd676f9754ffe726e18/Lib/http/client.py#L139) from http.client for header names and ensures we raise an `InvalidHeader` error consistently in all supported versions.
<removed>

Please refer to our [Stack Overflow tag](https://stackoverflow.com/questions/tagged/python-requests) for guidance.
    usuario@host:~$ cat /tmp/a este es un archivo de texto simple. usuario@host:~$ python xorfile.py /tmp/a Sup3rS3cr3t archivo borrado usuario@host:~$ cat /tmp/a <... algunos caracteres no imprimibles...> usuario@host:~$ python xorfile.py /tmp/a Sup3rS3cr3t archivo borrado usuario@host:~$ cat /tmp/a este es un archivo de texto simple. usuario@host:~$

46ccf99e | xxd -r -p > image1.jpg; file image.jpg
image1.jpg: JPEG image data
$ python XOR-hex.py hex 46ccf9ab | xxd -r -p > image2.jpg; file image.jpg
image.jpg: JPEG image data
$ python XOR-hex.py hex 46ccf9a571f0ffb17e41cb84 | xxd -r -p > image3.jpg; file image3.jpg
image3.jpg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, progressive, precision 8, 500x500, components

$ python XOR-hex.py hex 46ccf99e | xxd -r -p > image1.jpg; file image.jpg
image1.jpg: JPEG image data
Create Salhazry 4456

<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/80x15.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.
timeout parameter not applied to initial connection

If specifying the timeout parameter to requests.get(), an exception is raised after the specified timeout. If the code is executed without an internet connection, an exception is raised immediately. Python is at 3.10.

## Expected Result

It feels like the timeout should apply to this scenario as well.

## Actual Result

<!-- What happened instead. -->

## Reproduction Steps

```python
import requests

```

## System Information

    $ python -m requests.help

```json
{
  "paste": "here"
}
```

<!-- This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c). -->


Hi @chatumao, the timeout is for reads and connections. This isn't a wall clock, as specified in the docs, it's for each operation. If we cannot connect (and therefore cannot read), there's nothing for the timeout to apply to which is why we fail immediately.

I don't believe we intend to change this behavior at this time.

Ok, thanks for clearing that up.
Cleanup the docs sidebar

This PR should address some of the issues raised in #6142. The translation guides are mostly 10+ years out of date at this point. We're going to remove them from the sidebar for now since they're providing limited value.

We're also removing the algolia search bar as it's not functioning as intended. Several features have been added to RTD since we introduced algolia, and there's better options at this point. We may look at adding something like [this](https://docs.readthedocs.io/en/stable/guides/advanced-search.html) in a future PR, but for now the quicksearch at the bottom of the sidebar should be sufficient.
No module named 'urllib3.exceptions'

    from .exceptions import RequestsDependencyWarning
  File "/Users/ericli/venvs/venv3.10.3/lib/python3.10/site-packages/requests/exceptions.py", line 9, in <module>
    from urllib3.exceptions import HTTPError as BaseHTTPError
ModuleNotFoundError: No module named 'urllib3.exceptions'; 'urllib3' is not a package

Hi,
I'm using venv, and has already installed requests, but when I was trying to import requests, It always got the error,

No module named 'urllib3.exceptions'

(venv3.10.3) ➜  pyspider-urllib git:(main) ✗ pip list
Package            Version
------------------ -----------
certifi            2022.5.18.1
charset-normalizer 2.0.12
idna               3.3
pip                22.0.4
requests           2.27.1
setuptools         58.1.0
urllib3            1.26.9

Hi @doraspocket, I'd recommend verifying you don't have a file named `urllib3.py` or another relative directory named `urllib3` in your path for your project. This is a configuration or pathing issue on your machine/venv that unfortunately we won't be able to assist with on this issue tracker. We'd recommend posting on [StackOverflow](https://stackoverflow.com/) for further assistance.
Parts of the documentation still link to python-requests.org

Follow up to #6140 (which is locked)

In the sidebar the links to the docs in different languages point to e.g. `fr.python-requests.org`. These are specified in these files:

* https://github.com/psf/requests/blob/79f2ec3acc4e24fef6e6ce31ad7b1d4e2f77be31/docs/_templates/sidebarintro.html#L65
* https://github.com/psf/requests/blob/79f2ec3acc4e24fef6e6ce31ad7b1d4e2f77be31/docs/_templates/sidebarlogo.html#L60

While there are translations of the docs on readthedocs, at least [the German docs](https://requests.readthedocs.io/de/latest/) are woefully out of date. It might be better just to remove that section of the sidebar entirely.

----------------

The search box in the sidebar (which uses algolia search) has its links pointing to pages under `https://2.python-requests.org/en/master`. I can't find where this is configured. As it stands the search is non-functional.

Thanks for bringing this to our attention @domdfcoding. I've added #6144 which will be the first step in improving some of this. Many of those translations are now 8-10 years old, and given no one has volunteered to make updates in that window, I believe they're likely to remain that way.

I've removed the translations links from the sidebar as well as the algolia search. There should still be the quicksearch bar at the bottom of the sidebar for now and we'll want to look at leveraging RTDs search functionality going forward.

I'll close this out as the immediate issues are now resolved. Thanks again!
python-requests.org domain suspended.

http://python-requests.org/ is showing:

<img width="824" alt="Screenshot 2022-05-23 at 07 15 12" src="https://user-images.githubusercontent.com/227768/169755158-3c2b0013-b869-4a84-93f3-7bddeabd9548.png">

It's the same case for subdomains, including docs.python-requests.org, which is breaking the building of sphinx documentation which refer to `requests` docs: see https://github.com/brunns/mbtest/runs/6549875114?check_suite_focus=true#step:9:31 for an example.


Hi @brunns, thanks for bringing this to our attention. The domain is still owned by Kenneth and the maintainers haven't had access to it in quite some time.

We'll reach out to Kenneth regarding the domain verification. I wanted to quickly confirm though that https://docs.python-requests.org _is_ working for you. I'm still seeing the docs rendered correctly at that subdomain.

Strange - nope, I'm seeing that same error at https://docs.python-requests.org. I wonder if that's just me?

It may be a CDN caching issue. My hunch would be once those expire, I'll be seeing the same.

I can confirm this, [https://docs.python-requests.org](https://docs.python-requests.org) is not working for me, either.

A temporary and partial fix might be to stop http://requests.readthedocs.org/ redirecting to https://docs.python-requests.org/?

I can access https://docs.python-requests.org/en/latest/ fine, but our intersphinx links are failing due to cert domain mismatch

FWIW, https://docs.python-requests.org/en/latest/ redirects to https://docs.python-requests.org/en/latest/ for me, which 404s. 

For https://docs.python-requests.org/en/latest/user/quickstart/ I get an SSL error from Chrome:
```
This server could not prove that it is docs.python-requests.org; 
its security certificate is from *.ispapi.net. 
This may be caused by a misconfiguration or an attacker intercepting your connection.
```

...and then a 404 if I do get through.

Hi I am seeing the same... 
<img width="654" alt="image" src="https://user-images.githubusercontent.com/41849932/169803126-e439982c-0e30-4249-905f-25f9018f01c2.png">


Going to lock this discussion to contributors-only as the issue won't be fixed any faster with additional comments. If you're interested in following when the fix happens you can subscribe to this issue.

For those looking for alternatives, https://requests.readthedocs.io/en/latest/ should be returning docs correctly again. This has been the "official" location for some time as we lack controls on the python-requests domains. I'll update again once we receive a response from Kenneth.

Providing a quick update for users following the issue. I've spoken with Kenneth, but we do not have a clear path to recovering the domain. We'd advise anyone currently blocked by this (or anyone providing resources with links to requests) to use https://requests.readthedocs.io/en/latest/ as the primary domain. This matches what has been in the public docs since 2019 and represents what the current maintainers control.