Raibek/requests-github-issues huggingface dataset:
HTTPDigestAuth fails from secondary thread

Hi,

I am looking into this [Home Assistant issue 13524](https://github.com/home-assistant/home-assistant/issues/13524) where a request is prepared once at startup (on an initialisation thread), and then updated at regular interval from a thread pool. This is working fine with HTTPBasicAuth, but failing with HTTPDigestAuth.

Is this a bug? or is this by design? If it is by design, can you help me find a workaround for Home Assistant?

## Expected Result
```
{
  "authenticated": true,
  "user": "user"
}
```

## Actual Result
```
Traceback (most recent call last):
  File ".../python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File ".../python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "test.py", line 8, in do_update
    request
  File ".../python3.7/site-packages/requests/sessions.py", line 653, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File ".../python3.7/site-packages/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File ".../python3.7/site-packages/requests/auth.py", line 247, in handle_401
    if self._thread_local.pos is not None:
AttributeError: '_thread._local' object has no attribute 'pos'
```

## Reproduction Steps

```python
import requests
import threading

def do_update(name, request):
    print("do_update %s: starting", name)
    with requests.Session() as sess:
        response = sess.send(
            request
        )
    print(response.text)
    print("do_update %s: finishing", name)
    
method = 'GET'
resource = 'http://httpbin.org/digest-auth/auth/user/passwd'
auth = requests.auth.HTTPDigestAuth('user', 'passwd')
request = requests.Request(
    method, resource, auth=auth
).prepare()

#Works on main thread
do_update("main thread", request)

#Doesn't work on secondary thread
x = threading.Thread(target=do_update, args=("other thread",request,))
x.start()
x.join()

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "4.19.57+",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "101000bf"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": false
}

```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).
Set "Content-Disposition" header in `file` multipart failed

Summary.

- Source: [requests/models.py](https://github.com/psf/requests/blob/428f7a275914f60a8f1e76a7d69516d617433d30/requests/models.py)

```
    class RequestEncodingMixin(object):
        ...
        def _encode_files(files, data):
            ...
            rf = RequestField(name=k, data=fdata, filename=fn, headers=fh)
            rf.make_multipart(content_type=ft)

```
Variable `fh` holdes the 4th tuple item passed from

```
    files = {'settings': (filename, io.BytesIO(b'some,data,to,send\nanother,row,to,send\n'),
             'app/xml', {'Content-Disposition':'attachment'} )}

```
The `rf.header dict` get set passing `headers=fh` with `'Content-Disposition':....`
Calling `rf.make_multipart(content_type=ft)`, at the next line, only passing the 3trd tuple item.

The method [`make_multipart` - urllib3/fields.py](https://github.com/urllib3/urllib3/blob/ae7f8bbb57e74ffdb327c5e510963a886fa4ed8b/src/urllib3/fields.py) is defined as

```
    def make_multipart(
        self, content_disposition=None, content_type=None, content_location=None
    ):
        self.headers["Content-Disposition"] = content_disposition or u"form-data"
        ...

```
which replaces, as `content_disposition` is `None`, the allready set `self.headers["Content-Disposition"]` with the default `u"form-data"`.

## Expected Result
```
--e96a4935b8d5b2355f1da3070faa4b28
Content-Disposition: attachment; name="settings"; filename="settings.xml"
Content-Type: app/xml

some,data,to,send
another,row,to,send

--e96a4935b8d5b2355f1da3070faa4b28--

```

## Actual Result
```
--e96a4935b8d5b2355f1da3070faa4b28
Content-Disposition: form-data; name="settings"; filename="settings.xml"
Content-Type: app/xml

some,data,to,send
another,row,to,send

--e96a4935b8d5b2355f1da3070faa4b28--

```

## Reproduction Steps

```python
import requests

url = 'http://127.0.0.1:8080'
filename = "settings.xml"

files = {'settings': (filename, io.BytesIO(b'some,data,to,send\nanother,row,to,send\n'),
         'app/xml', {'Content-Disposition':'attachment'} )}

r = requests.post(url, files=files)

```

## System Information

Tested with Python: 3.5 - urllib3: 1.23 - requests: 2.19.1

## Possible Solution

Pass **all** three parameter explicit to `.make_multipart(...`, e.g.

      rf.make_multipart(content_disposition=fh.get('Content-Disposition'), 
                        content_type=ft, 
                        content_location=fh.get('Content-Location')



This isn't documented as intended to work. This it's effectively a feature request and the project is under feature freeze
Why is requests logging a 401 and 200 status code in the same request

I am running the following from the python repl with logs set to DEBUG:
```python
response = requests.get(
    url=endpoint_url,
    auth=requests.auth.HTTPDigestAuth(username, password),
)
```

This results in:
```
[2019-08-27 13:47:23,361 - DEBUG - connectionpool.py:815] Starting new HTTPS connection (1): my.url.com:port
[2019-08-27 13:47:23,532 - DEBUG - connectionpool.py:396] https://my.url.com:port "GET /some/important/endpoint?$skiptoken='12345' HTTP/1.1" 401 0
[2019-08-27 13:47:24,086 - DEBUG - connectionpool.py:396] https://my.url.com:port "GET /some/important/endpoint?$skiptoken='12345' HTTP/1.1" 200 951999
```

Why am I seeing a 401 response logged before the final 200?
Is there something wrong with the authentication? 

In addition to this I sometimes (very rarely) have a situation where the request never returns and only printed the 401 log. What is causing this error.

See also my [stackoverflow](https://stackoverflow.com/questions/57670089/why-is-python-requests-not-terminating-and-why-are-these-seperate-logs-printed) question for more details.

Hi @JimFawkes,

The 401 response is part of the way HTTP digest authentication, regardless of the http client (browser, curl, wget, requests, etc.) : the server provides the "WWW-Authenticate" header as part of the 401 response, which can then be used to generate and send the hash for the second attempts.

See [Digest access authentication](https://en.wikipedia.org/wiki/Digest_access_authentication) on Wikipedia

Hi @epenet,

thanks for your reply and for clearing this up. So basically I should expect this behaviour for HTTPDigestAuth.



Hi @JimFawkes, yeah epenet is correct. This is a challenge response mechanism used with DigestAuth, Requests will automatically try to authenticate itself against a 401 with your provided auth hook.

Regarding the second issue (that it never returned), this is a design choice by the requests module. If you do not provide a specific timeout, it will wait for ever for a response from the server. You should definitely add a timeout to your request:

```python
response = requests.get(
    url=endpoint_url,
    auth=requests.auth.HTTPDigestAuth(username, password),
    timeout=60,
)
```

Thank you very much for both of your responses and time.
Fix build - use python 2.7 compatible super() call.

Issue introduced here: https://github.com/psf/requests/pull/4983/files#diff-28e67177469c0d36b068d68d9f6043bfR425

@nateprewitt @kennethreitz friendly ping 🙏

@asherf, I think the first parameter should be the base class
```python
    super(SessionRedirectMixin, self).__init__()
```
not the current class
```python
    #Invalid
    super(Session, self).__init__()
```

Refs #5175 

> @asherf, I think the first parameter should be the base class
> 
> ```python
>     super(SessionRedirectMixin, self).__init__()
> ```
> 
> not the current class
> 
> ```python
>     #Invalid
>     super(Session, self).__init__()
> ```
> 
> Refs #5175

@epenet I looked in several places in the code:
https://github.com/psf/requests/blob/master/requests/adapters.py#L59
https://github.com/psf/requests/blob/master/requests/adapters.py#L123
https://github.com/psf/requests/blob/master/requests/exceptions.py#L25
the pattern that you suggest doesn't seem the one that is being used....

also if you look at the [python docs](https://docs.python.org/2/library/functions.html#super), the example there uses the 'current' class:
```python
class C(B):
    def method(self, arg):
        super(C, self).method(arg)
```

Thanks @asherf, there's a lot of contradictory information on super() out there, so I'm happy with your change.

It would be good for @nateprewitt or @kennethreitz to merge it as it's currently upsettingTravis CI on all other PR.

Hi @asherf, thanks for the PR. I’m not convinced we want the code that was merged in the referenced commit in the first place. I need to take a look through, see what testing would need to be added and what security implications it’s going to have.

I’ll leave this open until a decision is made but it’s likely we’ll revert the other commit.

I've pushed 09fd857 to resolve the issue. Resolving in favor of that.
SSL error ~5 times, then success ... any idea why?

Using requests 2.19.1; python 3.6; running on raspberry pi

I have a script to run on-boot and connect to my HTTPS server (certificate is Let's Encrypt). My script will continually attempt to connect until it is successful.

I often see several CERTIFICATE_VERIFY_FAILED in the logs before, suddenly, it connects just fine!

Any ideas why those CERTIFICATE_VERIFY_FAILED are thrown several times (like 3-10 times, attempts are made once every ~2 seconds), until finally it connects just fine?

Just a wild guess, maybe certificate store is made available in later boot stages?

Yeah I was looking for certificate caches, but I never found any. The `requests` library doesn't seem to cache them, and neither does `urllib3` as far as I could find.

Raspbian/Debian stores root certificates in `/etc/ssl/certs`. They don't ever get updated AFAIK (they come with the Raspbian image and are valid until 2030). More info at: [update-ca-certificates](http://manpages.ubuntu.com/manpages/xenial/man8/update-ca-certificates.8.html).

If you know of a place where, perhaps, _intermediate_ certificates are cached somewhere, please let me know. 

In fact, `requests` doesn't even use the root certificates in `/etc/ssl/certs`.

It instead uses the certificates which come with [certifi](https://certifiio.readthedocs.io/en/latest/).

So, yeah, not sure why I would get `CERTIFICATE_VERIFY_FAILED` a few times, then suddenly the connection succeeds. It makes me wonder if the real error is something like "cannot connect" or "network down" and that causes a `CERTIFICATE_VERIFY_FAILED` to be (misleadingly) raised.  (???)

It sounds like there may have been some sort of issue with certifi during boot but it's not immediately clear that this is related to Requests. Since we don't have hardware to debug this readily available, I'm not sure there's much we can do here. At a minimum we'd need a stack trace to better understand what's happening. 

Actually, I (finally) figured this out a few months ago. Although I forgot I had opened this ticket >2 years ago! Sorry about that.

The issue was that the RPi's clock was "in the past" (due to it being turned off for some amount of time), so it was rejecting certificates as being "from the future". Eventually, the clock would be updated via [NTP](https://www.ntp.org/) and then all would work fine after that.

Thanks for the follow up, @acu192! Glad you got it sorted.
urllib3.connection.VerifiedHTTPSConnection - not able to authenticate the company webpage 

I am trying to do a basic authentication using the code

import requests
from requests.auth import HTTPBasicAuth
import csv
import urllib3
urllib3.disable_warnings()
r = requests.get('https://partnermarketplace-qa-adm.dmzmo.xyz.corp/export/Orders_Entries.csv',stream = True , verify = False ,auth=('bwp', '00k'))

But I am getting the below error:

![image](https://user-images.githubusercontent.com/6632016/63288327-0855a380-c2da-11e9-9c07-5596a08a44d1.png)

The same code works on my friend's anaconda but I am getting the issue - not sure if it has something to do with proxy.
Please help



Hi @goldi28, 

This appears to be a request for help with your network and DNS configurations. This issue tracker is intended for reporting defects in Requests. Please open a question on Stackoverflow if you need further assistance. Thanks!
[IMPROVEMENT] Update monthly download count

Requests pulls over 50+ million downloads per month while the old figure is at 11.

https://pepy.tech/project/requests

> Hi @aadibajpai, thanks for the update :)

no problem, what do you think about a badge in the README too?

Like this [![Downloads](https://pepy.tech/badge/requests)](https://pepy.tech/project/requests).

@aadibajpai i love that!

@kennethreitz great!
SSLError when accessing *.dm5cdn.com

I got an `SSLError` when accessing `*.dm5cdn.com` while it works fine with `urllib.request.urlopen`. Here is an example:

```py
URL = "https://manhua1033-104-250-150-12.cdndm5.com/"

import urllib.request
try:
  urllib.request.urlopen(URL)
except Exception as err:
  print(err)

import requests
try:
  requests.get(URL)
except Exception as err:
  print(err)
```

## Expected Result

They should produce the same result.

## Actual Result

```
HTTP Error 403: Forbidden
HTTPSConnectionPool(host='manhua1033-104-250-150-12.cdndm5.com', port=443): Max
retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[S
SL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local is
suer certificate (_ssl.c:1076)')))
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "7",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010103f"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": false
}

```

After some searching, I found that [this solution](https://stackoverflow.com/questions/42981429/ssl-failure-on-windows-using-python-requests) works for me, but I still wonder why it failed. Does it mean that `certifi` needs to be updated? I do not know how SSL works.

Python x64 v3.7.4
certifi 2019.6.16

The server in question is issuing an unverifiable local cert which is why we're raising an exception. The reason urllib works is because it isn't do any checks to validate the connection is secure.
Missing `?` and extra `&` in the `response.url` when getting from a url of double http

The url of the following response is wrong.

```
baseurl = "http://www.google.com/?https//www.helloworld.com/query"
r = requests.get(url, params = {'abc':"1"})
print(r.url)
```

## Expected Result

'http://www.google.com/?https//www.helloworld.com/query?abc=1'

## Actual Result
Missing `?` behind the string `query` and extra `&` behind the string `query`

'http://www.google.com/?https//www.helloworld.com/query&abc=1'

## Reproduction Steps

```python
import requests
baseurl = "http://www.google.com/?https//www.helloworld.com/query"
r = requests.get(url, params = {'abc':"1"})
print(r.url)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.4"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010103f"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": false
}
```

i believe this is expected behavior. thanks for the writeup, though!

> i believe this is expected behavior. thanks for the writeup, though!

why is it an expected behavior? I am not familiar with web and html. Can you provide me some resources to read on this topic? Thanks.
Meta: website bugs - sidebar links to https://python-requests.org and http://www.git-legit.org/ broken

The following pages all link to http://www.git-legit.org/, which at the time of reporting returns a Turkish gambling spam page:
   * https://html.python-requests.org
   * https://2.python-requests.org
   * https://3.python-requests.org

Additionally, https://html.python-requests.org links to https://python-requests.org (note HTTPS), which returns an `SSL_ERROR_INTERNAL_ERROR_ALERT` error page in Firefox, and `curl: (35) error:14004438:SSL routines:CONNECT_CR_SRVR_HELLO:tlsv1 alert internal error` via cURL. See also https://github.com/psf/requests/issues/5138, where the Firefox error was mentioned.

send a PR!

The absence of a PR doesn't make this an invalid or fixed bug - please re-open.
requests.ConnectionError inherits only from OSError

`requests.ConnectionError` doesn't inherit from ConnectionError from std, which is confusing.

## Expected Result

```
(Pdb++) inspect.getmro(ConnectionError)
(<class 'requests.exceptions.ConnectionError'>, <class 'requests.exceptions.RequestException'>, <class 'ConnectionError'>, <class 'OSError'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>)
```

## Actual Result

```
(Pdb++) inspect.getmro(ConnectionError)
(<class 'requests.exceptions.ConnectionError'>, <class 'requests.exceptions.RequestException'>, <class 'OSError'>, <class 'Exception'>, <class 'BaseException'>, <class 'object'>)
```

## Reproduction Steps

```python
import requests
inspect.getmro(requests.ConnectionError)
issubclass(requests.ConnectionError, ConnectionError)

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.2"
  },
  "platform": {
    "release": "18.2.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000212f"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": false
}
```
can not use html charset decoding

### response header:
 'Content-Type': 'text/html'
no charset, but in html have:
```
print(r.encoding)
ISO-8859-1
<meta http-equiv="Content-Type" content="text/html; charset=gb2312" />
```
### code
```
import requests # pip install requests
r = requests.get(url)
unicode_str = r.text # may use `chardet` to auto-detect encoding
display:

'title': 'ÉÂÎ÷Ê¡ÈËÁ¦×ÊÔ´ºÍÉç»á±£ÕÏÌü',
```

Requests doesn't parse HTML or the contents of the body. If the information isn't specified in the Content-Type, we'll default back to ISO-8859-1. You are free to update the encoding as needed by setting `r.encoding` before calling text, you may also be interested in the util function `get_encodings_from_content`.
syntax error with last version of requests

Hi,
from HASSIO installation with python 3.7.3 I'm trying to launch this request:

python3 -c "import requests; print(requests.get(‘http://192.168.1.107/?action=command&command=value_temperature').text)"

but I obtain this error:

Command executed: python3 -c "import requests; print(requests.get(‘http://192.168.1.107/?action=command&command=value_temperature').text)": 1
  File "<string>", line 1
    import requests; print(requests.get(‘http://192.168.1.107/?action=command&command=value_temperature').text)
                                            ^
SyntaxError: invalid character in identifier

with previous versions all worked fine...

Anyone can help me? syntax is changed?

Hi @pilot1981, you have a `‘` at the beginning of your string. In the future please direct usage questions to Stackoverflow. That helps us triage bugs more easily and reduces noise to the people monitoring this list for bugs. Thanks!

Hi,
thanks for reply.

I have to post here?

https://stackoverflow.com/questions/tagged/python-requests

I tried to remove ' but it don't solve my error...now I have:

SyntaxError: invalid syntax




Yes, please post a question there if you’re continuing to see issues. This is not a Requests bug. Make sure you have the same character (`'`) on each end of the string.
Incorrect Content-Type Returned from Head

## Summary
When requesting the headers and then outputting the content-type for [this](https://www.arl.army.mil/arlreports/1999/ARL-CR-441.pdf) url, text/html; charset=utf-8 is output as the content-type, but running `curl -I https://www.arl.army.mil/arlreports/1999/ARL-CR-441.pdf` returns application/pdf

## Expected Result

application/pdf

## Actual Result

text/html; charset=utf-8

## Reproduction Steps

```python
import requests

url = 'https://www.arl.army.mil/arlreports/1999/ARL-CR-441.pdf'

headers = requests.head(url).headers
content_type = headers['Content-Type']
print(content_type)
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.7"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010103f",
    "version": "17.2.0"
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "101000af"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": true
}
```

The provided url (https://www.arl.army.mil/arlreports/1999/ARL-CR-441.pdf ) is not working properly.


@SatyajeetMahato Fixed the issue - seems the page blocks based on User Agent. Changing from the default resulted in "application/pdf" being returned as the content type
something strange about cookie in requests.session.

Summary.

## Expected Result
follow all 'set-cookie' actions from website.

## Actual Result
the website i request would set cookie for three or times to change a specific cookie or add a new cookie. and i found that first time the website set three cookies, that's ok, second time the website set another three cookies, and it failed, and third time the website tried to change three cookies, and it failed too. 
i printed session.cookies each time, it' ok, but cookies in  http packets that i captured did not match session.cookie, there were still the first three cookies.
by the way, my friend found that this situation also appear at other normal browser like Chrome and Firefox, but 'set-cookie' takes effect in Tor browser which is based on Firefox, maybe it's something new?


## Reproduction Steps

```python
import requests
session = requests.session()
session.get(url1)
session.get(url2)
session.get(url3)
```

## System Information

    $ python -m requests.help

```
<paste here>
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


forget to say how i solve it temporarily..cause i found that 'session.cookies' is ok, but cookies in http packets that i captured did not match session.cookie,  i tried to change cookieJar to a DICT and then change this DICT to cookieJar, and assign session.cookies with the new cookieJar before every http request.

thanks, i'll try this later...
Add headers to connect when using proxies?

Its possible with requests to add headers to connect? I need to add 2 headers
Host and proxy connection. But default connection when using proxy is witouth 2 those so i need to know if its in some way possible to add them. 



This is causing us some heartburn, too.

This isn't a great solution, but it could be helpful for diagnostics:

https://github.com/psf/requests/issues/2708

Being relatively new to python, this critical library seems very inconsistent.  Requests work one way if you call them directly, another way if you call them via sessions, another way if they're made via a PoolManager, and another way if using a ProxyManager.  I don't know how I would take the technique in 2708 and turn it into an easily repeatable solution.
Add a parameter so that URL encoding does not occur when sending URLs

URL encoding is not required when encountering special situations. So consider adding a URL that controls whether the URL sent is url-coded or not.

Use a connection adapter. :)

Sent from my iPhone

> On Aug 2, 2019, at 3:02 PM, 九世 <notifications@github.com> wrote:
> 
> URL encoding is not required when encountering special situations. So consider adding a URL that controls whether the URL sent is url-coded or not.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub, or mute the thread.


Kenneth is right, this would be best handled with a custom adapter. I don't see us adding this functionality directly to Requests at this time. Thanks for the suggestion!
how wrapper requests, to bypass WAF

how wrapper requests, to bypass WAF

WAF's incorporate a wide range of technologies which are usually highly customized for the application they are designed to protect.

It's impossible to answer your question as it involves a multitude of disparate technologies which are likely to employ very different forms of protection and traffic analysis.  There won't be a "one size fits all" solution to this and it's very likely Requests isn't the right tool for the job.

Besides if you own/manage the firewall you should be able to have your traffic whitelisted, right?  ;)
monkey patch requests.utils.requote_uri failed

I want to escape the {} symbol, but requests default to escaping the parameters of the url. I looked at the source code and the problem appears in the requote_uri of requests.utils , so I tried hooking this function.
```python
from urllib.parse import quote

import requests


def hook2(uri):
    print(11111111)
    safe_with_percent = "!#$%&'()*+,/:;=?@[]~{}^"
    return quote(uri, safe=safe_with_percent)


requests.utils.requote_uri = hook2
requests.get("http://127.0.0.1:8000", params='a=111&b=222!@$%^&({}')
```
But why did it fail?

@boy-hack this issue tracker is specifically for defects in Requests. Please direct questions to StackOverflow.com, thanks!

@boy-hack ive run into the same issue, did you get anywhere with monkey patching **requote_uri**
Fix typo in api.requests() docstring formatting

Fix a formatting typo which caused the documentation for `api.request()`'s `method` arg to be corrupted when rendered:

![2019-07-28-090608_764x154_scrot](https://user-images.githubusercontent.com/4450708/62008995-f2a7f080-b116-11e9-9531-50f2b2c0e87a.png)


Looks like this is a dupe of #5145
414 Request-URI Too Long

Using Python 3, and request, I'm able to POST some registers of measurements to my server, something like 20 records, everything works ok, but when I try to upload more registers the URL becomes too long and I have this response on my python linux terminal:

```
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>414 Request-URI Too Long</title>
</head><body>
<h1>Request-URI Too Long</h1>
<p>The requested URL's length exceeds the capacity
limit for this server.<br />
</p>
<hr>
<address>Apache/2.4.25 (Debian) Server at api.rookmotion.com Port 443</address>
</body></html>

```
The URL that I'm trying to post currently has a length of 333 char long.

```
r = requests.post(url, headers=headers, params = data)
print(len(r.text))
print(r.text)
```
       
But if I use the printed json on POSTMAN with the corresponding headers, POSTMAN works ok, so I don't know what should I configure additionally on my python request.


```
headers = {'user': self.user, 'key': self.password, 'content-type': 'application/x-www-form-urlencoded; charset=utf-8'}
       
url = "https://api.rookmotion.com/v1.0/..."


```

Thanks!

Why are you using a POST request and sending the data as params? Does the API specify this? If not, wouldn't you want to use `data=` to send the data rather than `params=`? Can you double-check which one Postman uses?

You're right params works ok but makes the URI so it works with few data only while sending as data allowed me to send all the records.

Thanks!

No problem, feel free to close the issue. Next time, you should post this sort of question on Stack Overflow, unless you know **for sure** it's a bug within Requests.
Change The Project Logo

The logo reminds us kenneth reitz and what he did, IMO it is better to remove the logo or replace it with something that is not specific to a person. 

That would make me sad.

> That would make me sad

Okey, then we can close? IMO you dont think this is serious but it is. 

I didn't think a `collaborator` will troll my issue but if that is the case, then i dont want to discuss it further. Lets rename the repo as `kenneth-reitzs-request` and close this issue
Client Side Certs with session

Summary.

Trying to get client side certificates to work with session but I'm seeing a difference between the results when making the call with a session and the helper get method.

## Expected Result
Calling get with a Session returns the same result as calling `requests.get`.

## Actual Result
Request fails as if the certificate was not passed.

## Reproduction Steps

![Screen Shot 2019-07-18 at 11 42 00 AM](https://user-images.githubusercontent.com/3720126/61476222-271af000-a952-11e9-9cd6-aaad69a45b94.png)

## System Information

    $ python -m requests.help

```
(panopta_env) ➜  ~ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "2.7.16"
  },
  "platform": {
    "release": "18.6.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1000212f"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": false
}
```


Looks like this is related to #4325.
Requests3 Docs: Add migration path for api from Requests2

Summary.

## Expected Result

Docs page which has migration for 1-2 also has for 2-3 ; 
https://github.com/kennethreitz/requests3/blob/master/docs/api.rst

## Actual Result

Only 1-2 is still on docs/api.rst, even on repo requests3 (which has no issues only PR's)

## Reproduction Steps

browse ^^

## System Information

    This is for dev docs


If right now everything is fully back compatible, I am happy to give a PR for that.
The only note then would be to ensure you are not using python  2, because Requests3 only supports python3.6+ per the readme.

Hi @AnneTheAgile, this reply is a bit later than it should have been. This project doesn't have any plans for Requests3 at this time, so I don't believe we'll be distributing documentation on migrating. Thanks for the suggestion though!
python-requests.org looks down

I saw #5079 but to me it is still down, even the domain itself doesn't resolve to an IP address.

Seems to be back.

Hi ! Thanks.

http://python-requests.org now redirects to https://2.python-requests.org/en/master/ and the content is ok

However notice that https://python-requests.org/ raise an `SSL_ERROR_INTERNAL_ERROR_ALERT` in my firefox browser.

@kennethreitz  : I don't understand why you close the issue as to me there is still an issue with https
Regards

@mhow2 he is just trolling, dont take his actions serious

@isidentical : maybe he is but in the meantime I still have the error above when accessing https://python-requests.org/
Is it possible to manually set the ecdh curve?

I am on ubuntu 18.04 and there seems to be a problem with the openssl version it ships with such that I cant successfully connect to our API unless I manually set the `set_ecdh_curve("secp384r1")`. Right now Im having to just use urllib and do something like:
```python
        ctx = ssl.create_default_context()
        ctx.set_ecdh_curve("secp384r1")
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE
```
I would like to switch from urllib to requests for my project but am unsure how to set this with requests. Is it possible, and if so how do I do it?


Thanks for opening this. This likely won't be implemented as a direct option as it's so low-level and already covered by `SSLContext`.
Workaround for ipv4 address validation

This fixes the is_ipv4_address function. Initially intended to be a workaround for Issue #5131 it is now a removal of an external dependency. Since the check is a simple one, there is no need to rely on external sources.

Just for a reference, the `socket.inet_aton` issue is being discussed on Python's bug tracker: https://bugs.python.org/issue37495

Now that we've dropped support for Python 2.7-3.6, I think we're more likely to rely on `ipaddress` than try to maintain this ourselves. I'm going to close this out in favor of that approach.
Need to send xlsx file and json data in a single POST requests

Please refer to our [Stack Overflow tag](https://stackoverflow.com/questions/tagged/python-requests) for guidance.
Trying following code. 
Error :-  ValueError: Data must not be a string.


data = open('migration_template.xlsx', 'rb')
files = {"Migration_Template_File ": data}
headers = {
    'accept': 'application/json',
    'Content-Type': 'multipart/form-data'
}

request_json = {
    "operationName": "LTE SRAN - Add BTS to NetAct DNS (optional)",
    "operationAlias": "Add the MRBTS FQDN and the BTSMED IP to the DNS in Netact",
    "operationAttributes": {
        "xlsxFileChooser": "Migration_Template_File"
    }
}

json_data = json.dumps(request_json)

response = requests.post(base_url, auth=(username, password), verify=False,
                             data= json_data,
                            files= files,
                             headers=headers
                             )
print(response.status_code)
print(response.content)


Are you sure you're running the most recent version of requests? I ran this with some dummy data and couldn't reproduce the error. If you don't want to mess with versioning, perhaps replace `data=json_data` with `data=request_json`. The data parameter is looking for a dictionary and `json.dumps` returns a JSON formatted string (https://docs.python.org/3/library/json.html)

Resolving this since it's been silent for almost a year and doesn't have the required info for reporting a defect. At first glance, this appears to be an outdated copy of Requests. Please use StackOverflow for general usages questions about the current version of Requests. Thanks!
[Question] Can we add cookies to the prepared request cookie jar when extending AuthBase?

Hi just a quick question, I have implemented a custom auth for our api which uses custom headers (user/password) by extending AuthBase, however I have another scenario where I want to auth (without those headers, but instead using cookies that I have derived from the browser upon logging in).

Can i add the cookies in the AuthBase __call__()?

a trivial example:

```python
class CookieAuth(AuthBase):
    """
    Custom auth using cookies from a selenium driver browser
    """
    def __init__(self, csrf_token, cookies):
        self.csrf_token = csrf_token
        self.cookies = cookies

    def __call__(self, r):
        r.headers['X-csrf-token'] = self.csrf_token
        # can i addcookies here to the prepared request?

    def __repr__(self):
        return "Authenticated using cookie auth"
```

_cookies is private and an instance of CookieJar? is it ok to add my cookies this way? or can it even be done by just adding them in?

n.b I don't want to just pass them into request using cookies=cookies because the wrapper to my api is like so:

myapi.v1.endpoint(CookieAuth(csrf_token='foo', cookies='bar').get_something()
proxies argument option: handles bytes literals in dict differently from other argument options

```python
https_proxy = b'http://localhost:8080'
http_proxy = b'http://localhost:8080'
proxy_dict = {b'https' : https_proxy, b'http' : http_proxy}

r = requests.request(url, proxies=proxy_dict, verify=False)
```
## Expected Result

The *proxies argument option* accepts bytes literals as arguments, and proxies are being used.

## Actual Result

The *proxies argument option* accepts bytes literals as arguments, but seems to disregard them internally: Proxies have no effect. However, no exception is raised either.

## Problem

The *proxies argument option* handles bytes literals differently compared to the other argument options, e.g.. *parameters* or *data*. The *proxies argument option* only works with ```str```:
```python
https_proxy = 'http://localhost:8080'
http_proxy = 'http://localhost:8080'
proxy_dict = {'https' : https_proxy, 'http' : http_proxy}
```
Exceptions are ignored. Using bytes literals does not raise an exception.




Going to check this out

I think we need to establish why byte literals should be accepted. I don't believe you can pass a bytes literal as a URL to requests.
update urllib3 raise proxy error

Summary.

Set requests on proxies, from urllib3(version 1.25.3) raise urllib3.exceptions.LocationParseError: Failed to parse: http://user:pass@xxx.xxx error

What you expected.

From urllib3 1.24.1 proxy set is work fine. urllib3 version 1.25.3 update url.py raise this error.

What happened instead.


```python
import requests

proxies = {
    'http': 'http://user:pass@127.0.0.1:1080',
    'https': 'http://user:pass@127.0.0.1:1080'
}

requests.get("http://www.bing.com", proxies=proxies)
```

## System Information


```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.7"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.2"
  },
  "platform": {
    "release": "7",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010103f",
    "version": "19.0.0"
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "101000af"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": true
}
```

Sorry for my bad English. Now I restore urllib3 to 1.24.1.

I'm not receiving your error:

```python
from urllib3.util import parse_url

parse_url("http://user:pass@127.0.0.1:1080")
Url(scheme="http", auth="user:pass", host="127.0.0.1", port=1080, path=None, query=None, fragment=None)
```

Is your `user:pass` example different from what you're actually using? For example do you have an `@` character in your username or password? Please post an example that raises the error you're seeing so I can help resolve it.

yeah,In my username contain @ character. Thanks for your reply.



发送自我的三星盖乐世智能手机。


-------- 原始信息 --------
发件人： Seth Michael Larson <notifications@github.com>
日期: 2019/7/3 23:40 (GMT+08:00)
收件人： kennethreitz/requests <requests@noreply.github.com>
抄送： wowopo <liukaijiang@outlook.com>, Author <author@noreply.github.com>
主题： Re: [kennethreitz/requests] update urllib3 raise proxy error (#5129)


I'm not receiving your error:

from urllib3.util import parse_url

parse_url("http://user:pass@127.0.0.1:1080")
Url(scheme="http", auth="user:pass", host="127.0.0.1", port=1080, path=None, query=None, fragment=None)

Is your user:pass example different from what you're actually using? For example do you have an @ character in your username or password?

―
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://github.com/kennethreitz/requests/issues/5129?email_source=notifications&email_token=AD3FW52WSQAPQKMGS5RX6EDP5TB5ZA5CNFSM4H5EOIPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZE3FRQ#issuecomment-508146374>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AD3FW53OVVDJ7X4G4XP6YA3P5TB5ZANCNFSM4H5EOIPA>.
Use urllib for chunked requests

Resubmit of [https://github.com/kennethreitz/requests/pull/4958](https://github.com/kennethreitz/requests/pull/4958 ) with up to date merge. Also addresses https://github.com/kennethreitz/requests/pull/4179. 

Specifically, in my use case, chunked requests when a proxy is in use results in failure to evaluate the proxy configuration and the request being sent direct. It appears that urllib3 supports chunking as an option so this fork to lower level urllib3 code seems to no longer be required.

Addresses [https://github.com/kennethreitz/requests/issues/3844](https://github.com/kennethreitz/requests/issues/3844)

fantastic. This was 100% my least favorite part of the codebase, and what I ended up doing in Requests III as well!

Hi @nateprewitt ,

I understand that this PR was reverted for 2.23.0 release. You mentioned on the revert that there might be outstanding work. Do you see anything remaining for this particular PR?

If this was referring to the open issues above (#3844 , #4402, #4894), I believe they are all closed by this PR. I personally run a test without/with the fix on my own test case and the fix worked fine (as commented on #3844).

Please let me know if I can help

Thanks!

What exactly is the outstanding issue with this PR?

By removing this from the master branch our dependent softawre build has incorporated a release missing this functionality. There was no notification to the PR author this happened... Did the reversion have "adequate review"?

I'll have to revert to fixing to an outdated commit or track a new fork with this functional fix re-applied. 

Hi !

Same problem here :( I've been debugging for hours a proxy issue on "chunked" requests where the proxy does not work and asks for authentication (407 http error code).

I had to fork this project to put back this feature because with urllib3, it works perfectly.

Can we plan to put this back ? @nateprewitt

Otherwise, we need to fix the old code because it's not working with authenticated proxies ... :(

Thanks

I've re-resubmitted this again (again) as [This PR](https://github.com/psf/requests/pull/5664
)
Can't connect to IPv6 addresses with zone ID

Requests fails on IPv6 URLs that contain a zone identifier when used with the current urllib3 (1.25).

## Expected Result

Prior to urllib3 v1.25, I could do this:
```python
import requests
requests.get("http://[fe80::145e:58ff:fec3:4698%enp0s20u1]:8080/")
```
And it worked as intended.

## Actual Result
Since I upgraded to urllib 1.25 and requests 2.22, this fails with `[Errno -2] Name or service not known`, because requests is escaping the % character and trying to access `[fe80::145e:58ff:fec3:4698`**%25**`enp0s20u1]` instead.

## Reproduction Steps
Install requests 2.22 or 2.21 and urllib 1.25 and try the two lines from "Expected Result", replacing the IP, zone ID and port. It might be a little tricky to reproduce if you do not have a device with a link-local address at hand.

## System Information
```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "5.1.14-arch1-1-ARCH",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010103f"
  },
  "urllib3": {
    "version": "1.25.3"
  },
  "using_pyopenssl": false
}
```


This is actually an issue with urllib3. I'm going to reopen this issue and update this one. :)

I've opened urllib3/urllib3#1641. This issue can be closed if you'd like.
upload file with tavern using content-type multipart/form-data

I am trying to use tavern for uploading file. Heres what my postman request looks like

[![enter image description here][1]][1]


using tavern heres what I have tried

    name: add data
    request:
      url: "server/runs/cube/64409f3e-d78c-4788-909e-e50518b48d8a/well/c4a1e3e9-c7f7-461a-a5aa-d10d09153bfd/data"
      method: POST
      data:
        type: TOPS
        file: data/test.csv
    response:
      status_code: 200



But I get a 500 internal server error. The request through postman works just fine. What am I missing?


  [1]: https://i.stack.imgur.com/MgMbk.jpg
  [2]: https://i.stack.imgur.com/VOwxn.jpg

I found tavern does not support in multipart/form-data format.
https://github.com/taverntesting/tavern/issues/233#issuecomment-453829482

Hi i4ali999@, this is an issue with Tavern and not requests. I’m going to resolve since there’s no additional steps to take here.
Add non-3XX redirects

Looking for early feedback.

`requests` should have a native solution to dealing with non-3XX redirects i.e. https://stackoverflow.com/questions/56366175/how-to-handle-redirects-that-do-not-use-3xx-status-codes. But should such a new feature be tied to a preexisting flag `allow_redirects`, or should there be a new flag such as `allow_meta_redirects`? I understand the use case between not allowing redirects and allowing redirects, but I am not sure if there is a use case where you want to handle 2XX redirects but do not want to handle 3XX redirects or vice versa.

Hi @alanyee, thanks for reaching out about the issue you’re seeing. To clarify from the stackoverflow question, therre isn’t anything in the http spec regarding “non-3XX” redirects. The functionality you’re experiencing is a JavaScript redirect.

Requests is an http client, not a browser, so it’s outside of our scope to support scripting. You may look at something like Selenium if you need full browser support.
how to set SIO_KEEPALIVE_VALS or call socket.ioctl in general?

related to the https://github.com/conan-io/conan/issues/5398
I need to set [SIO_KEEPALIVE_VALS](https://docs.microsoft.com/en-us/previous-versions/windows/desktop/legacy/dd877220(v%3Dvs.85)) on socket in order to use python.requests in Azure Cloud where Azure Load Balancer closes connections after 4 minutes of inactivity (https://github.com/uglide/azure-content/blob/master/includes/guidance-tcp-session-timeout-include.md). 

I have found some code to set socket options, however, I can't find relevant code to call `socket.ioctl` for underlying socket which request's session uses. could you help me with that?

Please refer to our [Stack Overflow tag](https://stackoverflow.com/questions/tagged/python-requests) for guidance.


related: https://github.com/requests/toolbelt/issues/98

You'll likely have to use a lower-level library like urllib3 and implement your own custom HTTPAdapter. Closing as nothing is actionable here by the Requests team.
Update sessions.py

Use `.format()` over `%` given PEP 3101

This is great for python3. @kenneth-reitz, I just want to make sure [requests](https://github.com/kennethreitz/requests) is continuing to spport python2.7. right? 

`.format()` should work for Python 2.7.

interesting fact: % is much faster (at least in my tests, years ago) than `.format`. I don't think we need to be concerned about that here though ;)