Raibek/requests-github-issues huggingface dataset:
Prefer Transfer-Encoding=chunked even if both Transfer-Encoding and Content-Length are set

RFC 7230 says (see section [3.3.3 Message Body Length](https://tools.ietf.org/html/rfc7230#page-32)):

> If a message is received with both a `Transfer-Encoding` and a `Content-Length` header field, the `Transfer-Encoding` overrides the `Content-Length`.

Since HTTP servers MUST ignore `Content-Length` if `Transfer-Encoding` is sent, **python-requests** MUST follow the same logic and send payload using `chunked` approach if both are detected in among request headers.

Normally, `Content-Length` MUST NOT be sent in the same request with `Trasnfer-Encoding`, however, it may happened both intentionally (e.g. malicious intent) and accidentally (e.g. proxying a request downstream).

Hi @ikalnytskyi, this was actually addressed in our 3.0.0 proposal branch with #3897. Instead of guessing, we would have explicitly prohibited the user from sending both. They need to make an explicit decision before the client will perform the request.

That said, it's unlikely this code will end up released anytime in the near future if ever. We're unable to make the change in a 2.x release due to it being breaking for existing users. In the event we ever release 3.0.0, we'll likely stick with the already merged PR. Thanks for the contribution though.

Hi @nateprewitt,

Thanks for stopping by and checking this.

> We're unable to make the change in a 2.x release due to it being breaking for existing users.

Can you please elaborate how it will break existing users? Whoever hit this bug is already broken. Receiving part does not understand what `requests` sends. That's how I found this in the first place: neither nginx nor uwsgi nor gunicorn are able to understand what was sent since once they see `Transfer-Encoding: chunked` they expect chunked data.

I still think something should be done. If you don't like respecting `Transfer-Encoding: chunked` supplied by user, can we consider resetting its value before sending out a request? 

> In the event we ever release 3.0.0, we'll likely stick with the already merged PR. 

I agree that explicit approach is better then *guessing*. It's just we already have *guessing* in 2.x, and the *guess* condition is wrong.
Open a stream from POST request

Hello,
I am trying to open a stream from a POST request. The idea is to open the stream and to attach the response to an SSE client.
is this the correct way to do it?
`response= requests.post(url_request , stream= True)`
If it is true, why it seems like the client is closing the socket at a certain point?



Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.
Allow further configuration of HTTPAdapter's PoolManager by passing pool_kwargs dict

There are various useful configurations of underlying urllib3 PoolManager not supported by the existing  HTTPAdapter, such as a global timeout, source_address, and many more https://github.com/urllib3/urllib3/blob/29b214a129883301f91ae4a74fd7ef2958cbf7b0/src/urllib3/poolmanager.py#L49-L75

It is trivial to support them directly. The construction of the HTTPAdapter's poolmanager at `HTTPAdapter.init_poolmanager(..., **pool_kwargs)` already contains this provision as the arbitrary `pool_kwargs` dict that is passed to the PoolManager constructor. All it takes is to also add this to the HTTPAdapter constructor, and pass it.

It also appears that this was the original intention that was somehow forgotten; this `**pool_kwargs` argument is not being passed by any caller of `init_poolmanager()`.

This project isn't accepting new features. You can easily accomplish this by calling `init_poolmanager()` after creating your `HTTPAdapter` instance. It will replace the previous poolmanger with your newly configured poolmanager. Is it extra work? Yes. Does it get you where you need this pull request to merge to achieve your deepest darkest pool configuration dreams? No.

I'm inclined to agree with Sigmavirus24 on this one. This is increasing surface area on a more or less frozen API for the adapters and is pretty trivial to configure with `init_poolmanager`. I think we'll pass on this, but thanks for the proposal!
Allow for autodection of encoding for scalar values

Since JSON RFC 7158 (and also in 7159 and 8259) scalar values are also valid JSON texts. The json decoder from the Python standard libs also accepts such scalar values.

The JSON Unicode detection algorithm in "requests" assumes that the first two characters are always ASCII. That is not true anymore.
Therefore the current detection fails in at least these two edge cases

- A single digit in UTF-16 encoding
- A quoted string with the first character having a Unicode value higher than 255 in UTF-16 encoding

Also autodetection by chardet returns the wrong encoding.

This change detects the encoding in case of those two edge cases.
Newly created Response object fails on boolean test

A newly created `Response` object fails boolean test in if statement (when under the hood `__bool__` method is called). Based on conventions used in Python, I expect an empty response object to be resolved to `False`. Although it is a design decision to which value to resolve, throwing an exception in this trivial case is unexpected behavior.

## Expected Result

```
2.23.0
Response is empty
```


## Actual Result

```
2.23.0
Traceback (most recent call last):
  File ".../local/reqeusts_error.py", line 6, in <module>
    if not response:
  File "/opt/miniconda/lib/python3.6/site-packages/requests/models.py", line 677, in __bool__
    return self.ok
  File "/opt/miniconda/lib/python3.6/site-packages/requests/models.py", line 703, in ok
    self.raise_for_status()
  File "/opt/miniconda/lib/python3.6/site-packages/requests/models.py", line 934, in raise_for_status
    if 400 <= self.status_code < 500:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'

Process finished with exit code 1
```

## Reproduction Steps

```python
import requests

print(requests.__version__)

response = requests.Response()
if not response:
    print('Response is empty')

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.3.1"
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.0"
  },
  "platform": {
    "release": "3.10.0-957.21.3.el7.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1000210f",
    "version": "18.0.0"
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1000215f"
  },
  "urllib3": {
    "version": "1.25.8"
  },
  "using_pyopenssl": true
}
```



No one uses the library this way in reality, so there's no harm in this behaviour

I agree with @sigmavirus24. Status code, headers, and some representation of a body (empty or otherwise) are all required parameters to successfully generate a Response in Requests. I don't think it's worth the time or verbosity to harden the code base against every possible mock when the state isn't reachable in production.
"407 Proxy Authentication Required" although it was provided in the request

I tried to run the following code

import requests

```
s = requests.Session()
s.trust_env = False

url="https://api.elsevier.com/content/abstract/scopus_id/85082366258"
headers={'X-ELS-APIKey': 'MY_API_KEY', 
'Accept': 'application/json', 
'User-Agent': 'pybliometrics-v2.5.0'}
proxies={'https': 'https://USERNAME:PASSWORD@proxy.thing.com:8080/'}
params={'view': 'META_ABS'}

resp = s.get(url, headers=headers, proxies=proxies, params=params)

print(resp.text)
```

But I get the following error message:

`requests.exceptions.ProxyError: HTTPSConnectionPool(host='api.elsevier.com', port=443): Max retries exceeded with url: /content/abstract/scopus_id/85082366258?view=META_ABS (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required')))`

My username and password are correct and don't contain any special characters, only alphanumeric ones.

What is going wrong?


Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

Hello

@sigmavirus24 There was a button which said "Ask for help" or something like that when I reported the issue

If you had read the template, you'd see the it points you to stack overflow as well. https://www.github.com/psf/requests/tree/master/.github%2FISSUE_TEMPLATE%2FCustom.md
One server fails ping -Max retries exceeded with url: /wsman (Caused by SSLError(SSLError(\"bad handshake: SysCallError(104, 'ECONNRESET')

New to Ansible, having trouble with one server failing ping: 

"msg": "kerberos: HTTPSConnectionPool(host='xxxxxl', port=5986): Max retries exceeded with url: /wsman (Caused by SSLError(SSLError(\"bad handshake: SysCallError(104, 'ECONNRESET')\",),)), ssl: HTTPSConnectionPool(host='xxxx', port=5986): Max retries exceeded with url: /wsman (Caused by SSLError(SSLError(\"bad handshake: SysCallError(104, 'ECONNRESET')\",),))",
    "unreachable": true,
    "changed": false,
    "skip_reason": "Host xxxx is unreachable"
}

I replaced host name with xxxx

We have several hundred servers, this is the only one giving this error.

I tried running the ConfigureRemotingForAnsible.ps1
Verified network settings
I can telnet to the host on port 5986 from another server
Verified IIScrypto has not disabled SSL's or TLS.
ran command - winrm e winrm/config/listener
which shows listener running on 5986
I am unable to open the URL of the server on port 5986


There is no bug here. The server or an intermediary is resetting the connection during the TLS handshake.

>  I am unable to open the URL of the server on port 5986

This is unclear if you mean you're generally unable to open it (i.e., in a browser, curl, or requests) or if you're only unable to open it with Requests. Either way, the TLS configuration (besides on/off) could be a problem and you should know what version of requests/urllib3 you're using. 

We do ping sweeps on our servers. One server in question is part of a group that fails the ping test, the code above is the output.

> We do ping sweeps on our servers. One server in question is part of a group that fails the ping test, the code above is the output.

That doesn't answer anything and doesn't negate that there is no bug here. The server is resetting the connection during the TLS handshake

The server's clock it set to UTC, which shows four hours in the future than Eastern Time Zone. The above issue does not occur when Eastern is set. So this appears to be either a bug, or Anisble does not like the time difference.

> The server's clock it set to UTC, which shows four hours in the future than Eastern Time Zone. The above issue does not occur when Eastern is set. So this appears to be either a bug, or Anisble does not like the time difference.

This really isn't appropriate for this project. You've given no evidence as to the fact that there's a bug in Requests. And now you're involving at least 1400 people in your stream of consciousness debugging. It's really not what those people are likely watching the project for. Please debug this with StackOverflow or the Ansible community. This isn't a Requests issue
Replace deprecated `setup.py test` with tox entry point

The seuptools test command has been formally deprecated since version
41.5.0 (27 Oct 2019). The project recommends replacing its use with tox
as a project level testing entrypoint.

Fixes the warning when running tests:

> WARNING: Testing via this command is deprecated and will be removed in
> a future version. Users looking for a generic test entry point
> independent of test runner are encouraged to use tox.

For more details, see the setuptools history:
https://setuptools.readthedocs.io/en/latest/history.html#v41-5-0

And the command's documentation:
https://setuptools.readthedocs.io/en/latest/setuptools.html#test-build-package-and-run-a-unittest-suite
module 'requests' has no attribute 'settings'

I want to print verbose logs from requests to see the headers and cookies in each request.
I followed [this page](https://requests.readthedocs.io/en/v0.6.6/user/advanced/), which says to run:

```
requests.settings.verbose = sys.stderr
````

I suspect this may be a case of outdated documentation, but I can't find anything more up to date.

## Expected Result

Verbose logging should be enabled.

## Actual Result

My IDE gives a warning about `settings not being a reference in __init__.pyi`. Running the program throws the exception `AttributeError: module 'requests' has no attribute 'settings'`

## Reproduction Steps

```python
import requests
requests.settings.verbose = sys.stderr
```


## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.9.2"
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.3"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "1010107f",
    "version": "19.1.0"
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1010107f"
  },
  "urllib3": {
    "version": "1.25.8"
  },
  "using_pyopenssl": true
}

```

The page you followed is for requests version 0.6 and you're using 2.23. Please read the correct documentation

Oh my bad - I just googled requests verbose logging. I can't seem to find the same documentation for the new version. Do you know where I can find this?

I just want to see the full HTTP POST/GET requests sent with the URL, headers and cookies used.

It doesn't exist. It was ripped out in v1. There's something to dump the request/response cycle in toolbelt.readthedocs.io though

For the other poor sods who run across this thread the same way I did. This is what I wanted to have found when I ran across this:

You want to use [requests_toolbelt](https://toolbelt.readthedocs.io/en/latest/)'s [dump](https://toolbelt.readthedocs.io/en/latest/dumputils.html) module.

Here's a quick example showing how to dump the requests and responses.

```
import requests
from requests_toolbelt.utils import dump

resp = requests.get('https://httpbin.org/redirect/5')
data = dump.dump_all(resp)
print(data.decode('utf-8'))
```

100,000% thank you for posting this.
Implicit JSON Content-Type is missing charset

This code in `requests/models.py`:

```python
            content_type = 'application/json'
            body = complexjson.dumps(json)
            if not isinstance(body, bytes):
                body = body.encode('utf-8')
# ...
            # Add content-type if it wasn't explicitly provided.
            if content_type and ('content-type' not in self.headers):
                self.headers['Content-Type'] = content_type
```

has an incomplete `Content-Type`.

## Expected Result

That `Content-Type` be `application/json; charset=utf-8`

## Actual Result

`Content-Type` is only `application/json`

## Reproduction Steps

Make any request passing `json={}` and without explicitly setting any headers.

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.3"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": false
}
```

To the best of my knowledge the charset is optional and not required when handling utf-8 thus this is perfectly fine.

Of course it's optional; the concern is that this leaves ambiguity that the far end needs to resolve. Requests even says that

```
            # No encoding set. JSON RFC 4627 section 3 states we should expect
            # UTF-8, -16 or -32. Detect which one to use
```

So if the far end doesn't know which encoding to use, it's forced to guess. UTF-8 is a safe guess, but a guess nonetheless.


Right, but the JSON spec does not define `charset` and thus specifying this ever is a violation of the RFC. In other words, the specification says "It should always be Unicode" and there are ways to distinguish where necessary, so this project should emphatically not specify a charset.

Further ammunition: https://stackoverflow.com/questions/4990095/json-specification-and-usage-of-bom-charset-encoding

Ah, that's what I was missing. Thank you.
Drop monkeypatching with pyopenssl

Monkey patching url3 with pyopenssl 3 is no longer a good idea nor even necessary, and actually has security drawbacks, as described in #5267.

This also undoes the changes I initially made in #1347, so I guess it means my previous contribution has completed its life cycle.

Hi @WhyNotHugo,

This was already addressed in #5443 and left patching as a potential fallback for now for older versions of Python 2.7. This patch has some additional removals from our debugging helper function that I don't think we want to remove. I think we'll pass on this for now.

Oh, I was under the impression that Python2 was being dropped anyway.
Problem with python3.6

When I try to send a request via python3.6 to this url: "https://www.khaneyeshoma.ir/", it waits until Timeout exception is raised. But when I try the same request via python2.7 it is successfully completed with status code: 200. Can you help me?
Thanks!

No. This is not a support forum. Go to stackoverflow.com. Provide the exact versions of python, your system dependencies, requests, etc. and the exact results you get.
about session?

If I generate a seesion for different domain names in the same program,
 for example :
    url1:
       session1 = requests.session ()
    url2:
       session2 = requests.session ()

What's the difference with me only generating a session for url1 and url2?
Will there be any difference in performance?
thanks!

### From the source code of request.sessions:
>This module provides a Session object to manage and persist settings across requests (cookies, auth, proxies).

An example of this is any cookies a request returns will be there whenever you call a request from that session you call the request from (if you made a session), however, if you didn't make a session (or used a different session, in your case) that different session would not have those cookies that were set.
(Also, correct me if I'm wrong, but "not making/using a session" just makes a session, sends the request, then forgets about the session.)

*I'm new to contributing to others' projects on GitHub, so PLEASE give me feedback or correct me if you can, thanks all :D*

Also, you should probably have asked this on a website like StackOverflow, as this isn't an issue with the library, rather a question on how it works.

Thanks for chiming in @ItzMeWilliam 😊

It looks like the initial question is covered so I'm going to resolve, and you're right these should usually be asked on Stackoverflow.
for non-english response, may add apparent_text for convenient

most non-English response has incorrect r.encoding so it need:
```
r.encoding = r.apparent_encoding
r.text
```

what about a new attribute like `r.apparent_text` or `r.text(chardect=True)`for the result?

Thanks!

https://github.com/psf/requests/issues/5084
https://github.com/psf/requests/issues/2086

New features are not being added

I have read that and I don't think this can count as a feature.

Adding new API surface area is a new feature. Further, if `r.encoding` is incorrect but `r.apparent_encoding` is correct that means the server is reporting the wrong thing and that's what the library is trying to use. In other words, there's a perfectly good work around here.

Hi @NewUserHa,

sigmavirus24 is correct that we aren't adding any new features currently which this would fall under. If chardet is incorrectly detecting an encoding, I would suggest reaching out to the [library](https://github.com/chardet/chardet) to see if you can help identify the issue. Otherwise, setting the encoding when the server returns an incorrect response is the best option to get around this.
Does `Response.content` decode the unicode character in JSON implictly?

Python 3.8.2, requests 2.23.0

Code for reproduce:

```python
headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0'
}
resp = requests.get('https://api.bgm.tv/subject/12', headers=headers)
print(resp.content)
```

Expect `resp.content` to be bytes containing the original payload bytes (https://requests.readthedocs.io/en/master/user/quickstart/#binary-response-content), but unicode character in the JSON is decoded as unicode representation with a prefix `\u`.


No, resp.content is bytes and resp.text is the decoded unicode text.

> No, resp.content is bytes and resp.text is the decoded unicode text.

Thanks for the reply. I decompressed the raw response body and figured out the `\uxxxx` transformation was made on the server side, so the issue has nothing to do with Requests.
Stale requests2 package on pypi

I'm not totally sure where the right place to report this is, but there is a confusing stale copy of requests on pypi, called requests2: https://pypi.org/project/requests2/

It is:
*  3 years old
* uploaded by the original maintainer
* showing a README with some healthy-looking badges
* listed as Development Status: 5 - Production/Stable in pypi
* getting used accidentally in new projects: https://libraries.io/pypi/requests2/usage
* easy to find when searching for requests:
![Selection_142](https://user-images.githubusercontent.com/412019/82508158-12027680-9ab9-11ea-9c41-45f7ee47d0d3.png)


I believe it should be [yanked](https://www.python.org/dev/peps/pep-0592/) and/or [replaced with a redirect](https://www.dampfkraft.com/code/how-to-deprecate-a-pypi-package.html#provide-a-redirect).


The pepole who own that project don't work on this project. You'll need to report this to PyPI

I'm not sure what you mean. I am not very familiar with the history of ownership/maintenance of requests, but it seems that over time it has been transferred from kennethreitz to the Python Software Foundation. If none of the current maintainers have access to the requests2 pypi package, I believe the correct method is to follow https://www.python.org/dev/peps/pep-0541/#how-to-request-a-name-transfer

@ccope Kenneth doesn't work on this project and isn't affiliated.

That may be the case, but the `requests2` package on pypi is an **exact**\* copy of this project at that version, so I don't believe it should be considered a separate project. I agree that there is a process to follow with pypi, but I believe it needs to be initiated by the current maintainers of this package, as they are the ones who should receive control over it.

\*: With the exception of a couple of metadata files which specify `requests2` instead of `requests`.

@ccope people are allowed to fork projects and re-upload them to PyPI. Entire companies do this by prefixing the names of dependencies they want exact-ish copies with their company name. You can't expect those maintainers to update those forks as well, somehow, can you? No. You can't expect these maintainers to attempt to maintain two packages, 1 of which isn't owned by anyone on the maintenance team.

Yeah, there's nothing we can really do about this. We don't own that package and names through at least Request5 are currently being squatted. The plan for the foreseeable to future is to keep the project (regardless of version) under just `requests`. Anything else should be treated as unrelated at this time.
`.json()` degrades `stream=True`

The current logic in `.json()` forces a load of `self.content`, removing any advantage from `stream=True`. For large JSON payloads this creates memory problems. The memory footprint could be halved(ish) by:

- Using `Content-Length` instead of `len(self.content)`
- Using `json.load(raw)` rather than `loads`, also avoiding `content.decode()`

## Expected Result

That the memory footprint of `.json()` be O(1) against input length, not counting the output object.

## Actual Result

The memory footprint is O(n) in input length due to reliance on `.content`.

## Reproduction Steps

Call `requests.get('something-large.json', stream=True).json()`

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.2"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "101000af"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": false
}
```


>     * Using `Content-Length` instead of `len(self.content)`

`Content-Length` only works if the content hasn't been encoded and had to be decoded, e.g., gzip compressed. It's not an accurate representation of what the JSON content length might be

> Using `json.load(raw)` rather than `loads`, also avoiding `content.decode()`

That might be an enhancement, if the project were accepting them.

This doesn't really matter. [`json.loads()` is defined](https://github.com/python/cpython/blob/b3c77ecbbe0ad3e3cc6dbd885792203e9e6ec858/Lib/json/__init__.py#L274-L296) as (paraphrased here)

```python
def load(fp, **kw):
    return loads(fp.read(), **kw)
```

so attempting to be fancy in `requests` wouldn't help at all. You would also need a streaming-supporting JSON parser.
Allow semicolon in the href of Link header

Semicolons are valid characters in URIs, splitting them while parsing the `Link` header results in unexpected token and wrong `url` attribute (notice `;oldid=934259284` portion the below example).

```
<https://en.wikipedia.org/w/index.php?title=COVID-19_pandemic&amp;oldid=934259284>; rel="original", <https://web.archive.org/web/timemap/link/https://en.wikipedia.org/w/index.php?title=COVID-19_pandemic&amp;oldid=934259284>; rel="timemap"; type="application/link-format"
```

This PR attempts to fix current parsing issue and also renames a variable to better reflect the purpose of it.

I would note here that this parser still is far from perfect and can be better implemented using state machine.

@nateprewitt is there anything we can to do to get it merged?
response header has encoding error

Summary. some http response header without English (using utf-8) has encoding error.
original text (2. 핵심정보를 담은 발표 형성평가 5월 19일) I uploaded this text on server.

and I responded to server with get function. (`requests.get()`)
but I got this text (2. íµì¬ì ë³´ë¥¼ ë´ì ë°í íì±íê° 5ì 19ì¼) from http response header.

I converted the text has encoding error to ascii to utf8 converter. It converts successfully.
maybe requests package gets http response header with ascii encoding.

**EDIT** I tried this code `req.encoding = 'utf-8'` but this code didn't work.
code:
```
headers = {
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7',
            'Content-type': 'text/plain; charset=utf-8'
        }

        req = requests.get(link, headers=headers, allow_redirects=True)
        req.encoding = 'utf-8'
print(req.headers['Content-Disposition']) # this code prints the text has encoding error
```

## Expected Result
핵심정보를 담은 발표 형성평가 5월 19일

## Actual Result

íµì¬ì ë³´ë¥¼ ë´ì ë°í íì±íê° 5ì 19ì¼

## Reproduction Steps

```python
import requests

```

## System Information

    $ python -m requests.help
I am using python 3.7.3, requests latest version (v2.23.0)
```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "101000af"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": false
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

thank you.

I fixed with code `req.headers['Content-Disposition'].encode("ISO-8859-1").decode("utf-8")`.
Python Requests normalizes the URL

When requesting a URL (get/post), Requests normalizes the URL.
For example:
```
r = requests.get("http://123.123.123.123/../../../../../../test")
print(str(r.status_code))
```

## Expected Result

```
GET /../../../../../../test HTTP/1.1
Host: 123.123.123.123
Connection: close
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.22.0
```

## Actual Result

```
GET /test HTTP/1.1
Host: 123.123.123.123
Connection: close
Accept-Encoding: gzip, deflate
Accept: */*
User-Agent: python-requests/2.22.0
```

## Reproduction Steps

```python
import requests
r = requests.get("http://123.123.123.123/../../../../../../test")
print(str(r.status_code))
```

## System Information

```
{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": "2.8"
  }, 
  "idna": {
    "version": "2.8"
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.17"
  }, 
  "platform": {
    "release": "5.3.0-51-generic", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "1010104f", 
    "version": "17.5.0"
  }, 
  "requests": {
    "version": "2.22.0"
  }, 
  "system_ssl": {
    "version": "1010100f"
  }, 
  "urllib3": {
    "version": "1.25.6"
  }, 
  "using_pyopenssl": true
}

```




Duplicate of #5289

Thanks.
Make content-type be multi-part without being given files

Currently, to post a multipart/form-data with boundaries, you either need to give a File or make the data File-like. This feature aims to allow uploading of data without uploading files.

I am working on a fix with @Marti2203 

When you want to send a form without a file but the form has to be of type 'multipart/form-data' we need to write down the explicit boundary in the content-type header, but this is generated by urllib3 and we cannot predict it or pregenerate it as it is supposed to be random, so we have to "spoof" the API by sending the data as a file.

There are other libraries that provide you a way to do this without changing the core functionality that millions of people rely on.

Here is example code that uses this "functionality".
```
import requests
url = "foo.bar"
payload = {'thing: (None,"""{'blah':  0 }""")}
response = requests.post(url, files = payload)
```
When inspecting [the logic for making a PreparedRequest](https://github.com/psf/requests/blob/9ed5db8ed28e816b597dafd328b342ec95466afa/requests/models.py#L502), it can be seen that if the user gives a content type, it will be placed, but to reach a proper multipart/form-data we need to know the boundary beforehand as the previous comment mentioned. We will make a pull request with a simple fix that changes the logic such that this issue is overcomed.

@sigmavirus24 We saw http.client and requests and requests is more simple. I just saw your comment

@Marti2203 there are already existing libraries you can use and should be using instead of trying to make this work in Requests core functionality. The project is basically in security-fix-only mode at this point

https://toolbelt.readthedocs.io/en/latest/uploading-data.html#streaming-multipart-data-encoder will allow you to do this (since you haven't looked very far on Google it seems)

Okay, that I had not known about the "security-fix-only mode"

Yeah, we have not looked too much into it because we needed something simple and postman recommended requests and http.client

"Requests has support for multipart uploads, but the API means that using that functionality to build exactly the Multipart upload you want can be difficult or impossible."(that speaks a lot about some parts of the elegance of the library)

@sigmavirus24 Apologies for not taking the time to Google very far. We weren't aware of [requests_toolbelt](https://toolbelt.readthedocs.io/en/latest/index.html) and assumed that all functionalities involved with making requests would be available in one package. Apparently, this wasn't the case and now we see why.

@sigmavirus24 On a separate note, it would be nice to have a reference from the **official repository** to [requests_toolbelt](https://pypi.org/project/requests-toolbelt/)

There was one once upon a time. I'm not a maintainer here though
Revert PR 1440, do not modify cookie value

Revert PR  #1440 as discussed in issue #3759 
It does not make sense to modify the cookie value arbitrarily.

At this point the behaviour is too ingrained to be reverted arbitrarily and would need to wait for a major version release which is not planned to the best of my knowledge

This functionality has been in the library for 7 years now, so we can't pull it out without breaking users. If we were to ever put another major version up, this could possibly be looked at, but I don't think we can merge this in 2.x. Thanks for the suggestion!
Possible bug or incorrect usage (on my end)

Hi so I'm using your requests module and everything was working fine when I was using it with python2. However, now that I'm on Mac OS Catalina and there's only python3 I've had to revise my code a little and now I'm getting some errors. So I have a little bit of code as follows:
response = requests.get(resource,
                                    headers={
                                        "Authorization": "Basic " + base64.b64encode(
                                            self.username + ":" + self.passwd
                                        ),
                                        "Accept": datatype
                                    },
                                    proxies={"https": None,
                                             "http": None})

And when I run my script it gives me an error:
File "/Users/dwalker/Documents/workspace/jamfapi/jamfapi_src/jamfapi/jss/JSSRestIface.py", line 48, in jamfApiCall
    self.username + ":" + self.passwd
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'

Now I had issues similar to this in another project when we converted from py2 to py3 and we resolved it by running .decode("utf-8") with the output object we were getting back from a command being run. However, I'm not sure if this is the same issue. Any ideas on what I'm doing wrong here or how I'm using requests incorrectly?

Thank you.


In Python3, the default encoding for strings changed. I believe if you want a Python2 style string, you need to prefix it with b as in `b"Byte text"`. Your issue is probably related to the distinction between string objects and bytes objects. Using decode, you can convert from bytes to string, and encode will convert from string to bytes. The docs probably have better information, with [strings documented here](https://docs.python.org/3/library/stdtypes.html#text-sequence-type-str) and [bytes documented here](https://docs.python.org/3/library/stdtypes.html#binary-sequence-types-bytes-bytearray-memoryview).

This is your incorrect usage. Also you don't need t oencode that yourself since `auth=(self.username, self.passwd)` would do the trick. You could also use `requests.auth.BasicAuth`.

sigmavirus24 so where would I use the auth=(self.username, self.passwd)? I'm a little confused.


First page of the docs demonstrates this https://requests.readthedocs.io/en/master/

Resolving since this seems to be addressed. 

Direct link to [auth docs](https://requests.readthedocs.io/en/master/user/authentication/#basic-authentication) for future readers.
Update Travis syntax

### This pull request  brings the following changes:

- Ordered the commands to follow Travis CI execution order: <https://docs.travis-ci.com/user/job-lifecycle/#the-job-lifecycle> 

- Adds explicit os and dist commands protecting from changes on Travis default distro

- Forces the build to use the latest minor version of each python release so build become fully reproducible

We don't want to pin to a minor version. Travis is supposed to auto-upgrade those for us

> supposed to auto-upgrade those for us

I cant revert that, what do you think about the other changes?

They're churn, functionally unnecessary, and in some cases useless (see apt caching)
urllib3.exceptions.ProxySchemeUnknown: Not supported proxy scheme None


What do your proxy settings look like? I suspect this is an error in how you're configuring them just like the issue you commented on and referenced

Yes. I have no clue what the right values would be for you though. This is not a bug in requests. Please close it. 

Then ask on StackOverflow for help learning how to use proxy servers. This isn't a tech support forum to help you, it's a defect tracker for this project and it's been identified that there is no defect in this report

Sorry, I misunderstood. I will delete this issue page.
How to set an effective useragent for Session?

I have set a useragent for the requests session:

session = requests.session()
headers = {}
headers['user-agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'
resp = session.get('http://www.gov.cn',headers=headers)
print(session.headers)

but in the result, the user-agent is 'python-requests/2.22.0'.
{'User-Agent': 'python-requests/2.22.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}

how can i set an effective useragent?


Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

@raisoon the `headers=headers` is only passed onto the get request itself... 

However if you want to set it permanently..

```python
import requests
from requests_toolbelt.utils import dump

session = requests.session()
session.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'
resp = session.get('http://www.gov.cn')

print(dump.dump_all(resp).decode('utf-8'))
```

I've included you a better debug to see the request sent as well :)

It looks like this ended up answered, going to resolve.
Modified file content after upload

Also I have been looking into documentation I didn't found the problem/explanation.

## Szenario:

- The server does use WebDav and I can use a PUT request for uploading
- In addition it must be digest authentication
- it works with curl using --digest and --user "xxxxxxx:yyyyyy"

The difference between curl and the Python script after here is
that the script seems to modify the content of the file while uploading.
If I look at http://internal-demo-url/uploads/demo.properties the
content looks like following:

## The uploaded file

```
--6cb2e8363c8d7f9191d2068b62828985
Content-Disposition: form-data; name="file"; filename="demo.properties"
Content-Type: text/plain

title=demo
context=upload
message=hello world 3!

--6cb2e8363c8d7f9191d2068b62828985--
````   

Of course I just would like to have the three lines in the middle.
**What did I wrong?**


## The Python code (upload.py)

```python
from requests import put
from requests.auth import HTTPDigestAuth

url = 'http://internal-demo-url/uploads'
with open('demo.properties', 'rb') as handle:
    headers = {}
    files = {'file':('demo.properties', handle, 'text/plain')}
    response = put(url, auth=HTTPDigestAuth('xxxxxx', 'yyyyyy'),
                        files=files, headers=headers)
    print(response.status_code)
```


Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

I'm cautious to say "it is a bug" however I see that the library is doing the stuff (while curl does not) and I don't see it documented on your side (except I missed it). From this I think the issue should be ok. Anyway I will ask on stackoverflow to make you happy ...

You've filed an issue that `files` is doing a `multipart/form-data` upload and that's not what cURL does with no description of how you're using curl or what format you expect the data to be in. `files` is documented to send data this way (that's its entire point) and thus either you've not read the docs and are using the wrong options (and need help from stackoverflow) or you think there's a bug in the encoding being used (which there decidedly is not).
utf-8 page wrongly detected as ISO-8859-1

A webpage has been detcected as being ISO-8859-1 encoded, even though it is encoded in utf-8.

## Expected Result

Correct classification as utf-8.

## Actual Result

utf-8 page detected as ISO-8859-1.


## Reproduction Steps

```
#!/usr/bin/python

import requests

# example url
url = "https://digitalezivilgesellschaft.org/"

# get the page and print the supposed encoding
response = requests.get(url)
print(response.encoding)
```
Compare that with

```
rm -f index.html; wget -nv https://digitalezivilgesellschaft.org/  2>/dev/null&& file index.html  | grep index | tail -1
```


## System Information

    $ python -m requests.help

```
explore_requests_bug$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.9"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.8.2"
  },
  "platform": {
    "release": "5.6.8-arch1-1",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1010107f"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": false
}
```


This concrete problem seems to be related to the more general issue
https://github.com/psf/requests/issues/2086

you can gei it like this:
```python
response.content.decode("utf8")
```
not
```python
response.text
```

Hi @klartext, if a webpage fails to specify its encoding, we'll default to ISO-8859-1 which has been the spec since RFC 7230. This is starting to change, but we're in an inconsistent state for now. We've kept the functionality for backwards compatibility and it still seems to be the correct case more often.

To resolve this issue you can either do as @GoddessLuBoYan suggests above, or set the `encoding` attribute on your Response object before calling `text`. This will do the decoding for you.
How to access token based API using requests in python?

I want to call one API from another API. So external API is having jwt token based authentication .
```
import requests
response = requests.get('http://host:port/api/users', auth= ("username","password"))
````
I am getting error :
```
{"error":"authentication failed: jwt parse error: token contains an invalid number of segments","code":16,"message":"authentication failed: jwt parse error: token contains an invalid number of segments","details":[]}
```
Or , First i need to call login API ,get the token and while calling another API apply that token in header.
Please suggest here.

Hi there! Thanks for opening this issue. Unfortunately, it seems this is a request for help instead of a report of a defect in the project. Please use [StackOverflow](https://stackoverflow.com) for general usage questions instead and only report defects here.

Please find the StackOverflow link for the same issue. 
https://stackoverflow.com/questions/61482014/how-to-access-token-based-api-using-requests-module-in-python

Looks like the StackOverflow issue has some answers in it, resolving this.
fix failed testcase in test_requests

when I run pytest in test_requests, an error occurred:

def test_conflicting_post_params(self, httpbin):
        url = httpbin('post')
        with open('Pipfile') as f:
>           pytest.raises(ValueError, "requests.post(url, data='[{\"some\": \"data\"}]', files={'some': f})")
E           TypeError: 'requests.post(url, data=\'[{"some": "data"}]\', files={\'some\': f})' object (type: <class 'str'>) must be callable

tests/test_requests.py:777: TypeError


This looks like a dupe of #5305. I'm going to resolve in favor of that. Thanks for the contribution though @sgy123456!