Raibek/requests-github-issues huggingface dataset:
Connection broken IncompletedRead

I'm crawling some websites. But I met a problem that makes me very depressed. 
The error:
('Connection broken: IncompleteRead(2048 bytes read)', IncompleteRead(2048 bytes read)) 

The code:    html=requests.post(url,data=json.dumps(payload),headers=headers,verify=False)

I have never met this problem before. Just one day, it came out.
I'm looking forward to your help.


It will be better if you can provide more info about this error.
What websites you are crawling?
Does it appear every time?
the Version of requests & python?

Or straightforwardly, could you provide a script that would recur this error?

> It will be better if you can provide more info about this error.
> What websites you are crawling?
> Does it appear every time?
> the Version of requests & python?
> Or straightforwardly, could you provide a script that would recur this error?

Thanks for your response!
I'm use the multiprocessing module and the proxies to crawler the websites.
Here is the code.
`html=requests.post(url,data=json.dumps(payload),headers=headers,verify=False,proxies=proxies)`
Here is the snapshot of the error. Unluckily, this error happens frequently, but not always.
<img width="770" alt="微信图片_20190412215135" src="https://user-images.githubusercontent.com/35482019/56042228-2c720880-5d6d-11e9-92a6-4ced472f5524.png">

In the processes, some works well, but some reports this error. 







The version of python : 3.7.1
The version of requests: 2.21.0

The issue you're hitting here @pyliangbo is that the server is failing to send the full payload before closing the connection. You'll need to handle this excption case if crawling arbitrary websites.
Error: Content is not allowed in prolog.. ParseError

Hi,

Basically if I run a command curl:

`curl -k -u user:pass --header 'Content-Type: application/xml' --request POST --data '<LocalIDs><LocalID ID="9128615"/></LocalIDs>' --url https://rhtpcapc001:8182/pc/center/webservice/datasources/dataSourceConsoleName/Data%20Aggregator%40RHTPCADA201/itemids`

I have the following answer:

`<?xml version="1.0" encoding="UTF-8"?><ItemIDResults><ItemIDResult LocalID="9528017" ItemID="828570"/></ItemIDResults>`

I made a script in python to run this command:

#!/usr/bin/python

import requests
import urllib3

urllib3.disable_warnings()

headers = {
    'Content-Type': 'application/xml',
}

data = {
  '<LocalIDs><LocalID ID': '"9528017"/></LocalIDs>'
}

response = requests.post('https://rhtpcapc001:8182/pc/center/webservice/datasources/dataSourceConsoleName/Data%20Aggregator%40RHTPCADA201/itemids', headers=headers, data=data, verify=False, auth=('user', 'pass'))

print (response.encoding)
print (response.headers['Content-Type'])
print (response.headers)
print (response.text)
print (response.content)
print (response.reason)

and gives me the following error:

ISO-8859-1
text/plain
{'Set-Cookie': 'JSESSIONID=md1waumsuoe81jceaxacbqgid;Path=/pc;Secure', 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'Content-Type': 'text/plain', 'Date': 'Tue, 09 Apr 2019 10:28:54 GMT', 'Content-Length': '173', 'Server': 'Jetty(9.3.5.v20151012)'}
JAXBException occurred : ParseError at [row,col]:[1,1]
Message: Content is not allowed in prolog.. ParseError at [row,col]:[1,1]
Message: Content is not allowed in prolog.. 
b'JAXBException occurred : ParseError at [row,col]:[1,1]\nMessage: Content is not allowed in prolog.. ParseError at [row,col]:[1,1]\nMessage: Content is not allowed in prolog.. '
Bad Request

I think the problem is the encoding. When I run only the command curl:

COMMAND:

`curl -k -u user:pass --header 'Content-Type: application/xml' --request POST --data '<LocalIDs><LocalID ID="9128615"/></LocalIDs>' --url https://rhtpcapc001:8182/pc/center/webservice/datasources/dataSourceConsoleName/Data%20Aggregator%40RHTPCADA201/itemids`

OUTPUT:

`<?xml version="1.0" encoding="UTF-8"?><ItemIDResults><ItemIDResult LocalID="9528017" ItemID="828570"/></ItemIDResults>`

in the answer we see encoding ="UTF-8"

in the python script with the response.encoding I see the encoding ISO-8859-1

thank you in advance for all the help.

BR

David
Support pytest 4

Fixes https://github.com/kennethreitz/requests/issues/5048

See https://docs.pytest.org/en/latest/deprecations.html#marks-in-pytest-mark-parametrize

Not sure about the CI here, it seems to be failing on master as well.
I've also tried to update the Pipfile.lock, but I have no idea how to do it properly without regenerating the entire Pipfile.lock. Tried some magic.

> Fix some issues to improve adherence to best practices

Facepalm, not the and not there wrote. Sorry.


I've dropped the pipfile changes, they conflicted.
Test failures with pytest 4

The tests are failing with pytest 4. The reason is https://docs.pytest.org/en/latest/deprecations.html#marks-in-pytest-mark-parametrize

Run `tox -re py37` to reproduce.

## Expected Result

Tests collect and pass.

## Actual Result

Tests fail to collect:

```
platform linux -- Python 3.7.3, pytest-4.4.0, py-1.8.0, pluggy-0.9.0
...
ERROR collecting tests/test_utils.py 
.eggs/pluggy-0.9.0-py3.7.egg/pluggy/hooks.py:289: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
.eggs/pluggy-0.9.0-py3.7.egg/pluggy/manager.py:68: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
.eggs/pluggy-0.9.0-py3.7.egg/pluggy/manager.py:62: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
.eggs/pytest-4.4.0-py3.7.egg/_pytest/python.py:238: in pytest_pycollect_makeitem
    res = list(collector._genfunctions(name, obj))
.eggs/pytest-4.4.0-py3.7.egg/_pytest/python.py:414: in _genfunctions
    self.ihook.pytest_generate_tests(metafunc=metafunc)
.eggs/pluggy-0.9.0-py3.7.egg/pluggy/hooks.py:289: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
.eggs/pluggy-0.9.0-py3.7.egg/pluggy/manager.py:68: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
.eggs/pluggy-0.9.0-py3.7.egg/pluggy/manager.py:62: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
.eggs/pytest-4.4.0-py3.7.egg/_pytest/python.py:141: in pytest_generate_tests
    metafunc.parametrize(*marker.args, **marker.kwargs)
.eggs/pytest-4.4.0-py3.7.egg/_pytest/python.py:1006: in parametrize
    function_definition=self.definition,
.eggs/pytest-4.4.0-py3.7.egg/_pytest/mark/structures.py:113: in _for_parametrize
    if len(param.values) != len(argnames):
E   TypeError: object of type 'MarkDecorator' has no len()
```

## Reproduction Steps

```console
$ tox -re py37
```

## System Information

```console
$ .tox/py37/bin/python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.3"
  },
  "platform": {
    "release": "5.0.5-300.fc30.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.21.0"
  },
  "system_ssl": {
    "version": "1010102f"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": false
}
```


I suppose https://github.com/kennethreitz/requests/blob/9742da7f91f595796a412a8150aacc00191be039/tests/test_utils.py#L36

Will try to craft a PR.
fix dependency resolution conflict and CI/CD

Removes the call to <4.1 pytest. Pytest 4.4 was released this week breaking CI/CD

Rebuild graph and Pipfile.lock.

Also removed `detox` as it's now built into tox :-)

I underestimated this fix! Python 2 failing because of a dependency on pathlib2 from Pytest.. 
Python3 failing because 1 of the tests depends on an old API

oh man :(
Add request hook

Add a request hook that fires when a request is sent. This resolves kennethreitz/requests#5037.

@Holzhaus I remember the hooks were removed in some old commit https://github.com/kennethreitz/requests/commit/ee6c638e9cfdbdf39b9e499d95c353460365f42c#diff-ab3246c102b4e3c7381cf4d1ea4b2bea for a reason, I don't know why though.

Seems like there was a conflict and the fix did not make it into the branch.
A similar issue ( #5037 ) states that it is fixed by this merge - but unfortunately does not seem to be in because of that conflict.
Would be great if this could be fixed. Any ETA ?

the current hook we have is the only necessary one, i determined. hence, the other ones were removed during the big refactor.
Incorporating a `__main__.py` file into request ?

Python supports the inclusion of a `__main__.py` file withing ones packages. I though it might be nice to include such a  file within requests to handle the more common tasks from the command line e.g. `GET`, `HEAD`, `PUT` and `POST`. I have included an example for both `HEAD` and `GET` though I'm not certain of their utility. 

More often then not it is necessary to download a file from a given url and I have added a `download` option to assist with that as well.

send this to the kennethreitz/requests3 repo!
unclosed socket

I am noticing this “warning” which making soap calls at high speed.

zeep/xsd/visitor.py:883: ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=(‘x.x.x.x’, 39092), raddr=('y.y.y.y', 9001)>
zeep/xsd/types/complex.py:159: ResourceWarning: unclosed <socket.socket

There is an explanation on requests library thread at: https://github.com/kennethreitz/requests/issues/3912

Is it something that can be handled at zeep?
Thanks

The stacktrace provided doesn't appear to be related to Requests. You may need to follow up with zeep.
Handling pound sign (#) within custom URL

I'm using requests to compile a custom URL and one parameter includes a pound sign. Can anyone explain how to pass the parameter without the pound sign changing to "%23"?

download URL: www.example.com/type=#results

currently returns: www.example.com/type=%23results

FWIW, this process works without using params within requests. I know it's weird having a pound sign in a URL but after multiple tests I've concluded that the only way of downoading the file I need is to include the pound sign.
change contentType from application/x-www-form-urlencoded to multipart/form-data

I am sending data to a webserver that does not understand application/x-www-form-urlencoded.
postman uses multipart/form-data to send the request and that works like a charm.

is there anyway to change the contentType so it will generate the needed boundary?
(if i just set the header myself it will not generate the boundary)

 


Hi,
Did you try:
`request.headers.update({'ContentType': multipart/form-data})`
If you use Session() simply substitute request to session.
Normally all the headers are kept.

Hi,
Thanks for the reply. somehow the updated header gets re-updated by the post itself. I found this article: 
https://franklingu.github.io/programming/2017/10/30/post-multipart-form-data-using-requests/
that pointed me to the solution. got it working now, all form fields are send in multipart/form-data (without any {dummy)files)

posted the link for others as there are quite a few with the same issue
{'error': "object of type 'filter' has no len()"} after a get request

Summary.

An error occurs when trying to run https://github.com/opencog/opencog/blob/master/examples/restapi/exampleclient.py

## Expected Result

A parseable JSON:
```
GET /api/v1.1/atoms/57:
{
  "outgoing": [],
  "incoming": [],
  "name": "giant_frog",
  "truthvalue": {
    "type": "simple",
    "details": {
      "count": "0.20000000298023224",
      "confidence": "0.0002499375259503722",
      "strength": "0.07999999821186066"
    }
  },
  "attentionvalue": {
    "lti": 0,
    "sti": 0,
    "vlti": false
  },
  "handle": 57,
  "type": "ConceptNode"
}
```

## Actual Result

An error occured:
>{'error': "object of type 'filter' has no len()"}

## Reproduction Steps

Follow the instructions in https://github.com/opencog/opencog/tree/master/examples/restapi/README.md (admittedly, you'd have to compile cogutil and atomspace, but I doubt they're the cause of the problem)



## System Information

    $ python -m requests.help
```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "2.1.4"
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.7"
  },
  "platform": {
    "release": "4.15.0-46-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "1010007f",
    "version": "17.5.0"
  },
  "requests": {
    "version": "2.21.0"
  },
  "system_ssl": {
    "version": "1010007f"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": true
}
```


This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


your error info for me, please.

Sorry, I've quoted the error message in my post. What would the error info look like?

This doesn't appear to be an issue with Requests, I'd recommend following up with opencog or opening a question on StackOverflow.
Add return to raise_for_status.

Add return to raise_for_status for writing `resp = requests.get('SOME_URL').raise_for_status()` and same code.

Any reason why this didn't make progress? I quite like this.

I don`t know why tests failing. I am usied wsl for testing and all good. Now I will try again.

How to run tests corectly on windows 10?

Do some test command with your changes 
also
take some auto test package and see what its result 
no matter what is your OS
read a python guide to know what your doing.

All tests comleat suddenly.

@kennethreitz, will be merged?

> 
> 
> @kennethreitz, will be merged?

Review your work 
something is not right

> something is not right

Please tell me what is not right?

How long does it take?

@sargun please add thumbs up for this pull request. @kenneth-reitz for some reason ignores it.

Can you rebase and squash?

IMO `raise_for_status` is more like a no-return function. Chaining evaluation is good but it returning the response surprised me.

As you propose to call the function?

I agree with @frostming.
data: URLs won't fetch with requests, work in regular Python urllib

Summary: [`data:` URLs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs) don't work, which is different from the Python 3 standard library.

## Expected Result

I expected the following to just work™:

```python
import requests
r = requests.get('data:text/plain;charset=utf-8,hello')
r.text  # 'hello'
```

## Actual Result

```
import requests
r = requests.get('data:text/plain;charset=utf-8,hello')

<the entire traceback omitted for brevity>

~\c\django-balkans\virtual\Lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
    638
    639         # Get the appropriate adapter to use
--> 640         adapter = self.get_adapter(url=request.url)

InvalidSchema: No connection adapters were found for 'data:text/plain;charset=utf-8,hello'
```

## Reproduction Steps

The following Python 3 code works as intended:

```python
import urllib.request
a = 'data:text/plain;charset=utf-8,Алоха, православныя!'
with urllib.request.urlopen(a) as response:
    print(response.read().decode('utf-8'))
```

As well as the following JavaScript code (which might appear unrelated, but is included to demonstrate that the `data:` URLs are widely supported by bog-standard tooling):

```javascript
const test = async () => {
    const res = await fetch('data:text/plain;charset=utf-8,Алоха, православныя!');
    const data = await res.text();
    console.log(data);
}
test()
```

I'm convinced that the requests library would benefit from supporting this also.

## System Information

    $ python -m requests.help

```json
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.0"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.21.0"
  },
  "system_ssl": {
    "version": "1010008f"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": false
}
```


Requests is an HTTP library, the `data` scheme isn't supported by requests or the underlying urllib3 library.

Seth is right here. Requests is an http client and not intended to have the full support of a browser. There is likely another library that may be better suited for your case if urllib isn't sufficient.
sessions: fix proxy

Take proxy configuration from session if not specify in args of request method

Working great here
Good job @prietopa 

This pull request change order to merge proxy, verify, cert, etc.
It's an error to modify proxy before merge with session.proxy

Hello,

Really appreciate for the fix.
But seems the bug fix not merged into master.
Would you please take a look at it?

Thank you,
Ken

Hi Ken,

When I push merge button, in github web page, it said:
-"Merging is blocked", Merging can be performed automatically with 1 approving review"-
Can you approved?
or
I can do a new pull request in the master branch.

Thank you
prietopa

Hello Prietopa,

Really appreciate for your time and your HOT fix.
I am not the contributor, seems that I don’t have access to approve your request.

@Gabronog Would you please take time to review Prietopa’s pull request.
Really appreciate for your help in advance.

Thank you,
Ken

> Hello Prietopa,
> 
> Really appreciate for your time and your HOT fix.
> I am not the contributor, seems that I don’t have access to approve your request.
> 
> @Gabronog Would you please take time to review Prietopa’s pull request.
> Really appreciate for your help in advance.
> 
> Thank you,
> Ken

Sure, it would be a pleasure :)
Thanks
Gabriel

@nateprewitt what are you thoughts on this?

let's just not change anything here. any delta in this area is potentially very dangerous for our users (unintended side-effects).

@kennethreitz  What I wanted to say is that request is not taking the proxy configuration, changing these lines of code fixes it as it was being override.
Post request not sending with big "data" parameter

Hi,

I'm sending some data formated like this : 

```python
data = {
            'encodedFile'   : base64.b64encode(fileContent),
            'priority'      : Config.cfg['OCForMaarch']['priority'],
            'status'        : Config.cfg['OCForMaarch']['status'],
            'type_id'       : Config.cfg['OCForMaarch']['type_id'],
            'format'        : Config.cfg['OCForMaarch']['format'],
            'category_id'   : Config.cfg['OCForMaarch']['category_id'],
            'subject'       : subject,
            'destination'   : destination,
            'address_id'    : contact['id'],
            'exp_contact_id': contact['contact_id'],
            'doc_date'      : date
}

res = requests.post(self.baseUrl + 'resources', auth=self.auth, data=data)`
```

My encoded file is a 12 pages PDF, pretty heavy. The problem is, the data isn't send at all. When I print what the API server receive, nothing of that array appear. If my 'encodedFile' is something like 'dGVzdA==', the API server receive all the informations.

Is there a limitation I had to check ?

Thanks in davance

> When I print what the API server receive, nothing of that array appear. If my 'encodedFile' is something like 'dGVzdA==', the API server receive all the informations.

What exactly is received on the API server endpoint when sending the entire file? Just no `body` ? Or no request at all?

> Is there a limitation I had to check ?

I'm not aware of any limitation requests has. However, you might want to check your API server: is it running behind nginx with a limit? 

> > When I print what the API server receive, nothing of that array appear. If my 'encodedFile' is something like 'dGVzdA==', the API server receive all the informations.
> 
> What exactly is received on the API server endpoint when sending the entire file? Just no `body` ? Or no request at all?
> 
> > Is there a limitation I had to check ?
> 
> I'm not aware of any limitation requests has. However, you might want to check your API server: is it running behind nginx with a limit?

The API received all the informations about auth etc.. But all the data were not sent. But finally I found the problem : 
The post_max_size parameters on my Apache2 php.ini. So closed the thread :) Thanks for your reply btw
In [21]: import requests /usr/lib/python3/dist-packages/cryptography/x509/base.py:16: RuntimeWarning: coroutine 'wait' was never awaited   from cryptography.x509.extensions import Extension, ExtensionType RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Summary.

## Expected Result

What you expected.

## Actual Result

What happened instead.

## Reproduction Steps

```python
import requests

```


```
/usr/lib/python3/dist-packages/cryptography/x509/base.py:16: RuntimeWarning: coroutine 'wait' was never awaited
  from cryptography.x509.extensions import Extension, ExtensionType
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Can you please fill out the template: describe what your expected and actual result was and how to reproduce.
And please paste the traceback (if valuable) in to the post body and no the title. Please use a descriptive title instead.

Thank you 🎉 

There isn't enough information here to evaluate what is happening. Given this is an issue in cryptography and not Requests directly, I'm going to resolve. I'd recommend checking the issue tracker with cryptography if you're still able to reproduce the issue.
certificate's issue for docs.python-requests.org

Summary.

## Expected Result

Make the docs page secure.

## Actual Result

SSL certification error. This is also pointed out kennethreitz/requests-html#276

![Screenshot from 2019-03-20 01-51-49](https://user-images.githubusercontent.com/8712070/54662118-480e2a00-4ab3-11e9-98c2-45da18317a2b.png)

## Reproduction Steps

Get 'not secure' message by accessing `https://docs.python-requests.org`.

This is because old CNAME record (requests.readthedocs.org) is configured.
 See [READTHEDOCS](https://docs.readthedocs.io/en/stable/custom_domains.html#custom-domain-ssl) 

Below shows lookup result.

```shell
$ dig  +noall +answer -t CNAME docs.python-requests.org
docs.python-requests.org. 396   IN      CNAME   requests.readthedocs.org.

```

## Solution

> Point the CNAME to readthedocs.io,

`https://requests.readthedocs.io` is secured right now.
Exception send post files open('ff.log', 'rb').tell() > 0

request reading from file pointer and silent corrupt fseek.

```
import requests
file = ('filename', open('file.txt', 'rb'))
r1 = requests.post('https://someurl.com', files=[file])  # normal send
r2 = requests.post('https://someurl2.com', files=[file])  # silent send zero length file
```

Because open('file.txt', 'rb') already read in requests(r1), and file seek in end file. Next read file, reading zero bytes.

This problem in practice.
If in first iterate requests is response not 201, and next iterate sending success with zero length file.

```
file = ('filename', open('file.txt', 'rb'))
while True:
    r = requests.post('https://someurl.com', files=[file])
    if r.status_code == 201:
        break
    time.sleep(5)
```

I suggest added to
https://github.com/kennethreitz/requests/blob/master/requests/models.py#L159

next code:
```
if fp.hasattr(fp, 'tell') and fp.tell() > 0:
    raise Exception('You send file seek not at the beginning file.')
```
I want change not obvious behaviour.


Hi @mrakob, thanks for the proposal. While we understand the pain this might cause, this is the normal behavior for dealing with file descriptors. If you read everything, future calls to read will return an empty byte-string. We cannot know what the user intends to do with the file and will only "rewind" it when a request has failed and engaged our internal retry mechanism.

A hard failure like the one proposed is a non-option because many workflows relying on sending partially read files. Given the lack of engagement on this issue, it doesn't appear to be a common stumbling block. I'd recommend either reading the file into a local variable (size permitting) if you need to resend, or writing a simple function that reopens the file before posting to each endpoint.
The reason that uploading file larger than 2GB will be truncated.

When I upload multipart-encoded file larger than 2GB in python2 environment, the body will be truncated. Server side can not read data completely. But the content-length value in header is expected. At client side, it do not send all data.

After tracing of function calls, it is caused by python2 socket library bug. Requests will use socket.sendall() to send body, but it use int type to keep bytes need be sent and has sent: [python2-socket-lib](https://github.com/python/cpython/blob/2.7/Modules/socketmodule.c#L2850). I also create a [issue](https://bugs.python.org/issue36337) to report this bug to python team.

Rather than waiting python team fix it, I thought Requests can fix quickly. After search it has also been reported many times.
Requests lib can split encoded string body into many segments each one smaller than 2GB, and send segments one by one. Or wrapping string body as as file-like object is also works, like this:
```python
from cStringIO import StringIO
body=StringIO(request.body)
```
https requests over proxy got 407

when the page_url use http protocol, it works well
``` python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import requests

page_url = "http://httpbin.org/ip"

proxy = "proxy_ip:proxy_port"

username = "username"
password = "password"

proxy_auth = {"http":  "http://%(user)s:%(pwd)s@%(ip)s/" % {'user': username, 'pwd': password, 'ip': proxy},
              "https": "https://%(user)s:%(pwd)s@%(ip)s/" % {'user': username, 'pwd': password, 'ip': proxy}}

res = requests.get(url=page_url, proxies=proxy_auth)
print(res.status_code)
print(res.content.decode('utf-8'))
```
if the page_url is `"https://httpbin.org/ip"`, got this:
``` python
requests.exceptions.ProxyError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /ip (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required',)))
```

I googled it and saw some similar issues in issues list, it does not seem to support https proxy ?
what would I do, please.


We can't help you debug your personal proxy, you'll need to do your own research or reach out to Stackoverflow for help.
Using specific SSL version TLSv1.2 in Python (mounting separate adapters not working)

I ran into the same problem mentioned in #3214 
However despite mounting my adapters separately I run into the same issues. I get the same error output in the thread mentioned above. 
```
class Myadapter(HTTPAdapter):
    def init_poolmanager(self, connections, maxsize, block=False):
        """Create and initialize the urllib3 PoolManager."""
        self.poolmanager = PoolManager(
            num_pools=connections, maxsize=maxsize,
            block=block, ssl_version=ssl.PROTOCOL_TLSv1_2)


def requests_session(session=None):
    try:
        adapter = Myadapter()
        session.mount('http://', adapter)
        session.mount('https://', adapter)
        return session
    except Exception:
        ('requests_session - Error Api calls')

data = {
    'username' : 
    'password' :
}
headers = {
    'Content-Type' : 'application/json'
}
session_id = requests.Session()
requests_r= requests_session(session=session_id)
resp= requests_r.post(url, data=data, headers=headers).json()
```

the Error is:
```
Max retries exceeded with url: some_random_api (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)",),))
```
Always get a CaseInsensitiveDict for headers.

If we look at how headers are merged from session and request, only if both are set the result will be a `CaseInsensitiveDict`. I propose to always use a case insensitive dict, even when one of the two is missing.

## Expected Result

Always get a `CaseInsensitiveDict` for headers.

## Actual Result

If only `session.headers` or `request.headers` are set, then headers are not case insensitive.

See also:

https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L55

I am not sure whether we should only do it `isinstance(dict_class, CaseInsensitiveDict)`, or whether it would make sense to always convert to the `dict_class`.

can i take up this issue @kootenpv 

Sure

Actually don't. We deliberately allow any mapping type to allow for a variety of use-cases. A change here won't be accepted
improve error message for invalid timeouts

Rearrange a try/except so that these two programs give similar error
messages.

Program 1:

```
import requests
requests.get("https://random.org, timeout=(True, True))
```

Program 2:

```
import requests
requests.get("https://random.org, timeout=True)
```

Before this commit, _Program 2_ raised an error in `urllib3` asking for
an "int, float or None" timeout.

Why is this closed & not merged?
CBOR support

Requests is not accepting feature requests at this time.

Still, I would like to open the issue to find out whether this use case is relevant for more people and worth consideration.

I would like to have a `cbor` parameter when making requests, similar to `json`, which would prepare the request body and set the `Content-Type` header.

**Cons:**
-  `requests` is HTTP for humans, CBOR is not human readable.
- It would add a parameter to the most used and most high-level functions -> extra complexity for the API.

**Pros:** 
- In this use case JSON and CBOR are very similar. When interacting with an API that accepts both encodings it results in unidiomatic code to handle both in different ways (using the parameter for JSON and manually doing this with CBOR)
- It is much more beautiful to use a parameter instead of preparing the request manually. The existince of the `json` parameter also speaks for this argument, with the difference JSON being much more popular than CBOR.

+1. I recently needed this and had to code it myself so yes, I do find it relevant.

CBOR is already "supported" in that you can use a CBOR library to encode and decode raw binary data over the wire. I don't think this is necessarily something Requests should be doing itself. Closing this for now.
Custom HTTP Non RFC Error Code Handling

Custom HTTP/Non-RFC error code handling is not supported - feature request to handle custom error codes.

## Expected Result

If an error code like 601 is thrown back instead of a typical RFC 404, 400, etc. then the requests library replies back with an HttpBadStatusLine

status_code response would be 601 or whatever the server sent back
## Actual Result

HttpBadStatusLine from urllib


## System Information

    Python v3

```
requests.put(someUrl, data=self.payload)
```

According to the HTTP protocol, I think those are invalid status codes.
Just out of curiosity - why would you want to use such status codes in the first place?

Apparently, in NodeJS you can craft custom Error Codes that python supports handling with the urllib.
My coworker is now presenting custom http codes to indicate different application statuses. Its common in other protocols in my workplace from what I'm seeing of the vendors where I'm at (well known video company).

I was thinking we extend the exception handling to allow custom error codes to be handled. As it stands if I use requests instead of the urllib I can handle these sorts of things, and I'd just like to see requests be able to do the same, natively, without refactoring.

Since this error is raised by the standard library, not Requests, there isn't anything we can do here. Servers should stick with compliant HTTP status codes.
Need a requests version number which OrderedDict is working on headers

OrderedDict not working on headers again on version: 2.18.4 and 2.21.0

OrderedDict still works fine on data in this version, but not working on headers again.

I know this won't be fixed for a long time.

So I need help with a version number which OrderedDict is working on headers.


My code:
headers01 = OrderedDict([("Connection", "close"), ("Upgrade-Insecure-Requests", "1"), ("User-Agent", "SomeAgent"), ("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8"), ("Accept-Encoding", "gzip, deflate"), ("Accept-Language", "Some Language")])

result:
Connection: close
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; InfoPath.2)
Accept-Language: en-US,en;q=0.5
Upgrade-Insecure-Requests: 1

There is a test for this in the test suite.

https://github.com/kennethreitz/requests/blob/a345b776fb2cf67d1a1810349e71fd60a83f53eb/tests/test_requests.py#L446

What happens if you run the tests on your machine?
OrderedDict not working on headers again

Summary.
OrderedDict not working on headers again.
I have tried both 2.21.0 and 2.18.4 
It works fine on data but not working on headers, headers still get reordered.
## Expected Result
headers01 = OrderedDict([("Connection", "close"), ("Upgrade-Insecure-Requests", "1"), ("User-Agent", "SomeAgent"), ("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8"), ("Accept-Encoding", "gzip, deflate"), ("Accept-Language", "Some Language")])
## Actual Result
Connection: close
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; InfoPath.2)
Accept-Language: en-US,en;q=0.5
Upgrade-Insecure-Requests: 1

## Reproduction Steps

```python
import requests

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "2.7.15"
  },
  "platform": {
    "release": "7",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020ff"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

BTW 

**AND I know there is no way to ask anyone to fix it since it's just a free project, and I do not Expected anyone to fix it.**
I just want someone to help me providing a requests **version number** which do not have this issue.
OrderedDict still works fine on data in this version.

dataxx = OrderedDict([("xxx", "xxx"), ("yyy", ''), ("zzz", ''), ("000", "222")])

Yea I just noticed this myself... only affect py2... i have gone back to 2.10 even and its still doing it... py3 on 2.12.4 is keeping the order

Referring back to here .. https://github.com/kennethreitz/requests/issues/2057

@alwaysnotworkingforme , if you setup a session first

`session = requests.session()` then do `session.headers=OrderedDict()` before you're get request the headers is defined as an __OrderedDict__ and stays as such..

@kenneth-reitz 



@VeNoMouS  - Thanks! This is exactly the solution I was looking for.

Confirmed this is working for requests version 2.22.0

Closing this issue as fixed.
Status_Code in Session

Hi,
it would be really nice to have access to the status code from the Session, such as:

```python
with session.get(url, params=params, timeout=g_timeout) as  r:
    if r.status_code == session.codes.ok: 
        return r.json()
```

Thank you.

What would be the benefit for you of having those accessible from a `requests.Session` over using the look ups from `requests.codes` ?

I use Requests under UNO (OpenOffice / LibreOffice), and I provide it with extension  [OAuth2OOo](https://github.com/prrvchr/OAuth2OOo/blob/master/README.md)  to allow OpenOffice which only supports python 2.X for now, to work.

The other extensions that require OAuth2 and Requests use only Session with the import of Request in the function providing the Session. Because UNO use lazy loading, I can only use dynamic import, the Requests module being provided by the extension OAuth2OOo is only available after the extension is loaded.

If the status_code are in Session then I do not need anything else ...(All in one)
In thinking where to put it, that I thought Session was the best.
Besides, I already allowed myself to patch, it's only a line:

`self.codes = codes`

Thank

Hi,
I confessed I was a little misguided, the good solution is to use Python dynamic typing and assign the status_code to an attribute of my Session variable, but certainly not to modify the source code of Requests ... Forgive me.
I close this issue
Why can't I use decorator with requests?

```
import requests

r = requests

def my_decorator(func):
    def wrapper(*args):
        print(func.hostname)
        print('sdasdas')
    return wrapper

say_wee = my_decorator(r.sessions.urlparse('http://google.com'))
print(type(say_wee))
r.sessions.urlparse = say_wee
r.get('http://google.com')
```
I want to add some actions to the default sessions.urlparse. I get error:
```
Traceback (most recent call last):
  File "res.py", line 14, in <module>
    r.get('http://google.com')
  File "/Library/Python/2.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 668, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 150, in resolve_redirects
    previous_fragment = urlparse(req.url).fragment
AttributeError: 'NoneType' object has no attribute 'fragment'
```

Hi, @marataziat 

`say_wee = my_decorator(r.sessions.urlparse('http://google.com'))`

You shouldn't use a decorator on a `ParseResult`.

What exactly are you trying to achieve? 

Your decorator takes a function as argument, but you actually pass it a `ParseResult`.
And your decorator doesn't call the original `requests.sessions.urlparse`, but you probably want to.

@timofurrer I want to add some additional layer of validating to the requests library. I don't want to rewrite all the functions so I made a decorator which just adds some code to the urlparse function. 

If you just want to extend the `r.sessions.urlparse` function your decorator should more look like the following: 

```python
import requests


def my_decorator(func):
    def wrapper(*args, **kwargs):
        print("Got r.sessions.urlparse *args:", args)
        print("Got r.sessions.urlparse **kwargs:", kwargs)
        return func(*args, **kwargs)
    return wrapper


requests.sessions.urlparse = my_decorator(requests.sessions.urlparse)
print(requests.get('http://google.com'))
```

Ohhh thank you
How to avoid SSRF when you use requests to parse untrusted URLs?

For an example I have a Flask app, it gets 'url' parameter from GET request. Then it do something like this:
```
requests.get(params['url']).text
```
So how to avoid localhost requests? Should I make a new network interface, which can access only external network and then send requests? I tried to search some answers for that and I found only https://github.com/JordanMilne/Advocate/ but the autor says that we shouldn't use it in production. So how to properly secure from SSRF in case like this? Can requests do requests through specified virtual network interfaces?

I'd say this isn't within the realm of requests responsibility. This is an input sanitization issue not an HTTP issue. My recommendation would be parsing the URL and maintaining an "allow list" of hosts and URL constructions that are acceptable to make requests to (scheme, w/o userinfo, etc).

I have found a better way: https://stackoverflow.com/questions/691045/how-do-you-determine-if-an-ip-address-is-private-in-python you can just check if IP is blacklisted by this or not. Idk how secure it is but I think it's better than nothing. 

```
>>> ip = '169.254.169.254'
>>> ipaddress.ip_address(ip).is_global
False
>>>
```
Lol, it's just a default python3 module and it can secure you even against aws metadata ip!! How can I make a module which will "proxy" the default requests module and add url ssrf protection?
Lock pytest-httpbin to <1.0

[pytest-httpbin](https://github.com/kevin1024/pytest-httpbin) release 1.0.0 updated the `subjectAltName` field to the certificate which our unit tests assume doesn't have an IP Address entry in order to test warning generation.  [Changelog for 1.0.0 even mentions this breakage in our unit tests.](https://github.com/kevin1024/pytest-httpbin/commit/be3c3205972195ca632949ed7af1dc7086cb7899)

So ran `pip freeze` after installing from pipenv and [we're still receiving pytest-httpbin==1.0.0](https://travis-ci.org/kennethreitz/requests/jobs/499055970#L551).

Don't know what to do from here. Pipenv bug?
`no_proxy` setting ignores the `proxies` dictionary

These work:

```
$ python -c "import os,requests;os.environ['HTTPS_PROXY']='https://b.r.o.k.e.n.com';requests.get('https://google.com', proxies={'no_proxy': 'google.com'})"
$ python -c "import os,requests;os.environ['HTTPS_PROXY']='https://b.r.o.k.e.n.com';os.environ['NO_PROXY']='google.com';requests.get('https://google.com')"
```

These do not work:

```
$ python -c "import requests;requests.get('https://google.com', proxies={'https': 'https://b.r.o.k.e.n.com', 'no_proxy': 'google.com'})"
$ python -c "import os,requests;os.environ['NO_PROXY']='google.com';requests.get('https://google.com', proxies={'https': 'https://b.r.o.k.e.n.com'})"
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.5"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.21.0"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.24.1"
  },
  "using_pyopenssl": false
}
```

cc @Lukasa @sigmavirus24 

Many issue (#4871, #879, #4600 ) is relative to `no_proxy`, and this is definitely a bug, even someone provide some patched code, why it's still unresolved?

Please fix the issue as it causes a slightly bigger issue - the session.send() method is ignoring the os.environ proxy settings so there is not way to specify proxy settings other than passing proxies explicitly, which doesn't bypass and ignores no_proxy.

A possible hint looking at the logic for "should we bypass proxies" in Requests, is that this logic is only checked on redirects…

https://github.com/psf/requests/blob/1b417634721ec377abb7f17bc1f215e07202c2f7/requests/sessions.py#L292

When I would expect it to _also_ be called on each request to decide "should we use proxies for this URL". Eg in `.get_adapter()`, or whichever place is responsible for saying "use the connection pool, or a proxy":

https://github.com/psf/requests/blob/1b417634721ec377abb7f17bc1f215e07202c2f7/requests/sessions.py#L718-L730

But I'm not 100% sure at which layer the proxies vs connection pool decision occur, eg if it's in the default `HTTPAdapter` or elsewhere (haven't dug that far into the code).

For what it's worth, the 2nd failing test case works as expected with [HTTPX](https://python-httpx.org):

```python
>>> import os, httpx
>>> os.environ['no_proxy'] = 'google.com'
>>> proxies = {'https': 'https://no.where.com'}
>>> c = httpx.Client(proxies=proxies)
>>> c.get('https://google.com')
<Response [200 OK]>
```

The 1st failing test case doesn't though, because `no_proxy` keys aren't taken into account yet by HTTPX (see https://github.com/encode/httpx/issues/1043).
Add project URLs with links to source code and docs

Warehouse now uses the `project_urls` provided to display links in the sidebar on [this screen](https://pypi.org/project/requests/), as well as including them in API responses to help automation tool find the source code for Requests. For example, see Django's [setup.py](https://github.com/django/django/blob/master/setup.py) and [PyPI listing](https://pypi.org/project/Django/).

fantastic!
utils.get_encodings_from_content regexps incorrect matches

`get_encodings_from_contents` uses regexps:

```
    charset_re = re.compile(r'<meta.*?charset=["\']*(.+?)["\'>]', flags=re.I)
    pragma_re = re.compile(r'<meta.*?content=["\']*;?charset=(.+?)["\'>]', flags=re.I)
    xml_re = re.compile(r'^<\?xml.*?encoding=["\']*(.+?)["\'>]')
```

I'm finding cases where this is matching "<meta foo="bar">random junk charset=something_weird".  A real-life example is at https://www.walmart.com/ip/108356879 where I get 7 matches.  The first one gives the desired "utf-8".  The next five are all "UTF-8\\".  The last one matches on a 24711 character match and produces a 1730 character gibberish result.

Locally I've fixed this by changing the regexp patterns, replacing the first ".*?" with "[^>\\]*?"

Would a PR implementing this in requests (and/or requests-toolbelt) be welcome?



A PR fixing this behavior would be welcome!