Raibek/requests-github-issues huggingface dataset:
requests get using proxy

Summary.

## Expected Result

What you expected.

utils.py proxy = proxies[proxy_key]
proxies is a set

What happened instead.

	ip = random.choice(p_list)
	proxy = {'http', 'http://{0}:{1}'.format(ip, port)}
	req = requests.get(url=target, proxies=proxy)

```python
Traceback (most recent call last):
  File "d:\spider\main.py", line 53, in get_list
    req = requests.get(url=target, proxies=proxy)
  File "C:\Python27\lib\site-packages\requests\api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "C:\Python27\lib\site-packages\requests\api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "C:\Python27\lib\site-packages\requests\adapters.py", line 390, in send
    conn = self.get_connection(request.url, proxies)
  File "C:\Python27\lib\site-packages\requests\adapters.py", line 286, in get_connection
    proxy = select_proxy(url, proxies)
  File "C:\Python27\lib\site-packages\requests\utils.py", line 643, in select_proxy
    proxy = proxies[proxy_key]
TypeError: 'set' object has no attribute '__getitem__'
```

## System Information
   python 2.7 win10
    $ python -m requests.help 

```
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).
Running tests with tarball from PyPi fails due to missing Pipfile

Summary.

## Expected Result

Tests complete successfully

## Actual Result

```
[  109s] ______________ TestRequests.test_POSTBIN_GET_POST_FILES_WITH_DATA ______________
[  109s] 
[  109s] self = <tests.test_requests.TestRequests instance at 0xb623c18c>
[  109s] httpbin = <function inner at 0xb691609c>
[  109s] 
[  109s]     def test_POSTBIN_GET_POST_FILES_WITH_DATA(self, httpbin):
[  109s]     
[  109s]         url = httpbin('post')
[  109s]         requests.post(url).raise_for_status()
[  109s]     
[  109s]         post1 = requests.post(url, data={'some': 'data'})
[  109s]         assert post1.status_code == 200
[  109s]     
[  109s] >       with open('Pipfile') as f:
[  109s] E       IOError: [Errno 2] No such file or directory: 'Pipfile'
[  109s] 
[  109s] tests/test_requests.py:738: IOError
```

## Reproduction Steps

Download tarball from PyPi:
https://pypi.org/project/requests/#files
Add official Python 3.7 support

Python 3.7 was just released [1]. This is a small change to
enable support in requests.

[1] https://docs.python.org/3.7/whatsnew/3.7.html

Signed-off-by: Eric Brown <browne@vmware.com>

Travis doesnt' appear to support 3.7 yet: https://travis-ci.org/requests/requests/jobs/397660384

Looks like it's being worked: https://github.com/travis-ci/travis-ci/issues/9815

I see nothing indicating that it's in progress on Travis' part other than an issue that non-Travis-CI employees are commenting on.

Hey @ericwb, we’ll likely want this enabled for appveyor as well before we claim support for 3.7.

I haven’t seen an update from them but that’d be something to look into before we merge this.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4717?src=pr&el=h1) Report
> Merging [#4717](https://codecov.io/gh/requests/requests/pull/4717?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/4d51e6d20179bdc906c45506c0817b6b677589ae?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4717/graphs/tree.svg?width=650&token=mZGswIiXLa&height=150&src=pr)](https://codecov.io/gh/requests/requests/pull/4717?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4717   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4717?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4717?src=pr&el=footer). Last update [4d51e6d...2aa29f8](https://codecov.io/gh/requests/requests/pull/4717?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


According to https://github.com/appveyor/ci/issues/2475, Appveyor now supports Python 3.7.

@jdufresne great to hear! Once we get a working build with that, I’m happy to merge.

Hey @ericwb, wanted to check in on this again if you get a minute. Otherwise, I'm going to look at cherry picking these onto master with the appveyor integration later this week. Thanks!

@ericwb would you be willing to rebase this?

And squash the commits too if you wouldn’t mind. Thanks!

@nateprewitt any release coming soon with python3.7 support?
Thanks in advance!

@mrkz, 2.19.1 should already support 3.7. This was just adding documentation stating that.
JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Hello ; please i run this code :

import json
import requests
url = "http://add_server"
data = json.dumps({'sl':5.84,'sw':3.0,'pl':3.75,'pw':1.1})
r = requests.post(url,data)
print(r.json)

and i get this error : 
<bound method Response.json of <Response [500]>>
 

and when i run this code : 
import json
from pprint import pprint
import requests
url = "http://add_server"
data = json.dumps({'sl':5.84,'sw':3.0,'pl':3.75,'pw':1.1})
r = requests.post(url,data)
pprint(r.json)

i get this error: 

JSONDecodeError Traceback (most recent call last)
in ()
4 data = json.dumps({'sl':5.84,'sw':3.0,'pl':3.75,'pw':1.1})
5 r = requests.post(url,data)
----> 6 print(r.json())

~/anaconda3/lib/python3.6/site-packages/requests/models.py in json(self, **kwargs)
890 # used.
891 pass
--> 892 return complexjson.loads(self.text, **kwargs)
893
894 @property

~/anaconda3/lib/python3.6/json/init.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
352 parse_int is None and parse_float is None and
353 parse_constant is None and object_pairs_hook is None and not kw):
--> 354 return _default_decoder.decode(s)
355 if cls is None:
356 cls = JSONDecoder

~/anaconda3/lib/python3.6/json/decoder.py in decode(self, s, _w)
337
338 """
--> 339 obj, end = self.raw_decode(s, idx=_w(s, 0).end())
340 end = _w(s, end).end()
341 if end != len(s):

~/anaconda3/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
355 obj, end = self.scan_once(s, idx)
356 except StopIteration as err:
--> 357 raise JSONDecodeError("Expecting value", s, err.value) from None
358 return obj, end

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Any help please. Thanks

Hey, I'll address both of the problems.
1. You write `print(r.json)` but `r.json` is a method, so instead you should write `print(r.json())`.
2. I got a little confused with what's going on here, but the thing is that the response is not a valid json (HTTP status code of 500), therefore you can't access `r.json()`.
**I think an informative exception should be raised in this case.**

Try to get a response with the HTTP status code of 200 and see if it works.

>  I think an informative exception should be raised in this case.

What's more informative than `JSONDecodeError` meaning we literally can not decode any JSON from this.

A response with a status code of 500 may return valid JSON. It may not. It's up to the user to know whether that's the case and call `response.json()` accordingly. `response.json()` being called with reckless abandon is about as likely to work as the user doing `import json; json.loads(response.text)` with the same amount of though.
Misspelled 'proxy' parameter in docstring

The 'proxy' parameter was misspelled as 'proxies' in the docstring.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4713?src=pr&el=h1) Report
> Merging [#4713](https://codecov.io/gh/requests/requests/pull/4713?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/9fc639d0306a8068057c9f5bcd34adf5f20fb083?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4713/graphs/tree.svg?token=mZGswIiXLa&width=650&height=150&src=pr)](https://codecov.io/gh/requests/requests/pull/4713?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4713   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4713?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/adapters.py](https://codecov.io/gh/requests/requests/pull/4713/diff?src=pr&el=tree#diff-cmVxdWVzdHMvYWRhcHRlcnMucHk=) | `70.18% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4713?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4713?src=pr&el=footer). Last update [9fc639d...5d90638](https://codecov.io/gh/requests/requests/pull/4713?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Remove unused httpbin parameters

httpbin is used to mock HTTP endpoints. In these methods, the parameter goes unused.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4712?src=pr&el=h1) Report
> Merging [#4712](https://codecov.io/gh/requests/requests/pull/4712?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/9fc639d0306a8068057c9f5bcd34adf5f20fb083?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4712/graphs/tree.svg?width=650&src=pr&token=mZGswIiXLa&height=150)](https://codecov.io/gh/requests/requests/pull/4712?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4712   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4712?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4712?src=pr&el=footer). Last update [9fc639d...d54aca4](https://codecov.io/gh/requests/requests/pull/4712?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Remove unused session variables

Removed local session variables that go unused during testing.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4710?src=pr&el=h1) Report
> Merging [#4710](https://codecov.io/gh/requests/requests/pull/4710?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/943a5c8e89db1758ae24adbbedacb3b05c32df4a?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4710/graphs/tree.svg?src=pr&token=mZGswIiXLa&width=650&height=150)](https://codecov.io/gh/requests/requests/pull/4710?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4710   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4710?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4710?src=pr&el=footer). Last update [943a5c8...28da72b](https://codecov.io/gh/requests/requests/pull/4710?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Update max Twitter characters to 280

Twitter expanded the maximum character count per tweet from 140 to 280 in November 2017.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4709?src=pr&el=h1) Report
> Merging [#4709](https://codecov.io/gh/requests/requests/pull/4709?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/d79187143ffa8e05a6cfb518b949954e1208bdfe?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4709/graphs/tree.svg?token=mZGswIiXLa&width=650&height=150&src=pr)](https://codecov.io/gh/requests/requests/pull/4709?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4709   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4709?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4709?src=pr&el=footer). Last update [d791871...0c036fe](https://codecov.io/gh/requests/requests/pull/4709?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Don't use depreciated `session()` method

- References to depreciated method `session()` now initialize `Session()` directly.
- Added test for depreciated `session()` method.

# [Codecov](https://codecov.io/gh/requests/requests/pull/4704?src=pr&el=h1) Report
> Merging [#4704](https://codecov.io/gh/requests/requests/pull/4704?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/cd97ce90ba04156a4aae2fd05aa22c790bec1a5c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4704/graphs/tree.svg?width=650&token=mZGswIiXLa&height=150&src=pr)](https://codecov.io/gh/requests/requests/pull/4704?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4704   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4704?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4704?src=pr&el=footer). Last update [cd97ce9...93ab15e](https://codecov.io/gh/requests/requests/pull/4704?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I don’t think this will necessarily hurt anything to migrate, but if we’re going to have the `session` method left in the API we should have at least some tests exercising it. If the day comes that we ever do remove `session` we will know very quickly that these tests need to be updated.

I’ll leave final judgement here to @sigmavirus24 but I’m not sure we need to do this now.

@nateprewitt In that case, I can add a test specifically for `session()` to ensure backwards compatibility.
response.json() is failing enconding for non ascii characters

Summary.
I'm executing a simple GET request with a list of data. But the .json() fails because there's a non ascii character and it can't encode it.

## Expected Result

[
  {
    "key": "value"
  }
]

## Actual Result
*** UnicodeEncodeError: 'ascii' codec can't encode character '\xfc' in position 329: ordinal not in range(128)


## Reproduction Steps

Try to pass this type of character (In this case is the "ü") in a request and then try to convert the response to json using response.json() 

```python
import requests

response = requests.request("GET", ulr, headers=headers)
response.json()
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.2"
  },
  "platform": {
    "release": "3.10.0-327.13.1.el7.x86_64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.19.1"
  },
  "system_ssl": {
    "version": "1000114f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Requests uses the standard `json` library. Your alternative here is to do

```py
json.loads(response.text)
```

That should exhibit exactly the same problem. This is not a bug in requests. Note that the `json()` method on a `Response` allows you to pass in extra keyword arguments. Those correspond to arguments one might pass to `json.loads()`.
Documentation and tests for form-encoded multivalued elements as lists


# [Codecov](https://codecov.io/gh/requests/requests/pull/4701?src=pr&el=h1) Report
> Merging [#4701](https://codecov.io/gh/requests/requests/pull/4701?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/4d51e6d20179bdc906c45506c0817b6b677589ae?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4701/graphs/tree.svg?height=150&width=650&token=mZGswIiXLa&src=pr)](https://codecov.io/gh/requests/requests/pull/4701?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4701   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4701?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4701?src=pr&el=footer). Last update [4d51e6d...9c92f75](https://codecov.io/gh/requests/requests/pull/4701?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Alright, I think I'm happy with this. Would you mind rebasing onto the master branch and then we can look at merging. Thanks @akaihola!

@nateprewitt, I now rebased on master.
Multivalued `data=` works when given as a list, but is not documented nor tested

For the `params=` request argument, [the documentation](http://docs.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls) explains how multivalued query parameters can be expressed as lists in the dictionary:

>You can also pass a list of items as a value:
>```python
>>>> payload = {'key1': 'value1', 'key2': ['value2', 'value3']}
>>>> r = requests.get('http://httpbin.org/get', params=payload)
>```

There's also a unit test for this:
```python
   def test_urlencoded_get_query_multivalued_param(self, httpbin):
        r = requests.get(httpbin('get'), params=dict(test=['foo', 'baz']))
```

For the `data=` request argument, only the "list of tuples" method for multivalued form-encoded values is shown in [the documentation](http://docs.python-requests.org/en/master/user/quickstart/#more-complicated-post-requests):

>You can also pass a list of tuples to the data argument. This is particularly useful when the form has multiple elements that use the same key:
>```python
>>>> payload = (('key1', 'value1'), ('key1', 'value2'))
>>>> r = requests.post('http://httpbin.org/post', data=payload)
>```

However, both `params=` and `data=` use the same `RequestEncodingMixin._encode_params()` helper for encoding the query parameters or form-encoded values. Indeed, using a list also works for `data=`:
```python
>>> requests.Request(url='http://foo', method='POST', data={'a': ['b', 'c']}).prepare().body
'a=b&a=c'
```

## Expected Result

The "list of items" method for specifying multivalued POST parameters should be documented and tested in the unit test suite.

## Actual Result

I couldn't find documentation or unit tests for the "list of items" method.

Resolved in #4701.
Test multi-class inheritance for BaseAdapter

Ensure that `BaseAdapter` supports multi-class inheritance. This means a subclass of `BaseAdapter` will invoke all parent classes when inheriting from multiple classes.

See: https://github.com/requests/requests/pull/4697#issuecomment-398814166

    class BaseAdapter(object):  # pass
        def __init__(self):
            super(BaseAdapter, self).__init__()

    class BaseAdapter(object):  # pass
        pass

    class BaseAdapter(object):  # fail
        def __init__(self):
            BaseAdapter.__init__(self)

    class BaseAdapter(object):  # fail
        def __init__(self):
            object.__init__(self)

    class BaseAdapter(object):  # fail
        def __init__(self):
            pass

# [Codecov](https://codecov.io/gh/requests/requests/pull/4699?src=pr&el=h1) Report
> Merging [#4699](https://codecov.io/gh/requests/requests/pull/4699?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/883caaf145fbe93bd0d208a6b864de9146087312?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4699/graphs/tree.svg?token=mZGswIiXLa&src=pr&width=650&height=150)](https://codecov.io/gh/requests/requests/pull/4699?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4699   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4699?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4699?src=pr&el=footer). Last update [883caaf...f043b4c](https://codecov.io/gh/requests/requests/pull/4699?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi @Stevoisiak, it looks like this test passes regardless of the existence of `__init__`. If that's the case, we're effectively testing a behaviour of the Python interpreter rather than any Requests code. I don't think we're likely to accept a patch around that unless there's a compelling reason to do so.

As I tried to state in the last PR, we really appreciate contributions to Requests but need to make sure they're providing benefit to our users and the code base. Test coverage is definitely an area we can do better in, but the tests should be focused on behaviours of Requests.

I'm going to close this out since I think it's unlikely to be merged.
Ignore '.idea' folder (PyCharm Settings)

When a project is opened in PyCharm, a '.idea' folder is automatically created to store user-specific settings. This can be safely ignored.

See https://rider-support.jetbrains.com/hc/en-us/articles/207097529-What-is-the-idea-folder-

# [Codecov](https://codecov.io/gh/requests/requests/pull/4696?src=pr&el=h1) Report
> Merging [#4696](https://codecov.io/gh/requests/requests/pull/4696?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/883caaf145fbe93bd0d208a6b864de9146087312?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4696/graphs/tree.svg?height=150&width=650&token=mZGswIiXLa&src=pr)](https://codecov.io/gh/requests/requests/pull/4696?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4696   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4696?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4696?src=pr&el=footer). Last update [883caaf...f287261](https://codecov.io/gh/requests/requests/pull/4696?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Please use a global gitignore. This has been proposed before and has been rejected. In the future please search open, closed, and merged PRs before opening new ones.
Documentation for response.json() wrong

## Summary

When calling response.json() on a response where the content is not valid json a JSONDecodeError bubbles up from the json library, although the documentation claims a ValueError should occur for that

## Expected Result

ValueError

## Actual Result

JSONDecodeError

## Reproduction Steps

```python
import requests

response = requests.Response()
response.json()
```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.3"
  },
  "platform": {
    "release": "4.9.0-6-amd64",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1010006f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}
```


Hi @l0rb,

`JSONDecodeError` is a subclass of `ValueError`. It will be returned by the `json` module response in Python3 in some cases, but doesn't exist in Python2. For code compatible across both versions, or Python 3 code that's compatible in all cases, you'd want to be excepting the `ValueError`.
Getting size of performed request

I want to find out what is the size of performed request - is it possible in requests library?

## Expected Result

in my dreams:

```
response.request.size()
```

## Actual Result

not apply

## Reproduction Steps

What I should consider manually computing this value. Header, querystring, body (if apply)?
Does below cod reflect bytes sent to server

```
sys.getsizeof(response.request)
```


## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.5"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020bf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

```

Hi @pbaranski,

Requests is in a feature freeze, so I don't know if we'd introduce a `size()` method to the `Request` object.

If your definition of "size" is bytes sent over the wire, I'm not sure that `sys.getsizeof(response.request)` expresses that. You can find the definition/layout of an http message in [RFC7230 section 3](https://tools.ietf.org/html/rfc7230#section-3). This is what will be sent when you issue the Request. A rough outline is:

```
GET /path/to/my/resource HTTP/1.1\r\n
Accept: */*\r\n
Content-Type: text/plain\r\n
Content-Length: 43\r\n
Connection: keep-alive\r\n
User-Agent: MyUserAgent1.2.3\r\n
\r\n
Dear Server,

I hate your stinking guts...
```

That should be 193 bytes across the wire including the request line, headers, and uncompressed body. There are a few different ways you could slice this depending on what you're trying to measure, but this is the general idea.

If you have further questions, let's move this to [StackOverflow](https://stackoverflow.com) with the `python-requests` tag. We typically reserve the github tracker for reports about defects in Requests. Thanks!

@nateprewitt Thanks for answer, is partially explains what I was after. Before i posted question here I made SO question what is available here  [https://stackoverflow.com/questions/50908813/python-requests-figuring-out-request-full-size](https://stackoverflow.com/questions/50908813/python-requests-figuring-out-request-full-size ) . I will try to post findings from your response to SO thread.
Cheers
Backslash char in Unicode encoded characters becomes a double backslash

We have an application which returns JSON responses. In such a response document, some string values could have encoded Unicode characters like `\u003C` (<) or `\u003E` (>). But `requests` seems to "process" this return value.

## Expected Result

Simplified version of my response JSON:

```
{ "key": "\u003CSome Text Here\u003E" }
```

## Actual Result

```
{ "key": "\\u003CSome Text Here\\u003E" }
```

The single backslash becomes a double backslash. I don't have this with plain `curl` or the Insomnia Mac REST client app.

## Reproduction Steps

Set up an nginx server serving document `encoded.json` with content:

```json
{
    "key": "\u003CSome text here\u003E"
}
```

Instructions:
```
$ docker pull nginx
$ docker run --name json-server -p:8080:80 -v `pwd`:/usr/share/nginx/html:ro -d nginx
```

In a Python shell:

```python
>>> import requests
>>> r = requests.Session()
>>> response = r.get('http://localhost:8080/encoded.json')
>>> response.headers
{'Server': 'nginx/1.15.0', 'Date': 'Tue, 19 Jun 2018 08:21:05 GMT', 'Content-Type': 'application/json', 'Content-Length': '44', 'Last-Modified': 'Tue, 19 Jun 2018 08:07:58 GMT', 'Connection': 'keep-alive', 'ETag': '"5b28b9de-2c"', 'Accept-Ranges': 'bytes'}
>>> response.encoding
>>> response.content
b'{\n    "key": "\\u003CSome text here\\u003E"\n}\n'
>>> response.text
'{\n    "key": "\\u003CSome text here\\u003E"\n}\n'
```

Note that `response.encoding` is not set.

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.4"
  },
  "platform": {
    "release": "17.6.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.19.1"
  },
  "system_ssl": {
    "version": "100020ff"
  },
  "urllib3": {
    "version": "1.23"
  },
  "using_pyopenssl": false
}
```

Also doesn't work with urllib 1.22 and requests 2.18.4.

well, that's not requests's fault, that's because you `print` it out, python escape that if you print it.
just write the content to file and you will get what you want.

@AndCycle thanks for putting me on the right track.

First of all, I find it very confusing that Python prints the double escaped version. This put me on the wrong track.

But eventually, I am using `response.content.decode('unicode-escape')` as my solution. `response.text` does not perform this type of decoding it seems.
`RequestException` does not contain `request` as a field in some situation.

`RequestException` does not contain `request` as a field in some situation.

## Expected Result

A `RequestException` (or its subclass) should contain a field named `request` if possible.

## Actual Result

I did not see any in 

https://github.com/requests/requests/blob/883caaf145fbe93bd0d208a6b864de9146087312/requests/models.py#L751-L756

Actually, I think it should be written like

```python
ChunkedEncodingError(e, request=self.request)
```

```python
ContentDecodingError(e, request=self.request)
```

```python
ConnectionError(e, request=self.request)
```

(Actually, the third line has another problem and had a fix in #4468 on version 3.0)

We omit it in these cases as you have a `response` which has `response.request` so if you're iterating over content you have the request that you can munge yourself if you're handling exceptions close to that response.

On top of that, those really aren't specific to a request as it is. Those happen after the response has been started so including the request may be misleading in those exceptions.

You are right since `RequestException.request` will be assigned in the following code if I already have `response` 

https://github.com/requests/requests/blob/883caaf145fbe93bd0d208a6b864de9146087312/requests/exceptions.py#L22-L24

However, what I want to do is just to log the information when an exception was raised. For example, when I trying to run the code like this
```python
try:
    requests.get("https://github.com/")
    requests.get("https://www.google.com/")
except RequestException as err:
    logging.warn('Failed in %s - %s', err.request.method, err.request.url)
```
All thing will work fine if `RequestException` was caused by `NewConnectionError` or `ConnectTimeoutError` in `urllib3`, but it will raise `AttributeError: 'NoneType' object has no attribute 'method'` if it was caused by `ReadTimeoutError` since `RequestException.request` was not initialized in such case.

What I want to express is that

- If a request was failed **in sending the request**, then I can get `RequestException.request` for logging

- If a request was failed **after receiving a bad response**, then I can get `RequestException.request` for logging

- However, if a request was failed **during the time receiving the content of a response**, then I get nothing but a `RequestException`

I don't know if it is reasonable, but I think in any case, I should get `RequestException.request` for logging or debugging
Multiple POST requests, second request gets a 404 error code

Hello, I am new to python and am having this same issue. I am trying to authenticate to a website that redirects you to a security question after the initial login. Both the initial login and subsequent page use the same "action URL" and on the second post request I am receiving a 404, here is my code, any help would be greatly appreciated:

```python
from bs4 import BeautifulSoup as bs
import requests
import time
sources = ["https://www.dandh.com/v4/view?pageReq=dhMainNS"]
req = requests.Session()


def login():
    authentication_url = "https://www.dandh.com/v4/dh"

    header = {"user-agent": "Mozilla/5.0 (Windows NT 10.0; rv:60.0) Gecko/20100101 Firefox/60.0"}

    payload = {"Login": "12345",
               "PW": "12345",
               "Request": "Login"}

    payload2 = {"securityAnswer": "12345",
                "Request": "postForm"}

    req.post(authentication_url, data=payload, headers=header)

    time.sleep(3)

    req.post(authentication_url, data=payload2, headers=header)

    time.sleep(3)
   

def print_object(sources):
    for url in sources:
        soup_object = bs(req.get(url).text, "html.parser")
        print(soup_object.get_text())


def main():
    login()
    print_object(sources)

main()
```


Hi @mwadiet,

This looks like it’s a request for help rather than a report of a defect in Requests. What you’re describing is a behavior of the server which isn’t related to Requests. If you open a question on [StackOverflow](https://stackoverflow.com), you should be able to get some guidance on how to approach the problem.

Thanks!

I appreciate the response, i only posted it on their because there was a previous post that the developers acknowledged as a bug in requests. Where doing a get or post would return a 200 status code and then a 404 immediately after on the second request

Sent from my iPhone

> On Jun 18, 2018, at 10:44 PM, Nate Prewitt <notifications@github.com> wrote:
> 
> Hi @mwadiet,
> 
> This looks like it’s a request for help rather than a report of a defect in Requests. What you’re describing is a behavior of the server which isn’t related to Requests. If you open a question on StackOverflow, you should be able to get some guidance on how to approach the problem.
> 
> Thanks!
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or mute the thread.


Nope. It was never acknowledged as a bug. That was a thing we were tracking and found a work around for with the specific service. This is likely a problem with the service you're talking to
requests.get is very slow while fetching websites

I am trying to build a simple terminal app to search stuff online through the terminal.
When I submit get requests through the library, I get very large wall times. I can't seem to pinpoint the problem.

Its fast when I directly type into the browser.

![screenshot from 2018-06-15 21-13-09](https://user-images.githubusercontent.com/24193921/41477203-3c595772-70e1-11e8-97a5-9fb24c62f660.png)


Hi there @sky-lynx 

Your output tells me that requests itself is plenty fast. The amount of CPU time it's taking to perform the syscalls (including opening a brand new socket) is taking less than 1/5th of a second. The rest of that time would appear to be coming from the network time. Unfortunately, we are not magical enough to improve the speed of the network you're using. Some of that time may be spent retrieving DNS records and some of it may be spent actually communicating with the external website. This slowness can also be aggravated by proxies sitting between you and the site. Sadly, there's little we can do here to help you and since this is not a bug in Requests itself, I'm afraid I'll be closing this issue.

Cheers

Apart from trying to get higher network speeds somehow, could you suggest some methods to deal with speeding up DNS records and proxies? 
Would skipping proxies help?
Response.content is wasteful in time and memory for large inputs

https://github.com/requests/requests/blob/883caaf145fbe93bd0d208a6b864de9146087312/requests/models.py#L827

iter chunk is 10K, let's say we have binary content that is about 10MB, the .join first converts the iter_content to a list of 1000 items of 10K each (if it's not chunked), then it creates a new large 10MB string, copies all of them to it, then returns it.

This both creates pressure on the allocator and at the peak takes * 2 the memory of the input.

Without resorting to C, and if we know the input size (i.e. not chunked), we can create a bytearray and fill it with one iteration at a time. On my computer it is about *2-3 faster, takes half the memory and causes 1/2 of page reclaims.

The problem is that is of course that it returns a bytearray and not a bytes type, maybe this should be a helper method for large inputs if people care about this stuff ?

```python
arr = bytearray(totalsize)
i = 0
for item in self.iter_content(CONTENT_CHUNK_SIZE):
  l = len(item)
  arr[i:i + l] = item
  i += l
return arr
```

Thanks for taking the time start this discussion, @tzickel! My initial thoughts here are mostly about usability and expectations around Requests' API. We deal predominantly in byte strings and strings when handling data. Implementing the suggested change would break that, and converting the `bytesarray` back to `bytes` nullifies the memory savings.

The general idea here is correct though, this is what we recommended for large body content. Using `iter_content` on streamed responses is much faster/more efficient. This approach also allows the user to choose whatever data structure or allocation best fits their needs. We could look at adding a helper function to the [Requests toolbelt](https://github.com/requests/toolbelt), but since this is a 5-line function that can be optimized for different use cases, it may be easier to leave this to users to implement.

My main issue is that I'm passing a large input (between a few MB to a few GB) to request toolbelt MultipartDecoder (because if it's not multipart I don't use .content, but stream it anyhow) and I'm looking for ways to optimize that workflow (considering I don't need the real input and that I need all the data in-memory). MultipartDecoder uses the request.content

https://github.com/requests/toolbelt/blob/6d74296f0f267d9e06ba7828585dac61276d4e2b/requests_toolbelt/multipart/decoder.py#L156

So I'm looking for ways to optimize it (without maintaining code outside).

I guess I should be using something like this:
https://github.com/requests/toolbelt/pull/222
And when I know the size of a part, I can directly stream into a preallocated bytearray (almost)
2.19.0 - TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

When running python program with command line arguments **-OO** the exception occurs. That issue can be reproduced since **requests** version **2.19.0**, since the logic inside module **requests/status_codes.py** has changed.

## Expected Result
python3.6 -OO -m my_script.py runs without any exceptions

## Actual Result

**TypeError** exception raises
```
Traceback (most recent call last):
  File "/Users/alexandromyshev/Downloads/send_requests.py", line 1, in <module>
    import requests
  File "/Users/alexandromyshev/venvs/rtest/lib/python3.6/site-packages/requests/__init__.py", line 114, in <module>
    from .models import Request, Response, PreparedRequest
  File "/Users/alexandromyshev/venvs/rtest/lib/python3.6/site-packages/requests/models.py", line 43, in <module>
    from .status_codes import codes
  File "/Users/alexandromyshev/venvs/rtest/lib/python3.6/site-packages/requests/status_codes.py", line 119, in <module>
    _init()
  File "/Users/alexandromyshev/venvs/rtest/lib/python3.6/site-packages/requests/status_codes.py", line 116, in _init
    __doc__ = (__doc__ + '\n' +
TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'
```
## Reproduction Steps

1) Create some simple script like
```import requests

def main():
    response = requests.get('http://google.com')
    print(response.status_code)
    return response


if __name__ == '__main__':
    main()
```
2) Run the command
```
python3.6 -OO send_requests.py
```

## System Information

    $ python3.6 -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.4"
  },
  "platform": {
    "release": "17.5.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.19.0"
  },
  "system_ssl": {
    "version": "100020df"
  },
  "urllib3": {
    "version": "1.23"
  },
  "using_pyopenssl": false
}

```

Fixed in #4683

2.19.1 is pushed to pypi, so this should be resolved now. Thanks everyone!
Do not alter status_codes docstring if it is None

Following https://github.com/requests/requests/pull/4394/files, with PYTHONOPTIMIZE=2, the current code breaks with TypeError due to None as the docstring. This broke dask tests https://github.com/dask/dask/pull/3594 .

# [Codecov](https://codecov.io/gh/requests/requests/pull/4683?src=pr&el=h1) Report
> Merging [#4683](https://codecov.io/gh/requests/requests/pull/4683?src=pr&el=desc) into [master](https://codecov.io/gh/requests/requests/commit/991e8b76b7a9d21f698b24fa0058d3d5968721bc?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/requests/requests/pull/4683/graphs/tree.svg?token=mZGswIiXLa&width=650&height=150&src=pr)](https://codecov.io/gh/requests/requests/pull/4683?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4683   +/-   ##
=======================================
  Coverage   66.79%   66.79%           
=======================================
  Files          15       15           
  Lines        1563     1563           
=======================================
  Hits         1044     1044           
  Misses        519      519
```


| [Impacted Files](https://codecov.io/gh/requests/requests/pull/4683?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [requests/status\_codes.py](https://codecov.io/gh/requests/requests/pull/4683/diff?src=pr&el=tree#diff-cmVxdWVzdHMvc3RhdHVzX2NvZGVzLnB5) | `0% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/requests/requests/pull/4683?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/requests/requests/pull/4683?src=pr&el=footer). Last update [991e8b7...8031ec7](https://codecov.io/gh/requests/requests/pull/4683?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@nateprewitt please release at your earliest convenience