Raibek/requests-github-issues huggingface dataset:
NTLM auth broken in 2.3.0?

Hi,

I am not sure exactly where this should go (here or https://github.com/requests/requests-ntlm/) so I will post it in both places. 

I have been using `requests-ntlm` (post-version 0.0.2.2) in conjunction with `requests 2.2.1`. Recently I upgraded to `requests 2.3.0` and found that NTLM authentication no longer worked; every request was rejected with 401 Unauthorized.

Side-by-side debugging of 2.2.1 vs 2.3.0 yielded the result that [this deletion](https://github.com/kennethreitz/requests/compare/v2.2.1...v2.3.0#diff-31f6e77c031977d33226530924b4337aL393) causes a connection not to be put into the pool. I can fix this by patching `requests-ntlm` but I am not familiar with either library, so while I know that this will solve my problem I wanted to mention this here in case it might be related to any other problems to do with `requests`.

I'm happy to provide debug logs or further details. 

Many thanks


This should be posted only on requests-ntlm. =)


Thanks, am going to repost now.
Verify https, without exception

I'm looking for a way to do a verify, but still continue with the actual request.
This way I can create a message, notifying the user, but returning the content.

See:
https://github.com/RuudBurger/CouchPotatoServer/issues/3357 or https://github.com/RuudBurger/CouchPotatoServer/issues/3369

If I check both 
http://www.sslshopper.com/ssl-checker.html#hostname=torrentshack.net
http://www.sslshopper.com/ssl-checker.html#hostname=nzbs.in
They seem fine to me, but they fail using requests

There are also a lot of users using selfsigned urls, giving just a warning would be good enough here also.
I've tried creating my own adapter, extending the httpadapter, but then I have to do a requests and if that fails on SSLError, remove verify=true and do the request again. Seems a bit overkill to me.

Hope you can help me out.


It's not entirely clear what you want here. You want to verify the certificate, but then continue even if the certificate is invalid?


Yeah, I just need to catch the error. Like a browser does, saying "maybe not safe, do you want to continue".


In the cases you're talking about they're clearly using SNI. Python 2 does not do SNI out of the box, so you need to [install these packages](https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484).

As for doing that, you should do exactly that: catch the error, and then make the request again. =)


Thanks, so I have this:

```
class HTTPSMaybeAdapter(HTTPAdapter):

    def send(self, *args, **kwargs):

        try:
            return super(HTTPSMaybeAdapter, self).send(*args, **kwargs)
        except SSLError as e:
            log.error('SSL verification failed, could be unsafe connection: %s', e)

            kwargs['verify'] = False
            return super(HTTPSMaybeAdapter, self).send(*args, **kwargs)
```

Thats what the best/simplest options is then?


The simplest option is genuinely at the top level:

``` python
s = requests.Session()

try:
    r = s.get(url)
except requests.exceptions.SSLError:
    r = s.get(url, verify=False)
```


Just wanted to mention that both implementations don't work. I always get `ConnectionError: HTTPSConnectionPool(host='hostenameofwebsite', port=443): Max retries exceeded with url: / (Caused by <class 'httplib.CannotSendRequest'>: )`.


That has nothing to do with HTTPS, and everything to do with the `httplib.CannotSendRequest` exception. I wonder if we leave the connection in a bad state if we fail the SSL validation.


Do you want me to open a new issue for this?


Not sure, let's sit on it here for a moment.


So I don't seem to be able to reproduce it, which is a little odd.


```
url = 'https://torrentshack.net'
s = requests.Session()

try:
    r = s.get(url)
except requests.exceptions.SSLError:
    r = s.get(url, verify = False)
```


Works fine for me. =)
Sending cookies even though I specify None

Sorry if this is a stupid question, but I _think_ it could be a bug. Is there any reason requests is sending cookies with this request?

``````
>>> headers = {'Cookie': None}
>>> r = requests.get('http://www.linkedin.com/in/xxx/', headers=headers)
>>> print r.request.headers
CaseInsensitiveDict({'Accept-Language': 'en-US,en;q=0.8', 'Cookie': 'bcookie="v=2&49d15313-08b9-45e7-894f-8f5c76614c04"; lidc="b=VB61:g=66:u=1:i=1401763223:t=1401849623:s=1820628467"; moflow_session_id=1465875475464192', 'Accept-Encoding': 'gzip, deflate, compress', 'Accept': '*/*', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D167 Safari/9537.53'})```
``````


The change in UA is to do with some other unrelated code, unrelated to cookies; just to be sure:

```
class Session(requests.sessions.Session):
    def __init__(self, *args, **kwargs):
        super(Session, self).__init__(*args, **kwargs)
        self.headers['Accept-Language'] = 'en-US,en;q=0.8'

        if os.path.isfile('useragents.txt'):
            self.headers['User-Agent'] = random.choice([l.strip() for l in open('useragents.txt')])
```


It looks to me like there may be a bug where the response's 'Cookie' field is found in request.header, rather than in the response header as expected.


No, neither of these are bugs. =)

The reason you're sending a cookie is because you got redirected: see `r.history` for evidence of it. This redirect set a cookie, which we are now sending back. As for there being a bug where the response's `Cookie` field is found on the request header, also no. The request that caused the specific response we're looking at _did_ have a cookie header on it.


@Lukasa Makes sense, thankyou.
Add HTTP 2.0 support

It would be great if requests could make HTTP 2.0 requests.


It can: http://hyper.readthedocs.org/en/development/quickstart.html#requests-integration


As a follow-on, I want hyper to be added to urllib3, which is being tracked in Lukasa/hyper#18. This'll give requests transparent HTTP/2 support.


Awesome, thanks.

On Mon, Jun 2, 2014 at 10:54 AM, Cory Benfield notifications@github.com
wrote:

> As a follow-on, I want hyper to be added to urllib3, which is being
> tracked in Lukasa/hyper#18 https://github.com/Lukasa/hyper/issues/18.
> This'll give requests transparent HTTP/2 support.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2082#issuecomment-44869767
> .

## 

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084
Cookie deletion test missing

Hello!

I _think_ Cookie deletion (servers sending "Set-Cookie: name=;") is not working in requests. I don't see any tests using "/cookies/delete?name" from httpbin so it may not be tested yet?

Thanks!


@sylvinus 100% of that level of cookie handling is handled by the standard library (cookielib). We simply implement a few extra methods on top of the standard Cookie Jar.

A quick test shows that this does work as expected:

``` pycon
>>> import requests
>>> s = requests.Session()
>>> s.get('https://httpbin.org/cookies/set?name=value')
<Response [200]>
>>> s.cookies
<<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='name', value='value', port=None, port_specified=False, domain='httpbin.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={}, rfc2109=False)]>
>>> s.get('https://httpbin.org/cookies/delete?name')
<Response [200]>
>>> s.cookies
<<class 'requests.cookies.RequestsCookieJar'>[]>
```

Do you have a problem with some other website?


Okay thanks for the sample! I think this should still be included in requests tests since it's a behaviour that depends on the Session object.

That was actually the source of my confusion:

```
>>> requests.get("https://httpbin.org/cookies/set?name=value").history[0].cookies
<<class 'requests.cookies.RequestsCookieJar'>[Cookie(version=0, name='name', value='value', port=None, port_specified=False, domain='httpbin.org', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={}, rfc2109=False)]>
>>> requests.get("https://httpbin.org/cookies/delete?name").history[0].cookies
<<class 'requests.cookies.RequestsCookieJar'>[]>
```

I would have expected in the second case to have a way to see an empty cookie was sent (besides using a Session or manually inspecting the headers, both of which I don't want to do).


Uh...how would you expect to see a _deleted_ cookie? I mean in terms of interface?


I guess that's an open question :-) I manage a list of cookies separately and need a way to see if the remote server tried to delete one of them. Maybe the only way to do that properly is to write a custom cookie jar? 


@sylvinus what you seem to need is to either use a lower level solution so you can pass the proper class to the cookielib methods or to write a custom cookie jar that will be able to use a Response instance from requests retrieve all of the information it needs. If you need further help with this, I would suggest asking a question on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests).

Regarding adding a test I'm -0, because the tests are already slow enough.


I think a custom cookie jar is the way forward here. =)
Basic Auth handler sets unicode header value on Python 2

Observe:

``` python
>>> import requests
>>> requests.__version__
'2.3.0'
>>> r = requests.Request('GET', 'http://www.google.com/', auth=('user', 'pass'))
>>> p = r.prepare()
>>> p.headers['Authorization']
u'Basic dXNlcjpwYXNz'
```

This is wrong: all header keys and values should be native strings. Easily fixed though. =)


Interesting...


Alright, fix is now attached.


:cake:
How to run app on Windows?

How to run the Flask app on Windows? (I want to test my fork)

Following the `Procfile` I first tried `gunicorn httpbin:app -w 6`, but that failed because Gunicorn doesn't run on Windows.

Next I tried `python httpbin/core.py` but that failed too 

> Traceback (most recent call last):
>   File "httpbin\core.py", line 24, in <module>
>     from . import filters
> SystemError: Parent module '' not loaded, cannot perform relative import

Any ideas?


I think you meant to open this on the httpbin repository, not this one. =)

Nevertheless, try `python -m httpbin.app`.


Oops, yes. Thanks, `python -m httpbin.core` worked. What does the `-m` flag do?


Stack Overflow will explain better than I could: https://stackoverflow.com/questions/7610001/what-is-the-m-switch-for-in-python
Newline problems in Windows

Calls to `requests.put([...], data=file_obj)` send invalid requests on Windows when `file_obj` is a file object that contains Windows-style newlines (`\r\n`). This does not repro in Mac.

The problem is that the `Content-Length` header is determined through the `super_len` function, which in turn calls `os.fstat(o.fileno()).st_size`. This return value, which comes from the actual file size in the disk, does not need to match the length of the contents that get read through the file object, since Python will perform newline normalization in Windows:

``` python
>>> import os
>>> f = open('test.md') # this file contains Windows-style newlines
>>> f.fileno()
3
>>> os.fstat(3).st_size
4031L
>>> c = f.read(4031)
>>> len(c)
3983
>>> c[:50]
'# Cosmos "Little Joe" (1G) Hardware spec.\n\nThe fir'
```

In Mac, on the other hand, the content is read without newline normalization:

``` python
>>> f = open('test.md') # this file contains Windows-style newlines
>>> c = f.read(4031)
>>> c[:50]
'# Cosmos "Little Joe" (1G) Hardware spec.\r\n\r\nThe f'
```

The consequence of this mismatch is that the HTTP request will have a body which is smaller than what the Content-Length says, so the server will wait for the rest of the content and it will eventually give up.


You can repro this on a Mac by forcing Python to normalize newlines through the `'U'` file mode (https://docs.python.org/2/library/functions.html#open): `open('test.md', 'rU')`


Thanks for raising this issue!

This is not entirely surprising: you've not passed Requests the file in its on-disk form. Instead, you're applying transformations to it (newline normalisation is one, but decoding to unicode is another). This is problematic because it means the information about the file is now untrue.

Bear in mind that we send the file on the network _as bytes_. Given that the file exists on disk _as bytes_, the way to send it across the network intact is to send it _as bytes_. Use `open('test.md', 'rb')`.


Many thanks for the ultra-quick reply! The solution you propose works like a charm! :+1:

It could be nice to raise an error on the `requests` side if the amount of bytes read from the file object does not match the expected length. It'll be easier for other devs to pinpoint what went wrong if that kind of error is added.

Thanks again for your help!


@edrevo That's a great idea. Unfortunately, it's quite challenging to do, as we don't normally read from the file ourselves, we pass it down to the lower layers. An alternative might be to adjust [this section of the docs](http://docs.python-requests.org/en/latest/user/advanced/#streaming-uploads) to use `'rb'` as well, to reduce the risk of people getting this wrong.


Sounds good!


Ok, fixed up in 785e4ab3606be275f6887aab718ab7cda4a8938d. Thanks again for reporting this!
On Python 3, sending bytes automatically choses 'x-www-form-urlencoded' as the content_type

Is this on purpose. I would have assumed bytes to mean "send this unknown chunk of data as-is".


@Lukasa this is the actual behaviour. I tested on Python 2 and 3 and this only happens on Python 3 due to [these lines](https://github.com/kennethreitz/requests/blob/6c72509f5bb0e9cd8fad64a44ba99687ed044771/requests/models.py#L434..L439) I'm pretty sure we don't want this behaviour by default. I'm going to work on this some tonight and post my progress before heading to sleep.


#2072 fixes this issue. Thanks for reporting it @miracle2k! :cake:
Document and initialise Response.request

In response to #2066. Weirdly, we weren't initialising the `.request` property in the constructor. Nothing terrible there, but we need to do it to get it documented, so let's do it.


:clap: 


If this were only a documentation change I'd be pressing "Merge" right now. :+1: :shipit: 


Haha! I can't believe we haven't done this before :)

:cake:
Allow override of Authorization header in request

I have encountered a situation similar to #927.

I am creating a new request with an explicit `Authorization` header, e.g. 

```
posted = post('…', headers={'Authorization': '…'}, data=…)
```

Requests is finding and using an entry from a `.netrc` file I didn’t know existed; it appears to have been generated by the Heroku toolbelt. This overrides the Authorization header supplied in the `headers` dictionary.

Can the default behavior instead allow environmental factors to be overridden?

Also, can requests make this behavior in some way easier to discover? I am unable to find a way to retrieve the request headers actually sent; they would have provided a valuable clue about the mystery username & password.


Both of these problems in this case have solutions. To avoid using `.netrc`, use a `Session` and set `Session.trust_env` to `False`. To see what headers were sent _by requests_ (noting that the lower layers may add more), check `response.request.headers`.


This is a duplicate of #2062 and as such, I'm closing this


Thanks. @Lukasa, can `response.request` be added to the [Response documentation](http://docs.python-requests.org/en/latest/api/#requests.Response)?


@migurski Good idea, and done. See #2069.
Fix Python 3.2 compatibility in tests

The FAQ says that requests is compatible with Python 3.2. However, I could not execute the tests using that Python version. This PR fixes this and updated the FAQ to match.


I should have mentioned this above: I noticed that `tox.ini` was added to `.gitignore` in 7e825acd9b730632623984f14cd0f3f2c0cbf45f. I guess the author of that commit used `tox` himself to run the test suite when he decided to remove Python 3.1 and 3.2 from the trove classifiers.

If `tox` itself is unwanted, then I'll be happy to redo the changes without `tox` — the important point for me was to get support for Python 3.2 back to the test suite.


I've updated the pull request by taking out the tox related changes (the branch name should be the only trace now).


:+1: I'm in favor of this change. The only thing that would be better would be if @kennethreitz added support to Jenkins for 3.2, 3.4, and pypy 2.2 :)


@sigmavirus24 noted.
request headers are incomplete

Requests automatically adds a host header if the user didn't specify one. That header is not set in the request object. Example:

```
>>> import requests
>>> ret = requests.get('http://www.google.com')
>>> ret.request.headers['host']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/requests/structures.py", line 77, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'host'
```

This means that the headers in this object do not actually match what was sent on the wire, which is magic. Inside of poolmanager in _set_proxy_headers the accept and host headers are set, is there some way to make them set in that request object as well?


Thanks for raising this issue!

Actually, `_set_proxy_headers` is only used when routing via a proxy, and implements custom logic for that case. The section of the code that normally sets these headers is actually down in `httplib`, way down in the standard library.

We've never guaranteed that the request object represents exactly what was sent on the wire, and we cannot meaningfully do it. Requests is abstractions built on abstractions, and at least one of those abstraction layers is outside of our control (`httplib`). We simply cannot prevent `httplib` changing our requests however it sees fit.

We could take ownership of the `Host` header, but I'm not really convinced it gains us much, aside from allowing us to introduce bugs.


Ah, that makes some sense. Well, if the http library doesn't expose the headers it sent then there really isn't a good way of doing this, just means I'll have to add some logic into my testing code that someone magically adds some headers.
remove unused IteratorProxy

It's undocumented, unused and basically uninteresting. It looks like it was added in error in this commit https://github.com/kennethreitz/requests/commit/ef8563ab36c6b52834ee9c35f6f75a424cd9ceef. Or at least the interesting bit was factored out into `super_len`.


Yup, that looks unneeded to me. Thanks! :cake:


:+1: 


:camel: 

Not a cake. I'm sure why.
SSLError on GAE

Hi,

When I try to run this, I get an SSL error:

``` python
import requests
full_url = u'https://qascore.percolate.com/api/pull_request/'
requests.get(url=full_url)
```

```
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    requests.get(url=full_url)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 382, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 8] _ssl.c:507: EOF occurred in violation of protocol
```

I'm using `requests==2.3.0`.
I don't have the error when using http of course.


It appears this does not only happen on GAE. Investigating...


```
~$ openssl s_client -connect qascore.percolate.com:443
CONNECTED(00000003)
12470:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:/SourceCache/OpenSSL098/OpenSSL098-50/src/ssl/s23_lib.c:182:
```

OpenSSL seems to be having trouble with this site too although curiously Python 3 can get a response easily.


This is an SNI bug. Follow the steps [here](https://stackoverflow.com/questions/18578439/using-requests-with-tls-doesnt-give-sni-support/18579484#18579484) and you should be fine. =)


Thanks, `ndg-httpsclient`and `pyasn1` where enough to fix the pb.
Exceptions have other exceptions as messages.

I could be wrong but this seems very strange behaviour to me. I'm referring to the end of `adapters.py` that passes `urllib3` exceptions to `requests` exceptions as the first argument.

This gets very confusing since `exception.message` is another exception. In my case it broke error reporting tools that were expecting a string.

```
ipdb> exception
ConnectionError(MaxRetryError("HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /users/authentication (Caused by <class 'socket.error'>: [Errno 61] Connection refused)",),)
ipdb> exception.message
MaxRetryError("HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: /users/authentication (Caused by <class 'socket.error'>: [Errno 61] Connection refused)",)
ipdb> type(exception.message)
<class 'requests.packages.urllib3.exceptions.MaxRetryError'>
```

It would seem much more appropriate to me to replace `raise ConnectionError(e)` with `raise ConnectionError(e.message)`. This provides all the same information while still conforming to the Python convention of `raise Exception("string message")`.


It is not the most common behaviour ever but it is in fact a very deliberate design decision. We wrap all errors provided to us by urllib3 (and even some that bubble up from underneath urllib3) and present them to the user in a predictable way. However, for the more advanced user, being able to inspect those lower level error messages (and in fact when trying to help users it is very useful for us as well) is invaluable. You very easily ensure that `e.message` is a string by calling `str` on it if you can manage that yourself. That aside, I consider it your error reporting tools fault for not being more paranoid about the data it collects from user's code.

I appreciate that you took the time to raise this issue @fletom but I don't think we're going to change how we handle exceptions.

Cheers!


I understand that it's useful to retain the original exception. In my opinion, it would be much nicer to have this appear as a secondary attribute on the exception object (e.g. `e.original_exception`) instead of breaking Python exception conventions. Semantically speaking, a "message" is obviously a string, not another exception.

But I'll leave it at that. Thanks for taking time to write a detailed response.
Python 3 - TypeError: decoding str is not supported

I am getting the above error thrown when I attempt to get the request text when the content is a dict-string (ie. '{}'). I've found that it is this line, but I am not sure how to fix it at this point:

https://github.com/kennethreitz/requests/blob/master/requests/models.py#L738

If you were to remove the `errors` keyword, it works fine. I believe it's because the content is not returning as bytes, as `str` expects.

https://docs.python.org/3.3/library/stdtypes.html#str

Not sure how to fix this yet.

Note: if you want to replicate this, use Python 3 and set content to '{}' and try to return `text`


Thanks for raising this!

Content must be bytes, always. If you're writing Transport Adapters or generally tweaking requests, you must ensure that content will always be bytes.


I'm seeing this when using requests-oauthlib with facebook. Is this
something that will be fixed in requests? Might even look into it myself
tomorrow.
On May 20, 2014 12:11 AM, "Cory Benfield" notifications@github.com wrote:

> Thanks for raising this!
> 
> Content must be bytes, always. If you're writing Transport Adapters or
> generally tweaking requests, you must ensure that content will always be
> bytes.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/kennethreitz/requests/issues/2052#issuecomment-43508779
> .


Again, let's be clear: the idea that content will be bytes is a deliberate design decision, and so will not be 'fixed'. After all, it's intended behaviour. If there's a bug it's in requests-oauthlib.


No problems. I'll get a PR to requests-oauthlib. Thanks for clarifying it.
Content length value is written as unicode rather than bytes

Using the simple following snippet:

```
>>> files = {'myfile': ('filename.zip', file('some.zip', 'rb'), 'application/octet-stream')}
>>> r = requests.Request('POST', URL, files=files).prepare()
>>> r.headers
{'Content-Length': u'395', 'Content-Type': 'multipart/form-data; boundary=ec38bda3760c46aea548602228efdd47'}
```

This is an issue because httplib will consider it must decode the body as well before sending it (at least in Python 2.7.1, I assume with 3.x it's all bytes anyway). This fails when content is binary.

Massaging the Content-Length to revert it to a byte string does work well.

This was with requests 1.1.0.


This is long-since fixed, with the fix originally in version 2.1.0. This is mentioned in [the release notes](https://github.com/kennethreitz/requests/blob/master/HISTORY.rst#210-2013-12-05), was tracked under issue #1688 and was fixed in issue #1699. =)


My bad.


No worries @Lawouach ! We appreciate that you took the time to report it


Thanks. My previous comment was a bit short, sorry. Thanks for the swift reply and, really, I should have looked at the release notes. Thanks guys :)
add equivalent to adapter 'init_poolmanager' method for proxy-based connections

I'm trying to make a HTTPAdapter which injects `source_address` - and after doing my own found https://github.com/kennethreitz/requests/issues/2008#issuecomment-40793099 which it is virtually identical to.

However this fails when proxy configuration is in use as `self.poolmanager` is not used.  I suspect overriding `get_connection` is required to accomplish this now; it would be nice if https://github.com/kennethreitz/requests/blob/59c8d81/requests/adapters.py#L210-L215 could become another method similar to `init_poolmanager`, so proxy-based connections could be more easily 'adapted'.

Happy to attempt a PR if this idea has merit / is considered 'in-scope' :-)


I think there's no good reason not to do this, and it would definitely make this easier. @sigmavirus24?


I would go one step further than the lines you highlighted. That code in `get_connection` could read:

``` python
        if proxy:
            proxy = prepend_scheme_if_needed(proxy, 'http')
            proxy_manager = self.proxy_manager_for(proxy)
            conn = proxy_manager.connection_from_url(url)
```

With the method

``` python
def proxy_manager_for(self, proxy):
    if not proxy in self.proxy_manager:
        proxy_headers = self.proxy_headers(proxy)
        self.proxy_manager[proxy] = proxy_from_url(
                                        proxy,
                                        proxy_headers=proxy_headers,
                                        num_pools=self._pool_connections,
                                        maxsize=self._pool_maxsize,
                                        block=self._pool_block)
    return self.proxy_manager[proxy]
```


And yes I'm :+1:. Go for it @codedstructure
Incorrect encoding detected when no encoding is given

On https://mozorg.cdn.mozilla.net/media/caldata/FrenchHolidays.ics, the encoding is set to `ISO-8859-1`, although no encoding was given in the headers, which is why i think it should be `None` instead.


This is well-known behaviour in requests, and comes right out of RFC-2616. We go into it [here](http://docs.python-requests.org/en/latest/user/advanced/#encodings) in the documentation, but in short: RFC 2616 says that if a Content-Type is received that is of type `text/*` (any text content type) but no charset is present, we must assume `ISO-8859-1`.


Is there a way to detect this fallback as a user of requests?


Sure:

``` python
fallback_to_latin1 = (not 'charset' in r.headers['Content-Type']) and (r.headers['Content-Type'].startswith('text/')
```


Okay, thanks!


With http://evertpot.com/http-11-updated/, could there be any change on this?


@untitaker Way ahead of you, see #2086. =)
Update urllib3 to 4fb351cd2


:+1: :cake:

@kennethreitz Just so you're aware, the reason this has been opened is because of shazow/urllib3#385, which is a nastly TLS-related bug on Python 3.4.1. This will be a big problem for `pip` in addition to our users, so we'll want to think hard about pushing a new release. Let me know if you decide to do that soon and @sigmavirus24 and I will tidy up the code ready for a release.


Yes, please. Right now 3.4.1 is in rc1 and I'd really like to get a new pip release out before that lands with the fixes in this so that 3.4.1 can ship with this fixed.


:sparkles: :cake: :sparkles:


@Lukasa @sigmavirus24 alright, let's get a release ready. :)
content is different: r.json() and r.text 

Hi,
a question shows in following, content of r.json() is not the same with r.text (only the last element compared to r.text in this example).
I thought that it may comes from streaming, but it still the same even I set stream=False. 
Do I have any misunderstanding to this method?

``` python
>>> r.text
u'{"responseHeader":{"status":0,"QTime":646},"success":{"127.0.1.1:8983_solr":{"responseHeader":{"status":0,"QTime":87}},"127.0.1.1:8983_solr":{"responseHeader":{"status":0,"QTime":42}},"127.0.1.1:8983_solr":{"responseHeader":{"status":0,"QTime":36}},"127.0.1.1:8983_solr":{"responseHeader":{"status":0,"QTime":128}},"127.0.1.1:8983_solr":{"responseHeader":{"status":0,"QTime":123}}}}\n'
>>> r.json()
{u'responseHeader': {u'status': 0, u'QTime': 646}, u'success': {u'127.0.1.1:8983_solr': {u'responseHeader': {u'status': 0, u'QTime': 123}}}}
```


No, you've just been sent some awkward JSON. =)

Let me format the JSON to show you what I mean:

```
{
  "responseHeader": {"status":0,"QTime":646},
  "success":
  {
    "127.0.1.1:8983_solr": 
    {
      "responseHeader": {"status":0,"QTime":87}
    },
    "127.0.1.1:8983_solr":
    {
      "responseHeader":{"status":0,"QTime":42}
    },
    "127.0.1.1:8983_solr":
    {
      "responseHeader":{"status":0,"QTime":36}
    },
    "127.0.1.1:8983_solr":
    {
      "responseHeader":{"status":0,"QTime":128}
    },
    "127.0.1.1:8983_solr":
    {
      "responseHeader":{"status":0,"QTime":123}
    }
  }
}
```

Note that all four keys in the 'success' dictionary are the same. The JSON specification does technically allow this, but most implementations don't. Python turns a JSON 'object' into a dictionary, which cannot have repeated keys, so r.json() doesn't have them either. [More info here](http://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object).


Got it, thanks a lot.
Does not work with Microsoft ISA Firewall Client for ISA Server

`Requests` does not work if I try to use it in a corporate network with Microsoft ISA Server when I use ISA Firewal Clent. I do not set any proxy settings, but `requests` despite it have looked proxy. This behavior has been defined by default for `trust_env` in the module `sessions` in the line range 423-427 (https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L423). 

``` python
proxies = proxies or {}

        # Gather clues from the surrounding environment.
        if self.trust_env:
            # Set environment's proxies.
            env_proxies = get_environ_proxies(url) or {}
            for (k, v) in env_proxies.items():
                proxies.setdefault(k, v)
```

In my network function `get_environ_proxies` returns a list of proxy servers, and it does not work for me. For correct work via ISA Firewal Clent, dict of proxies must be empty always.

For example, this behavior prevents to use `pip` (see https://github.com/pypa/pip/issues/1182#issuecomment-41282819).


Let me try to understand what your problem is. You want requests not to explicitly consult a proxy (presumably because the ISA server proxy is transparent).

The environment proxies are gathered in two ways: firstly, by consulting environment variables (`HTTP_PROXY` and `HTTPS_PROXY`), and then by consulting the OS configuration. You have the following options:
1. If you have `HTTP_PROXY` or `HTTPS_PROXY` set, unset them.
2. Set the `NO_PROXY` environment variable to include hosts you'd like us not to proxy to (which ought to fix most of `pip`).
3. Set `trust_env` to `False`.

Note that `get_environ_proxies` ends up calling `urllib.getproxies()`. If that's returning any proxies on your system, it means your system is explicitly configured to talk to those proxies.


@Lukasa Yes, I want requests not to explicitly consult a proxy. I use ISA Client for proxy-transparency.

I do not set the environment variables `HTTP_PROXY` or `HTTPS_PROXY`. Other programs work transparently with ISA Server proxy via ISA Client.

Thank you for info about `NO_PROXY=1`. It works! My troubles are ended for "python networking". :)


Glad to hear it. =)

I wonder whether ISA Client adds information to the registry that's returned by `getproxies()`. Something worth investigating in the future.


You may download the tar.gz/zip file and pass it to the pip install
command.
(it's my workaround behind corp proxy)

On Mon, Oct 26, 2015 at 12:43 PM, Fan Du notifications@github.com wrote:

> @espdev https://github.com/espdev Hi, I've been reading all posts about
> this problems by you. Followed your instructions but anyhow couldn't make pip
> install work behind NTLM. :(
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/kennethreitz/requests/issues/2036#issuecomment-151121746
> .
redirect will not keep the Cookie field in headers

i set cookie in headers, and pass the headers argument to requests.get, but when it happens redirect, the Cookie in headers will be deleted(i found it in the source code)


We don't expect cookies to be set that way, we expect you to use the `cookies` argument to the request or to use the `Session.cookies` property. =)


got it, i know this usage, but want one more choice~


There should be one obvious way to do it, and preferably only one way to do it. In other words, I'm 100% against adding another way to set cookies. 


by the way 'Cookie' is  just one of the header fields, same as other header fields ,  I think it's usage should be more flexible, I just express my mind, don't be angry with that :) 


The policy here is: Requests should treat headers as opaque. We assume that if you set any header yourself you are going to take control of knowing what it means and how to work with it. =)


@paramiao I'm not angry with you voicing your opinion. I was just voicing mine. I'll elaborate a bit more:

Cookies can be set during a redirect. We keep track of those and send them on subsequent requests (if appropriate). We cannot do that regardless of whether you set the prior cookie header or not. We construct the `Cookie` header ourselves when necessary and happily delete it on redirects only to reconstruct it.

If you want the header you set to persist, I suggest you take control of redirects using `allow_redirects=False`. This is the only way requests can and will work the way you expect it to.
Requests 2.3 not available on pypi.

Hi.

It looks like requests 2.3 is done. The [RTD](http://docs.python-requests.org/en/latest/) page refers to 2.3 and the __init__ file has been on version 2.3 for the last 3 months.  Maybe 2.3 is not done and the version increment was wrong? Either way it'd be good to have clarity on this.


The version increment was wrong, and slipped through in an over-keen pull request. We initially elected to let that go believing that we were going to release soon, but it's been more than a month now so I'll change that back.

For future reference, we will never 'ship' a release that doesn't go out on PyPI. Whatever you get from PyPI _is_ the most recent release. =)


Thanks for the rapid clarification. I thought you might have somehow created a release and forgotten to ship it. Weird, but I suppose possible. 

We had a similair issue in [PRAW](https://github.com/praw-dev/praw) where a change in documentation mismatched the actual latest version. This happened when we added a feature with documentation, "latest" on RTD then showed documentation about a non-existing feature. Which was silly. We fixed this by creating a "release" branch, that always matched the last shipped version (part of deploy script) and pointing RTD at it. Made all problems like these disappear. Maybe something for you as well?


Done. =)


Yeah, that's just not how requests does it. We used to, but haven't done in a long time: it adds lots of extra process to the code itself.

I do it in [hyper](http://hyper.rtfd.org/en/latest), but the overhead here is just not something we want. =)


Well that's up to you, just wanted to suggest it. Anyway thanks for the rapid response and fixing.
Unable to Post the Json Object

Hi I am trying to post the request using python. I would like to what is wrong with my code. Can you let know what is correct way of doing it

Python Code :(Does not work)
data ={'object':{'login':{'username':'nsroot','password':'nsroot'}}}
headers = {'Content-Type': 'application/x-www-form-urlencoded'}
response =requests.post(url,data=json.dumps(data))

Response :+1: 
Invalid POST request

Rest Call : (Works)
POST /nitro/v1/config/ HTTP/1.1
Host: 192.168.1.11
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded

object = {'login':{'username':'nsroot','password':'nsroot'}}


Thanks for raising this issue!

You're not POSTing quite the same thing as your example. Your example is actually a x-www-form-urlencoded body which happens to have a JSON value in one of the values, but you're posting a wholly-JSON encoded body. Try this:

``` python
data = {'object': json.dumps({'login':{'username':'nsroot','password':'nsroot'}})}
response = requests.post(url, data)
```


Thanks..  It works   :)-Pa1

Date: Fri, 2 May 2014 23:56:03 -0700
From: notifications@github.com
To: requests@noreply.github.com
CC: strap.16@hotmail.com
Subject: Re: [requests] Unable to Post the Json Object (#2033)

Thanks for raising this issue!

You're not POSTing quite the same thing as your example. Your example is actually a x-www-form-urlencoded body which happens to have a JSON value in one of the values, but you're posting a wholly-JSON encoded body. Try this:

data = {'object': json.dumps({'login':{'username':'nsroot','password':'nsroot'}})}
response = requests.post(url, data)

—
Reply to this email directly or view it on GitHub.
Session proxies not used

In:

s = requests.Session()
s.proxies["http"] = ""
s.proxies["https"] = ""
s.get("http://server/",proxies=s.proxies)

I'd expect "proxies=s.proxies" to be redundant.


What makes you think it isn't?


Guess ...


I shouldn't have to guess. =) Bug reports are not a guessing game, they're a report of a fault: you should say what you did, what you expected to happen, and what actually happened.


I thought I did exactly that.
Without the proxies it would use my environment variables.


Right, there we go. You didn't say that, you just wrote some code down and then said you expected it to be redundant. That doesn't tell me what was _wrong_. =)

Yes, this has been identified already in the very poorly named issue #2018. I'm planning to fix it this weekend.

Thanks for raising the issue! I'm closing to focus discussion on #2018.


Thanks
Is it possible to specify an SSL timeout?

Not really an issue I guess,

But I'm noticing on some logs that I'm receiving an SSLError 'The read operation timed out' just over a second after all the data has been read and uploaded from a file during a PUT upload request.

I've tried increasing the timeout by passing a higher value requests.put(url, filename, timeout=60), but I guess this isn't getting passed to the SSL part of the underlying library? as it still times out at just over a second as it did before.

Is there a way to increase the timeout for the SSL part of the request from 1 second (it appears to be 1 second anyway)


I've honestly got no idea. @t-8ch, does the underlying ssl library not time out based on the socket timeout?


@GP89 which SSL backend and Python interpreter are you using?
I can't reproduce a SSLError on read timeout, only with a handshake using the stdlib. The handshake with PyOpenSSL never times out...


I'm not sure, how would I check? PyOpenSSL I would assume though.
That's odd, I was seeing it a lot and the timeouts were always raised just over a second after the call was made, which made me think it was likely that there was a timeout value set somewhere.


Sorry for the long delay, I am travelling...
Ha, a usecase for #2023.

Could you show me the result of running https://gist.github.com/t-8ch/4ecfd78a502b7a616028?

I used `ncat -kl --ssl 0.0.0.0 4000` and `requests.get('https://localhost:4000')` to reproduce your issue but I could not get the same error as you. Could you try those commands and tell me the outcome?


Closing for inactivity.
Chunk Size - Question -- Not issue

Is there any guideline on selecting chunk size?

I tried different chunk size but none of them give download speed comparable to browser or wget download speed

here is snapshot of my code

r = requests.get(url, headers = headers,stream=True)

total_length = int(r.headers.get('content-length'))
if not total_length is None: # no content length header
        for chunk in r.iter_content(1024):
            f.write(chunk)

Any help would be appreciated.?


Hi @gauravp2003,

Much of the documentation around requests, including the [README](https://github.com/kennethreitz/requests#contribute), suggests that the issue tracker is for bugs or feature requests, not questions. If you have questions, the should be asked on [StackOverflow](https://stackoverflow.com/questions/tagged/python-requests). @Lukasa, someone else, or I will answer it there.

Thanks for your interest in requests,
ImportError: No module named 'requests.packages.urllib3.util'

I'm using Python 3.4. I get this error from `import requests`:

```
Traceback (most recent call last):
  File "devwiki.py", line 4, in <module>
    import requests
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.3.0-py3.4.egg/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.3.0-py3.4.egg/requests/utils.py", line 25, in <module>
    from .compat import parse_http_list as _parse_list_header
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.3.0-py3.4.egg/requests/compat.py", line 7, in <module>
    from .packages import chardet
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.3.0-py3.4.egg/requests/packages/__init__.py", line 3, in <module>
    from . import urllib3
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.3.0-py3.4.egg/requests/packages/urllib3/__init__.py", line 16, in <module>
    from .connectionpool import (
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.3.0-py3.4.egg/requests/packages/urllib3/connectionpool.py", line 36, in <module>
    from .connection import (
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/requests-2.3.0-py3.4.egg/requests/packages/urllib3/connection.py", line 43, in <module>
    from .util import (
ImportError: No module named 'requests.packages.urllib3.util'
```


@zakcalrissian how did you install requests? Your traceback seems to say that you have version 2.3.0 installed when no such version has been released.


I got an email about this as well. Installing directly from Github reveals this problem, but it's very specific: you need to install the package and then run from somewhere _other_ than the requests directory.

The problem is that #2017 updated urllib3 to a new version that includes a `util` module that is a directory, not a file. However, we didn't update `setup.py` to ensure that the new directory is installed, so import fails. I'll fix this up and raise a pull request shortly.


I had this problem too. Installing 2.2.1 from Pypi has resolved the problem for the moment.


Ok, the fix attached to this issue should fix the problem. =)


:+1: This is due to a refactor pull request introduced upstream.


It works in local. Thanks. 


:cake:


It works. Thank you!


Couldn't we just use [`setuptools.find_packages`](https://pythonhosted.org/setuptools/setuptools.html#using-find-packages) instead of the long list of subpackages?
JSON Upload

We download it but don't upload it. Asymmetrical. 

```
r = requests.post('http://httpbin.org/post', json={'life': 42'})
```


This is really easy to do. I'm curious though, what made you change your mind about this?


Tagging this as contributor friendly because it's perfect for @sigmavirus24 to pair with a new developer on.


cool!


Did anyone already start working on it ? If not, I would be interested in giving it a try ...


@willingc and I are working on this.


Is anyone still working on it? Otherwise I will be happy to work on this as my first issue ever. Please let me know if I can contribute. :)


## Yes they are. Please be patient. 

Sent from my Android device with K-9 Mail. Please excuse my brevity.


I've just been doing `requests.post('http://httpbin.org/post', data=json.dumps({'life':42}))`; is there some limitation I'm missing about doing that way?


@gbromios That's the only way to do it at the moment. Note, however, that the Content-Type is wrong. =)


ah, so convert to a string and override the Content-Type header in one step. That seems reasonable.


Yeah — basically if we do it on one side (download), we should also do it on the other (upload). Nice and symmetrical.