Raibek/requests-github-issues huggingface dataset:
TypeError in MockRequest when 'Host' header set to bytestring

Issue is specific to Python3

An incorrect assumption is made in requests.cookies.MockRequest on Python3, [see here](https://github.com/kennethreitz/requests/blob/master/requests/cookies.py#L52):

```
class MockRequest(object):
    ...
    def get_full_url(self):
        # Only return the response's URL if the user hadn't set the Host
        # header
        if not self._r.headers.get('Host'):
            return self._r.url
        # If they did set it, retrieve it and reconstruct the expected domain
        host = self._r.headers['Host']
        parsed = urlparse(self._r.url)
        # Reconstruct the URL as we expect it
        return urlunparse([
            parsed.scheme, host, parsed.path, parsed.params, parsed.query,
            parsed.fragment
        ])
    ...
```

In Python2 this works fine where str and unicode objects are allowed interchangeably in urlunparse.

In Python3 this is an issue because `host = self._r.headers['Host']` may return a bytestring and `parsed = urlparse(self._r.url)` returns a ParseResult with str objects. Combining these two causes urlunparse to throw a fit.

Steps to reproduce:

```
>>> import requests
>>> a = requests.Request('GET', 'http://www.google.com', headers={'Host': b'www.google.com'})
>>> b = requests.cookies.MockRequest(a)
>>> b.get_full_url()
```


Yup, this looks wrong to me. It does look like we need to ensure that we have a match between the two types. In fact, we should probably ensure we're always returning the correct type for the platform.


@Lukasa 
I've attempted a patch of this issue in PR #3598 
Would appreciate your input on the patch.

:smile:


Will follow up with a fix for the dependency on httpbin.org. Please leave this open until then.
TooManyRedirects  error in session.py

i dont know if this has to do anything with this problem  ... but i'm kinda sending request to my own app from localhost to server (basically its the same app on both side  1 on the server and 1 on the localhost)

im using requests 2.11.1

```
    resp = requests.get('http://apineginpay.in/api/init')
    return HttpResponse( resp.text )
```

> TooManyRedirects at /pointer/init
> Exceeded 30 redirects.
> Request Method:   GET
> Request URL:  http://localhost:8000/pointer/init
> Django Version:   1.9.6
> Exception Type:   TooManyRedirects
> Exception Value:  
> Exceeded 30 redirects.
> Exception Location:   c:\Python344\lib\site-packages\requests\sessions.py in resolve_redirects, line 110
> Python Executable:    c:\Python344\python.exe
> Python Version:   3.4.4
> Traceback Switch to copy-and-paste view
> 
> c:\Python344\lib\site-packages\django\core\handlers\base.py in get_response
>                     response = self.process_exception_by_middleware(e, request) ...
> ▶ Local vars
> c:\Python344\lib\site-packages\django\core\handlers\base.py in get_response
>                     response = wrapped_callback(request, _callback_args, *_callback_kwargs) ...
> ▶ Local vars
> C:\wamp\www\djangoapp\pointer\views.py in init
>     resp = requests.get('http://apineginpay.in/api/init') ...
> ▶ Local vars
> c:\Python344\lib\site-packages\requests\api.py in get
>     return request('get', url, params=params, *_kwargs) ...
> ▶ Local vars
> c:\Python344\lib\site-packages\requests\api.py in request
>         return session.request(method=method, url=url, *_kwargs) ...
> ▶ Local vars
> c:\Python344\lib\site-packages\requests\sessions.py in request
>         resp = self.send(prep, **send_kwargs) ...
> ▶ Local vars
> c:\Python344\lib\site-packages\requests\sessions.py in send
>         history = [resp for resp in gen] if allow_redirects else [] ...
> ▶ Local vars
> c:\Python344\lib\site-packages\requests\sessions.py in <listcomp>
>         history = [resp for resp in gen] if allow_redirects else [] ...
> ▶ Local vars
> c:\Python344\lib\site-packages\requests\sessions.py in resolve_redirects
>                 raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects, response=resp) 


i solved this by setting user-agent in the headers (i wasn't sending any headers before )
Fix RequestsCookieJar specific update call

This should solve the issue in #3579. `merge_cookies` performs the same `update` call inside of a try/except block which will fallback to the same logic that [`update`](https://github.com/kennethreitz/requests/blob/master/requests/cookies.py#L347-L353) itself implements for non-`RequestsCookieJar`-like objects.


Ok, per my note above I'm going ahead and merging this. Thanks @nateprewitt!
"LookupError: unknown encoding: idna" in worker threads, under embedded Python 3.5

The following snippet raises an exception on Windows, using the [embedded distribution of Python 3.5.2 x64](https://www.python.org/downloads/release/python-352/):

``` python
from concurrent.futures import ThreadPoolExecutor
import requests
def task(_):
    return requests.get("https://example.org").reason

executor = ThreadPoolExecutor(max_workers=2)
for x in executor.map(task, [(1,), (1,),]):
    print(x)
```

```
OK
Traceback (most recent call last):
  File "repro.py", line 7, in <module>
    for x in executor.map(task, [(1,), (1,),]):
  File "concurrent\futures\_base.py", line 556, in result_iterator
  File "concurrent\futures\_base.py", line 398, in result
  File "concurrent\futures\_base.py", line 357, in __get_result
  File "concurrent\futures\thread.py", line 55, in run
  File "repro.py", line 4, in task
    return requests.get("https://example.org").reason
  File "K:\PR\pybug\requests\api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "K:\PR\pybug\requests\api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "K:\PR\pybug\requests\sessions.py", line 461, in request
    prep = self.prepare_request(req)
  File "K:\PR\pybug\requests\sessions.py", line 394, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "K:\PR\pybug\requests\models.py", line 294, in prepare
    self.prepare_url(url, params)
  File "K:\PR\pybug\requests\models.py", line 361, in prepare_url
    host = host.encode('idna').decode('utf-8')
LookupError: unknown encoding: idna
```

This problem seems specific to the standard library being inside a zip file in the embedded distribution, as I couldn't reproduce this in regular distributions, or when Python35.zip is extracted out to Lib.

`import encodings.idna` succeeds. Doing this in the main thread prior to launching the workers will avoid this problem, as per http://stackoverflow.com/a/13057751. Though it looks like the `str.encode()` call is the culprit, changing my task function to be `'foo'.encode('idna')` doesn't trigger the exception.

I've tested this using Requests 2.10.0 and 2.11.1. This is also reproducible using `multiprocessing.pool.ThreadPool`.


@rbcarson Can you confirm for me: does adding `import encodings.idna` to your `task` function in your example also fix the problem?


@Lukasa Yes, it does.


Ok, so I think that means the solution is to have a no-op import. =)


If we put that import in the `models.py` file I think that fix will be sufficient. Let's put a comment on it though so we remember why we did it!


I've already started down this path in my own code. It seemed a bit shady to me, but if it's good enough for upstream... :smirk: 

Do you think this problem lies with the interpreter, and thus worth reporting to python.org?
Fix encoding issue in test_response_reason_unicode_fallback

`test_response_reason_unicode_fallback` is currently failing in Python 2. The original proposed fix in #3557 unfortunately didn't address the underlying issue.

While this patch should fix the test now, it may be worth taking a brief moment to discuss how we envision the original PR #3554 working. Currently an error retrieved from an `except` clause can't be cast as `str`, `unicode`, written to a file, or easily decoded in Python 2. It prints to the console fine, but that's about it. This seems like a semi-serious issue for making this exception usable. I need to run a few more tests this evening or tomorrow, so we can sit on this for the weekend unless someone has some obvious insight I'm missing.


Ok, I'm gonna call this good to go I guess. I feel like the behavior of the unicode string in an Exception is less than ideal but it's apparently one of the many long running unicode issues in Python 2.x.

For anyone reading this commit, trying to figure out how to do anything useful with the exception in Python 2, both of these should work:

``` python
unicode(e.message)  # unicode
unicode(e.message).decode('utf-8') # str
```


@sigmavirus24 Would you like to take a look at this?


@Lukasa I would like to pull it and run the tests locally. I suspect we might have a locale mismatch which is why this _appears_ to work for @nateprewitt and not for me. This is why I was using bytes explicitly previously.


Hey @sigmavirus24, while this could be a locale mismatch, there are 5 other tests in the same file using unicode characters in unicode strings like this (including \xf6). I _think_ I figured out why we were getting different results, and [detailed it](https://github.com/kennethreitz/requests/pull/3557#issuecomment-246908451), along with most of my other findings in #3557. I had @Lukasa confirm he could reproduce this issue before I opened the PR. Is this patch not working on your system?


Quoting myself:

> I would like to pull it and run the tests locally. 


> ...which is why **this** _appears_ to work for @nateprewitt and not for me.

Sorry @sigmavirus24, I interpreted "this" as meaning "this patch". I was just trying to clarify.


Ok, so this test as written passes for me, as do all the tests. My locale is:

```
(env) cory@heimdall:requests/ % locale
LANG="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_CTYPE="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_ALL=
```


Thanks @nateprewitt. Care to update proposed/3.0.0 too then?


Sure thing, I'll move the changes over sometime today.
added results of benchmark test about using session


Thanks for this! I'm inclined to want to avoid having a specific benchmark, and instead to speak in general terms. Something general about approximate improvement would be better. Are you interested in that change?


When I first time read this documentation with words `significant performance` about session my thoughts was `eh maybe ten milliseconds`, I did not know that for every https call without session server must send certificate to client and get response and that this is seconds but not milliseconds, I was surprised when maked lazy test of it and get this numbers. Why are you against benchmarks? It is good illustration, I love benchmarks in any kind :)


I'm not against benchmarks at all: I'm against benchmarks in _narrative documentation_. I'd be happy to have the benchmark in a `.py` file in the docs tree and then have the narrative docs just say what you said above with a link to that benchmark. It just doesn't need to be inline on the page, that's all.


There's been no response to this pull request since feedback was provided. We can reopen this if interest in it is renewed at any point.
convert all headers to bytes before invoking urllib3/httplib

related to #3177 and shazow/urllib3#855

It's really easy to get this wrong. See sam-washington/requests-aws4auth#24 for an example.


There is no requirement for us to pass urllib3 bytes. urllib3's interface has always allowed native strings on the platform, which includes unicode on Python 3. The specific error you're looking at is one that we have already fixed in urllib3, so a fix will be coming down the pipe in the next few months. =)


@Lukasa what is the change in urllib3 that will solve this issue?


It's shazow/urllib3#719
Swallow SSL errors raised by eventlet

Fixes #3104 (or the weird mutant version of it that happens if you use eventlet).

When eventlet monkey-patches ssl, the resulting `socket.read` calls throw an instance of the same class (`SSLError`) with a different message (`'timed out'` instead of `'read operation timed out'`). This causes the SSLError to not get swallowed by urllib3 or requests.

Please let me know if you'd prefer the issue to be fixed in another way, happy to adjust--this was just the clearest place to fix it.


Urllib3 is vendored with Requests, you'll want to submit your PR to [their repo](https://github.com/shazow/urllib3).


Yup, as @drpoggi said, Requests doesn't carry any patches to urllib3, we just use it as is. Please make this change to urllib3 and open a PR there. =)
ValueError encountered

https://dpaste.de/t0kx

More info available if needed.


@antlong Can you provide your code, please? Can you also tell us what version of Requests you're using, and where you got it from?


https://dpaste.de/zUtN

Requests 1.2.3 from pip.

Note: The urls in the file contain special characters.


So Requests 1.2.3 is _ancient_: I'm almost certain that we fixed this in the last two years. Please upgrade your Requests. =)


Ahhhh forgot i was on an older centos dev box running this. Thanks!
socks5 does not work well 

Hi. When I'm using proxy = {'http': 'socks5://IP:PORT', 'https': 'socks5://IP:PORT'}, I got "AssertionError: Not supported proxy scheme socks5". I had upgraded my requests( version:2.11.1)  and env is mac osx.


```
proxies = {'http': 'socks5://127.0.0.1:9050',
                   'https': 'socks5://127.0.0.1:9050'}
requests.get(url, proxies=proxies)
```

it works for me. I suggest you debug to find error.


@PegasusWang This suggests that your PySocks isn't working well. `pip install -U pysocks==1.5.6` should resolve your problem.
Start without network hang-up

Hi, 
I am trying to handle and recovery intermittent-network conditions while using requests.

I tried this very-simple script:
https://gist.github.com/lrodorigo/30b00c20f5994aaf2911d7443b8f1f15

And i noticed as follows:
- if the script is started with _wifi-on_ it works as expected (Connection Alive and exception traceback printed coherently with network status)
- if the script is started with **wifi-off** it continously raise **ConnectionError: HTTPConnectionPool(host='www.google.com', port=80): Max retries exceeded** exception, **even if I connect to the internet while it's running**.

I'm running on Arch Linux Kernel 4.5.4-1-ARCH, Python 2.7.11 x64, Requests v2.10.0

Anyone experienced this kind of issue?


This seems like it may be related to #3223. I don't think there was any resolution with that issue though.


It's my strong belief that this is a behaviour outside of Requests manifesting. If you aren't using a `Session`, there is essentially no persistent state across requests.  Can you try your script with urllib2 to confirm whether you experience the same issue?


I am not using Session.
I experience same issue also with urllib2.

I just work-rounded with a custom error exit-code  and a shell script that
monitor it and restarts my program in a loop until, but it's really ugly.

2016-09-11 19:15 GMT+02:00 Cory Benfield notifications@github.com:

> It's my strong belief that this is a behaviour outside of Requests
> manifesting. If you aren't using a Session, there is essentially no
> persistent state across requests. Can you try your script with urllib2 to
> confirm whether you experience the same issue?
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/kennethreitz/requests/issues/3567#issuecomment-246191585,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AIJ30Pwwu4PL2X0CjVW7uRVOMTfnUI-wks5qpDcigaJpZM4J6C7F
> .

## 

_Luigi R._


If you experience the same issue with urllib2 then there is very little we can do to help you. Given that I tried to reproduce this on OS X and was unable to, it seems to be a problem with the way the Linux networking stack is configured. You'll need to pursue debugging it with the Linux folks or your OS vendor.
behavior of encoding

Hello,

I am having a terrible time figuring out what requests is doing here.

  ```  r = requests.get('http://0.0.0.0:5006/buy', params=payload)
   print(r.url) #debug
   print(r.encoding)
   print(r.text)
   print(r.headers)

``````
debugging commands in console produce:
```payload from cli is {"key1": "foo", "key2": "bar"}
payload in function is {"key1": "foo", "key2": "bar"}
http://0.0.0.0:5006/buy?%7B%22key1%22:%20%22foo%22,%20%22key2%22:%20%22bar%22%7D
ISO-8859-1```

The URL looks wrong to me, is it somehow being created incorrectly?

``````


The URL is right; its urlencoding the string payload you gave it. You cannot just send arbitrary data in a URL: as it is structured, certain bytes have semantic meaning. If you don't want that meaning, you need to encode those bytes using what is called urlencoding. That's what we have done here.
get request with a payload param key having a '-' dash in the word, does not work

example:
url_payload = { 'username': user, 'severity': 'critical', 'job-type': 'spark', 'finished-time-begin': START_TIME, 'finished-time-end': FINISH_TIME}

results in non-200 code while testing with curl or postman or a browser with a same parameter with '-' in it, yields correct results.
Any clues?


We need a lot more details about the actual code before being able to be helpful.
use enforce_content_length in Requests

This is a **DO NOT MERGE** until we bring urllib3 1.17 into the Requests `proposed/3.0.0` branch, which should likely happen before 3.0.0 gets out the door.

This is the final step for the work in shazow/urllib3#949 and addresses the issues originally raised in shazow/urllib3#723 and #2833.


@Lukasa, here's a bit of a hiccup I'd appreciate some input on. Currently all data reads in Requests end up using `itercontent`. 

Currently, regardless of whether or not the content is streamed, we end up initially running it through the `generate` function inside `itercontent`. This function assumes we're using Transfer-Encoding: chunked if it's called, and raises `ChunkedEncodingError` wrapping urllib3's `ProtocolError`. This is clearly incorrect because we're not chunked for responses with Content-Length but I'm not sure how best to catch this. 

We could parse the exception string for 'IncompleteRead', but that seems clunky to me. We could also add a conditional inside to raise `ProtocolError` on `stream=False` and `ChunkedEncodingError` on `stream=True` but that may be subtle enough to confuse since it's the same base error. The last option I can think of is removing `ChunkedEncodingError` completely and just raising the unmodified `ProtocolError` instead. Thoughts?


...since when does `iter_content()` make any assumptions about the transfer-encoding of the response? Can you point to where in the code you believe this assumption is made?


Oh, nevermind, I see! There's no need for all that, we can just check the response headers for a transfer encoding. =)


This is [the block](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L678-L679) I'm hitting now that we're raising the appropriate `ProtocolError` in urllib3. Perhaps this is simply a wording issue on the [exception](https://github.com/kennethreitz/requests/blob/master/requests/exceptions.py#L88-L89) though. When I read "chunked encoding", I immediately go to Transfer-Encoding, but perhaps that's me misreading things.


Ok, great! To clarify, are you suggesting a conditional in the `except ProtocolError` checking the headers? Is raising the `ProtocolError` otherwise appropriate?

Edit: The reason I ask is it would adversely affect `resolve_redirect` [here](https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L106-L107).


We must raise a Requests exception.


Alright, I'll run with that then. Thanks!


@Lukasa we'll need to merge master into Proposed/3.0.0 to run the tests normally, but with 1.19 in Requests 2.12 this should be ready for a look.


That merge is done. =)


Thanks @Lukasa! I'll merge it in now.


Branch updated and tests are running clean. I think this is ready for a review whenever you find a spare moment.


Ok, I think this looks good! Thanks so much @nateprewitt! :sparkles:
Same error here " bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)

import urllib3
import urllib3.contrib.pyopenssl
import certifi

urllib3.contrib.pyopenssl.inject_into_urllib3()

http = urllib3.PoolManager(
    cert_reqs='CERT_REQUIRED', # Force certificate check.
    ca_certs=certifi.where(),  # Path to the Certifi bundle.
)
# You're ready to make verified HTTPS requests.

try:
    r = http.request('GET', 'https://devapi.trustmile.com/static/index.html')
    print r.status
except urllib3.exceptions.SSLError as e:
    print e


This is a bug report for urllib3, but you have issued it against Requests. =) Please open this bug report on the urllib3 repository.
Allow use of 'no_proxy' in the proxies argument

Add the ability to add 'no_proxy' and a value to the 'proxies'
dictionary argument.

https://github.com/kennethreitz/requests/issues/2817
Closes gh-2817


Alright, good start @JohnVillalovos! I've left some feedback in the diff that would be good to have addressed, thanks. =D


Closing due to inactivity (not sure if that's our fault or not, sorry!). Please re-submit if you're still interested in contributing this code :)

As far as I know the submitted patch is still valid, though I haven't dug into it.

I thought I had responded to all comments and latest version took them into account.

@JohnVillalovos that's true but we cannot reopen this since your repository seems to have been deleted.

Okay. Looks like I have some merge conflicts. I will create another pull request after I fix it.

Thanks @JohnVillalovos 

Created new pull request: https://github.com/kennethreitz/requests/pull/3865
Utils.check_header_validity fails on integer values

I am working on an app for ThreatConnect using their python SDK. If I use requests v2.11.0 it works fine, but testing against v2.11.1 caused an error in check_header_validity. This issue is easily reproducible with any call against requests where you specify a Content-Length header using an integer.

The check_header_validity method in requests/utils.py currently expects values to be either byte or string types using an isinstance check against the type 'bytes'. If an integer value was specified for a header (i.e. Content-Length) it would fail the isinstance check and is then assumed to be a string.

The call to re.match requires an input type of byte/string so when an integer is passed in it fails throwing an InvalidHeader exception.

Considering that this check is in place to ensure that there is no leading spaces, it should be acceptable to positively validate all numerical values without a re.match call. Alternatively you could do a hard conversion 'value = str(value)'.

ThreatConnect request object

```
{'_content_type': None, '_request_uri': '/v2/indicators/files', '_headers': {'Content-Length': 142, 'Content-Type': 'application/json'}, '_remaining_results': 1, '_path_url': None, '_description': u'adding indicator 868CB6B12D319973349D8A7752C23C05.', '_body': '{"size": "6657", "rating": 5, "confidence": 95, "sha1": "19D049110800D02F15956CE90E76CE429FC4DB9F", "md5": "868CB6B12D319973349D8A7752C23C05"}', '_payload': {'createActivityLog': 'false'}, '_resource_type': <ResourceType.FILES: 535>, '_owner': None, '_success_callback': None, '_result_start': 0, '_http_method': 'POST', '_failure_callback': None, '_owner_allowed': True, '_track': False, '_resource_pagination': False, '_result_limit': 0}
```

Trace:

```
Traceback (most recent call last):
  File "dg_tc.py", line 173, in <module>
    "TLP Red")
  File "dg_tc.py", line 78, in process_data
    indicator.commit()
  File "/home/tester/.local/lib/python2.7/site-packages/threatconnect/IndicatorObject.py", line 1193, in commit
    api_response = self._tc.api_request(ro)
  File "/home/tester/.local/lib/python2.7/site-packages/threatconnect/ThreatConnect.py", line 289, in api_request
    request_prepped.prepare_headers(ro.headers)
  File "/home/tester/.local/lib/python2.7/site-packages/requests/models.py", line 409, in prepare_headers
    check_header_validity(header)
  File "/home/tester/.local/lib/python2.7/site-packages/requests/utils.py", line 800, in check_header_validity
    "not %s" % (value, type(value)))
requests.exceptions.InvalidHeader: Header value 142 must be of type str or bytes, not <type 'int'>
```


Hey @napalmer7, this is the intention of this check. All header values must be passed as strings in Requests. There are unfortunately more complicated datatypes than `int`, such as `datetime`, which won't format correctly in a `str()` call. You can find relevant discussions and reasoning in #3366 and #3477.


@napalmer7 **Please** search closed issues for relevant discussions before opening new ones.
Using Python's requests lib throwing ProxyError

When I use requests.get('http://www.google.com')
It throws a `ProxyError`:

```
ProxyError: HTTPConnectionPool(host=My vps's IP , port=80): Max retries exceeded with url: http://www.google.com/ (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x03EB66F0>: Failed to establish a new connection: [Errno 10061] ',)))
```

When I use the HTTP Protocol request website, the host will turn into my vps's IP.  Although I shutdown the VPN. But userHTTPS Protocol will not  appear this issue.


@coderfreedom You almost certainly have the `HTTP_PROXY` environment variable set. Why is that set if you don't want HTTP tools to use your VPS?


@Lukasa   But I close it in IE settings. 
sometimes I need VPN to climbe the Great Fire Wall    : ) you know, a lot of  websites I can't visit. Sometimes visit github I also need use VPN.


Hrm. Can you run this for me?

``` python
import requests
print requests.utils.get_environ_proxies('http://www.google.com')
```


@Lukasa 

```
{'http': '107.160.9.10'}
```

It was my vps's IP, why????


I don't know, but it's somewhere in your system: either the HTTP_PROXY environment variable or elsewhere in your system configuration. Either way, we're just doing what your system tells us to do. You can probably override that by setting `proxies={'http': None}`.


@Lukasa I use this way before. But sometimes  I need use othre lib which can't set proxies, such as `builtwith.parse('http://example.webscraping.com')`  it also through `Error: <urlopen error [Errno 10061] >`


@coderfreedom In that case, this is a configuration issue on your machine. I'm afraid I can't really help beyond that. =(


@coderfreedom you can use `trust_env=False` to avoid using any proxies discovered in your environment as a work around until you figure out your system configuration problem.


@Lukasa  Anyway, thank you for your help  : )
This problem has troubled me for months, It drives me crazy "_"
I think it's time to use Mac OS.  : )


@sigmavirus24  But another lib will still not work. 


@Lukasa  @sigmavirus24  Does python has a global variable to carvery it???


@coderfreedom : Did you solve this problem? I also met VPN related issue.


it read your IE's proxy setting, you can try unset proxy setting in inetcpl.cpl.
ISO-8859-1 fallback for reason decoding

This implements the proposed solution to #3538 by falling back from Unicode to ISO-8859-1 for raw reason decoding.

This is a relatively trivial fix, and I wasn't sure if you wanted to waste bandwidth fixing, @mitsuhiko. I had a brief chat with @Lukasa who said to toss this up, but I'll gladly drop it in favor of #3538, if you had plans to update.


@sigmavirus24 are you happy with this?


Yep!
preserving {0} in requests

```
def buy_fortune():
   value1 = 'riddles'
   payload = { 'fortunefile': value1}
   url = server_url+'buy?payout_address={0}'
   print(url)
   r = requests.get(url, params=payload)
   print(r.url)
```

produces:

```

http://localhost:5000/buy?payout_address={0}
http://localhost:5000/buy?payout_address=%7B0%7D&fortunefile=riddles
```

I want to preserve the payout address parameter as {0) rather than %7B0%7D
test_lowlevel.py is failing in a Python 2 environment

If the Test test_lowlevel.py is being run by pytest in a Python 2 environment, the Testing fails at line 55 with an assertion error: assert len(fake_proxy.handler_results) == 1 is false because fake_proxy.handler_results is empty. The Problem could be as discussed on IRC, that a broken import is causing this failure, because pysocks for python 2 is not imported correct.

Running the same test file within the Python 3 environment, doesn't fail, the whole requests test suite is passed without an issue.

System: Arch Linux x64
Requests Version for Python 2 and 3: 2.11.1
PySocks Version for Python 2 and 3: 1.5.7  

![2016-09-05-181625_1245x1033_scrot](https://cloud.githubusercontent.com/assets/2480847/18253462/200a75b0-7395-11e6-9604-4ed3ebeb0ffb.png)


This error is a direct result of PySocks version 1.5.7, which when a SOCKS proxy is listening on only an IPv4 or IPv6 address but has routes available for both, connection setup can fail. This is fixed in the current master of PySocks, and can also be resolved by downgrading to PySocks 1.5.6 which does not support IPv6 addresses at all.

The best fix we can make here is the same as shazow/urllib3#964. Are you interested in contributing such a fix?


Yes I would love to but, I need some direction what to do exactly


@Trodis Have you seen the fix in the linked PR?
requests.get doesn't fail HTTP requests with verify option

When you use the verify parameter on requests.get, the function is supposed to check that the webpage certificate is valid. Obviously, if the webpage is not HTTPS, the webpage has no certificate.

I think it would be nice if in that case the request fails. I mean, imagine we have Google's cert on /srv/google-cert:

```
$ cat /srv/google-cert
-----BEGIN CERTIFICATE-----
MIIDVDCCAjygAwIBAgIDAjRWMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMDIwNTIxMDQwMDAwWhcNMjIwNTIxMDQwMDAwWjBCMQswCQYDVQQG
EwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UEAxMSR2VvVHJ1c3Qg
R2xvYmFsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2swYYzD9
9BcjGlZ+W988bDjkcbd4kdS8odhM+KhDtgPpTSEHCIjaWC9mOSm9BXiLnTjoBbdq
fnGk5sRgprDvgOSJKA+eJdbtg/OtppHHmMlCGDUUna2YRpIuT8rxh0PBFpVXLVDv
iS2Aelet8u5fa9IAjbkU+BQVNdnARqN7csiRv8lVK83Qlz6cJmTM386DGXHKTubU
1XupGc1V3sjs0l44U+VcT4wt/lAjNvxm5suOpDkZALeVAjmRCw7+OC7RHQWa9k0+
bw8HHa8sHo9gOeL6NlMTOdReJivbPagUvTLrGAMoUgRx5aszPeE4uwc2hGKceeoW
MPRfwCvocWvk+QIDAQABo1MwUTAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTA
ephojYn7qwVkDBF9qn1luMrMTjAfBgNVHSMEGDAWgBTAephojYn7qwVkDBF9qn1l
uMrMTjANBgkqhkiG9w0BAQUFAAOCAQEANeMpauUvXVSOKVCUn5kaFOSPeCpilKIn
Z57QzxpeR+nBsqTP3UEaBU6bS+5Kb1VSsyShNwrrZHYqLizz/Tt1kL/6cdjHPTfS
tQWVYrmm3ok9Nns4d0iXrKYgjy6myQzCsplFAMfOEVEiIuCl6rYVSAlk6l5PdPcF
PseKUgzbFbS9bZvlxrFUaKnjaZC2mqUPuLk/IH2uSrW4nOQdtqvmlKXBx4Ot2/Un
hw4EbNX/3aBd7YdStysVAq45pmp06drE57xNNB6pXE0zX5IJL4hmXXeXxx12E6nV
5fEWCRE11azbJHFwLJhWC9kXtNHjUStedejV0NxPNO3CBWaAocvmMw==

-----END CERTIFICATE-----
```

Then right now this is what happens:

```
>>> import requests
>>> requests.get('https://www.google.com', verify='/srv/google-cert')
<Response [200]>
>>> requests.get('http://www.ull.es', verify='/srv/google-cert')
<Response [200]>
```

I'm proposing a change: the second get call (which is HTTP) should fail.

I've tested this on requests 2.11.1.


Thanks for this suggestion! However, I'm afraid I disagree. =) That represents a pretty substantial change in the behaviour of the verify argument, and is well outside what is currently documented. Additionally, it raises some extremely difficult questions. For example, what if the request is redirected from HTTPS to HTTP: should that fail?

But fundamentally it just punishes users too heavily for simple errors. `verify` has a very simply defined behaviour [as shown here](http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification): whenever set to a validation mode, requests will validate _HTTPS_ requests, but not others.

You can achieve your desired behaviour (no HTTPS support) by reconfiguring your Session somewhat:

``` python
s = requests.Session()
del s.adapters['https://']
```

This will make Requests entirely unable to make HTTP requests.
Error received after updating the requests package

@shazow @Lukasa @mcescalante @borbamartin

Hi guys;

I just tried to extract data by using the URL of the webservices; moreover, I used the requests package however it returned the error "TypeError: **str** returned non-string (type SysCallError)" according to that; I just read the section for the issue at GitHub and upgraded the requests package. However, it returned another error. Before updating the code worked for an hour and stopped and gave the error as I shared "TypeError: **str** returned non-string (type SysCallError)" however right now it does not work; moreover jit just returns the traceback:

Traceback (most recent call last):
  File "C:\Users\may\Desktop\Documents\BOUN-CSE\Master Thesis\Project Code Files\pubchem\PubchemMapping.py", line 35, in <module>
    activities = new_client.activity.filter(target_chembl_id__in=[target['target_chembl_id'] for target in targets])
  File "C:\Python27\lib\site-packages\chembl_webresource_client\query_set.py", line 114, in next
    self.chunk = self.query.get_page()
  File "C:\Python27\lib\site-packages\chembl_webresource_client\url_query.py", line 390, in get_page
    res = session.post(self.base_url + '.' + self.frmt, data=data, timeout=self.timeout)
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 522, in post
    return self.request('POST', url, data=data, json=json, *_kwargs)
  File "C:\Python27\lib\site-packages\requests_cache\core.py", line 126, in request
    *_kwargs
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Python27\lib\site-packages\requests_cache\core.py", line 97, in send
    response, timestamp = self.cache.get_response_and_time(cache_key)
  File "C:\Python27\lib\site-packages\requests_cache\backends\base.py", line 70, in get_response_and_time
    if key not in self.responses:
  File "C:\Python27\lib_abcoll.py", line 388, in **contains**
    self[key]
  File "C:\Python27\lib\site-packages\requests_cache\backends\storage\dbdict.py", line 163, in **getitem**
    return pickle.loads(bytes(super(DbPickleDict, self).**getitem**(key)))
  File "C:\Python27\lib\copy_reg.py", line 50, in _reconstructor
    obj = base.__new__(cls, state)
TypeError: ('dict.**new**(HTTPHeaderDict): HTTPHeaderDict is not a subtype of dict', <function _reconstructor at 0x03039570>, (<class 'requests.packages.urllib3._collections.HTTPHeaderDict'>, <type 'dict'>, {'date': ('date', 'Thu, 01 Sep 2016 08:24:03 GMT'), 'transfer-encoding': ('transfer-encoding', 'chunked'), 'vary': ('vary', 'Accept'), 'server': ('server', 'gunicorn/19.1.1'), 'x-chembl-retrieval-time': ('x-chembl-retrieval-time', '0.00829792022705'), 'connection': ('connection', 'Keep-Alive'), 'cache-control': ('cache-control', 's-maxage=30000000, max-age=30000000'), 'x-chembl-in-cache': ('x-chembl-in-cache', 'True'), 'content-type': ('content-type', 'application/json')}))

Can you help to solve the problem please ?

All the Best;
Aziz


This error is coming from requests_cache. I recommend contacting that project for assistance. =)
requote_uri modifies URI

I would like to fetch
http://git.mycompany.com/api/v3/projects/my%2Ename%2Fmyproject/repository/commits?ref_name=master
but requote_uri transforms this URI to
http://git.mycompany.com/api/v3/projects/my.name%2Fmyproject/repository/commits?ref_name=master
what is an invalid URI for gitlab rest api.


Well, in the first instance you should consider contacting GitLab, because they clearly have a problem with their infrastructure. The period is a safe character in the path component of a URL, and as a result a URI that contains an unescaped period in the path must be treated exactly the same as one that contains an escaped period.

As to your issue, you can address this by using the [PreparedRequest flow](http://docs.python-requests.org/en/master/user/advanced/#prepared-requests), which will allow you to force the URL to any form you like without requote_uri getting in your way.


Thank you!
proxies has no effect

Suppose my IP is `1.1.1.1` and an anonymous proxy is `2.2.2.2:2222`. I write code like this:

``` python
while True:
    proxy = '2.2.2.2:2222'
    session = requests.Session()
    session.headers['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0'
    session.proxies = {
        'http': proxy,
        'https': proxy,
    }
    print session.get('http://members.3322.org/dyndns/getip').content
    print session.get('http://members.3322.org/dyndns/getip', proxies=session.proxies).content
```

The output for one loop should be:

```
2.2.2.2
2.2.2.2
```

Firstly, I ran this program, the result was the same as expected.
However, several days later, I suddenly found that the output for one loop became:

```
1.1.1.1
2.2.2.2
```

I killed the process and ran it again, the output was still wrong. (repeated several times)
Then I restart the computer and ran it again. The output became right, again.

In fact, I can hardly reproduce the issue. But it did happened.

```
OS: Windows 10 home (version 1511, os version 10586.545).
Python 2.7.11
requests 2.11.0 (installed by pip)
```


Without the ability to reproduce this, I don't think we can really take any action on this. =(
Header doesn't accept int values already

Hi,
I discovered that new version (2.11.1, build:0x021101) doesn't accept int type in headers.
Before update (2.3.0, build:0x20300) everything was ok. 

```
import requests
headers = {'exampleHeader': 10} # raise InvalidHeader: Header value 10 must be of type str or bytes...
# headers = {'exampleHeader': '10'}  # for new version of library
requests.request("PUT", http://google.com, headers=headers)
```


This was an intentional change.

You will find many other closed issues discussing this since 11 came out.

You will need to convert to string before using as a header.

On Fri, 26 Aug 2016, 6:21 PM Mateusz notifications@github.com wrote:

> Hi,
> I discovered that new version (2.11.1, build:0x021101) doesn't accept int
> type in headers.
> Before update (2.3.0, build:0x20300) everything was ok.
> 
> import requests
> headers = {'exampleHeader': 10} # raise InvalidHeader: Header value 10 must be of type str or bytes...
> 
> # headers = {'exampleHeader': '10'}  # for new version of library
> 
> requests.request("PUT", http://google.com, headers=headers)
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/kennethreitz/requests/issues/3540, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AGk_7boWiOSD9tXu8NMDEBLGVkyZ9MoBks5qjqIggaJpZM4Jt2QH
> .


Yup, that's correct. =) Thanks @TetraEtc!
Add persistent examples

Learning python and Requests it was not immediately clear that many arguments can also be made persistent using the Sessions class.


@Lukasa I was certain this was described elsewhere, am I wrong?


[The first sentence of the Session docs](http://docs.python-requests.org/en/master/user/advanced/#session-objects) reads:

> The Session object allows you to persist certain parameters across requests.

Is there a reason we need this change? Do we need to improve the Session docs directly?


While searching on how to use an SSL client certificate the [developer interface documentation](http://docs.python-requests.org/en/master/api/) did not help at all. SSL client certificates are only mentioned as a parameter for requests.request(). 

The [Session Object documentation](http://docs.python-requests.org/en/master/user/advanced/#session-objects) does have a [section](http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification) covering SSL client certificates but only as a parameter to requests.get(). Additional examples for using persistent parameters would have really helped me.


Well, the scope of the changes here is pretty small, so I don't object to merging this for now. =)


Thanks @RichieB2B!
('Connection aborted.', error(105, 'No buffer space available'))

I have some code that calls like

r = session.get(url, stream=False, timeout=1)
data = r.content

This is called in a loop for 150 urls constantly. Usually this works but after a while I get errors like

('Connection aborted.', error(105, 'No buffer space available'))

This is over an ipsec tunnel so I suspect the bug is with the tunnel but wanted to make sure.

Thanks


Do you have a more complete traceback for that? This error _generally_ indicates a problem with the socket buffer itself: that is, the buffer is overflowing. It'd be good to see where it's coming from, but _probably_ it's related to the ipsec tunnel: certainly many similar bugs seem to be raised that involve ipsec.


I solved it in the by doing
echo "65535" > /proc/sys/net/ipv4/xfrm4_gc_thresh

I found the solution https://wiki.strongswan.org/issues/1169

The default on centos 7 was quite low.

Thanks
Test case for requests getting stuck on post redirect with seekable streams

https://github.com/kennethreitz/requests/issues/3079

This is a test case that depends on httpbin 0.5.0 to show the bug.


Thanks for these tests! They do, as expected, fail.

What we need now, I think, is for someone to step up and write a solution. Any volunteers?


I guess this issue can only be fixed inside urllib3 since it handles the actual redirects.


@tzickel At a quick glance, I don't think that urllib3 has any knowledge of whether our request is a redirect or not. We pass each redirect call to urllib3 as if it were a single request.

#3655 addresses the problem posed in your second test, I'm not claiming it's the best solution though. Any input there or clarification on what I'm missing in a urllib3 implementation would be appreciated :)


@Lukasa tzickel's commit was merged into master with #3655, so this can probably be closed.


Yup. Thanks for the work @tzickel!
avoid use of getvalues in super_len

This is a follow up on @jseabold's work in #3339. These last minor changes should fix the issues with return values of `seek` between Python 2 and Python 3.


It'd be really nice to have some more tests around this. Ideally tests that go into each of the branches. @nateprewitt, are you open to adding those?


Yep, I'll throw a few more together for `TestSuperLen`.


@Lukasa, tests are updated with custom classes. I think things should be good to go with tests.


Small notes about the tests, but this is looking really good!


Great! Changes in last notes have been updated.


Ok, cool, I like this. @sigmavirus24?


@sigmavirus24, wanted to ping in case this fell off the queue. No rush, just checking in :)


I have one suggestion but it's optional. 


I don't think I've got anything else, so I think we're good to merge from my end. Let me know if you have anything else @lukasa.

We can probably start discussing #3545 too.


Ok then, let's do this! =D
Segfault while using requests in GPIO Callback on Raspberry Pi 3

I'm getting a segfault sometimes after pressing the Button.
This little script can be used to reproduce the error. (If you don't get a segfault after 20 button presses, restart the script, sooner or later you'll get one).

requests Version is 2.4.3
Python Version is 3.4.2

Perhaps someone could give me a hint what/how to find out what the problem is. 

```
import time
from time import mktime, time, sleep
import RPi.GPIO as GPIO
import requests

def button_press_isr1(channel):
    GPIO.remove_event_detect(18)
    sleep(0.0001)
    cnt = 0
    edge_start1 = time()
    while (time() - edge_start1) <= 0.035 : # 35 mSec max, 15mSec margin
        sleep(0.0015) # 1.5 mSec
        if GPIO.input(18) == 0 : # 0/False for Falling, 1/True for Rising
            cnt += 1
        else: # when we have captured a glitch, start all over again
            cnt = 0
        if cnt == 7 : # 7? lucky number
            break
    if cnt == 7 :
        response = requests.post(url="http://192.168.1.5:8080")
    GPIO.add_event_detect(18, GPIO.FALLING, callback=button_press_isr1)


def init():
  #GPIO.setwarnings(True)
  GPIO.setmode(GPIO.BCM)
  GPIO.setup(18, GPIO.IN)

  GPIO.add_event_detect(18, GPIO.FALLING, callback=button_press_isr1)

def main():
  try:
    while True:
      sleep(0.03)

  except KeyboardInterrupt:
      # exit with CTRL+C
      print("CTRL+C used to end Program")
  finally:
    GPIO.cleanup()


if __name__ == "__main__":
    init()
    main()

```


Thanks for reporting this!

It's not possible to segfault the interpreter from inside Requests, because Requests is written in pure-Python. That suggests that the issue is inside the GPIO calls. Unfortunately, to be more specific you'll need to use a debugger like GDB to catch the segfault and print the C stack for that call.

Support for that problem is outside the scope of this bugtracker, I'm afraid.


Ah great to know that there is only one option left :-), thank you
Passing seekable objects without len to super_len causes .getvalue() which copies them just to get the length

This is an performance issue, especially when passing large objects. This can be easily triggered with a data post body:

bigdata = 'a' \* 100000000
stream = io.BytesIO(bigdata)
a = requests.post('http://localhost', data=stream)

will call super_len on stream, which will call stream.getvalue() which will actually copy the 100MB in memory, get it's length, and discard the copy.

we should check first if the object has 'seek' and 'tell' and use them (much quicker) to get the length instead.

https://github.com/kennethreitz/requests/blob/52b15f811f8ef52a144c96b8b742d734dd39d693/requests/utils.py#L59


Thanks for the report! This is a known issue and we're working on a fix: see #3339.
3518 Bug Fix and Response Header to Python equivalents

I'm fairly new to open source contributions, so any feedback is appreciated. This PR
1) Addresses issue #3518 by iterating over ALL_PROXY last. 
2) Elaborates some of the most commonly used response headers into their corresponding Python equivalents for convenience, namely r.date() and r.last_modified() return datetime objects and r.content_length() and r.age() return ints when possible, None otherwise
3) Added corresponding tests for the above 


Thanks for this @michelsch92!

Some notes. Firstly, I'm afraid to say that #3518 was already resolved by #3527. I'm sorry about that, the timing was just a bit unlucky.

Secondly, the response helpers, I think we don't want to put them into the core requests library itself. Generally speaking we try to keep the surface area of the API down, because the more methods and properties that there are on the object the harder it gets to understand. However, @sigmavirus24 may be interested in having those properties in a headers helper object in the requests-toolbelt.

Unfortunately, that means this patch won't be accepted at this time. This has nothing to do with the quality of your work: altogether it was high quality work, and we'd have been pleased to accept it. Please do keep looking around for ways to contribute, because you're most welcome here!
Fixed bug to give scheme proxy priority over "all"

Addresses bug called out in issue #3518.


This seems reasonable to me. @Lukasa thoughts?


This looks really good to me, I'm totally happy with it! Thanks so much @shellhead! :sparkles: :cake: :sparkles:


Shouldn't this line also be swapped to close out this issue? https://github.com/kennethreitz/requests/blob/master/requests/sessions.py#L229


Yup, looks like!


Thanks for pointing this out.

I'm also thinking I missed this as well https://github.com/kennethreitz/requests/blob/master/requests/utils.py#L620

Is there any reason to not use `select_proxy` in sessions? It seems strange to have two different ways of selecting the proxy.


`select_proxy` does something subtly different. In this case in the session we are trying to recompute the proxies from the environment, but otherwise not to do anything.