Raibek/requests-github-issues huggingface dataset:
Max_redirect = 0; Location header missing

Hi,
Requests ver: 2.11.1

I'm using a session object (s) to make a POST to a specific URL:
r=s.send(prepared)

I expect this to return 302, which is happening.
I set s.max_redirects=0 because I don't want Requests to automatically visit the new URL.
So, I promptly receive a TooManyRedirects exception, which is good.
However, in the exception handler, I'm unable to get hold of the header 'Location'.
The response object itself is empty.

I debugged it and saw that the Location header was captured correctly. But, when the exception is thrown, it's all gone.

(Pdb) n
> /Library/Python/2.7/site-packages/requests/adapters.py(275)build_response()
-> return response
(Pdb) response
<Response [302]>
(Pdb) response.__dict__
{'cookies': <RequestsCookieJar[]>, '_content': False, 'headers': {'Date': 'Fri, 02 Dec 2016 22:27:44 GMT', 'Content-Length': '0', 'Location': 'http://XXX:8080/openam?goto=https:YYY', 'Server': 'Apache-Coyote/1.1'}, 'url': u'https://XXX:8443/proxyplus/nnl/reg', 'status_code': 302, '_content_consumed': False, 'encoding': None, 'request': <PreparedRequest [POST]>, 'connection': <requests.adapters.HTTPAdapter object at 0x1053ae1d0>, 'elapsed': datetime.timedelta(0), 'raw': <requests.packages.urllib3.response.HTTPResponse object at 0x1055e3550>, 'reason': 'Found', 'history': []}

As you can see, 'Location' is found at this stage in 'response', but when the exception is thrown, response becomes empty.

Thanks!


Hey @arao-nnl, in order to get the behaviour you're looking for, you'll need to use a slightly different mechanism. `max_redirects` is intended to raise an exception halting the request in the event of a redirection loop, or excessive redirects.

For your issue of trying to simply halt before a redirect is performed, I suggest using the `allow_redirects` parameter. This will retrieve the 302 and then leave the object untouched until you decide how to proceed (such as only reading the Location header).

```python
s = requests.Session()
r = requests.Request('GET', 'http://httpbin.org/redirect-to?url=get')
resp = s.send(r.prepare(), allow_redirects=False)
loc = resp.headers['Location']
```
No tests being run in CI against new pull requests

It looks like there aren’t any tests being run in CI against pull requests, so there’s a bit of extra work for reviewers to get the patch locally and run the tests themselves. And running tests locally is more susceptible to environment-specific test quirks.

It looks like there used to be CI on older pull requests from <http://ci.kennethreitz.org>, but that seems to be unresponsive now. 😢 

Would you be interested in a patch that adds some config for Travis CI (and/or another CI service of your choice)?

Sorry Alex, but this happens to all of us:

Closing as a dupe of #3739.

And not even an old issue either! Wow, I’m really not with it today.

:wink: We all have off days, nothing to be worried about there. Just thought it was funny.
Rebase #2431 against proposed/3.0.0

This is really quite unexciting: #2431 had a tentative +1 from @Lukasa, pending a rebase and retargeting against 3.0.0, but it’s been sat unloved for nearly eight months. 😢 

This patch does the required rebase and targets the 3.0.0 branch.

Woah, zombie PR back from the dead!  I haven't worked in Python for over a year now, but I'll see if I can address the suggestions this weekend.

I had a go at addressing the review markups, but I feel like more testing is required.

I feel like there’s enough subtle behaviour here that we’d be better off with a whole stack of tests (probably using `@pytest.mark.parametrize`) that tests a whole variety of cases here. Which I may or may not get around to writing sometime this week. At a minimum, the cases described in [this comment](https://github.com/kennethreitz/requests/pull/2431#issuecomment-72333964).

Yeah, I think more testing here would be fab. Not sure how we're going to get it though, especially if @alexwlchan doesn't have time to do it.

I’ve stuck a note in my todo list to come back and write more tests here. I’ve built up a bit of a backlog of unfinished PRs, but hopefully I’ll find time to get back to this over Christmas.

Closing this out due to lack of activity. Feel free to re-open once you get a chance!
Requests won't follow some redirects

for example:

```
requests.get('http://archopht.jamanetwork.com/article.aspx?doi=10.1001/archopht.125.1.128')
```
Won't work ^

however you can do it with urllib, urllib2

```
urllib2.urlopen('http://archopht.jamanetwork.com/article.aspx?doi=10.1001/archopht.125.1.128')
```
or with urllib3
```
 http = urllib3.PoolManager()
 http.request('GET', 'http://archopht.jamanetwork.com/article.aspx?doi=10.1001/archopht.125.1.128')
```

Is there any way to accomplish this with requests?

Hey @timothyjlaurent, the issue you're hitting is with the server rather than Requests. It looks like the base domain jamanetwork.com (where the redirect points you) may not want it crawled. The server has blacklisted any request with a User-Agent containing the word "python" or "curl".

Given their robots.txt doesn't specify a restriction on your request though, I think we can shrug and work around it.  Simply supplying a custom 'User-Agent' header, whether it's your own or from a standard browser, will fix this.
```python
url = 'http://archopht.jamanetwork.com/article.aspx?doi=10.1001/archopht.125.1.128'
headers = {'User-Agent': 'undercover-agent/1.0'}
requests.get(url, headers=headers)
```

Nicely caught @nateprewitt 

Thanks @nateprewitt
The new 2.12.2 breaks certain docker-py's build's argument

see the issue in https://github.com/docker/docker-py/issues/1325, although I am not sure this is a right place to post.

As per #3734 this is already getting worked on. The relevant issue to follow in Requests is #3735 

Also - If you had locked in docker/docker-py you'd see that they had already been talking about this over there in [docker/docker-py/#1321](https://github.com/docker/docker-py/issues/1321)

Yup, this is a duplicate. Closing in favour of #3735.
Is there a specific reason why there is no CI (Travis/Appveyor)?

I was surprised to see that requests is not making use of Travis/Appveyor CI. Is there a specific reason for that?

I would be willing to send a PR implementing it, if the only reason is that no one got around to it yet.

I also saw that `tox.ini` is in the .gitignore file - so is tox used locally by some devs but not part of the project? If so: what are the reasons for not having a `tox.ini` as part of the project? I would use tox.ini to integrate the project with the CI systems and also add a convenient way to run the tests on different interpreters.

Hey @obestwalter, I talked to @Lukasa about this a few weeks ago. To quote a bit of his response:

>Requests moved away from Travis because it made a lot of live network requests in its test suite and Travis had frequent network outages that caused real problems with the test run. 

While a lot of these issues have been resolved, I think the general sentiment is Travis generates a fair amount of noise for the repo owner and has historically had quirks. Until recently, Requests ran CI on a Jenkins server, but that's since been removed.

As for tox, I think most of Requests' contributors are running a local tox.ini file for testing changes. If you run that in conjunction with something like [pyenv](https://github.com/yyuu/pyenv), you have an easy local equivalent of a Travis environment which has worked for the most part the last several months.

Note this may no longer be the state of things, I just wanted save Lukasa from having to reiterate what he's conveyed recently via other channels.

Here's the last couple of Kenneth's comments on the state of Requests' CI/Travis in 2016 too. [[Feb](https://github.com/kennethreitz/requests/pull/2991#issuecomment-178813436)] [[Apr](https://github.com/kennethreitz/requests/pull/3096#issuecomment-211701179)]

@obestwalter All of these decisions are fundamentally @kennethreitz's: the other maintainers have no objection to re-adding Travis support and a toxfile.

Hello @Lukasa and @nateprewitt,

thanks for the explanation. Why is the tox.ini then not part of the project? Is there a problem to get it working for everyone involved? My naive thinking would be that once you have tox properly configured for the project it works for everyone involved. 

So, if there are any concrete problems/bugs that prevent your project from adopting tox properly, just let us know here: https://github.com/tox-dev/tox (yes, we finally moved to Github :)).

We have adopted tox heavily, but Kenneth didn't want a toxfile checked into the repository: that's why it's not part of the project. =) Once again, these particular decisions are made by Kenneth.

Thank you - I was pointed to the relevant discussions. I should have looked a bit more closely before asking stupid questions. Sorry for the noise.
empty username and password no longer generate basicauth authstring

Possibly just a previously quirkly behavior that intentionally no longer works.

For `requests<2.12.2` having `None` for username and password still resulted in a BasicAuth authstr being created. Starting in `2.12.2` it will raise an exception.

```Traceback (most recent call last):
  ...
  File "/blah/lib/python2.7/site-packages/requests/auth.py", line 38, in _basic_auth_str
    b64encode(b':'.join((username, password))).strip()
TypeError: sequence item 0: expected string, NoneType found
```

here's the change, (which certainly seems desirable), but caused our upstream (`python-gitlab`) to break.
https://github.com/kennethreitz/requests/commit/329a5cfdd4500fb1ce405a3242ef95f9197d24ba

Uh...what basic auth string did it generate?

```
(Pdb) username is None
True
(Pdb) password is None
True
(Pdb) authstr
'Basic Tm9uZTpOb25l'
```
yea i'm fully aware that this doesn't make much sense and closing wontfix seems entirely reasonable. Just figured I'd call it out in case anyone else was similarly relying on the behavior.

Heh, yeah, that's the base64 encoded form of `None:None`. I think we can call this wontfix: that should never have worked, and it's quite surprising.

@hummus should we file a ticket at [python-gitlab](https://github.com/gpocentek/python-gitlab/issues)'s issue tracker? I would do it myself but I don't know enough about the problem
Requests v2.12.2 does not add parameters to URLs with schemes it does not understand.

Follows from #3734.

When we patched to ignore all unrecognised schemes instead of just those that didn't begin `http`, we stopped handling parameters for those URLs. This may break some more benign uses such as `http+unix`, which wanted to add parameters to their URLs.

While this is inline with our intended policies (we do not understand URLs that have schemes we don't know anything about), this, along with the IDNA behaviour in v2.12.{0,1} will probably have broken a whole bunch of people using these non-standard URL schemes.

We should consider whether Session objects should have a registry of places to look for URL preparation based on scheme. This will allow people to opt-in to the HTTP-like processing of URLs, as well as to register their own.

I should note that we have had this conversation with the docker-py folks before: see #1879.

Ultimately, there is a problem with how much people want us to understand URLs for non-standard HTTP schemes. People would like us to understand them enough to add query parameters, but not so much that we look for a hostname to internationalize.

I am as conflicted as I was three years ago.

Thanks @Lukasa for chiming in and looking into this!

Being able to rely on `requests` to make docker-py work has been a great boon for us, especially the ability to handle different transport formats (HTTP/s, UNIX sockets, and more recently Windows named pipes) through the same interface is a powerful feature for us.

If you see any way we could keep this flexibility while making things easier on the requests library to support our use-case(s), I'd be happy to hear them and consider any changes.

Thanks again for your help.

In the short term, I think we need to back out the breakage: kicking things over with @graingert convinced me that ultimately, whether we ever intended docker-py's usage of Requests to actually work, it *has* worked for more than three years. If we're going to break that, we should break it on purpose, rather than by accident.

So this issue remains worth discussing because we should solidify our position on this kind of thing for 3.0.0. Either we'll want to add explicit support for having HTTP-like schemes, or we'll want to add support for custom schemes to register URL processing handlers, or we'll want to drop support entirely as we (accidentally) did in v2.12.2. We should discuss what of those we think we want to do, and make sure there is a transition plan in place for projects like docker-py.

> In the short term, I think we need to back out the breakage

https://github.com/kennethreitz/requests/pull/3738
remove weird characters from README.rst


There's no reason to remove these. What are you trying to achieve here?

What are they? And what are they for? They are not displayed well in my computer.

From Readme in GitHub
![captura de pantalla de 2016-11-29 10 05 31](https://cloud.githubusercontent.com/assets/14212780/20703284/76a287dc-b61b-11e6-8e4c-155b3a940b57.png)
From pip
![captura de pantalla de 2016-11-29 10 05 43](https://cloud.githubusercontent.com/assets/14212780/20703288/7a051336-b61b-11e6-8007-63c3aa0c15fc.png)


They are [emoji](https://en.wikipedia.org/wiki/Emoji). They're pictograms that have been assigned Unicode glyphs. If your computer cannot render them you can either elect to resolve that by installing an emoji font, or you can simply ignore them: you aren't losing out on any critical data. The picture you can't see, for the record, is a slice of cake. 

The cake is a lie.
CLOSE_WAIT Issue

Hi,

We are developing a python application using tornado framework. The application consumes a REST end point exposed by splunk system to get some metrics data. A scheduled background process runs and populates our application local data-store.  While tornado serves data to third party using the local data-store. 

We are using Requests==2.9.1 to communicate with Splunk

Code snippet using which we are communicating to Splunk:
------------------------------------------------------------------
                      try:
                            resp = requests.get(splunkapi,params = params, verify = False, headers = header )
                            if resp.status_code == 200 and (resp.content not in ['""','"None"']) :
                                results = json.loads(resp.json())
                                with open("outfile.json" , 'w') as outfile:
                                    json.dump(results, outfile,indent=4)
                            else:
                                with open("outfile.json" , 'w') as outfile:
                                    json.dump(dict(), outfile,indent=4)
                      except Exception,e:
                            with open("outfile.json" as outfile:
                                json.dump(dict(), outfile,indent=4)


CLOSE_WAIT Issue:
---------------------
Step 1: I bring up Tornado Application.
Step 2: I bring up splunk system.
Everything works fine
Step 3: For some reason, the sys admins have to reboot the machine where tornado application is running.
Lot of connections in CLOSE_WAIT status at splunk side.
Step 4: Restart Splunk.
Everything is cleaned up. The system continues to run smoothly.
Step 5: Reboot the system where tornado application is running.
Again same issue. Lot of CLOSE_WAIT connections at splunk side. 

Are the CLOSE_WAIT connections a result of "Requests" not communicating properly with splunk?

No. 

CLOSE_WAIT is the TCP state that is entered by the TCP peer that initiates a connection close. It cannot be avoided: it exists to ensure that delayed or retransmitted packets belonging to a TCP connection are not lost.

When the Tornado application is closed down, Requests' pooled sockets all get closed by the Python runtime or the OS. That means your Tornado application initiated the close, and so all those pooled sockets will enter TIME_WAIT. This is not a problem: it is expected behaviour for a TCP client.

Ah, apologies, I see you're getting the CLOSE_WAIT on the other machine. That seems like Splunk is initiating an idle timeout. What Python interpreter are you using?

We are using Python 2.7.5 for our tornado application (dockerized). The splunk version on the other machine is 6.4.0 which is packaged with python 2.7.11.

Thanks!

Are you familiar with Wireshark? It would be interesting to see Wireshark traffic from the application. In particular, I'm interested to see whether the TCP connections your server sets up to splunk are cleaned up promptly in use or not.
URL has an invalid label when requesting https://r8---sn-q4f7snss.googlevideo.com

```python
>>> import requests
>>> requests.get("https://r8---sn-q4f7snss.googlevideo.com")
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/requests/models.py", line 370, in prepare_url
    host = idna.encode(host, uts46=True).decode('utf-8')
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/idna/core.py", line 355, in encode
    result.append(alabel(label))
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/idna/core.py", line 276, in alabel
    check_label(label)
  File "/usr/local/lib/python3.4/dist-packages/requests/packages/idna/core.py", line 253, in check_label
    raise InvalidCodepoint('Codepoint {0} at position {1} of {2} not allowed'.format(_unot(cp_value), pos+1, repr(label)))
requests.packages.idna.core.InvalidCodepoint: Codepoint U+0027 at position 2 of "b'r8---sn-q4f7snss'" not allowed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 474, in request
    prep = self.prepare_request(req)
  File "/usr/local/lib/python3.4/dist-packages/requests/sessions.py", line 407, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/usr/local/lib/python3.4/dist-packages/requests/models.py", line 302, in prepare
    self.prepare_url(url, params)
  File "/usr/local/lib/python3.4/dist-packages/requests/models.py", line 372, in prepare_url
    raise InvalidURL('URL has an invalid label.')
requests.exceptions.InvalidURL: URL has an invalid label.
```

requests 2.12.0

When reporting bugs in open source projects *please* check the open issues first before opening a new one. If you had done that you'd have found literally the text "URL has an invalid label" next to the word "Bug" four entries down.

Duplicate of #3683, closing accordingly.
Old "cryptography" package causes crash

As of `requests==2.12.1`, `requests[security]` depends on `cryptography>=1.3.4`.  That's fine.  However, `requests` (without `[security]`) crashes if `cryptography==1.0.2` is installed.  This is a perfectly valid configuration of packages according to `pip`; as a user, I would expect it to behave as if I had not specified `[security]` (because I didn't specify it).

The specific failure is discussed a bit more here:

http://discuss.flexget.com/t/unhandled-error-in-plugin-configure-series-solved/2900

The crash was resolved, also as described at the above link, by running:

```
pip install --upgrade cryptography
```

Uninstalling `cryptography` would probably have worked fine too.

The crash appears to be caused because `requests` internally tests for `[security]` by importing a thing which imports its bundled `pyopenssl` which imports `cryptography`; if that succeeds, the extra is used.  But with a too-old version of `cryptography`, it imports just fine but it constructs objects that are missing required fields.

I can think of a couple approaches here:

* Catch this failure and print an error message that more-helpfully explains that your `cryptography` package needs updating
* Validate the version/functionality of the `cryptography`package more carefully, and don't enable the `[security]` extra's features if an old version happens to be installed

I'd be happy to post a PR for either approach.  Right now I'm looking for feedback on whether it's worth addressing this (I'd certainly appreciate it) and on what approach you'd prefer.

Thanks for this!

I think this is actually fundamentally a problem with urllib3: urllib3's contrib module should be more aggressively checking for whether it possesses a suitable cryptography module before injecting itself. If you open an issue on that repository, we can take it forward from there.

Thanks again for the report!

Thanks @Lukasa !  Posted an issue in (what I hope is) the main urllib3 repo.  

Gentle *ping* here -- this has been fixed upstream; is there anything I can/should be doing to pull the fix in here?

Nope. Updates to urllib3 are brought in as part of our release process. It'll get merged when it gets merged.
Restrict URL preparation to HTTP/HTTPS

Requests treats all URLs starting with the string 'http' as HTTP URLs.
Preparation with IDNA breaks non-standard URIs like http+unix. Requests
now prepares only URLs with prefix http:// and https://.

Signed-off-by: Christian Heimes <christian@python.org>

http+unix is used in Docker, Custodia and SSSD, e.g. https://github.com/latchset/custodia/blob/master/custodia/client.py#L60 with URI like ```http+unix://%2Fvar%2Frun%2Fcustodia.socket/secrets``` for HTTP over Unix sockets on socket file ```/var/run/custodia.socket```.

This looks entirely reasonable to me, thanks @tiran!

One note though: can we get a test for this?

Cool, thanks @tiran! All tests pass, so it seems like it's good to go. Thanks so much! :sparkles: :cake: :sparkles:
Older changelog entries don't specify urllib3 versions

Some of the older changelog entries, starting at 2.4.0 and working backward, mention that the urllib3 version was bumped but don't say what version it was bumped to.

This can cause mildly annoying dependency resolution problems, especially if you partake in unsavory activities such as:

* Installing requests from an RPM source that doesn't vendor urllib3
* Running extremely outdated versions of requests/urllib3

It would be useful if the changelog included these details (as newer changelog entries do).

I'm happy to put in the time to do this, but probably not immediately.

So, uh, this won't work as well as you think.

Prior to v2.6.2 the Requests team didn't necessarily update to specific urllib3 versions but instead updated to the state of the urllib3 master branch. That means there often *is no* urllib3 version that directly matches what Requests was using. So my question is: why does your RPM source not appropriately state this dependency?

That’s unfortunate.

I’m not entirely sure where the RPM came from (somebody else put it in the codebase and didn’t update the urllib3 RPM). We’ve managed to resolve the current issue by updating to a newer-ish version, which seems to have resolved the current issues.

Since there isn‘t a nice mapping of requests versions::urllib3 versions, I guess I’ll leave this for now.
Indna causes issues with Jython due to uts46data.py

Unable to use requests in Jython 2.7 without some patches to Indna.  I've made these at https://github.com/METAVERO/idna and notified main repo owners.

Thanks for letting us know. You should strongly consider opening a pull request on the idna repository instead of just an issue.


Hi Lukasa, I did open a pull request on my repository and then closed it.  A little new to Github -- do I need to open another pull request or are you able to review the history in that repo?


It's harder to review the history. It would be easier if you opened a pull request against the _original_ repository though. 


Please review - https://github.com/METAVERO/idna/pull/1

I re-did the repo via a fork from original since it wasn't setup properly before.  There is now an open pull request to the main Idna project.


@johnabooth It doesn't look like you've actually opened a pull requests against the upstream repo [kjd/idna](https://github.com/kjd/idna). You'll need to visit that repo, use the "New Pull Request" button and select your branch `johnabooth-jython27-fix`. The maintainer of that repo should be able to perform a review on your proposed changes.

https://github.com/kjd/idna/pull/33 was merged. It splits the large piece of code to smaller ones and should fix the Jython issue.

`idna` released `https://github.com/kjd/idna/releases/tag/v2.2`, so if the dependency is adjusted, #3711 should be resolved. Should the entry in `setupy.py`'s `extras_require['security']` be increased to 2.2?

I don't think we need to do that, no. Jython users will simply get the newest version as appropriate.

Thanks all.

Sent from my iPhone

> On Dec 21, 2016, at 7:26 AM, Cory Benfield <notifications@github.com> wrote:
> 
> I don't think we need to do that, no. Jython users will simply get the newest version as appropriate.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
>
'Extensions' object has no attribute 'get_extension_for_class'

This occurred after I upgraded form 2.11.1 to 2.12.1. The error occurs arbitrarily for the URLs I tried.

> requests.get('http://google.com') # works
> requests.get('http://www.google.com') # works
> requests.get('https://google.com') # does not work
> requests.get('https://www.google.com') # does not work
> requests.get('http://amazon.com') # sometimes works
> requests.get('http://www.amazon.com') # does not work
> requests.get('https://amazon.com') # does not work
> requests.get('https://www.amazon.com') # does not work

Below is the error:

```python
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.pyc in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    348         # Trigger any extra validation we need to do.
    349         try:
--> 350             self._validate_conn(conn)
    351         except (SocketTimeout, BaseSSLError) as e:
    352             # Py2 raises this as a BaseSSLError, Py3 raises it as socket timeout.

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.pyc in _validate_conn(self, conn)
    833         # Force connect early to allow us to validate the connection.
    834         if not getattr(conn, 'sock', None):  # AppEngine might not have  `.sock`
--> 835             conn.connect()
    836 
    837         if not conn.is_verified:

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connection.pyc in connect(self)
    328         elif context.verify_mode != ssl.CERT_NONE \
    329                 and self.assert_hostname is not False:
--> 330             cert = self.sock.getpeercert()
    331             if not cert.get('subjectAltName', ()):
    332                 warnings.warn((

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.pyc in getpeercert(self, binary_form)
    322                 (('commonName', x509.get_subject().CN),),
    323             ),
--> 324             'subjectAltName': get_subj_alt_name(x509)
    325         }
    326 

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/contrib/pyopenssl.pyc in get_subj_alt_name(peer_cert)
    169     # faster than looping in Python)
    170     try:
--> 171         ext = cert.extensions.get_extension_for_class(
    172             x509.SubjectAlternativeName
    173         ).value

AttributeError: 'Extensions' object has no attribute 'get_extension_for_class'
```

Tried it on Ubuntu/Python 2.7.10 and macOS/Python 2.7.11. I have OpenSSL 1.0.2 for both. Same error on both. 

What version of `cryptography` do you have installed please?


I have cryptography 1.0.2. Upgrading to the latest 1.5.3 solved the problem. Thanks!


No problem! For anyone looking at this issue in future, our minimum cryptography version is currently 1.3.4, and that is expressed in the `'security'` extra for requests at this time.


May the gods bless you for this post

@andreleotorres 🙏🏼
Added new function that returns json output in a pretty printed string form.

The function sjson in the Response class will return a json object in string form. Unless provided arguments, it will return a pretty printed string, when printed is human readable. You can look at the docs for the json module's dumps object to know which arguments to send to the sjson function for custom output. 

![out](https://cloud.githubusercontent.com/assets/3707038/20460056/30a5c37a-aea5-11e6-923f-ecf4277e1907.png)
This is what the output should look like.


I believe requests is in an API freeze at this point, especially when user code could do the same thing with `r.content` and their preferred indentation level and separators. Contributions are always welcome, but API changes are the type of thing that would be better served by raising an issue and getting feedback on the interface, before you spend your time writing the code. 

EDIT: Of course I'm not a requests maintainer, I'm just trying to give feedback based on what I've seen. Some interface of this type may well get accepted. 


@drpoggi Can you explain how users can create the same output with r.content.  The output from r.content is different from r.json().  Also how do you provide preferred indentation level and separators. As far as I know it doesn't take any arguments. You can provide preferred indentation level and separators to r.sjson through key value arguments. You can look at the documentation for the dumps function in the json module to give custom arguments.  The whole point of creating the function was so that users can get the json object in a human readable form with simple readable code. Also how do you know there is a freeze in requests. Was there a post somewhere, or is it in the documentation somewhere?

EDIT: Also thanks for the feedback.


@rmhasan, thanks for your interest in contributing to Requests. You can find information about the freeze and other contributing topics in Requests [Contributor's Guide](http://docs.python-requests.org/en/master/dev/contributing). The specific information you're looking for is under [feature requests](http://docs.python-requests.org/en/master/dev/contributing/#feature-requests).


@nateprewitt Thanks. 


Thanks for this, but ultimately this duplicates the logic of the `pprint` module in the standard library. Because of that, and because the potential user base is fairly limited, I don't think this feature warrants an exception to the API freeze.

Thanks for your contribution though!
Allow domains instead urls

```
requests.get('google.com')
requests.exceptions.MissingSchema: Invalid URL 'google.com': No schema supplied. Perhaps you meant http://google.com?
```

Why does it not allow use domains? It looks as the simple task and I can do it. What do you think? I searched similar issues but did not find.

It's impossible to know whether you wanted us to use HTTP or HTTPs, and in the face of uncertainty (especially when privacy is concerned) it's significantly more correct to not guess.

You can absolutely implement whatever behaviour you're thinking of in a wrapper. There's no need and no desire for it to live in Requests.
bug in PreparedRequests without simplejson library (under jython 2.7 only?)

I noticed the following bug when running requests under jython 2.7 where I had not yet installed simplejson:

In requests/models.py, the method PreparedRequest.prepare_body has the following code:

````
        if not data and json is not None:
            # urllib3 requires a bytes-like body. Python 2's json.dumps
            # provides this natively, but Python 3 gives a Unicode string.
            content_type = 'application/json'
            body = complexjson.dumps(json)
            if not isinstance(body, bytes):
                body = body.encode('utf-8')
````

`json` in this case, is defined from the .compat file:

````
try:
    import simplejson as json
except (ImportError, SyntaxError):
    # simplejson does not support Python 3.2, it throws a SyntaxError
    # because of u'...' Unicode literals.
    import json
````

If simplejson is not installed under jython, the code in models.py throws an exception because dumps is not defined in the json module.


In this instance, I think I'm inclined to suggest that this is a JSON problem, not a Requests problem. json.dumps is not a new interface. And without it, our json parameter cannot possibly work anyway.


@j-Christ, Jython 2.7 should support dumps without issue. I'd suggest checking your local directory or python path for another json.py file.


@nateprewitt Thanks!  That was the issue...somewhere in the jython path was a json package by patrickdlogan@[...].com, and requests was finding that one instead.
clarify that the `chunk_size` is optional when streaming to a file

Tripped me up briefly...figured others might run into the same thing. Thanks!

It's probably important to note that you _should_ provide a chunk_size here despite it being optional. The default of 1 is likely not what the user wants for any non-trivial file.


@nateprewitt Yeah, thought about that...might make sense to give a value of `128` or something reasonable in the example, and just say that it can be changed. I'm no expert on this, so whatever y'all think!


Great! How does changing the example to:

```python
with open(filename, 'wb') as fd:
    for chunk in r.iter_content(chunk_size=128):
        fd.write(chunk)
```
and then that line saying you can adjust the chunk_size to a number that fits your requirements sound? This will probably better serve the user than pointing them towards the default behaviour of iter_content. That's just one guys thoughts though :)

I'd be happy to merge that docs change. 


@afeld if you think this change is agreeable and have the inclination, feel free to throw the PR together. Otherwise, I'll try to open something next week.
SSLError: [SSL] PEM lib (_ssl.c:2603) when I want to connect to a server with TLS

Hi, 
I can't connect myself to an Apple server using requests.
I can connect myself to this same server using openssl directly or using socket directly
For instance, I'm doing that in such way with openssl
`openssl s_client -CAfile GeoTrust_Global_CA.pem -connect api.push.apple.com:443
`
My code is pretty basic:
`s = requests.session()
    s.cert = "GeoTrust_Global_CA.pem"
    s.port = 443
    url = 'https://api.push.apple.com'
    r = s.get(url)`

@darakt You're using the `cert` argument, but you want to be using the `verify` argument.


`requests.get('url', verify=True)` is causing this issue for me.
Adding 'import requests' to readme code snippet.

The interpreter snippet at the top of the readme.rst is the first snippet of code that users will see for using requests. I think an import statement should be shown at the top of the code snippet, so a novice or even intermediate python user will know to import the module first before attempting to use it. It makes the snippet more complete. I've seen other github projects include the import statement in snippets like the one in the readme. I changed the doc/index.rst file for consistency, since they are the same code snippet.

For instance [tensorflow](https://github.com/tensorflow/tensorflow)'s readme has a similar code snippet and it has the import statement at the top of the code snippet. 


Thanks for this!


undo


Hi @kennethreitz , I was wondering why my pull request got reverted.


I prefer it the way it is, and it was made that way intentionally. 

I understand why you made this suggestion, and I appreciate it — but this is the way I want to present the project, for aesthetic reasons.
Buggy servers with SNI give unrecognized_name

When trying to open https://www.scbt.com with requests, an SSLError is raised, even though verify=False. Apache HttpClient has the same problem, but browsers open the page without any complaint. I was hoping requests would be more tolerant with such flawed servers, especially if the caller indicates that security is not of importance.
[stacktrace.txt](https://github.com/kennethreitz/requests/files/596393/stacktrace.txt)

Thanks!

Felix

Requests cannot be more tolerant of that server in the default configuration. The `unrecognized name` error you're seeing is an alert sent by the _remote server_: that is, our inability to set SNI means that the remote server is simply refusing to negotiate TLS with us. That means we cannot possibly be "tolerant" of that error: the remote server is being intolerant, not us.

I should note that the remote server's intolerance is _entirely_ reasonable. SNI is not an optional feature on the modern web, it's mandatory. For most other users I'd say that you should install the `requests[security]` extra, but right now that won't work because it relies on a CFFI extension and Jython has no support for CFFI at this time.

Unfortunately, that means that there's simply nothing we can do for you. Either Jython needs to add support for SNI, in which case Requests will automatically start using it, or Jython needs to add support for CFFI. Otherwise, however, we simply cannot help your use case.


Thanks very much for your quick reply! I was hoping to be able to imitate what the browser does, but it seems quite difficult indeed. Other servers using https work fine, but that particular one seems quite "tight". :-)


Sadly, the browser sends SNI, so it doesn't have this problem. 😞
A whitespace at the start of Url causes error on request

    In [1]: import requests

    In [2]: requests.get("http://stackoverflow.com/")
    Out[2]: <Response [200]>

    In [3]: requests.get(" http://stackoverflow.com/")
    # ... stacktrace skipped ...
    InvalidSchema: No connection adapters were found for ' http://stackoverflow.com/'

Url should have been stripped. But I may be missing something here.

I'm open to a PR that will cause us to strip URLs, that seems reasonable enough to me.


Sure.


I have created a pull request #3697. Please have a look.


PR is merged. Closing it as it's fixed.
Long timeout

I need check my service is ok or not。 sometimes it will breakdown.

### how to reproduce the issue
```
 try:
    r = requests.get("http://xxx.com/",timeout=1)
    print(r.status_code)    
except requests.exceptions.Timeout:
	print 'timeout'
except requests.exceptions.RequestException as e:	
    print(e)
   
```
it always let me wait for 30 seconds

`I hope it only waits one second, if server doesn't response or the link is unreachable, throw exception`

below are exception message

> Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x3d344d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))





The requests timeout parameter is _per connection_, and Requests will make as many connection attempts as there are domain names. However, Requests cannot control the timeout of the DNS lookup, which is the problem here. Unfortunately, Python doesn't allow us any way around this problem: there is just no cross-platform way to do a DNS lookup with a timeout using the things installed by default.

Sorry!
security extras need to be updated

Starting with `urllib3` 1.17, `ndg-httpsclient` and `pyasn1` are no longer required to inject `pyopenssl` into `urllib3`. `cryptography` and `idna` are required instead. The result of this is if an older version of `pyopenssl` is installed and you run `pip install requests[security]`, you will see import errors when trying to inject `pyopenssl`.

Yup, you're entirely correct.


Given that we're shipping a patch release, I may just fix this myself and ship it today.
v2.12.0: URL has an invalid label error

When url contains an underscore the request fails with 'URL has an invalid label error'

Seems like a result of updating to urllib3 to 1.19.

While underscores in the hostname would not meet strict RFC requirements it would still work in many scenarios, so I would propose to remove the check

Example:
```
requests.get('http://test_t')
*** InvalidURL: URL has an invalid label.
```

Thanks for opening this issue, @afaicode. This is currently being tracked in #3683 which will be the official issue for further discussion. I believe @Lukasa has stated underscores in host names are not behaviour we likely want to support, but you're welcome to raise objections in the appropriate issue.


Yup, closing in favour of #3683.
NewConnectionError  [Errno 10060]

I tried to use the `Geocoding API` of goole maps to convert the address to coordinates, but it failed which seems to be related to `requests` when I tried the sample code like:
```
gmaps=googlemaps.Client(key='MY_KEY')
geocode_result=gmaps.geocode('1600 Amphitheatre Parkway, Mountain View, CA')
print geocode_result
```
but I got an exception like 
```
googlemaps.exceptions.TransportError: HTTPSConnectionPool(host='maps.googleapis.com', port=443): Max retries exceeded with url: /maps/api/geocode/json?address=1600+Amphitheatre+Parkway%2C+Mountain+View%2C+CA&key=AIzaSyBYdEWfcUNHUP3zLtc17nX-3aE4uPUkFnY (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x0000000016239278>: Failed to establish a new connection: [Errno 10060] ',))
```
BTW, I am work on PyCharm with python version 2.7.12 and I am using a VPN, I wonder if it's related to this.
Any help is appreciated, thanks in advance.


This is almost certainly related to your networking setup. Essentially, "failed to establish a new connection" is telling you that we weren't able to connect to `maps.googleapis.com`. Exactly why and how to fix it is a different matter: there are lots of possible problems here.

What kind of VPN are you using? What version of Requests is installed?


thanks you @Lukasa ,I use `shadowsocks` and the version of my requests is `2.10.0` while it's used to be `2.9.1` and updated with the install of google map api;
btw, I can get a correct json format result if I hardcode the url `https://maps.googleapis.com/maps/api/geocode/json?address=1600+Amphitheatre+Parkway,+Mountain+View,+CA&key=AIzaSyBYdEWfcUNHUP3zLtc17nX-3aE4uPUkFnY` in my web browser.


So you should consider upgrading to Requests 2.12.0: it's possible that you're bumping into problems with our SOCKS detection. The other possibility is that you're bumping into a Requests limitation, which is that we currently always do local DNS lookups rather than getting the proxy to do the DNS lookup.


However, @googlemaps only support Requests <= `2.11.1`

@fixablecar That seems like it's their problem, not ours.
Amplifies the timeout warning in the docs.

Partially addresses #3070 until a fix can come in the next major version.

Thanks @mlissner! :sparkles:


Sorry, I don't mean to be a squeaky wheel. Isn't this already stated two sentences later? We may want to delete one so we're not saying the same thing twice.

> If no timeout is specified explicitly, requests do not time out.


I don't think I mind the duplication here: they say the same thing in different language, which is potentially useful in a case like this.


I'm fine with removing the second mention if folks want, but I agree with @Lukasa too. I didn't realize for ages that a request not timing out meant my program would hang until I killed it. It's a pretty critical issue in my opinion, so being super-duper, repetitively explicit doesn't seem like a bad thing to me.
decoding fix for Streaming Requests docs

This will address #3675 with 71b8941. I've added an additional commit (d81ad06) noting a suggested best practice of providing an encoding fallback when using the `decode_unicode` parameter. This is an optional improvement that I can strip if we only want the bare bones fix.

LGTM, let's do this! Thanks @nateprewitt! :sparkles: :cake: :sparkles:
[Errno -3] Temporary failure in name resolution

While trying to get access token from api I getting the follwoing error,

ConnectionError: HTTPSConnectionPool(host='', port=443): Max retries exceeded with url: /oauth2/access_token/ (Caused by <class 'socket.gaierror'>: [Errno -3] Temporary failure in name resolution)

Is there any way I can resolve this error?

Thanks.


This error means that a DNS problem occurred and we couldn't resolve the IP address. If you see this problem you should simply retry the request.


Already I have tried that, but it's not working for me. Shall I try with IP address Or is there any other solution for this?


Can you print the URL you're trying to reach?


I tried the same API URL in postman, its working fine. But  in my request method, it throws the above error.


So this strongly suggests you have a problem with your OS or Python configuration, or possibly your URL, because it seems like the Python code cannot resolve the domain name.


I am getting the right URL from requests. Following warning error message is displayed while running the script. Is there any issues?

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning


Those error messages aren't going to be relevant, no. Can you set `allow_redirects=False` to see if you're being redirected away?


it's working now. problem from dns system. thank you


how did u resolve dns problem???

I face the same problem, and solved by restart computer

I fixed this by disabling my VPN

You have issues with your DNS. Check on your system that public interface is configured to use correct DNS servers.

For me, it was the firewall that was blocking the connection. I'm using ufw .. the following command fixed it for me.
`sudo ufw allow out to any port 53`
don't forget to reload ufw
`sudo ufw reload`
Streaming Requests documentation error for Python 3

The [documentation](http://docs.python-requests.org/en/master/user/advanced/#streaming-requests) for streaming requests supplies a code snippet (below) that hits #3359 which is also experienced in `iter_lines` due to its use of `iter_content`. 

```python
r = requests.get('http://httpbin.org/stream/20', stream=True)

for line in r.iter_lines():
    if line:
        print(json.loads(line))
```

Since `iter_lines` default is `decode_unicode=False`, it returns bytes causing Python 3 to barf on `json.loads(lines)`. Unfortunately, `decode_unicode=True` doesn't work either because httpbin.org doesn't provide a charset in the Content-Type header.

As for fixing the example, we can manually decode `line` before passing it into json.loads. This exposes some of our dirty laundry though, and may give the user the wrong idea that they need to manually decode their strings in all cases.

Alternatively, we update httpbin to pass a charset with the Content-Type and change the example to use `decode_unicode=True`.

Edit: Moving semi-related comment to appropriate issue. (#3359)

I view this as strictly a Python 2 vs Python 3 problem, rather than a problem with `iter_lines`. `iter_lines` returns bytes by default, that's a good sensible default, so we should just decode those bytes ourselves.


Alright, I'll update the example accordingly.


Resolved with #3678.