verify=False and requests.packages.urllib3.disable_warnings()

As of 1.9 of `urllib3`, the following warning appears once per invocation:

```
/usr/local/lib/python2.7/site-packages/requests-2.4.0-py2.7.egg/requests/packages/urllib3/connectionpool.py:730: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html (This warning will only appear once by default.)
  InsecureRequestWarning)
```

When using `verify=False` would it be useful to also set `requests.packages.urllib3.disable_warnings()`?

I understand that this is a design decision that not everyone might agree with. :)


I think you can disable these at a global level with the `warnings` module. Further, to work with logging (if I remember correctly) you need to reach into `urllib3` (and we document that), so I'm not against documenting this for users who are not going to use certificate verification for HTTPS connections.


I remain strongly in favour of leaving these warnings in place. Yes, they're annoying, but they're also there for a reason. If anything, I'm tempted to switch it off and replace it with one of our own! =P


At this point it's fairly obvious that @Lukasa and I are -1 on this feature. @kennethreitz @shazow any opinions?


To some extent, I agree that the warnings are important, but I think that there are multiple factors that might need to be considered.

From a developer perspective, know that I know about this, I can just turn this off if I feel inclined. I'm newer to packages, so it when I read the docs, in the warning, that solution didn't actually work. I like the idea that @Lukasa presented about making something that 's specific to `requests`. 

From a user perspective, I installed `pyvmomi` with `pip`today, which uses `requests` in its internals. It's really a non-transparent error that gets emitted back to the user in a case where `requests` is a silent supporting library.


Yes, `requests.packages.urllib3.disable_warnings()` is a shortcut for turning it off using the warnings module's filtering.

I'd strongly recommend having some kind of warning to this effect. +0.5 on having the urllib3 one propagate, +1 if you want to put in the effort and adding a requests-specific one. -1 on having no warning.

If you'd like, we can make the urllib3 warning message configurable and you can override it so you can piggyback on the same logic otherwise.


Once again, I don't consider this message user-hostile, I consider it extremely valuable. You now know that pyvmomi has turned off TLS certificate verification, which is fairly important information!

That said, I don't object to us having a more requests-y way to silence it.


Yeah, really I think this is a bug in `pyvmomi`. If they don't want to be embarrassed like this, they should be disabling warnings in their tool. It isn't our job to _not_ warn users that the connections some tool is making could expose them to MITM attacks because the tool is not performing certificate verification.


Thanks for the discussion folks! I do appreciate everyone commenting, and helping me see the value in the way that things are done, and why. I was having a hard time seeing the more general cases! :)

Will work on a patch today, as time allows, to have a 'requests-y' way to silence. Feedback will be welcome!


@invisiblethreat feel free to hop into IRC if you have questions


Just wondering if you considered the case where requests is used in a webhook. I have to suppress the warning so it doesn't pollute the JSON output of the script (or am I missing something?)


@macterra I'm not sure I understand. Are you looking for alternative strategies for disabling the warning or ...?


It's also extremely unclear to me why your webhook is disabling certificate verification. I'd be wanting to leave that on.


Further, if you're piping stdout from some script to get the results, warnings will come out from stderr so they should not pollute your JSON output.


Right, if the warning is on stderr then no problem. I couldn't tell from looking at the output in the console, my mistake.


I think we should customize this to point to the requests documentation instead of urllib3's documentation.


I'm not sure I understand what this warning feature is meant to achieve and how the warning is meant to be controlled.

I have a module which uses `requests` and has to make requests with the `verify=False` argument. This causes the users of my module to see the unnecessary warning. Unnecessary warnings make important warnings more difficult to see,

It would make sense for my module to disable the warning, but then _I would be disabling the warning also in any other module_ using `requests` in the application! 

Things are not better if I need to instruct the user of my module to disable the warning: The fact that I'm using `requests` is normally just an invisible implementation detail the user shouldn't need to know. And the user still would only have the option to silence everything or noting.

I don't think global warnings are going to be helpful.

I could subclass `urllib3.HTTPSConnectionPool` and override `_validate_conn()` and make `requests` use that in my module to avoid hiding warnings from other modules, but that seems to be too much work for a simple thing.


> I'm not sure I understand what this warning feature is meant to achieve
> 
> causes the users of my module to see the unnecessary warning.

When you set `verify=False` you are no longer securing your network connections. That, IMHO, is _not_ an unnecessary warning, it's a hugely relevant warning. Your users, who previously had no idea that you weren't verifying certificates, now know this to be true.

If the warning is not valuable for your module, it is not likely to be valuable for any other module in the application (once you've thrown away security in one network point there's no point crowing about it everywhere else). I don't really see a problem with disabling it globally.


When the user explicitly requests `verify=False` for a particular request, I don't see the value of showing a warning. When I as a module author set `verify=False`, I set it by user's request (or I'm being malicious, but the warning doesn't help there either, because I can silence the warnings). Indeed, I want to avoid being malicious and don't want to globally silence warnings, because that takes away the useful warning for those requests some other part of the application is unknowingly making insecurely.

Also, having the warning on for those requests where verification is explicitly turned off by by the user also hides the requests where the user would want the warning, since the warning is given only once. The warning is also not really helpful for the user, since it doesn't mention the URL of the particular request.

I don't agree that throwing away security in one network point somehow makes any security checks useless in the application, neither do any browser vendors. Browsers let me bypass security checks for individual URLs but keep checking the rest and I like that.

When I have a tool which talks to an internal server with a self-signed cert in an internal network, but also communicates with outside hosts, I do want to verify the outside communication. This would be the situation where I would like as a user to see warnings about accidentally unsecured requests.

Consider I'm making `service_foo` and someone uses it in an app:

``` python
import service_foo
import requests

session = service_foo.Session('https://10.0.0.1', verify=False)
data = session.get_data()
requests.put('https://example.com/submit', data=data)
```

I have 2 options for `service_foo`:
1. I keep global security warning on
   - User always gets a warning when the application talks to `https://10.0.0.1`
   - User never gets a warning even if request to `https://example.com/submit` is not safe
2. Turn off global security warning:
   - User never gets a warning even if request to `https://example.com/submit` is not safe

I don't think either option is good, but option 1 is worse, because it is giving false alarms. But I'm not comfortable having security checks turned off for the user as a side-effect of using my module.

If I would do this with a shell script, the user would be happier and safer:

```
curl --insecure -o data https://10.0.0.1/get_data
curl --upload-file data https://example.com/submit
```

To me it only makes sense to give a warning if the configuration of the Python platform is broken. The page https://urllib3.readthedocs.org/en/latest/security.html linked to in the `InsecureRequestWarning` message is indeed geared to show how to fix problems in the platform. If the user requests to skip verification, there shouldn't be a warning like there isn't a warning if the user requests an `http` URL instead of an `https` one.


> When the user explicitly requests verify=False for a particular request, I don't see the value of showing a warning.

Who is the 'user'? Throughout your post this question kept coming to my mind, because I believe you're conflating two audiences.

> When I as a module author set verify=False, I set it by user's request (or I'm being malicious).

Or you're being negligent. You have had had users complain that they couldn't interop with their self-signed certs and so you turned cert verification off, despite the fact that turning off cert verification is _not_ the way to deal with that problem.

> that takes away the useful warning for those requests some other part of the application is unknowingly making insecurely

This sentence baffles me. It suggests that it is acceptable to warn when an application _unknowingly_ makes insecure requests, but that an application _knowingly_ making them is somehow OK. I do not see how knowingly making insecure requests should in any way be considered 'more secure' than unknowingly doing so.

> Also, having the warning on for those requests where verification is explicitly turned off by by the user

Which user? How are we to distinguish between the module author and the 'user', whoever they may be?

> The warning is also not really helpful for the user, since it doesn't mention the URL of the particular request.

The warning should not mention the URL of the request, because that would risk generating warning spam. We warn _once_, saying 'this application is at risk', not 'this particular communication is at risk'.

> Browsers let me bypass security checks for individual URLs but keep checking the rest and I like that.

Browser vendors _warn_ when you access a URL with an invalid cert! They print dialog boxes and highlight the URL bar in red! That's _exactly_ what we're doing. We aren't stopping you from doing anything, we're just saying "hey, this is bad!" What you're asking us to do is the same as asking browser vendors to allow users to turn off that red warning for particular URLs, and I guarantee that they will refuse to do so because the security implications are monstrous.

> When I have a tool which talks to an internal server with a self-signed cert in an internal network, but also communicates with outside hosts, I do want to verify the outside communication.

No, you want to verify _all_ the communication. Verify the self-signed cert! Validate that you got the certificate you expected to get. `verify=False` should be considered a sledgehammer approach to security, effectively saying "screw security, just make it work". That's absolutely fine, you have a right to say that, but we have an obligation to call it out as insecure.

> I don't think either option is good, but option 1 is worse, because it is giving false alarms.

Option 1 is not giving false alarms, it's giving real alarms. The communication to 10.0.0.1 is _insecure_, and we should not be pretending otherwise.

> If I would do this with a shell script, the user would be happier and safer.

The user may well be happier, but they wouldn't be safer. They'd be exactly as safe as they were before. You seem to be under the impression that turning off this warning somehow magically makes certificate verification go away, and it does not. I'll touch on this again at the end of this response.

> To me it only makes sense to give a warning if the configuration of the Python platform is broken. 

No, we should fail hard if the configuration of the Python platform is broken and you didn't ask for unverified requests. If your platform cannot make safe TLS connections then we absolutely should not be making them, except in the situation where our user expressly tells us not to care (by setting `verify=False`), in which case we should warn that what you're about to do is dangerous.

I think you're labouring under a misapprehension, so I'd like to make something very clear: there is no way to make an unvalidated HTTPS request with requests without either a) setting `verify=False` (our warn behaviour) or b) deliberately sabotaging the `ssl` module. We cannot catch b) and do not warn on it. This is the only situation that would fall under the notion you raised of "problems with the platform". The advice on urllib3's help page does not apply to us because we take all the necessary platform related steps, including bundling trusted certs and manually verifying certificates.

There is a dangerous view in the web community that you should only verify certificates signed by trusted root certificates. This view is utterly misguided. If you're encountering self-signed certificates you should damn well validate them. That's totally do-able! Add the self-signed cert to a `.pem` file and pass it as an argument to `verify`!

If you're having trouble with combining that with the bundled `.pem` file, please let me know and I'll enhance mkcert.org to enable you to concatenate your own certificates with the trusted roots. But please, don't pretend that setting `verify=False` is safe: it simply isn't.


> Also, having the warning on for those requests where verification is explicitly turned off by by the user also hides the requests where the user would want the warning, since the warning is given only once.

This is also a bit baffling. By setting `verify=False` you maybe explicitly turning it off for just that request, but there is no way to convey that beyond the point where we construct our request. There's also no reason to convey it beyond that point because you've disabled certificate verification. The context in which you did so is of no consequence to us or anyone using your app.


> What you're asking us to do is the same as asking browser vendors to allow users to turn off that red warning for particular URLs, and I guarantee that they will refuse to do so because the security implications are monstrous.

My browsers allow me to permanently accept an unverified certificate, which is "monstrously insecure".

> The communication to 10.0.0.1 is insecure, and we should not be pretending otherwise.

The connection is insecure in the way that you can't verify a digital certificate, but verifying a certificate doesn't really tell you whether the server you are talking to is secure. But when I'm talking to a server in a closed network I can really make sure the security of the server.

> I think you're labouring under a misapprehension, so I'd like to make something very clear: there is no way to make an unvalidated HTTPS request with requests without either a) setting verify=False (our warn behaviour) or b) deliberately sabotaging the ssl

I'm trying to wonder how I could be a good citizen in my module by honoring the user's wish to ignore certificate checks and warnings for the URL they give to me. And what value the warning model adds. What is the case where a request with `verify=False` should show a warning to the user?

I don't see how the warning mechanism can catch negligent code, since it can't distinguish whether a request is made because of sloppy coding or because a user requested so. I don't think a module like `requests` should dictate a security policy either. I've understood warnings are usually aimed for developers so they can fix incorrect code, but this warning isn't like that. If the warning is just for the general education of the user, there should be an easy way for the user to hide it.

Getting the warning isn't only cosmetic either, because it messes the output of a program.

I only see negative value of the warning so I'm turning it off in my module even if I hate to hide such a global policy change there.

> There is a dangerous view in the web community that you should only verify certificates signed by trusted root certificates. This view is utterly misguided.

Didn't know there's a view like that. A cert signed by a root cert doesn't really prove anything about the security of the site. It's cheap to get an anonymous cert if you want to do bad things.

> If you're encountering self-signed certificates you should damn well validate them. That's totally do-able! Add the self-signed cert to a .pem file and pass it as an argument to verify!

The user would need a secure channel to obtain the cert, like an internal trusted network. But if the server is itself in the same internal network, there is not much gained. But this is in any case something the user decides, I can't impose a policy in my module.


I agree with @kankri, for the most part. That was the original design intention.

I propose something — that we disable by default but have our own function for re-enabling it, or documenting how to turn it on. I don't want out users to have to go out of the way to use the code as intended. `verify=False` is a feature, albeit not a best-practice one. That's none of our business. 


I agree that `verify=False` is a feature, but I don't agree that it's a feature on the same level as `params=` or `cert=`. It's a feature that defaults to a secure value and may be set to an insecure one. It's a giant, tempting option for people to throw security out the window for the sake of expediency, and I think that impulse should be resisted (but not disallowed). I will always lean towards the 'you must explicitly be insecure' school of thought, and I don't care if that means flicking two switches and not one.

Regardless, this is your call not mine. =)


Just wanted to say I agree with @kankri and that @kennethreitz's remark

> verify=False is a feature, albeit not a best-practice one. That's none of our business. 

sums it up well.


For those who also want to disable the warnings, this is how to do it. You need to use the [warnings modules](https://docs.python.org/2/library/warnings.html), which is part of the standard library:

```
import warnings
import requests
from requests.packages.urllib3 import exceptions

with warnings.catch_warnings():
    warnings.simplefilter("ignore", exceptions.InsecureRequestWarning)
    warnings.warn('a non-requests warning is not blocked')
    print requests.get('https://rsa-md5.ssl.hboeck.de/', verify=False)
```

This configures a warning filter that will ignore any warning of category `InsecureRequestWarning`. The output looks like:

```
test.py:46: UserWarning: a non-requests warning
  warnings.warn('a non-requests warning is not blocked')
<Response [403]>
```

(The test site happens to return a 403 Forbidden page, but that's not important here.)

Note that you need to use the class from the bundled `urllib3` package, and not the class from a top-level `urllib3` package, if you happen to have one installed.

You can (and probably should) make a small function that uses the context manager in the smallest possible region of code:

```
def silent_unverified_get(*args, **kwargs):
    kwargs['verify'] = False
    with warnings.catch_warnings():
        warnings.simplefilter("ignore", exceptions.InsecureRequestWarning)
        return requests.get(*args, **kwargs)
```


Or simply do this:

``` python
requests.packages.urllib3.disable_warnings()
```


@Lukasa 

> Or simply do this:
> 
> ```
> requests.packages.urllib3.disable_warnings()
> ```

Except that I find no mention of this function in the requests manual.

While it is far from everybody who knows about it, I would argue that the `warnings` module _is_ the standard tool a Python programmer should look to when he wants to disable warnings. It is part of the standard library and is well documented.

I suggest putting a reference to `warnings` into the `requests` documentation — or to the convenience `disable_warnings` function if you like, as long as there is a corresponding `enable_warnings` function (it [seems that there isn't such a function](https://urllib3.readthedocs.org/en/latest/security.html#insecurerequestwarning)).


Once again: I don't want to disable warnings in general. I just want this particular warning to go away when I _explicitly_ set verify=False in my code. There may be other, useful, warnings, unlike this particular, useless, warning. What is so difficult to understand about this?!


@zaitcev At the risk of repeating myself:

``` python
requests.packages.urllib3.disable_warnings()
```


And if even that is too broad for you:

``` python
from requests.packages.urllib3.exceptions import InsecureRequestWarning

requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
```