HSTS support

Hello!

Not sure this falls in the scope of requests, but nobody seems to have mentioned it so far so here it goes :)

I think it would be great to have an opt-in flag for requests that reads the [HSTS preload list](https://hstspreload.org/), and automatically switches to https if there's a match, just like a browser would.

This would save users from reimplementing this logic when they want to avoid useless http requests (they could already have known they would result in a redirect to https).

Additionally, it could be great to store HSTS headers from the server responses locally (just like a cookie jar?)

Thanks!

Hi there!

This is a substantial feature request, and it's one we do want to support, but there's a big chunk of work involved. There has been some initial progress made on shazow/urllib3#608, which could be used to bootstrap this work into urllib3 (which is where it would have to go).

However, some of these things have got real challenges. For the HSTS preload list, there are two issues. First, the preload list is a nearly 2-megabyte JSON text file that needs to be deserialized. We could attempt to process that, but it's not going to be the fastest thing to do and it'll expand to many more megabytes of memory. This is all stuff I'd not want to ship in Requests by default. Second, the preload list is regularly updated, but Requests often isn't. That means we'd have to take an approach like certifi, which is not the end of the world, but definitely troublesome.

Anyway, all of this is to say: yeah, we can do this. But it isn't a high priority right now. Anyone who wants to work on it should feel free to pick up and run with the urllib3 work, but we have other priorities that we'd like to address before this.

Thanks for the suggestion though!

@Lukasa thanks for the feedback!

I understand the concerns for performance (I care more about being able to do one less request than the security aspects to be frank). I think we could use a disk-based format that makes the lookups fast and memory-efficient?

@sylvinus So, the most appropriate way to do it would be to use a sqlite DB, but ultimately that's putting the cart before the horse. If we can get support for HSTS *without* preload that persists to disk, then essentially the preload list becomes nothing more than a priming of the HSTS DB.

Again, I recommend you look at the direction that was taken in shazow/urllib3#608 to get an idea of what we were looking at. If you're interested, I'm sure you can pick the code up and try to bring it back up to date.

Okay, thanks! I'll have a look.

Hi, I'd say HSTS primarily concerns returning to website using HTTPS. I think that could easily be implemented in requests, just as a cache of websites that have redirected to HTTPS and set Strict-Transport-Security header. HSTS preload is completely different story. Right now I'd have to resort to manually flagging a domain name in my application so subsequent requests would use HTTPS by default.

@laurivosandi Maybe I'm not understanding you, but I don't think any of what I said above is materially different to what you're saying here. I said above that HSTS preload is a complex problem but that HSTS without preload could be done, but it is not currently a high priority for any of the maintainers. As always, I've linked to the initial prototyping work in urllib3: if you want to resurrect that work, you're more than welcome.