Repository: encode/httpx

Issue:
Consider dropping HSTS preloading
Prompted by https://github.com/encode/httpx/issues/1025#issuecomment-661269344

Issues like #1025 should prompt us to reconsider whether using HSTS preloading from a server-side client (introduced via #151) is a sensible thing to do at all.

Key points of reasoning here:

- While it mimics the way browsers behave, and [hstspreload](https://github.com/sethmlarson/hstspreload) is a super valuable library, from a UX perspective it can lead to ambiguous behaviors (#896) and cases when you _really_ don't want HSTS preload to kick in (like in #1025).
- Crucially, from #124 it seems that HSTS preloading was introduced as a way to tackle "schema-less URLs". As of today HTTPX _requires_ a schema on URLs, so if it was the initial motivation then it is now irrelevant. Besides, we can assume that developers are generally very aware of their choices to use `http://` vs `https://` on the server-side (as opposed to browser users just entering the domain name without a schema).

Some options are:

- Add a toggle so that HSTS preload becomes opt-in or opt-out.
- Drop HSTS functionality entirely.

Given all the context points above, we might want to just go with option 2).
Document HSTS is still honored if allow_redirects=False
I tried to disable the redirection with latest httpx but it is always redirection the URL:

```python
>>> import httpx
>>> r = httpx.head("http://www.python.org",  allow_redirects=False)
>>> r.url
URL('https://www.python.org')
```

status code is also 200 and not 301 as expected.


Discussion:
My vote is for `Add a toggle so that HSTS preload becomes opt-in`, though for 1.0 it might be better to reduce the surface area and drop it entirely.
so it's probably happening because the domain is using HSTS: 
https://github.com/encode/httpx/blob/94323f98ac812031dcf0bfecf6227beb2afd9c41/httpx/_client.py#L213-L222

I will leave it open as I expected that allow_redirects=False would prevent that. At least the current behaviour which is corrent could be documented :)
Thanks, just reproduced quickly locally and it does look like we're just honoring HSTS, but no HTTP redirection is being made. I think your suggestion to document this behavior makes sense, so I updated the issue title to reflect this…
Here's the `TRACE` to confirm that no redirect was made, we just made the request using HTTPS in the first place:

```console
$ HTTPX_LOG_LEVEL=trace python -m asyncio
>>> import httpx
>>> async with httpx.AsyncClient() as c:
...     r = await c.head('http://www.python.org', allow_redirects=False)
...
TRACE [2020-04-07 11:55:59] httpx._config - load_ssl_context verify=True cert=None trust_env=True http2=False
TRACE [2020-04-07 11:55:59] httpx._config - load_verify_locations cafile=/Users/florimond.manca/.pyenv/versions/3.8.1/lib/python3.8/site-packages/certifi/cacert.pem
TRACE [2020-04-07 11:55:59] httpx._dispatch.connection_pool - acquire_connection origin=Origin(scheme='https' host='www.python.org' port=443)
TRACE [2020-04-07 11:55:59] httpx._dispatch.connection_pool - new_connection connection=HTTPConnection(origin=Origin(scheme='https' host='www.python.org' port=443))
TRACE [2020-04-07 11:55:59] httpx._dispatch.connection - start_connect tcp host='www.python.org' port=443 timeout=Timeout(timeout=5.0)
TRACE [2020-04-07 11:55:59] httpx._dispatch.connection - connected http_version='HTTP/1.1'
TRACE [2020-04-07 11:55:59] httpx._dispatch.http11 - send_headers method='HEAD' target='/' headers=Headers({'host': 'www.python.org', 'user-agent': 'python-httpx/0.12.0', 'accept': '*/*', 'accept-encoding': 'gzip, deflate', 'connection': 'keep-alive'})
TRACE [2020-04-07 11:55:59] httpx._dispatch.http11 - send_data data=Data(<0 bytes>)
TRACE [2020-04-07 11:55:59] httpx._dispatch.http11 - receive_event event=NEED_DATA
TRACE [2020-04-07 11:56:00] httpx._dispatch.http11 - receive_event event=Response(status_code=200, headers=[(b'connection', b'keep-alive'), (b'content-length', b'49040'), (b'server', b'nginx'), (b'content-type', b'text/html; charset=utf-8'), (b'x-frame-options', b'DENY'), (b'via', b'1.1 vegur'), (b'via', b'1.1 varnish'), (b'accept-ranges', b'bytes'), (b'date', b'Tue, 07 Apr 2020 09:56:00 GMT'), (b'via', b'1.1 varnish'), (b'age', b'1149'), (b'x-served-by', b'cache-bwi5131-BWI, cache-cdg20742-CDG'), (b'x-cache', b'HIT, HIT'), (b'x-cache-hits', b'1, 7'), (b'x-timer', b'S1586253360.041811,VS0,VE0'), (b'vary', b'Cookie'), (b'strict-transport-security', b'max-age=63072000; includeSubDomains')], http_version=b'1.1', reason=b'OK')
DEBUG [2020-04-07 11:56:00] httpx._client - HTTP Request: HEAD https://www.python.org "HTTP/1.1 200 OK"
TRACE [2020-04-07 11:56:00] httpx._dispatch.http11 - receive_event event=EndOfMessage(headers=[])
TRACE [2020-04-07 11:56:00] httpx._dispatch.http11 - response_closed our_state=DONE their_state=DONE
TRACE [2020-04-07 11:56:00] httpx._dispatch.connection_pool - release_connection connection=HTTPConnection(origin=Origin(scheme='https' host='www.python.org' port=443))
TRACE [2020-04-07 11:56:00] httpx._dispatch.connection - close_connection
TRACE [2020-04-07 11:56:00] httpx._dispatch.http11 - send_event event=ConnectionClosed()
```

Resolving patch:
diff --git a/httpx/_client.py b/httpx/_client.py
--- a/httpx/_client.py
+++ b/httpx/_client.py
@@ -2,7 +2,6 @@
 import typing
 from types import TracebackType
 
-import hstspreload
 import httpcore
 
 from ._auth import Auth, BasicAuth, FunctionAuth
@@ -209,15 +208,7 @@ def _merge_url(self, url: URLTypes) -> URL:
         Merge a URL argument together with any 'base_url' on the client,
         to create the URL used for the outgoing request.
         """
-        url = self.base_url.join(relative_url=url)
-        if (
-            url.scheme == "http"
-            and hstspreload.in_hsts_preload(url.host)
-            and len(url.host.split(".")) > 1
-        ):
-            port = None if url.port == 80 else url.port
-            url = url.copy_with(scheme="https", port=port)
-        return url
+        return self.base_url.join(relative_url=url)
 
     def _merge_cookies(
         self, cookies: CookieTypes = None
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -56,7 +56,6 @@ def get_packages(package):
     zip_safe=False,
     install_requires=[
         "certifi",
-        "hstspreload",
         "sniffio",
         "chardet==3.*",
         "idna==2.*",


Test patch:
diff --git a/tests/client/test_client.py b/tests/client/test_client.py
--- a/tests/client/test_client.py
+++ b/tests/client/test_client.py
@@ -175,25 +175,10 @@ def test_base_url(server):
 
 
 def test_merge_url():
-    client = httpx.Client(base_url="https://www.paypal.com/")
-    request = client.build_request("GET", "http://www.paypal.com")
-    assert request.url.scheme == "https"
-    assert request.url.is_ssl
-
-
-@pytest.mark.parametrize(
-    "url,scheme,is_ssl",
-    [
-        ("http://www.paypal.com", "https", True),
-        ("http://app", "http", False),
-        ("http://192.168.1.42", "http", False),
-    ],
-)
-def test_merge_url_hsts(url: str, scheme: str, is_ssl: bool):
-    client = httpx.Client()
-    request = client.build_request("GET", url)
-    assert request.url.scheme == scheme
-    assert request.url.is_ssl == is_ssl
+    client = httpx.Client(base_url="https://www.example.com/")
+    request = client.build_request("GET", "http://www.example.com")
+    assert request.url.scheme == "http"
+    assert not request.url.is_ssl
 
 
 def test_pool_limits_deprecated():