Repository: encode/httpx

Issue:
HTTPX claims Python 3.6+ support when asyncio backend TCPStream.start_tls explicitly does not support Python 3.6
As stated, HTTPX explicitly does not support Python 3.6 [here](https://github.com/encode/httpx/blob/master/httpx/concurrency/asyncio.py#L61) but declares Python 3.6 support [here](https://github.com/encode/httpx/blob/master/setup.py#L40). This has caused problems with users of a package of mine that is supposed to support Python 3.6.


Discussion:
Unfortunately Python 3.6 doesn't ship with start_tls() support on the asyncio event loop so you'll only hit this issue when using HTTP proxies. :( If you're using Trio then you can support this use-case, there's just no way to do it in asyncio+3.6.
I understood why the check and exception existed. My main point is that if httpx doesn't fully support 3.6, it really should be 3.7+. 

My choices as a developer are to use a different back end (which is of no consequence), make my packages 3.7+ because they don't technically support 3.6, or go use a different client library again.
Well, it would be a shame to declare only 3.7+ is supported when actually the vast majority of the features (all except proxies, actually) are available on 3.6, wouldn’t it?

Note if this isn’t mentioned in the proxy docs we should probably add a note. :)

(Perhaps there *might* be a way to create or push for a backport of start_tls on 3.6?)
I'm absolutely okay with us supporting 3.6, but not providing full `proxies` support in that case. Ideally we'd have a top-level error, rather that waiting until `start_tls` is called.

I'm also somewhat suprised tho - our proxy tests don't exclude 3.6 - presumably there's succeeding because we're not actually making a live connection?

Are there alternatives here?

* Backporting `start_tls`? (Sounds fiddly)
* Issuing a new connection, rather than upgrading?


> Well, it would be a shame to declare only 3.7+ is supported when actually the vast majority of the features (all except proxies, actually) are available on 3.6, wouldn’t it?

There are no degrees of compatibility here. This is actually the first time I've ever encountered this kind of situation that wasn't a mistake. While one way to think about it is that it's "just proxy support". That also makes it quite difficult to discover. An old debate trick, go to the extreme.

> Note if this isn’t mentioned in the proxy docs we should probably add a note. :)

This doesn't really help with discovery here. Note that, in my case, my code does not handle proxies itself and it's entirely synchronous. I don't use a proxy myself with the packages in question. And I have long-since moved on from 3.6. So, why would I even think to check the aysnc backend code or proxy docs to see that httpx isn't actually Python 3.6-compatible in that particular situation like it claims. I only discovered it when a user who happens to use Python 3.6 and a proxy ran into an issue. How many others might run into this again or have already but were silent about it.

A special note is that I believe this code was added **after** I had ported my code to httpx, so it's not like I could've known beforehand.

> I'm absolutely okay with us supporting 3.6, but not providing full `proxies` support in that case. Ideally we'd have a top-level error, rather that waiting until `start_tls` is called.

I'm actually kinda surprised that there's discussion of leaving things as-is. I don't think you'd claim Python 3.4/3.5/2.7 support because part of the code (even a significant part) supported those versions. I don't want to move my packages to 3.7+ just yet either. But that would be my move unless httpx implemented ``start_tls`` itself for 3.6 with asyncio backend until 3.6 support is dropped. If httpx claims Python 3.6 compatibility when that's not true, I have to change my packages to 3.7+. If httpx moves to 3.7+, I have to change my packages to 3.7+. But, if all the code doesn't support a version of Python, it shouldn't claim it supports that version of Python. So that's what I'd do.
And, if a different perspective helps, let's talk in terms of packages ``httpx`` is dependent on. ``httpx`` has ``h11`` pinned to ``0.8.*``. I don't know if ``0.9.*`` would require any code changes in ``httpx``, but let's say it would. There are 3 traditionally viable options:

* Keep ``httpx`` pinned to ``0.8.*`` and make no code changes.
* Make code changes and pin ``h11`` to ``0.9.*``.
* Add a compatibility later in ``httpx`` to support both ``0.8.*`` and ``0.9.*`` and pin to both those versions.

Would you consider making code changes to ``httpx`` to support ``h11 0.9.*``, have part of ``httpx`` no longer work on ``h11 0.8.*``, and tell users to just not use that part of ``httpx`` unless you have ``h11 0.9.*``?
So I remembered another major reason that I wanted to continue supporting 3.6 despite start_tls() not being implemented: With uvloop installed you can get start_tls() to work with Python 3.6.

So our compatibility table looks like this:

| Python Version | Using Asyncio? | Uvloop Installed? | All Proxy Feautres Work |
|----------------|----------------|-------------------|--------------|
| >=3.7          | N/A            | N/A               | Yes          |
| 3.6            | No             | N/A               | Yes          |
| 3.6            | Yes            | Yes               | Yes          |
| 3.6            | Yes            | No                | No           |

If your user can install uvloop it'll fix their issue, not sure if that's viable for them.
@sethmlarson Shouldn't that be a conditional requirement for httpx for Python instead of pushing that onto users? At this point you are asking a user of a user to manually install something that is required for some functionality of httpx. That should be invisible to them. So what you are asking is for me and other direct httpx users to add that conditional dependency to all the projects we use httpx in that support Python 3.6. And that would involve us discovering that it is necessary for some functionality in the first place.
I agree with you it's a sub-optimal situation. I'm not in favor making a huge set of changes to our dependencies because I have a feeling that this will all be moot in a month.

Here are how I see things currently:
- We're supporting the last 2 stable Python releases. (I don't consider 3.8 stable until it has a bugfix release)
- Python 3.6 accounts for about [~10% of our users](https://pypistats.org/packages/httpx).

Here are my thoughts on a proposed solution:

- Leave our dependencies as they are.
- For the HTTPX 0.8 release we drop support for Python 3.6.
- Wait to release 0.8 until CPython 3.8.1 has a release candidate [which is scheduled for a month from now](https://www.python.org/dev/peps/pep-0569/#schedule) so we continue to support 2 stable Python versions. 
- Optionally make a change to the exception to point to documentation about how to handle the `start_tls()` situation on Python 3.6+asyncio.
I certainly wouldn't classify as adding a uvloop dependency for Python 3.6 a huge set of changes. Though I just remembered that uvloop doesn't support Windows anyway, so that would be an incomplete solution as well.

I am a bit surprised at httpx only supporting 2 major releases of Python. Python 3.6 still has 2 years of life left. Even as someone who jumps to the next on my personal machine as soon as releases drop. And Python is changing to a 1 year release cycle, so is that going to change anything? That does force dependent packages to follow suit when not necessary. Obviously dropping Python 3.6 has some good reasons, this issue among them. But what if that is not the case?
Also, I'm not sure what the definition of supporting a Python version is. If httpx isn't supposed to support Python 3.8, why is it not constrained that way in setup.py?
> I certainly wouldn't classify as adding a uvloop dependency for Python 3.6 a huge set of changes. Though I just remembered that uvloop doesn't support Windows anyway, so that would be an incomplete solution as well.

Yeah, that's why I'm not a fan of adding uvloop as a dependency.

> I am a bit surprised at httpx only supporting 2 major releases of Python.

We aren't claiming we're only supporting 2 major releases, my above bullet point is that we're currently supporting 2 stable releases by saying we support Python 3.6+. No one made the decision that that's what we're doing to do as a matter of policy, we'll support as many Python versions as we can as long as it's still viable.

> Also, I'm not sure what the definition of supporting a Python version is. If httpx isn't supposed to support Python 3.8, why is it not constrained that way in setup.py?

HTTPX works with and supports Python 3.8. I'm claiming that Python 3.8 isn't a "stable" release yet because it hasn't had a bug-fix release.


Basically what I'm trying to portray in my above comment is that if we drop Python 3.6 right now we'll only be supporting one stable Python release (3.7) which forces our users into a single bucket.
And my point is that httpx doesn't actually support Python 3.6 now, even if it only affects a small number of users. I've asked dozens of people around the office just to make sure I wasn't off base about this. Even linking this issue to them. To a person they are as baffled as I am about this.

But from this discussion, it seems my only option will be to drop Python 3.6 from any projects that I use httpx in and use Python 3.7+ for any future projects in httpx's stead.

I wish Python version usage data could be trusted. We don't really know if Python 3.6 is actually only 10% of users or not.
As @tomchristie suggested, IMO this:

> Issuing a new connection, rather than upgrading?

ie looking at if we can provide a downgraded version of the feature on 3.6 is probably the most sensible next step out of this.

(This and *for now* clarifying that proxies aren’t working (partially?) on 3.6.)
Issuing a new connection that doesn't go through the proxy tunnel could violate a user's privacy by having the DNS request for the requested domain be emitted from their machine, rather than the proxy.
We've been working on the basis of 3.6+ support from the start. Turns out one of the very newest features only supports 3.7+, which I hadn't fully appreciated, but isn't a deliberate documentation omission. (httpx is only a few months old, and the proxy support feature is only a few weeks old.)

Constructively our options are:

* Clearly document `proxies` support as 3.7+ only. Make sure it's usage under 3.6 raises a clear error. _(Not ideal, but reasonable enough)_
* Include uvloop as a dependency which only installs on 3.6 *nix platforms, and clearly document `proxies` support as not supported on Windows under 3.6. _(Would be perfectly reasonable, and probably? the best option.)_
* Some alternative technical solution to bringing full proxies support to 3.6. _(If there is an option there, that'd be wonderful.)_
* Drop 3.6 support. _(Unneccessarily blunt.)_

Personally, I'm in favor of option 2:
> - Include uvloop as a dependency which only installs on 3.6 *nix platforms, and clearly document proxies support as not supported on Windows under 3.6. (Would be perfectly reasonable, and probably? the best option.)

Certainly easy to do with constraints in `setup.py`:
`"uvloop; sys_platform == 'linux' and python_version < '3.7'"`
> * Some alternative technical solution to bringing full proxies support to 3.6. _(If there is an option there, that'd be wonderful.)_

Just found this: https://github.com/urllib3/urllib3/issues/1323#issuecomment-362494839 🎉

Resolving patch:
diff --git a/httpx/concurrency/asyncio.py b/httpx/concurrency/asyncio.py
--- a/httpx/concurrency/asyncio.py
+++ b/httpx/concurrency/asyncio.py
@@ -58,16 +58,49 @@ async def start_tls(
         self, hostname: str, ssl_context: ssl.SSLContext, timeout: TimeoutConfig
     ) -> "SocketStream":
         loop = asyncio.get_event_loop()
-        if not hasattr(loop, "start_tls"):  # pragma: no cover
-            raise NotImplementedError(
-                "asyncio.AbstractEventLoop.start_tls() is only available in Python 3.7+"
-            )
 
         stream_reader = asyncio.StreamReader()
         protocol = asyncio.StreamReaderProtocol(stream_reader)
         transport = self.stream_writer.transport
 
-        loop_start_tls = loop.start_tls  # type: ignore
+        if hasattr(loop, "start_tls"):
+            loop_start_tls = loop.start_tls  # type: ignore
+        else:
+
+            async def loop_start_tls(
+                transport: asyncio.BaseTransport,
+                protocol: asyncio.BaseProtocol,
+                sslcontext: ssl.SSLContext = None,
+                *,
+                server_side: bool = False,
+                server_hostname: str = None,
+                ssl_handshake_timeout: float = None,
+            ) -> asyncio.Transport:
+                """Python 3.6 asyncio doesn't have a start_tls() method on the loop
+                so we use this function in place of the loop's start_tls() method.
+                Adapted from this comment:
+                https://github.com/urllib3/urllib3/issues/1323#issuecomment-362494839
+                """
+                import asyncio.sslproto
+
+                waiter = loop.create_future()
+                ssl_protocol = asyncio.sslproto.SSLProtocol(
+                    loop,
+                    protocol,
+                    sslcontext,
+                    waiter,
+                    server_side=False,
+                    server_hostname=server_hostname,
+                    call_connection_made=False,
+                )
+
+                transport.set_protocol(ssl_protocol)
+                loop.call_soon(ssl_protocol.connection_made, transport)
+                loop.call_soon(transport.resume_reading)  # type: ignore
+
+                await waiter
+                return ssl_protocol._app_transport
+
         transport = await asyncio.wait_for(
             loop_start_tls(
                 transport=transport,


Test patch:
diff --git a/tests/test_concurrency.py b/tests/test_concurrency.py
--- a/tests/test_concurrency.py
+++ b/tests/test_concurrency.py
@@ -1,5 +1,3 @@
-import sys
-
 import pytest
 import trio
 
@@ -31,9 +29,6 @@ async def test_start_tls_on_socket_stream(https_server, backend, get_cipher):
     See that the concurrency backend can make a connection without TLS then
     start TLS on an existing connection.
     """
-    if isinstance(backend, AsyncioBackend) and sys.version_info < (3, 7):
-        pytest.xfail(reason="Requires Python 3.7+ for AbstractEventLoop.start_tls()")
-
     ctx = SSLConfig().load_ssl_context_no_verify(HTTPVersionConfig())
     timeout = TimeoutConfig(5)