Repository: dask/distributed

Issue:
Token based security for workers
This came up when working with @mrocklin on daskernetes.

It'd be great if when launching the scheduler I can generate a cryptographically good-enough random token, and all workers then need to provide this token to connect. This allows us to use this as a simpler form of security (just disallow unauthenticated endpoints) than setting up a full blown mutually authenticated TLS setup. 




Discussion:
cc @pitrou

On Wed, Jan 10, 2018 at 8:57 PM, Yuvi Panda <notifications@github.com>
wrote:

> This came up when working with @mrocklin <https://github.com/mrocklin> on
> daskernetes.
>
> It'd be great if when launching the scheduler I can generate a
> cryptographically good-enough random token, and all workers then need to
> provide this token to connect. This allows us to use this as a simpler form
> of security (just disallow unauthenticated endpoints) than setting up a
> full blown mutually authenticated TLS setup.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1686>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszJ0qNq0B6d2m929fz2huvV1BnPgSks5tJXiZgaJpZM4RaP9J>
> .
>

Did anything ever come of this idea?
@darenr not that I know of.  I suspect that if someone chooses to work on this then they'll continue the conversation here.  I think that this would be a great feature!
cc @rcthomas @shreddd , you were asking about this.

@jcrist @jacobtomlinson do you have thoughts on how hard this would be to implement?
I think this would have to be done somewhere in the comm protocol, but haven't thought too much about this.

FWIW in cases like this I usually mint self-signed certs and secure the cluster with TLS. In this case the certs function much like a token, but we can offload the security onto another system. If you're using token based security you're going to want to have encryption as well to not leak the token. Minting certs per cluster handles both these cases just fine.

Is this something that we should be providing automatically for users who are more security minded, but don't have a proper CA around?
(also, your response was exactly the suggestion of @rcthomas (NERSC) in a recent phone call)
I'm not sure. Maybe a utility function for creating the certs. Moving certs around will also need to be done in a secure manner and would be backend specific. I think there's a way we could do this automatically for most backends (I at least know how to do this for dask-yarn and dask-kubernetes). Perhaps a `security` kwarg to a cluster manager:

```
security : bool or Security, optional
    A security object to use to secure the cluster with TLS.
    Set to True to have self-signed certs created automatically.
    Default is False for no TLS.
```
OK, so maybe we try this out for LocalCluster as an example.  Get the various cluster manager folks to sign off on it, and then raise an issue on each of the dask-foo repositories?
Sure, we could do that.
@rcthomas you had some preferred way to generate certs.  Would you have time to share it here?
This is what we use in dask-gateway: https://github.com/dask/dask-gateway/blob/master/dask-gateway-server/dask_gateway_server/tls.py
https://github.com/jupyterhub/jupyterhub/pull/2055 Is what we do in JupyterHub now
https://github.com/LLNL/certipy looks awesome! (Used in the PR @yuvipanda mentioned)
@mrocklin sorry, it was indeed certipy
@jcrist I think that a meeting or two ago you may have informally volunteered to investigate this.   Is this still something that you're interested in?
Sorry, yes, hopefully will get to it by tomorrow.

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -523,8 +523,10 @@ class Client(Node):
         Claim this scheduler as the global dask scheduler
     scheduler_file: string (optional)
         Path to a file with scheduler information if available
-    security: (optional)
-        Optional security information
+    security: Security or bool, optional
+        Optional security information. If creating a local cluster can also
+        pass in ``True``, in which case temporary self-signed credentials will
+        be created automatically.
     asynchronous: bool (False by default)
         Set to True if using this client within async/await functions or within
         Tornado gen.coroutines.  Otherwise this should remain False for normal
@@ -659,8 +661,15 @@ def __init__(
             if security is None:
                 security = getattr(self.cluster, "security", None)
 
-        self.security = security or Security()
-        assert isinstance(self.security, Security)
+        if security is None:
+            security = Security()
+        elif security is True:
+            security = Security.temporary()
+            self._startup_kwargs["security"] = security
+        elif not isinstance(security, Security):
+            raise TypeError("security must be a Security object")
+
+        self.security = security
 
         if name == "worker":
             self.connection_args = self.security.get_connection_args("worker")
diff --git a/distributed/deploy/local.py b/distributed/deploy/local.py
--- a/distributed/deploy/local.py
+++ b/distributed/deploy/local.py
@@ -56,7 +56,10 @@ class LocalCluster(SpecCluster):
         like ``['feed', 'run_function']``
     service_kwargs: Dict[str, Dict]
         Extra keywords to hand to the running services
-    security : Security
+    security : Security or bool, optional
+        Configures communication security in this cluster. Can be a security
+        object, or True. If True, temporary self-signed credentials will
+        be created automatically.
     protocol: str (optional)
         Protocol to use like ``tcp://``, ``tls://``, ``inproc://``
         This defaults to sensible choice given other keyword arguments like
@@ -122,7 +125,15 @@ def __init__(
 
         self.status = None
         self.processes = processes
-        security = security or Security()
+
+        if security is None:
+            # Falsey values load the default configuration
+            security = Security()
+        elif security is True:
+            # True indicates self-signed temporary credentials should be used
+            security = Security.temporary()
+        elif not isinstance(security, Security):
+            raise TypeError("security must be a Security object")
 
         if protocol is None:
             if host and "://" in host:
diff --git a/distributed/security.py b/distributed/security.py
--- a/distributed/security.py
+++ b/distributed/security.py
@@ -1,3 +1,7 @@
+import datetime
+import tempfile
+import os
+
 try:
     import ssl
 except ImportError:
@@ -76,6 +80,67 @@ def __init__(self, **kwargs):
         self._set_field(kwargs, "tls_worker_key", "distributed.comm.tls.worker.key")
         self._set_field(kwargs, "tls_worker_cert", "distributed.comm.tls.worker.cert")
 
+    @classmethod
+    def temporary(cls):
+        """Create a new temporary Security object.
+
+        This creates a new self-signed key/cert pair suitable for securing
+        communication for all roles in a Dask cluster. These keys/certs exist
+        only in memory, and are stored in this object.
+
+        This method requires the library ``cryptography`` be installed.
+        """
+        try:
+            from cryptography import x509
+            from cryptography.hazmat.backends import default_backend
+            from cryptography.hazmat.primitives import hashes
+            from cryptography.hazmat.primitives import serialization
+            from cryptography.hazmat.primitives.asymmetric import rsa
+            from cryptography.x509.oid import NameOID
+        except ImportError:
+            raise ImportError(
+                "Using `Security.temporary` requires `cryptography`, please "
+                "install it using either pip or conda"
+            )
+        key = rsa.generate_private_key(
+            public_exponent=65537, key_size=2048, backend=default_backend()
+        )
+        key_contents = key.private_bytes(
+            encoding=serialization.Encoding.PEM,
+            format=serialization.PrivateFormat.PKCS8,
+            encryption_algorithm=serialization.NoEncryption(),
+        ).decode()
+
+        dask_internal = x509.Name(
+            [x509.NameAttribute(NameOID.COMMON_NAME, "dask-internal")]
+        )
+        altnames = x509.SubjectAlternativeName([x509.DNSName("dask-internal")])
+        now = datetime.datetime.utcnow()
+        cert = (
+            x509.CertificateBuilder()
+            .subject_name(dask_internal)
+            .issuer_name(dask_internal)
+            .add_extension(altnames, critical=False)
+            .public_key(key.public_key())
+            .serial_number(x509.random_serial_number())
+            .not_valid_before(now)
+            .not_valid_after(now + datetime.timedelta(days=365))
+            .sign(key, hashes.SHA256(), default_backend())
+        )
+
+        cert_contents = cert.public_bytes(serialization.Encoding.PEM).decode()
+
+        return cls(
+            require_encryption=True,
+            tls_ca_file=cert_contents,
+            tls_client_key=key_contents,
+            tls_client_cert=cert_contents,
+            tls_scheduler_key=key_contents,
+            tls_scheduler_cert=cert_contents,
+            tls_worker_key=key_contents,
+            tls_worker_cert=cert_contents,
+        )
+
     def _set_field(self, kwargs, field, config_name):
         if field in kwargs:
             out = kwargs[field]
@@ -84,12 +149,16 @@ def _set_field(self, kwargs, field, config_name):
         setattr(self, field, out)
 
     def __repr__(self):
-        items = sorted((k, getattr(self, k)) for k in self.__slots__)
-        return (
-            "Security("
-            + ", ".join("%s=%r" % (k, v) for k, v in items if v is not None)
-            + ")"
-        )
+        keys = sorted(self.__slots__)
+        items = []
+        for k in keys:
+            val = getattr(self, k)
+            if val is not None:
+                if isinstance(val, str) and "\n" in val:
+                    items.append((k, "..."))
+                else:
+                    items.append((k, repr(val)))
+        return "Security(" + ", ".join("%s=%s" % (k, v) for k, v in items) + ")"
 
     def get_tls_config_for_role(self, role):
         """
@@ -106,14 +175,41 @@ def get_tls_config_for_role(self, role):
 
     def _get_tls_context(self, tls, purpose):
         if tls.get("ca_file") and tls.get("cert"):
-            ctx = ssl.create_default_context(purpose=purpose, cafile=tls["ca_file"])
+            ca = tls["ca_file"]
+            cert_path = cert = tls["cert"]
+            key_path = key = tls.get("key")
+
+            if "\n" in ca:
+                ctx = ssl.create_default_context(purpose=purpose, cadata=ca)
+            else:
+                ctx = ssl.create_default_context(purpose=purpose, cafile=ca)
+
+            cert_in_memory = "\n" in cert
+            key_in_memory = key is not None and "\n" in key
+            if cert_in_memory or key_in_memory:
+                with tempfile.TemporaryDirectory() as tempdir:
+                    if cert_in_memory:
+                        cert_path = os.path.join(tempdir, "dask.crt")
+                        with open(cert_path, "w") as f:
+                            f.write(cert)
+                    if key_in_memory:
+                        key_path = os.path.join(tempdir, "dask.pem")
+                        with open(key_path, "w") as f:
+                            f.write(key)
+                    ctx.load_cert_chain(cert_path, key_path)
+            else:
+                ctx.load_cert_chain(cert_path, key_path)
+
+            # Bidirectional authentication
             ctx.verify_mode = ssl.CERT_REQUIRED
+
             # We expect a dedicated CA for the cluster and people using
             # IP addresses rather than hostnames
             ctx.check_hostname = False
-            ctx.load_cert_chain(tls["cert"], tls.get("key"))
+
             if tls.get("ciphers"):
                 ctx.set_ciphers(tls.get("ciphers"))
+
             return ctx
 
     def get_connection_args(self, role):


Test patch:
diff --git a/distributed/deploy/tests/test_local.py b/distributed/deploy/tests/test_local.py
--- a/distributed/deploy/tests/test_local.py
+++ b/distributed/deploy/tests/test_local.py
@@ -246,6 +246,15 @@ def test_Client_twice(loop):
             assert c.cluster.scheduler.port != f.cluster.scheduler.port
 
 
+@pytest.mark.asyncio
+async def test_client_constructor_with_temporary_security(cleanup):
+    async with Client(
+        security=True, silence_logs=False, dashboard_address=None, asynchronous=True
+    ) as c:
+        assert c.cluster.scheduler_address.startswith("tls")
+        assert c.security == c.cluster.security
+
+
 @pytest.mark.asyncio
 async def test_defaults(cleanup):
     async with LocalCluster(
@@ -695,10 +704,12 @@ def test_adapt_then_manual(loop):
                 assert time() < start + 5
 
 
-def test_local_tls(loop):
-    from distributed.utils_test import tls_only_security
-
-    security = tls_only_security()
+@pytest.mark.parametrize("temporary", [True, False])
+def test_local_tls(loop, temporary):
+    if temporary:
+        security = True
+    else:
+        security = tls_only_security()
     with LocalCluster(
         n_workers=0,
         scheduler_port=8786,
@@ -712,7 +723,7 @@ def test_local_tls(loop):
             loop,
             assert_can_connect_from_everywhere_4,
             c.scheduler.port,
-            connection_args=security.get_connection_args("client"),
+            connection_args=c.security.get_connection_args("client"),
             protocol="tls",
             timeout=3,
         )
@@ -722,7 +733,7 @@ def test_local_tls(loop):
             loop,
             assert_cannot_connect,
             addr="tcp://127.0.0.1:%d" % c.scheduler.port,
-            connection_args=security.get_connection_args("client"),
+            connection_args=c.security.get_connection_args("client"),
             exception_class=RuntimeError,
         )
 
@@ -977,8 +988,12 @@ async def test_repr(cleanup):
 
 
 @pytest.mark.asyncio
-async def test_capture_security(cleanup):
-    security = tls_only_security()
+@pytest.mark.parametrize("temporary", [True, False])
+async def test_capture_security(cleanup, temporary):
+    if temporary:
+        security = True
+    else:
+        security = tls_only_security()
     async with LocalCluster(
         n_workers=0,
         silence_logs=False,
diff --git a/distributed/tests/test_security.py b/distributed/tests/test_security.py
--- a/distributed/tests/test_security.py
+++ b/distributed/tests/test_security.py
@@ -379,3 +379,42 @@ def check_encryption_error():
                 handle_comm,
                 connection_args=sec2.get_listen_args("scheduler"),
             )
+
+
+def test_temporary_credentials():
+    sec = Security.temporary()
+    sec_repr = repr(sec)
+    fields = ["tls_ca_file"]
+    fields.extend(
+        "tls_%s_%s" % (role, kind)
+        for role in ["client", "scheduler", "worker"]
+        for kind in ["key", "cert"]
+    )
+    for f in fields:
+        val = getattr(sec, f)
+        assert "\n" in val
+        assert val not in sec_repr
+
+
+@gen_test()
+def test_tls_temporary_credentials_functional():
+    pytest.importorskip("cryptography")
+
+    @gen.coroutine
+    def handle_comm(comm):
+        peer_addr = comm.peer_address
+        assert peer_addr.startswith("tls://")
+        yield comm.write("hello")
+        yield comm.close()
+
+    sec = Security.temporary()
+
+    with listen(
+        "tls://", handle_comm, connection_args=sec.get_listen_args("scheduler")
+    ) as listener:
+        comm = yield connect(
+            listener.contact_address, connection_args=sec.get_connection_args("worker")
+        )
+        msg = yield comm.read()
+        assert msg == "hello"
+        comm.abort()