Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
To disable the initContainer running as root to block an insecure IP to expose by default?
We currently have `singleuser.cloudMetadata.enabled=false`, which mean we actively disable access to a metadata REST API endpoint on k8s nodes on the cloud that can expose some potentially sensitive stuff. This is an endpoint that we already disable access to by default in our k8s NetworkPolicy for the singleuser pod's.

The drawback of disabling this in the way we do it, is that we require to startup an extra initContainer for the user pods, and it requires a lot of privileges compared to normal pods, which is the cause of #1408 for example.

I think it could make sense to disable this fix by default now, but a drawback is that the NetworkPolicy resources while created by default, isn't enforced without a NetworkPolicy controller available in the k8s cluster, and that isn't installed by default in all k8s clusters.

Overall though, I lean towards thinking we should stop having a default that adds this privileged initContainer to the user pods in favor of relying on the NetworkPolicy we create.
Blocking access to cloudmetadata endpoint requires "unsecure" PodSecurityPolicy
Fix of issue [#407 Block access to cloud metadata endpoint by default](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/407) unfortunately introduced the requirement to grant `NET_ADMIN` capability and `privileged: true` to the user Pod to successfully start up as the `initContainer` manipulates `iptables`. 

A better approach would be to use Egress rules in a `NetworkPolicy` as usually cluster admins that use PSPs also have NetPols in place.

So for example if JupyterHub is running in `Namespace` jhub a possible Network Policy could look like this:

```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-cloudmetadata
  namespace: jhub
spec:
  podSelector:
    matchLabels:
      app: jupyterhub
      component: singleuser-server
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 169.254.169.254/32
```

Couldn't check yet but will do asap.


Discussion:
I agree with disabling it by default, and documenting that if your cluster doesn't support egress NetworkPolicies you many need to enable the initContainer.
It's already in the default singleuser network policy: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/9c15a42b1227f3b54826f273f1689e4dc8c8e12e/jupyterhub/values.yaml#L212-L217
Note there's an open PR to turn on NetworkPolicies by default: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/pull/1271 so that might be the appropriate place to remove the `iptables` container
The privileged container disappears if I __enable__ cloud_metadata. Is that a bug?
@dimm0 that's intentional, **enabling** cloud metadata means **no** network restrictions are needed since it's an internal web service which many cloud providers enable by default.
I'm running in my own k8s cluster on my own hardware... Why am I __enabling__ it? :)
History 😀. This was originally written to work with Google Cloud, and over time support was added for other public and private k8s deployments. Changing it is a breaking change so has to balanced against the impact to existing users, but if you're interested in following this up maybe start a new issue?
No, I'm good.. Will leave as is
It's just confusing, and not well described

Resolving patch:
diff --git a/jupyterhub/files/hub/jupyterhub_config.py b/jupyterhub/files/hub/jupyterhub_config.py
--- a/jupyterhub/files/hub/jupyterhub_config.py
+++ b/jupyterhub/files/hub/jupyterhub_config.py
@@ -435,7 +435,7 @@ def camelCaseify(s):
 
 cloud_metadata = get_config('singleuser.cloudMetadata', {})
 
-if not cloud_metadata.get('enabled', False):
+if cloud_metadata.get('block') == True or cloud_metadata.get('enabled') == False:
     # Use iptables to block access to cloud metadata by default
     network_tools_image_name = get_config('singleuser.networkTools.image.name')
     network_tools_image_tag = get_config('singleuser.networkTools.image.tag')


Test patch: