internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pypa/setuptools

Issue:
Using `setuptools.extern.importlib_metadata.EntryPoints.names`  is incompatible with older versions of `importlib_metadata` and/or `importlib.metadata`
`importlib_metadata.EntryPoints.names` was introduced in 3.6.0 and it is currently being used in https://github.com/pypa/setuptools/blob/963dbd25cb64898b6f3ee247440db0421f16ca18/setuptools/dist.py#L538

This potentially cause problems when `MetaPathFinder`s in Python 3.8 that use `importlib.metadata` APIs to implement [`find_distributions`](https://docs.python.org/3.8/library/importlib.metadata.html#extending-the-search-algorithm).

This problem was identified in https://github.com/pypa/pyproject-hooks/pull/195#issuecomment-2088695300.


Discussion:

Resolving patch:
diff --git a/setuptools/dist.py b/setuptools/dist.py
--- a/setuptools/dist.py
+++ b/setuptools/dist.py
@@ -535,7 +535,8 @@ def warn_dash_deprecation(self, opt, section):
 
     def _setuptools_commands(self):
         try:
-            return metadata.distribution('setuptools').entry_points.names
+            entry_points = metadata.distribution('setuptools').entry_points
+            return {ep.name for ep in entry_points}  # Avoid newer API for compatibility
         except metadata.PackageNotFoundError:
             # during bootstrapping, distribution doesn't exist
             return []


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Add Jupyter structure for docs
- [x] Add readthedocs.yml
- [x] move environment.yml inside source directory
- [ ] add project jupyter copyright


Discussion:

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -50,9 +50,9 @@
 master_doc = 'index'
 
 # General information about the project.
-project = 'Zero to Jupyterhub'
-copyright = '2017, Chris Holdgraf'
-author = 'Chris Holdgraf'
+project = u'Zero to JupyterHub with Kubernetes'
+copyright = u'2016, Project Jupyter team'
+author = u'Project Jupyter team'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
add link to github repo on website
I think there should be an obvious link on the website to the repo / issue tracker so that people can leave feedback more easily.


Discussion:
Just to clarify @amueller, a link from the docs back to the GitHub issue tracker.
yes

Resolving patch:
diff --git a/doc/source/conf.py b/doc/source/conf.py
--- a/doc/source/conf.py
+++ b/doc/source/conf.py
@@ -95,13 +95,23 @@
 # further.  For a list of options available for each theme, see the
 # documentation.
 #
-# html_theme_options = {}
+html_theme_options = {
+    'show_powered_by': False,
+    'github_user': 'jupyterhub',
+    'github_repo': 'zero-to-jupyterhub-k8s',
+    'github_banner': False,
+    'show_related': False
+}
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
 html_static_path = ['_static']
 
+html_sidebars = {
+    'index': ['sidebarintro.html'],
+}
+
 
 # -- Options for HTMLHelp output ------------------------------------------
 


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
add handling of init_containers to the default jupyterhub_config.py
While experimenting a bit with lifecycle hooks and init containers with kubespawner, I noticed that while lifecycle hooks are captured in the jupyterhub_config.py [here](https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/images/hub/jupyterhub_config.py#L111-L113), there's not out-of-the-box handling for init-containers.

Does it make sense to add this to jupyterhub_config.py?
```
init_containers = get_config('singleuser.init-containers')
if init_containers:
    c.KubeSpawner.singleuser_init_containers = init_containers
```
Let me know and I can send a PR.


Discussion:
Yup, go for it!

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -114,6 +114,10 @@ def get_secret(key, default=None):
 if lifecycle_hooks:
     c.KubeSpawner.singleuser_lifecycle_hooks = lifecycle_hooks
 
+init_containers = get_config('singleuser.init-containers')
+if init_containers:
+    c.KubeSpawner.singleuser_init_containers = init_containers    
+
 # Gives spawned containers access to the API of the hub
 c.KubeSpawner.hub_connect_ip = os.environ['HUB_SERVICE_HOST']
 c.KubeSpawner.hub_connect_port = int(os.environ['HUB_SERVICE_PORT'])


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
nodeSelector support
I'd like to use node pools on GKE to have a cheaper nodes like `n1-standard-1` host hub related pods and a more expensive nodes host user pods. Is there support for `nodeSelect` pod configuration for the hub and singleuser pods?

https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

My use case is a cluster for long term student projects over the course of the semester, and I'd like to use autoscaler on the expensive node pool to go make it go to zero noes when no students are using the cluster, while leaving the hub up on something cheaper while waiting for a user to log in.


Discussion:

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -71,6 +71,7 @@ def get_secret(key, default=None):
 c.KubeSpawner.singleuser_uid = get_config('singleuser.uid')
 c.KubeSpawner.singleuser_fs_gid = get_config('singleuser.fs-gid')
 
+c.KubeSpawner.singleuser_node_selector = get_config('singleuser.node-selector')
 # Configure dynamically provisioning pvc
 storage_type = get_config('singleuser.storage.type')
 if storage_type == 'dynamic':


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Allow multiple extraConfig
Binder relies on `hub.extraConfig` to configure the Spawner. If we want to customize a given Binder *deployment* via `hub.extraConfig`, we have to copy/paste the hub.extraConfig from the binderhub chart and add to it.

It would be handy if extraConfig were something that could be added to without having to copy the previous value. Are dicts the only structure that can do this?


Discussion:
Dicts I think are the only structure that can do this. We can and should easily add this to the chart.

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -1,4 +1,5 @@
 import os
+import glob
 import sys
 import yaml
 from tornado.httpclient import AsyncHTTPClient
@@ -295,6 +296,6 @@ def generate_user_name(spawner):
     # Set default to {} so subconfigs can easily update it
     c.KubeSpawner.singleuser_extra_pod_config = {}
 
-extra_config_path = '/etc/jupyterhub/config/hub.extra-config.py'
-if os.path.exists(extra_config_path):
-    load_subconfig(extra_config_path)
+extra_configs = sorted(glob.glob('/etc/jupyterhub/config/hub.extra-config.*.py'))
+for ec in extra_configs:
+    load_subconfig(ec)


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Allow making JupyterLab default thing to launch
Is there a way to make JupyterLab come up by default when new users connect?

Is there a way to get the JupyterHub control panel from JupyterLab?


Discussion:
https://github.com/jupyterhub/jupyterlab-hub for accessing JupyterHub control panel from Lab I think.


We should definitely document how to configure the hub so that lab starts by default
@mrocklin if you follow the instructions in the repo that @yuvipanda linked, can you report back if it worked? If so I'll make a PR to z2jh to add that in
I was wondering the same thing.  I followed the instructions there, but it does not have links back to the control panel like JupyterNotebook.

IIRC it is missing something like: https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/singleuser.py#L173
I think the way to do this is allow setting default_url in the helm chart, and document that if you want default to be lab you need:

1. Set `singleuser.defaultUrl`
2. Install the jupyterhub-labhub package in your image

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -248,6 +248,9 @@ def generate_user_name(spawner):
 if cmd:
     c.Spawner.cmd = cmd
 
+default_url = get_config('singleuser.default-url', None)
+if default_url:
+    c.Spawner.default_url = default_url
 
 scheduler_strategy = get_config('singleuser.scheduler-strategy', 'spread')
 


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
allow setting service account from config.ayml
We currently disable [service accounts](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) for all user pods, disallowing kubernetes API access. This is the correct default for us, but we should allow users to easily override it if they need.


Discussion:

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -72,6 +72,10 @@ def get_secret(key, default=None):
 c.KubeSpawner.singleuser_uid = get_config('singleuser.uid')
 c.KubeSpawner.singleuser_fs_gid = get_config('singleuser.fs-gid')
 
+service_account_name = get_config('singleuser.service-account-name', None)
+if service_account_name:
+    c.KubeSpawner.singleuser_service_account = service_account_name
+
 c.KubeSpawner.singleuser_node_selector = get_config('singleuser.node-selector')
 # Configure dynamically provisioning pvc
 storage_type = get_config('singleuser.storage.type')


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Block access to cloud metadata endpoint by default
Currently, we expect users to do extra steps to secure their clusters from users accessing the cloud metadata endpoints (https://zero-to-jupyterhub.readthedocs.io/en/v0.5-doc/security.html#audit-cloud-metadata-server-security). IMO, we should instead do that by default and allow users to opt out of it. Most users won't actually be doing the blocking right now, and run insecure clusters...


Discussion:

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -1,6 +1,7 @@
 import os
 import glob
 from tornado.httpclient import AsyncHTTPClient
+from kubernetes import client
 
 from z2jh import get_config, get_secret
 
@@ -104,7 +105,7 @@
 
 init_containers = get_config('singleuser.init-containers')
 if init_containers:
-    c.KubeSpawner.singleuser_init_containers = init_containers
+    c.KubeSpawner.singleuser_init_containers.extend(init_containers)
 
 # Gives spawned containers access to the API of the hub
 c.KubeSpawner.hub_connect_ip = os.environ['HUB_SERVICE_HOST']
@@ -252,6 +253,30 @@ def generate_user_name(spawner):
 if default_url:
     c.Spawner.default_url = default_url
 
+cloud_metadata = get_config('singleuser.cloud-metadata', {})
+
+if not cloud_metadata.get('enabled', False):
+    # Use iptables to block access to cloud metadata by default
+    network_tools_image_name = get_config('singleuser.network-tools.image.name')
+    network_tools_image_tag = get_config('singleuser.network-tools.image.tag')
+    ip_block_container = client.V1Container(
+        name="block-cloud-metadata",
+        image=f"{network_tools_image_name}:{network_tools_image_tag}",
+        command=[
+            'iptables',
+            '-A', 'OUTPUT',
+            '-d', cloud_metadata.get('ip', '169.254.169.254'),
+            '-j', 'DROP'
+        ],
+        security_context=client.V1SecurityContext(
+            privileged=True,
+            run_as_user=0,
+            capabilities=client.V1Capabilities(add=['NET_ADMIN'])
+        )
+    )
+
+    c.KubeSpawner.singleuser_init_containers.append(ip_block_container)
+
 scheduler_strategy = get_config('singleuser.scheduler-strategy', 'spread')
 
 if scheduler_strategy == 'pack':


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Github Organization as whitelist in config.yaml
I am wanting to whitelist members of a github organization (be it a team within the organization or the whole organization). I have seen that this is possible using standard Jupyterhub but am not sure how to go about it using just the k8s version. I've read through docs and comments [here](https://github.com/jupyterhub/oauthenticator/pull/58) and am not sure how to do the same using the config.yaml file. Any help or direction appreciated!


## Contents of `config.yaml`
```
proxy:
  secretToken: "SOMESECRET"
  https:
    hosts:
      - SOMEDOMAINNAME
    letsencrypt:
      contactEmail: SOME@EMAIL.COM
singleuser:
  image:
    name: SOME/IMAGE
    tag: latest
  lifecycleHooks:
    postStart:
      exec:
        command: ["/bin/sh", "-c", "test -f $HOME/.copied || cp -Rf /srv/app/src/. $HOME/; touch $HOME/.copied"]
  memory:
    limit: 3G
    guarantee: 3G
  cpu:
    limit: 1
    guarantee: 0.4
  storage:
    type: none
  initContainers:
    - name: block-metadata
      image: minrk/tc-init:0.0.4
      # Block access to GCE Metadata Service from user pods
      command: ["iptables", "-A", "OUTPUT", "-p", "tcp", "-d", "SOMEIP", "-j", "DROP"]
      securityContext:
        runAsUser: 0
        capabilities:
          add: [ "NET_ADMIN" ]
auth:
  admin:
    users:
      - twanito
  whitelist:
    users:
      - twanito
  type: github
  github:
    clientId: "SOMECLIENTIDE"
    clientSecret: "SOMECLIENTSECRET"
    callbackUrl: "https://SOMEDOMAINNAME/hub/oauth_callback"
```



Discussion:
Thank you for opening the issue! Can you point us to docs on making it happen for generic JupyterHub? Should be able to figure it out from there!

Thanks!

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -136,6 +136,10 @@
     c.GitHubOAuthenticator.oauth_callback_url = get_config('auth.github.callback-url')
     c.GitHubOAuthenticator.client_id = get_config('auth.github.client-id')
     c.GitHubOAuthenticator.client_secret = get_config('auth.github.client-secret')
+    org_whitelist = get_config('auth.github.org_whitelist', [])
+    if len(org_whitelist) != 0:
+        c.GitHubOAuthenticator.github_organization_whitelist = org_whitelist
+        c.GitHubOAuthenticator.scope = ['read:org'] # required for private membership
 elif auth_type == 'cilogon':
     c.JupyterHub.authenticator_class = 'oauthenticator.CILogonOAuthenticator'
     c.CILogonOAuthenticator.oauth_callback_url = get_config('auth.cilogon.callback-url')


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
'pack' scheduling of nodes should have affinity for the hub
The 'pack' scheduling strategy for nodes should create a very high affinity for the hub, as this will be the node to scale down last of all.

Learning more, will send a PR when I got it right.
'pack' scheduling of nodes should have affinity for the hub
The 'pack' scheduling strategy for nodes should create a very high affinity for the hub, as this will be the node to scale down last of all.

Learning more, will send a PR when I got it right.


Discussion:

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -285,7 +285,19 @@ def generate_user_name(spawner):
         'affinity': {
             'podAffinity': {
                 'preferredDuringSchedulingIgnoredDuringExecution': [{
-                    'weight': 100,
+                    'weight': 50,
+                    'podAffinityTerm': {
+                        'labelSelector': {
+                            'matchExpressions': [{
+                                'key': 'component',
+                                'operator': 'In',
+                                'values': ['hub']
+                            }]
+                        },
+                        'topologyKey': 'kubernetes.io/hostname'
+                    }
+                }, {
+                    'weight': 5,
                     'podAffinityTerm': {
                         'labelSelector': {
                             'matchExpressions': [{


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Cull_idle fails for some users with non ascii usernames
At the moment Cull Idle queries the API for the users with inactive running servers and sends a `DELETE` to the endpoint users/USERNAME/server. https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/706a8f7637881ace06b66c90c5267019d75d2cf0/images/hub/cull_idle_servers.py#L69 This works for most users, but fails with users with non ASCII usernames as the api responds with a 404.

By url-encoding the username we can make sure it works for every user.
Cull_idle fails for some users with non ascii usernames
At the moment Cull Idle queries the API for the users with inactive running servers and sends a `DELETE` to the endpoint users/USERNAME/server. https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/706a8f7637881ace06b66c90c5267019d75d2cf0/images/hub/cull_idle_servers.py#L69 This works for most users, but fails with users with non ASCII usernames as the api responds with a 404.

By url-encoding the username we can make sure it works for every user.


Discussion:

Resolving patch:
diff --git a/images/hub/cull_idle_servers.py b/images/hub/cull_idle_servers.py
--- a/images/hub/cull_idle_servers.py
+++ b/images/hub/cull_idle_servers.py
@@ -31,6 +31,11 @@
 import json
 import os
 
+try:
+    from urllib.parse import quote
+except ImportError:
+    from urllib import quote
+
 from dateutil.parser import parse as parse_date
 
 from tornado.gen import coroutine
@@ -66,7 +71,7 @@ def cull_one(user, last_activity):
         # shutdown server first. Hub doesn't allow deleting users with running servers.
         if user['server']:
             app_log.info("Culling server for %s (inactive since %s)", user['name'], last_activity)
-            req = HTTPRequest(url=url + '/users/%s/server' % user['name'],
+            req = HTTPRequest(url=url + '/users/%s/server' % quote(user['name']),
                 method='DELETE',
                 headers=auth_header,
             )


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Should allow spawning single-user container with extra resources like nvidia.com/gpu
In our case, the GPU is a necessary resource, but the current version doesn't support requesting GPU or other non-built-in resources.

I made a patch to solve this issue, see details in my pr: #793, it was tested and running on our production environment now.


Discussion:

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -125,6 +125,8 @@
 c.KubeSpawner.mem_guarantee = get_config('singleuser.memory.guarantee')
 c.KubeSpawner.cpu_limit = get_config('singleuser.cpu.limit')
 c.KubeSpawner.cpu_guarantee = get_config('singleuser.cpu.guarantee')
+c.KubeSpawner.extra_resource_limits = get_config('singleuser.extra_resource.limits')
+c.KubeSpawner.extra_resource_guarantees = get_config('singleuser.extra_resource.guarantees')
 
 # Allow switching authenticators easily
 auth_type = get_config('auth.type')


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Annotations on single user notebook
I'm trying to add an annotation to all notebooks which are created, however I can't seem to find where to configure this in the chart config. The hub, proxy and ingress all have annotation options, but the singleuser section doesn't.

Am I missing something or has this not been implemented yet?

_NB if I could do this dynamically based on username that would be even better. But that may be a bigger ask and invoke KubeSpawner._


Discussion:
@yuvipanda do you have any thoughts on this?
KubeSpawner has `extra_annotations`: https://github.com/jupyterhub/kubespawner/blob/v0.9.0b2/kubespawner/spawner.py#L379 (and it supports `{username}` too)
@manics that's great. How does that translate through to the `values.yaml` file in here?

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -57,6 +57,8 @@
 
 c.KubeSpawner.events_enabled = get_config('singleuser.events', False)
 
+c.KubeSpawner.extra_annotations = get_config('singleuser.extra-annotations', {})
+
 c.KubeSpawner.extra_labels = get_config('singleuser.extra-labels', {})
 
 c.KubeSpawner.uid = get_config('singleuser.uid')


Test patch:
Repository: jupyterhub/zero-to-jupyterhub-k8s

Issue:
Supporting launching companion Pods
We're starting work on a new project for cybersecurity education building on the [TryCybSI](http://try.cybersecurity.ieee.org/trycybsi/) environment.  The basic premise is that students are able to spin up a multi-pod sandbox environment to be able to walk through various security concepts.  The primary environment will be Jupyter, but we need to be able to launch companion pods with each notebook environment.  For example, in the [ArpSpoof](http://try.cybersecurity.ieee.org/trycybsi/help/arpspoof) case we'd need the "hacker" environment 
 (Jupyter) alongside "victim" (client, maybe Jupyter) and "server" (webserver) environments. These would need to be separate Pods with a common network/security policy, since it wouldn't be interesting to do this on locahost (i.e., a single pod). I've seen the `hub.extraContainers` configuration and mention of potentially allowing users to "[do additional things in their namespaces (like spawning other clusters)](https://github.com/jupyterhub/kubespawner/issues/76).  Is this type of scenario Z2JH can support (or would be interested in supporting, if we were to work on a way)?



Discussion:

Resolving patch:
diff --git a/images/hub/jupyterhub_config.py b/images/hub/jupyterhub_config.py
--- a/images/hub/jupyterhub_config.py
+++ b/images/hub/jupyterhub_config.py
@@ -129,6 +129,10 @@
 if init_containers:
     c.KubeSpawner.init_containers.extend(init_containers)
 
+extra_containers = get_config('singleuser.extra-containers')
+if extra_containers:
+    c.KubeSpawner.extra_containers.extend(extra_containers)
+
 # Gives spawned containers access to the API of the hub
 c.KubeSpawner.hub_connect_ip = os.environ['HUB_SERVICE_HOST']
 c.KubeSpawner.hub_connect_port = int(os.environ['HUB_SERVICE_PORT'])


Test patch: