Repository: jupyterhub/jupyterhub

Issue:
Enhancement: automatically create a directory for the user after successful login
We are using jupyterhub with LDAPAuthenticator and DockerSpawner. 

We would like to mount a host volume into the docker container, which is not a problem if the directory to be mounted as volume exists before running the container using docker. If the directory to be mounted as volume does not exists beforehand, the docker daemon will create it "on demand" - but then, that created directory will be chown'ed by root:root - the user jovyan inside of the container will not be able to write into her own directory /home/jovyan/work as the host directory is owned by root.

There are some workarounds using docker, especially to write an ENTRYPOINT which executes a shellscript to re-chown the directory to the user. However, it would be a lot of work to enhance all of the docker images we'd like to use just for this case. And yes, we do not know who will be loggin in into the jupyterhub in the first place, as everyone of a certain LDAP-group will be allowed to login. 

The idea is as follows:

* Some user logs in into jupyterhub using one of the authenticators, in our case, the LDAP Authenticator
* After a successful login, the jupyterhub process will create a directory (to be specified in the configuration) on the host, if not already exists. Then, the directory will not be automatically owned by root as in the case when docker will create the directory during the first volume-mount process. Therefor, one additional traitlet will be made available to specify which directory is to be created. The traitlet will allow the variable {username} to be substituted by the name of the user who has just logged in.
* the Dockerspawner will mount the volume /home/jovyan/work to the just created directory, which is an already available dockerspawner feature. 

In case this idea finds some community support, I'd be willing to implement the optional directory creation after succesful login. 

I'm happy if you share your thoughts on this. Thank you.

Regenschein 


Discussion:
I think this is a sensible feature for DockerSpawner that people have asked for several times. The only question for me is: what should the options be?

1. should the owner of the directory be configurable (only works if the Hub is itself root)?
2. permissions?
3. initial contents?

Good questions. 

If we would create some "post-login-hook" class (or pre-spawner-hook) that could be implemented in different ways, we could give the adaptors of jupyterhub some freedom to implement own processes.

For our special case, it would be sufficient to just create the directory as the user the jupyterhub server itself is running as, then the jovyan user of group "users" would be able to write/read into the mounted directory. That could be a very simple implementation (just createdir if not exists).

I am unsure whether this hook would be so docker-specific that it would be better to be put into the docker-spawner? I have not looked into all the other types of spawners and authenticator/spawner combinations yet.

Regenschein
Authenticators already have a pre_spawn_start hook that fires prior to spawner start.

The permissions issue with nonexistent mounts is pretty docker-specific. I don't think I've heard of people encountering issues with directories and permissions in other cases yet.
Thanks for your hint, I'll look into that and give it a try.
I have reflected a little bit about the pre_spawn_start hook which is present in the Authenticator since 0.4.0.

Of course, I could subclass the LDAPAuthenticator and implement the pre_spawn_start hook to create directories and copy initial content into it etc., but when I'd switch to another type of Authenticator, everything would be lost.

The most common usage for the pre_spawn_hook at the moment is to do authentication related setup and teardown, for example for the PAM-sessions. That seems to be ok as the responsibility for the authentication process itself is transparent. Or the authenticator might want to inject credentials into the spawner, that would be the typical glue between an authentication and spawned notebook process.

The other use cases:

- Create a directory for the user
- Copy initial content into that directory
- Possibly do chowns and other host system related stuff 

do not really fit into "authentication"; these cases do more sound like an initial "bootstrap" process pre-spawning and might be more related to the spawner itself. 

So I found the following possibilities to do a more or less clean implementation:

* Either allow the Spawner to define an own "bootstrap"-Hook. Then all people like me who'd like to do some magic before the - in my case - DockerSpawner starts, could subclass the DockerSpawner and implement an own bootstrap routine. That would be convenient for the python developer, but not necessarily for the system administrator or devop. 

* Or add another abstract SpawnerBootstrap-class that could be realized in the jupyterhub codebase as BootstrapScriptTemplateRunner which takes as argument a conventional shell script as a template. Easily for everone, a directory could be created, you could copy initial content like tutorials etc. That could be as complex as using wget, git clone, unzip, etc.

* That might be also useful for other Authenticator/Spawner pairs (ability to copy content etc.), and nothing of the authentication or spawning-processes themselves have to be changed.

The BootstrapScriptTemplateRunner which would implement the SpawnerBootstrap-class would get a traitlet for the script template. Inside the script template, the adaptor could use variables like {username} which are substituted before execution of the script.

Does that sound too weird? Any other suggestions?

Best wishes + thank you
Regenschein
Additional configuration options could be:

- run once or always (once: only for the first notebook to be spawned, always: before every notebook spawn)
- ignore errors (to spawn a notebook no matter what the bootstrap process has ended in)

Regenschein
It's almost complete in that fork:
https://github.com/CRegenschein/jupyterhub/tree/master/examples/bootstrap-script

@cregenschein cool! I don't think the bootstrap class should be added to this repo as something importable (and thus adding to the Hub's public API), but maybe as an example, demonstrating how to do such things.

If anything should be added to the API, I think it should only be a hook on Spawner that calls a Python function, not invoking a script or a new configurable class. That Python function can invoke a script if appropriate, but there should not be an assumption that it is a script.

Of course, this is all already possible with the existing Python configuration files, but it requires writing a subclass to write the hook, rather than a simple assignment. I do want to encourage subclasses as a normal thing to do in configuration,  but I acknowledge that it's more than some folks are comfortable doing, so a hook may be appropriate. Of course, if a hook is appropriate here, here are several other places for similar hooks to go as well.
This would be a very useful hook for me as well for similar reasons - I want to pre-populate a users environment with some basic info / notebook tutorials etc.

The `bootstrap_class` seems a more complex solution than necessary - a simple `pre_spawn_func` attribute which took a Python function would be simpler and just as flexible.

Unfortunately, in my situation JupyterHub won't be running on the same computer as the `DOCKER_HOST` so it makes things a bit more complicated. Nevertheless, it doesn't hurt to have the `pre_spawn_func` available and it's up to the user to make sure what they do with it makes sense.
I just override `start` in my jupyterhub_config, do the things i need to do and then launch. I do things like create a virtualenv there, seems to work ok. https://github.com/wikimedia/puppet/blob/production/modules/jupyterhub/files/jupyterhub_config.py is overriding authenticator, but could just as easily do it to spawner.

Thanks to all for your feedback! 

Does this sound like the plan to go for:

* Add a configurable "pre_spawn_bootstrap"-Hook into the Base-Spawner (as suggested by @minrk, that would be different from the example by @yuvipanda where the bootstrapping takes place in the add_user() method)

* The parameter passed to the pre_spawn_bootstrap is the name of the user (str). Is there anything else that could be useful?  

* Provide an example on how to implement one pre_spawn_bootstrap using the jupyter_config.py configuration file, i. e. create a directory the pythonic way 

* I'll put my script-runner implementation and example script into another github repository for extended reference. 

If no one disagrees, I'll work on that :-D

best wishes
Regenschein
> Add a configurable "pre_spawn_bootstrap"

I would call it `pre_spawn_hook`, but yes.

> The parameter passed to the pre_spawn_bootstrap is the name of the user (str)

I would pass the Spawner itself, that way it has access to everything, and we don't need to be concerned about API breakage if we decide these hooks need access to more later.

> Provide an example on how to implement one pre_spawn_bootstrap using the jupyter_config.py configuration file, i. e. create a directory the pythonic way

:+1:

> I'll put my script-runner implementation and example script into another github repository for extended reference.

👍  or feel free to invoke a shell script via this mechanism in an example here. Something like:

```python
# jupyterhub_config.py

from subprocess import check_call
def my_hook(spawner):
    username = spawner.user.name
    check_call(['/path/to/myscript', username])

c.Spawner.pre_spawn_hook = my_hook
```


Ok. Perfect. 
Here are the critical code line for a review:

in spawner.py
https://github.com/CRegenschein/jupyterhub/blob/master/jupyterhub/spawner.py#L352
https://github.com/CRegenschein/jupyterhub/blob/master/jupyterhub/spawner.py#L546

in user.py
https://github.com/CRegenschein/jupyterhub/blob/master/jupyterhub/user.py#L285

And the readme / examples:
https://github.com/CRegenschein/jupyterhub/tree/master/examples/bootstrap-script

Happy to hear your critical feedback before i request a pull.

Thanks.

Regenschein

Hi @CRegenschein,

Thanks for your work on this. 

Would you mind creating a Pull Request for you suggested changes? You can mark it [WIP] in the PR title. It's a bit easier for reviewers to make suggestions on the PR itself since we have the lines of code around it for context :smile: If you have any questions on doing this, just ask 👍 
Done :)


Resolving patch:
diff --git a/examples/bootstrap-script/jupyterhub_config.py b/examples/bootstrap-script/jupyterhub_config.py
new file mode 100644
--- /dev/null
+++ b/examples/bootstrap-script/jupyterhub_config.py
@@ -0,0 +1,26 @@
+# Example for a Spawner.pre_spawn_hook
+# create a directory for the user before the spawner starts
+
+import os
+def create_dir_hook(spawner):
+    username = spawner.user.name # get the username
+    volume_path = os.path.join('/volumes/jupyterhub', username)
+    if not os.path.exists(volume_path):
+        os.mkdir(volume_path, 0o755)
+        # now do whatever you think your user needs
+        # ...
+
+# attach the hook function to the spawner
+c.Spawner.pre_spawn_hook = create_dir_hook
+
+# Use the DockerSpawner to serve your users' notebooks
+c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
+from jupyter_client.localinterfaces import public_ips
+c.JupyterHub.hub_ip = public_ips()[0]
+c.DockerSpawner.hub_ip_connect = public_ips()[0]
+c.DockerSpawner.container_ip = "0.0.0.0"
+
+# You can now mount the volume to the docker container as we've
+# made sure the directory exists
+c.DockerSpawner.volumes = { '/volumes/jupyterhub/{username}/': '/home/jovyan/work' }
+
diff --git a/jupyterhub/spawner.py b/jupyterhub/spawner.py
--- a/jupyterhub/spawner.py
+++ b/jupyterhub/spawner.py
@@ -349,6 +349,25 @@ def _deprecate_percent_u(self, proposal):
         """
     ).tag(config=True)
 
+    pre_spawn_hook = Any(
+        help="""
+        An optional hook function that you can implement to do some bootstrapping work before
+        the spawner starts. For example, create a directory for your user or load initial content.
+        
+        This can be set independent of any concrete spawner implementation.
+        
+        Example:
+        
+        from subprocess import check_call
+        def my_hook(spawner):
+            username = spawner.user.name
+            check_call(['./examples/bootstrap-script/bootstrap.sh', username])
+
+        c.Spawner.pre_spawn_hook = my_hook
+        
+        """
+    ).tag(config=True)
+
     def __init__(self, **kwargs):
         super(Spawner, self).__init__(**kwargs)
         if self.user.state:
@@ -522,6 +541,11 @@ def get_args(self):
         args.extend(self.args)
         return args
 
+    def run_pre_spawn_hook(self):
+        """Run the pre_spawn_hook if defined"""
+        if self.pre_spawn_hook:
+            return self.pre_spawn_hook(self)
+
     @gen.coroutine
     def start(self):
         """Start the single-user server
diff --git a/jupyterhub/user.py b/jupyterhub/user.py
--- a/jupyterhub/user.py
+++ b/jupyterhub/user.py
@@ -141,7 +141,6 @@ def __init__(self, orm_user, settings=None, **kwargs):
         )
 
     # pass get/setattr to ORM user
-
     def __getattr__(self, attr):
         if hasattr(self.orm_user, attr):
             return getattr(self.orm_user, attr)
@@ -165,7 +164,7 @@ def running(self):
         if self.server is None:
             return False
         return True
-    
+
     @property
     def server(self):
         if len(self.servers) == 0:
@@ -283,6 +282,9 @@ def spawn(self, options=None):
         if (authenticator):
             yield gen.maybe_future(authenticator.pre_spawn_start(self, spawner))
 
+        # run optional preparation work to bootstrap the notebook
+        yield gen.maybe_future(self.spawner.run_pre_spawn_hook())
+
         self.spawn_pending = True
         # wait for spawner.start to return
         try:


Test patch: