Repository: bokeh/bokeh

Issue:
Custom models distribution without compiler on the server side
Running Bokeh in the cloud environment (like Google App Engine) gives this error:

```
An internal error occurred:
node.js is needed to allow compilation of custom models ("conda install -c bokeh nodejs" or follow https://nodejs.org/en/download/)
```

The setup, it is working fine on my local computer, but on cloud version looks like in the latest version 0.12.3. some extra dependency is added to node.js that was not in the previous version.

It will be nice if the error message is giving more hints about which custom model is creating all the issues. In my case I have several Bokeh plots (and some of them have a custom model like https://github.com/bokeh/bokeh/issues/4690#issuecomment-229694803).

This compilation problem creates challenges for distribution when the package is only managed by pip install -r requirements.txt and no option to add conda install -c bokeh nodejs.

The dependency is very subtle and not well documented.



Discussion:
The change to pre-compilation on the python was necessary in order to allow custom extensions to inherit from other custom extensions. This is needed both for future planned facilities outside the core library, and also was very widely asked for in a community discussion. We did mention in the release migration notes that `bokeh-compiler` had been removed. However, we probably could have elaborated on that more. Bokeh is a large project with a small team and we are often working completely at the limits of our available human resource capabilities, so things sometimes slip through, despite our best intentions. However, I will try to make a point to expand our documentation to be more explicit about the new requirement for nodejs when using custom extensions. 

@mattpap would it be possible to "pre-pre-compile" custom extensions? That is, could we have some way to create a python module on one system with node, that could be transferred and imported for use on another system without node?

I agree that @bryevdv 's suggestion would be nice. Yesterday I wrote an extension coffeescript and ran it locally, then grepped the output html document for the compiled JS and copied that into a new **implementation** for production. It could image something like:

```
from bokeh.util.compiler import Compile

js_implementation = Compile(MyCustomModel)
```

@canavandl I might even suggest a `bokeh` command line tool for this:

```
bokeh compile extension_source.py -o extension.py
```

I can't say I'd be thrilled about this, but it would provide some kind of release valve for anyone that wan't to use extensions but can't install node on a server or whatever. 

Should this problem go away if I include javascript code rather than coffeescript?

I would expect it to, but I have not had a chance to try. 

At first glance this does not appear to be the case.  Anything special I should do to inform Bokeh that the text I'm sending in JavaScript rather than CoffeeScript?

possibly yes, try 

```
__implementation__ = JavaScript(js_code)
```

where `JavaScript` is in `bokeh.util.compiler`

The problem persists even with this change.

Well, then I was wrong, and more investigation will be required.

ping @mattpap 

Looking at the implementation in `bokeh/util/compiler.py` it looks like Bokeh doesn't currently check on the language until _after_ it has checked for the presence of nodejs.

nodejs may be needed to resolve any import/require statements in any case. We will have to get some feedback from @mattpap 

The choice of language doesn't make any difference when it comes to whether nodejs is needed or not. Even with plain JavaScript we have to resolve dependencies (`require()`  and `import/export`) and those dependencies may not be JavaScript, but whatever we support (CSS, Less, CoffeeScript, Eco, TypeScript). To get a sane implementation we have to go all the way and support everything we do support, especially when it comes to custom models. With CustomJS we default to plain JavaScript and no dependency resolution for backwards compatibility and because CustomJS is usually simple code.

This issue is blocked on #5315. When this is fully resolved, bokeh will store either final bundle or individual compiled sources, and check SHA1 signatures to decide if recompilation is needed or not. This way this issue will boil down to a packaging problem. We have to address this anyway, because currently custom model generation slows down bokeh considerably, because they are recomputed every time resources are requested.

When trying to deploy my app on heroku, i ran into same issue... hope for this bug to be fixed soon

We cache extensions after first use but still do need an option to distribute in cases where to other users or systems where nodejs is not present. Pre-compiled extensions should be locked to the version they were created for, though. 
What would it take to make this happen? I'd be interested on working on this since we have a number of custom models now and compiling them each time is pretty slow.
>it would provide some kind of release valve for anyone that wan't to use extensions but can't install node on a server or whatever.

It seems like supporting precompiling would help not just in cases where node can't be installed, but in any cases where custom models are in heavy use, which is becoming a limiting factor for our work.  I'm happy to volunteer Philipp for this, as he's the one most directly affected by the slow current approach. :-)  But it may be something most easily addressable by another Bokeh dev.
I think @philippjfr could tackle this handily, and given that he has a concrete use-use, will have the best opinions regarding it. AFAICT the gist is:

* Have a method that can save out locally compiled code, along with any useful or necessary metadata (e.g. Bokeh versions) A JSON file is probably simplest.

* Have a method that can ingest/install the output file into the compiled module cache. It should probably error e.g. if Bokeh versions in the metadata do no match. 

* Docs/examples

Resolving patch:
diff --git a/bokeh/util/compiler.py b/bokeh/util/compiler.py
--- a/bokeh/util/compiler.py
+++ b/bokeh/util/compiler.py
@@ -248,7 +248,7 @@ class TypeScript(Inline):
         .. code-block:: python
 
             class MyExt(Model):
-                __implementation__ = TypeScript(""" <TypeSctipt code> """)
+                __implementation__ = TypeScript(""" <TypeScript code> """)
 
     '''
     @property
@@ -256,14 +256,14 @@ def lang(self):
         return "typescript"
 
 class JavaScript(Inline):
-    ''' An implementation for a Bokeh custom model in JavaSript
+    ''' An implementation for a Bokeh custom model in JavaScript
 
     Example:
 
         .. code-block:: python
 
             class MyExt(Model):
-                __implementation__ = Javacript(""" <Javactipt code> """)
+                __implementation__ = Javacript(""" <JavaScript code> """)
 
     '''
     @property
@@ -364,10 +364,26 @@ def dependencies(self):
     def module(self):
         return "custom/%s" % snakify(self.full_name)
 
-def bundle_models(models):
-    """Create a bundle of `models`. """
-    custom_models = {}
+def _model_cache_no_op(model, implementation):
+    """Return cached compiled implementation"""
+    return None
+
+_CACHING_IMPLEMENTATION = _model_cache_no_op
+
+def get_cache_hook():
+    '''Returns the current cache hook used to look up the compiled
+       code given the CustomModel and Implementation'''
+    return _CACHING_IMPLEMENTATION
 
+def set_cache_hook(hook):
+    '''Sets a compiled model cache hook used to look up the compiled
+       code given the CustomModel and Implementation'''
+    global _CACHING_IMPLEMENTATION
+    _CACHING_IMPLEMENTATION = hook
+
+def _get_custom_models(models):
+    """Returns CustomModels for models with a custom `__implementation__`"""
+    custom_models = {}
     for cls in models:
         impl = getattr(cls, "__implementation__", None)
 
@@ -377,19 +393,11 @@ def bundle_models(models):
 
     if not custom_models:
         return None
+    return custom_models
 
+def _compile_models(custom_models):
+    """Returns the compiled implementation of supplied `models`. """
     ordered_models = sorted(custom_models.values(), key=lambda model: model.full_name)
-
-    exports = []
-    modules = []
-
-    def read_json(name):
-        with io.open(join(bokehjs_dir, "js", name + ".json"), encoding="utf-8") as f:
-            return json.loads(f.read())
-
-
-    bundles = ["bokeh", "bokeh-api", "bokeh-widgets", "bokeh-tables", "bokeh-gl"]
-    known_modules = set(sum([ read_json(name) for name in bundles ], []))
     custom_impls = {}
 
     dependencies = []
@@ -402,13 +410,35 @@ def read_json(name):
 
     for model in ordered_models:
         impl = model.implementation
-        compiled = nodejs_compile(impl.code, lang=impl.lang, file=impl.file)
+        compiled = _CACHING_IMPLEMENTATION(model, impl)
+        if compiled is None:
+            compiled = nodejs_compile(impl.code, lang=impl.lang, file=impl.file)
 
         if "error" in compiled:
             raise CompilationError(compiled.error)
 
         custom_impls[model.full_name] = compiled
 
+    return custom_impls
+
+def bundle_models(models):
+    """Create a bundle of `models`. """
+
+    custom_models = _get_custom_models(models)
+    if custom_models is None:
+        return
+
+    exports = []
+    modules = []
+
+    def read_json(name):
+        with io.open(join(bokehjs_dir, "js", name + ".json"), encoding="utf-8") as f:
+            return json.loads(f.read())
+
+    bundles = ["bokeh", "bokeh-api", "bokeh-widgets", "bokeh-tables", "bokeh-gl"]
+    known_modules = set(sum([ read_json(name) for name in bundles ], []))
+    custom_impls = _compile_models(custom_models)
+
     extra_modules = {}
 
     def resolve_modules(to_resolve, root):


Test patch: