Repository: bokeh/bokeh

Issue:
Expose all JSON representations in one place
In order to make life much simpler for other language bindings, publish a single plain-text JSON file that has default representation of every model in one place. 

Additionally, it would be good to generate another file in some format that specifies the type for each model property as well. 

cc. @hafen


Discussion:
My initial thought was to publish these in the static docs site, but it occurs that it might be better to make a separate repository to hold these assets. 
Having a formal location for this that is kept up-to-date would be awesome. Based on some initial scraping and work I have done along these lines for rbokeh, here is a proposed JSON schema example:

```
{
  "LayoutDOM": {
    "name": "LayoutDOM",
    "bases": ["bokeh.model.Model"],
    "desc": "An abstract base class for layout components. LayoutDOM is not generally useful to instantiate on its own."
    "attrs": [
      {
        "name": "css_classes",
        "type": "Seq ( String  )",
        "desc": "A list of css class names to add to this DOM element. Note: the class names are simply added as-is, no other guarantees are provided.",
        "default_value": {}
      },
      {
        "name": "disabled",
        "type": "Bool",
        "desc": "Whether the widget will be disabled when rendered. If True, the widget will be greyed-out, and not respond to UI events.",
        "default_value": false
      },
      {
        "name": "height",
        "type": "Int",
        "desc": "An optional height for the component (in pixels).",
        "default_value": {}
      },
      {
        "name": "sizing_mode",
        "type": "Enum ( SizingMode  )",
        "desc": "How the item being displayed should size itself. Possible values are \"fixed\", \"scale_width\", \"scale_height\", \"scale_both\", and \"stretch_both\". \"stretch_both\" elements are completely responsive (independently in width and height) and will resize to occupy all available space, even if this changes the aspect ratio of the element. This is sometimes called outside-in, and is a typical behavior for desktop applications. \"fixed\" elements are not responsive. They will retain their original width and height regardless of any subsequent browser window resize events. \"scale_width\" elements will responsively resize to fit to the width available, while maintaining the original aspect ratio. This is a typical behavior for modern websites. For a Plot, the aspect ratio plot_width/plot_height is maintained. \"scale_height\" elements will responsively resize to fit to the height available, while maintaining the original aspect ratio. For a Plot, the aspect ratio plot_width/plot_height is maintained. A plot with \"scale_height\" mode needs to be wrapped in a Row or Column to be responsive. \"scale_both\" elements will responsively resize to fir both the width and height available, while maintaining the original aspect ratio.",
        "default_value": "fixed"
      },
      {
        "name": "width",
        "type": "Int",
        "desc": "An optional width for the component (in pixels).",
        "default_value": {}
      }
    ],
    "json": "{\n  \"css_classes\": null,\n  \"disabled\": false,\n  \"height\": null,\n  \"id\": \"373908d6-3b02-4dfd-a5e6-1e84d8a927fa\",\n  \"js_callbacks\": {},\n  \"name\": null,\n  \"sizing_mode\": \"fixed\",\n  \"tags\": [],\n  \"width\": null\n}\n\n\n  "
  } ,
  "Plot": {
    ...
  },
  ...
}
```

I have been using a schema like this populated by scraping the docs to auto-generate R class code and tests (see [here](https://github.com/hafen/rbokeh-models/blob/master/R/bokeh_models.R) and [here](https://github.com/hafen/rbokeh-models/blob/master/tests/testthat/test-models.R)).

There are some things about this schema that could be different. For example, as @bryevdv suggests, we can put the JSON prototypes in a separate file. Also, since the "default_value" fields are populated from parsing the JSON prototypes, they are redundant and not necessary. So basically, in addition to a JSON file providing all of the prototypes populated with defaults, if we have a JSON file that has an entry for each model, gives its base classes and a description, and then lists all the attributes with types and descriptions, then we are in good shape.

Note that there may be a question about how to represent types for the attributes. On my end, in R, I have already gone through all the possible types and as represented above as strings (e.g. Enum (SizingMode), Seq (String), etc. - there are a lot after looking at every attribute for every model) and figured out how to represent / validate / resolve any permutation of those in R, so a simple string representation of the Python type is perfectly fine.

I like your format with one big file with more information. This should be easy to create. I will probably start with a utility script then we can figure out how/where to publish it later. 
Just checking in on this. Now that 0.12.5 is out, it might be a good time to get this in place.
@hafen thanks for reminding me, I have scheduled it for this release cycle. 
@hafen I plan to have this in place by next week. I think it might need one more level, so that the acceptable values of enums can be documented as well:
```
{
    "enums" : {
        "SizingMode": [....],
    },

    "models": {
         "LayoutDOM": {

         }
    }

}
```

Another question, where to publish? Somewhere on pydata.org? A separate GH repo?
@hafen Here is a tentative script that will print all this information:

```
from datetime import date
import json

import bokeh.models as bm; bm
import bokeh.models.widgets as bmw; bmw

from bokeh.model import Model

data = {}
for name, m in sorted(Model.__class__.model_class_reverse_map.items()):
    item = {
        'name'  : name,
        'bases' : [str(base)[8:-2] for base in m.__bases__],
        'desc'  : m.__doc__.strip() if m.__doc__ is not None else '',
    }
    props = []
    for prop_name in m.properties():
        descriptor = m.lookup(prop_name)
        prop = descriptor.property
        
        detail = {
            'name'    : prop_name,
            'type'    : str(prop),
            'desc'    : prop.__doc__.strip() if prop.__doc__ is not None else '',
        }
        
        default = descriptor.instance_default(m())
        if isinstance(default, date):
            default = str(default)
            
        if isinstance(default, Model):
            default = str(default)
            
        if isinstance(default, list) and any(isinstance(x, Model) for x in default):
            default = [str(x) for x in default]
        
        detail['default'] = default
        
        props.append(detail)
    
    item['props'] = props
    
    data[name] = item
    
print(json.dumps(data, indent=2))
```

Two comments:

* listing enums is separately does not seem necessary, in this output they show up as, e.g. 
   ```
   "type": "Enum('stretch_both', 'scale_width', 'scale_height', 'scale_both', 'fixed')",
   ```

* Need to improve how instance defaults are handled. This code results in 
  ```
   "default": [
          "AdaptiveTicker(id='335fd24b-a160-4480-adbd-f42f37a347fc', ...)",
          "AdaptiveTicker(id='fb6d80ab-a5fc-4985-8d6e-4d09a6b1447b', ...)",
          "AdaptiveTicker(id='68b29354-8818-4590-96f6-054bebcd53b1', ...)",
          "DaysTicker(id='23314e89-2317-40d0-9310-81fc4a1cf07b', ...)",
          "DaysTicker(id='a12a336a-ed64-411b-b97f-d6d22324cbe2', ...)",
          "DaysTicker(id='530914bc-408a-47f4-aec9-b5c66407f762', ...)",
          "DaysTicker(id='e9e979ef-192c-451d-ab90-df099ba683e9', ...)",
          "MonthsTicker(id='2344a2c9-79dd-4019-9715-bb0fe656b421', ...)",
          "MonthsTicker(id='5c089dec-223c-47b1-98ac-c14e6f8bb795', ...)",
          "MonthsTicker(id='0f5af923-38f6-442e-b708-d480524cbfce', ...)",
          "MonthsTicker(id='a249b692-5fd1-4787-a45e-542bd0b417be', ...)",
          "YearsTicker(id='5b9828a1-f153-4bae-90e7-74e140af7522', ...)"
   ]
   ```

   Do you have any input into what would be better for your purposes? Such instances are not necessarily default constructed so we will need to extract more info... If you let me know exactly what you'd like to see I will change it to accommodate.

Also ping @canavandl @hsparra 
Here is a gist of the current output:

https://gist.githubusercontent.com/bryevdv/de62a68029661a6e44169c17a34966f5/raw/997a5f1e7f92fea86b273a5c7c8bfaf246760d1e/gistfile1.txt
I will also say that currently some of the docs contain python specific information. We could consider splitting the doc into a generic intro/description and then any python specific bits, so that this output would only contain the generic description... but that's a pretty big chunk of work so it might have to happen separately/later. 
Excellent!! Thanks Bryan. Some responses:

- I'm indifferent to where it is hosted as long as it is consistent and officially tied to each release.
- The docs having python-specific info isn't a big deal for what I have in mind. I'm planning on placing any docs info inline with the R class definitions mostly for context. The higher-level interface will have its own docs which may borrow from these docs but this will be done manually.
- For each model class, can we also include a string in the JSON output that contains a JSON prototype for the class? I see that your output has defaults, so maybe that is sufficient and I can just use that to build a default prototype, but having a string to compare to for testing makes me feel a little more comfortable since JSON parsers in R sometimes do strange things in special cases.
@hafen I had another thought which was that instead of publishing the file, we could simply provide the script. That puts a little larger burden on the language binding implementors (you've got to run the script yourself) but would save us from having to complicate the automated build, which is at least worth considering. It also means you can generate the spec in between releases and that might be valuable. 

I can definitely add the existing "json prototype", tho to be clear, the real true "default prototype" is one with completely empty attrs. That is, if a value is not present, the BokehJS default value is the one used. You should really only set attribute values when you want something *different* from the BokehJS default. The existing current "json prototype" was mostly just to have a list of the *possible* attributes, in JSON format. I think the current script output has all that info contained, which is why I wanted to double check that the old style "prototype"  is really useful and needed. 

I don't have strong feelings about either of the above things, so if you do please let me know and I will implement accordingly. 

Do you any thoughts about the last question (how instance defaults are handled)?
ping @mattpap as well in case there are considerations for the scala bindings that can be taken into account. 
Scala bindings can't be generated automatically, because there are a lot of things that have to be adjusted, especially around the type system, because either we want to squeeze the most out of it or its limitations force us to do so (e.g. lack of native union types).
> `"default": null`

Does this mean the default is actually `None` or it wasn't provided?
@bryevdv it is fine with me either way for how you'd like the JSON to be generated.

As for adding the JSON prototype for each model to the JSON file, I understand that prototypes contain defaults and that we don't need to generate JSON for the defaults, but it would be nice for me to have just as a sanity check that my code is generating the proper JSON. Ideally I would test against something of more substance than the default, but since the default is already available, that will be good enough. The main thing is whether I'm properly preserving the JSON scalar/array/object types which I can get with the default. You might say that I could just generate the JSON prototype from the data that's already there, but I'd rather check against something I know to be correct. For example, the NumberSpec defaults provided in the JSON file are scalars (e.g. `1.0`), but in the JSON prototype (which I look up in the online docs) they are translated (e.g. `{ value: 1.0 }`). Basically I want to have a sanity check that the JSON generated through Python matches the JSON generated through R.

BTW, I have a lot of code built upon this current setup, so if things change with the script (for example, how you handle instance defaults), please let me know.
Doing something better for instance defaults is the only real *change* I would envision. I will try to have that added as well as a default JSON prototype (which should be a non-breaking additional field) by early next week. 

Resolving patch:
diff --git a/scripts/spec.py b/scripts/spec.py
new file mode 100644
--- /dev/null
+++ b/scripts/spec.py
@@ -0,0 +1,52 @@
+from datetime import date
+import json
+
+import bokeh.models as bm; bm
+import bokeh.models.widgets as bmw; bmw
+
+from bokeh.model import Model
+
+def _proto(obj, defaults=False):
+    return json.dumps(obj.to_json(defaults), sort_keys=True, indent=None)
+
+data = {}
+for name, m in sorted(Model.__class__.model_class_reverse_map.items()):
+    item = {
+        'name'  : name,
+        'bases' : [base.__module__ + '.' + base.__name__ for base in m.__bases__],
+        'desc'  : m.__doc__.strip() if m.__doc__ is not None else '',
+        'proto' : _proto(m(), True),
+    }
+    props = []
+    for prop_name in m.properties():
+        descriptor = m.lookup(prop_name)
+        prop = descriptor.property
+
+        detail = {
+            'name'    : prop_name,
+            'type'    : str(prop),
+            'desc'    : prop.__doc__.strip() if prop.__doc__ is not None else '',
+        }
+
+        default = descriptor.instance_default(m())
+        if isinstance(default, date):
+            default = str(default)
+
+        if isinstance(default, Model):
+            default = _proto(default)
+
+        if isinstance(default, (list, tuple)) and any(isinstance(x, Model) for x in default):
+            default = [_proto(x) for x in default]
+
+        if isinstance(default, dict) and any(isinstance(x, Model) for x in default.values()):
+            default = { k: _proto(v) for k, v in default.items() }
+
+        detail['default'] = default
+
+        props.append(detail)
+
+    item['props'] = props
+
+    data[name] = item
+
+print(json.dumps(data, indent=2))


Test patch: