Repository: scikit-hep/pyhf

Issue:
add versioning to schema
# Description

For future-proofing we need to add versioning to the schemas. I would propose that we have a simple string-based version

```
version: "1.0"
.. remaining keys possibly version specific ..
```

and the following strategy

* when loading the model, it is validated against schemas of the declared version. 

* within the code, a given in-memory representation i.e. the `Workspace` class can be 
instantiated using specs of a subrange of versions `[v_min,v_max]`. possible the ctor will forward/backward evolve to a common format ( I suspect usually forward, i.e. accept a v1 representation but convert it into a v2 representations internally

* a given `pyhf` version  can read a subset of pyhf schema versions. The goal here should be to be as backward and forwards compatible as possible
  * forward compatible: validating the version number itself should be optional (i.e. a v2 spec might be structurally compatible with v1, e.g. if we add new optional fields) so it should still validate as a v1 model even though it has "version: v2" in the spec.
  * backward compatible: future pyhf versions should carry all older schemas as well so older models can be validated. if needed we will add converters for forward evolution to a more recent version once a model is validated in a older version
  * probably would be nice to have `pyhf` be able to output what version range is supported
  * to the extend that pyhf has translators from one version to another, a cli to do this conversion might be nice in order to normalize let's say multiple worksapces defined in different versions for a combination 

* if backward compatibility is not possible, this means a older version of the spec can only be read in a older version of pyhf. This is fine per se, this necessarily will require a new major release, but we should really try to avoid it.




Discussion:
I have a couple of thoughts on this. Given that the spec loading is simply to flesh out the workspace/model - perhaps we can move the Workspace class into a versioned piece of python code -- but this means adding tests and ensuring all the various forward/backward compatibilities that come with it.

I'm ok with versioning the schema alone - but we really should not be changing it too much after this point in time to be honest -- since we're already able to round-trip results.
yes I don't expect backward-incompatible changes math-wise, but could imagine that we move things around in the schema to accomodate future features. Given the first point I think the chance that we can forward translate old schemas is quite high since I don't think we'll drop any features.
> yes I don't expect backward-incompatible changes math-wise, but could imagine that we move things around in the schema to accomodate future features. Given the first point I think the chance that we can forward translate old schemas is quite high since I don't think we'll drop any features.

It's a good point. This is the part that I'm still thinking about how we can structure things. One can version explicitly -- or we could keep around older versions:

```
- pyhf/schemas/latest/ --> v2/
- pyhf/schemas/v1/
- pyhf/schemas/v2/
```

as an example of versioning implicitly. Then we're keeping around all versions of it. Similarly, if one does it this way, then we can just provide helper utilities to convert from one version to another.

Similarly, if we version explicitly, with the version number, we can write a utility that handles the conversions entirely.

Resolving patch:
diff --git a/pyhf/pdf.py b/pyhf/pdf.py
--- a/pyhf/pdf.py
+++ b/pyhf/pdf.py
@@ -176,9 +176,10 @@ class Model(object):
     def __init__(self, spec, **config_kwargs):
         self.spec = copy.deepcopy(spec)  # may get modified by config
         self.schema = config_kwargs.pop('schema', 'model.json')
+        self.version = config_kwargs.pop('version', None)
         # run jsonschema validation of input specification against the (provided) schema
         log.info("Validating spec against schema: {0:s}".format(self.schema))
-        utils.validate(self.spec, self.schema)
+        utils.validate(self.spec, self.schema, version=self.version)
         # build up our representation of the specification
         self.config = _ModelConfig(self.spec, **config_kwargs)
 
@@ -483,9 +484,10 @@ def __init__(self, spec, **config_kwargs):
         self.spec = spec
 
         self.schema = config_kwargs.pop('schema', 'workspace.json')
+        self.version = config_kwargs.pop('version', None)
         # run jsonschema validation of input specification against the (provided) schema
         log.info("Validating spec against schema: {0:s}".format(self.schema))
-        utils.validate(self.spec, self.schema)
+        utils.validate(self.spec, self.schema, version=self.version)
 
         self.measurement_names = []
         for measurement in self.spec.get('measurements', []):
@@ -512,7 +514,7 @@ def get_measurement(self, **config_kwargs):
 
         """
         m = self._get_measurement(**config_kwargs)
-        utils.validate(m, 'measurement.json')
+        utils.validate(m, 'measurement.json', self.version)
         return m
 
     def _get_measurement(self, **config_kwargs):
diff --git a/pyhf/readxml.py b/pyhf/readxml.py
--- a/pyhf/readxml.py
+++ b/pyhf/readxml.py
@@ -270,6 +270,7 @@ def parse(configfile, rootdir, track_progress=False):
         'measurements': process_measurements(toplvl),
         'channels': [{'name': k, 'samples': v['samples']} for k, v in channels.items()],
         'data': {k: v['data'] for k, v in channels.items()},
+        'version': utils.SCHEMA_VERSION,
     }
 
     utils.validate(result, 'workspace.json')
diff --git a/pyhf/utils.py b/pyhf/utils.py
--- a/pyhf/utils.py
+++ b/pyhf/utils.py
@@ -10,16 +10,23 @@
 
 SCHEMA_CACHE = {}
 SCHEMA_BASE = "https://diana-hep.org/pyhf/schemas/"
+SCHEMA_VERSION = '1.0.0'
 
 
-def load_schema(schema_id):
+def load_schema(schema_id, version=None):
     global SCHEMA_CACHE
+    if not version:
+        version = SCHEMA_VERSION
     try:
-        return SCHEMA_CACHE["{0:s}{1:s}".format(SCHEMA_BASE, schema_id)]
+        return SCHEMA_CACHE[
+            "{0:s}{1:s}".format(SCHEMA_BASE, os.path.join(version, schema_id))
+        ]
     except KeyError:
         pass
 
-    path = pkg_resources.resource_filename(__name__, os.path.join('schemas', schema_id))
+    path = pkg_resources.resource_filename(
+        __name__, os.path.join('schemas', version, schema_id)
+    )
     with open(path) as json_schema:
         schema = json.load(json_schema)
         SCHEMA_CACHE[schema['$id']] = schema
@@ -30,8 +37,8 @@ def load_schema(schema_id):
 load_schema('defs.json')
 
 
-def validate(spec, schema_name):
-    schema = load_schema(schema_name)
+def validate(spec, schema_name, version=None):
+    schema = load_schema(schema_name, version=version)
     try:
         resolver = jsonschema.RefResolver(
             base_uri='file://{0:s}'.format(


Test patch: