Repository: bids-standard/pybids

Issue:
Can no longer specify paths to custom schemas
Previously, you could use `config` to pass the path of a `.json` config file to define custom entities. This no longer works, because `config` takes a list of _names_ and not paths. Passing in a path will not work as the path of pybids configs will be appended to it.

This might be a problem for some users, and we could potentially add this functionality back in.

@effigies this affects fitlins. can you live without this feature? it looks like you pass a custom set of entities only when using `BIDSLayout` for outputting new paths, in which case you could potentially just use a basic grabbit `Layout` which still accepts full paths in its API.


Discussion:
I would prefer to have it in `BIDSLayout`. Is there a reason that it was removed, or was it just an oversight during refactoring?
It was a deliberate decision in order to simplify the interface and prevent confusion (actually, I'm not sure why the `config` argument is still in there at all—I think my plan was to get rid of it entirely; not sure what work it's currently doing). While we could technically allow custom paths in `config`, because of the way things are set up, they would only apply to the root project, not to any derivatives. And I really don't want to go back to manually specifying a full root-to-config mapping; that would sort of defeat the point of the `BIDS`-specific abstraction and/or complicate the way arguments are passed to grabbit.

Remember, you can still add customization via `layout.json` files, so all you have to do is drop whatever config file you're trying to pass at initialization into the root of any folder(s) you want to use it. That's a little more work, but I think it leaves the initialization API uncluttered, and makes the scope of a config file clearer. Can you work with that?
I think the `config` argument is still there so that you can apply more than just `bids` to the base directory, without having to also list it as a `derivative`. Useful when reading just a derivative. 

I think we could work with that, I'm just thinking more long term, if ppl are going to use `BIDSLayout`s to save files, it shouldn't be too hard to do that (although I guess my question is, how many people are going to use custom specs, rather than just the base derivatives spec, which you can access easily) 
Actually, I take that back; you *can* control what `config` applies to a derivative folder if you make explicit `add_derivatives()` calls after initializing the `BIDSLayout` (rather than passing `derivatives=[...]`). So that mitigates at least one of my concerns. It's still the case though that passing extra values in `config` when initializing a `BIDSLayout` will result in them *only* applying to the root project and not to any derivatives. I suppose we could document this behavior in the docstring.

An alternative approach we could consider, and that I'd be fine with, is to allow users to load in new named config files at the module level. I kind of like that because it would make it very clear in the code that the user is overriding the default behavior. So it would be something like:

```python
bids.layout.add_config({"my_extras": "my_extras.json"})
BIDSLayout('/bids', ..., config=['bids', 'my_extras'])
```

Thoughts?

[EDIT] On further thought, this still doesn't solve the problem of `config` not applying to derivatives... you'd still need to call `add_derivatives()` explicitly. But I think I still like this approach better.
That would work for me.

The alternative is to allow `config` to take either keywords (`bids` or `derivatives`) or paths. If an elemnt of the list is not one of the reserved keywords, it can be assumed to be a full path. I suppose that makes the API a *bit* more complex, but it seems like a reasonable amount. 

As an aside, I don't think its a big deal to have to call `add_derivatives` if you want more control as to how derivatives are added (whereas `derivatives=` is really just a "short cut" for a default configuration for derivatives). 
Let's go with the module-level `add_config()` approach; I think it makes sense to be explicit about something like this, and the pybids codebase is already pretty bloated with type-checking and validation. Want to open a PR?

Resolving patch:
diff --git a/bids/config.py b/bids/config.py
--- a/bids/config.py
+++ b/bids/config.py
@@ -11,7 +11,12 @@
 
 _config_name = 'pybids_config.json'
 
-_default_settings = {}
+conf_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
+                         'layout', 'config', '{}.json')
+_default_settings = {
+    'config_paths': {
+        name: conf_path.format(name) for name in ['bids', 'derivatives']}
+}
 
 
 def set_option(key, value):
diff --git a/bids/layout/__init__.py b/bids/layout/__init__.py
--- a/bids/layout/__init__.py
+++ b/bids/layout/__init__.py
@@ -1,3 +1,3 @@
-from .layout import BIDSLayout
+from .layout import BIDSLayout, add_config_paths
 from .validation import BIDSValidator
-__all__ = ["BIDSLayout", "BIDSValidator"]
+__all__ = ["BIDSLayout", "BIDSValidator", "add_config_paths"]
diff --git a/bids/layout/layout.py b/bids/layout/layout.py
--- a/bids/layout/layout.py
+++ b/bids/layout/layout.py
@@ -1,8 +1,10 @@
 import os
+import re
 import json
 import warnings
 from io import open
 from .validation import BIDSValidator
+from .. import config as cf
 from grabbit import Layout, File
 from grabbit.external import six, inflect
 from grabbit.utils import listify
@@ -10,8 +12,7 @@
 from collections import defaultdict
 from functools import reduce, partial
 from itertools import chain
-import re
-
+from bids.config import get_option
 
 try:
     from os.path import commonpath
@@ -26,6 +27,25 @@ def commonpath(paths):
 __all__ = ['BIDSLayout']
 
 
+def add_config_paths(**kwargs):
+    """ Add to the pool of available configuration files for BIDSLayout.
+    Args:
+        kwargs: each kwarg should be a pair of config key name, and path
+
+    Example: bids.layout.add_config_paths(my_config='/path/to/config')
+    """
+
+    for k, path in kwargs.items():
+        if not os.path.exists(path):
+            raise ValueError(
+                'Configuration file "{}" does not exist'.format(k))
+        if k in cf.get_option('config_paths'):
+            raise ValueError('Configuration {!r} already exists'.format(k))
+
+    kwargs.update(**cf.get_option('config_paths'))
+    cf.set_option('config_paths', kwargs)
+
+
 class BIDSFile(File):
     """ Represents a single BIDS file. """
     def __init__(self, filename, layout):
@@ -57,7 +77,7 @@ def image(self):
         """
         try:
             return nb.load(self.path)
-        except Exception as e:
+        except Exception:
             return None
 
     @property
@@ -122,16 +142,20 @@ def __init__(self, root, validate=True, index_associated=True,
 
         target = os.path.join(self.root, 'dataset_description.json')
         if not os.path.exists(target):
-            raise ValueError("'dataset_description.json' file is missing from "
-                          "project root. Every valid BIDS dataset must have "
-                          "this file.")
+            if validate is True:
+                raise ValueError(
+                    "'dataset_description.json' is missing from project root."
+                    " Every valid BIDS dataset must have this file.")
+            else:
+                self.description = None
         else:
             with open(target, 'r', encoding='utf-8') as desc_fd:
                 self.description = json.load(desc_fd)
-            for k in ['Name', 'BIDSVersion']:
-                if k not in self.description:
-                    raise ValueError("Mandatory '%s' field missing from "
-                                     "dataset_description.json." % k)
+            if validate is True:
+                for k in ['Name', 'BIDSVersion']:
+                    if k not in self.description:
+                        raise ValueError("Mandatory '%s' field missing from "
+                                         "dataset_description.json." % k)
 
         # Determine which subdirectories to exclude from indexing
         excludes = {"code", "stimuli", "sourcedata", "models", "derivatives"}
@@ -147,11 +171,8 @@ def __init__(self, root, validate=True, index_associated=True,
         # Set up path and config for grabbit
         if config is None:
             config = 'bids'
-        config = listify(config)
-        conf_path = os.path.join(os.path.dirname(os.path.abspath(__file__)),
-                                 'config', '%s.json')
-        bids_conf = [conf_path % c for c in config]
-        path = (root, bids_conf)
+        config_paths = get_option('config_paths')
+        path = (root, [config_paths[c] for c in listify(config)])
 
         # Initialize grabbit Layout
         super(BIDSLayout, self).__init__(path, root=self.root,
@@ -162,11 +183,11 @@ def __init__(self, root, validate=True, index_associated=True,
         # Add derivatives if any are found
         self.derivatives = {}
         if derivatives:
-            if derivatives == True:
+            if derivatives is True:
                 derivatives = os.path.join(root, 'derivatives')
             self.add_derivatives(
                 derivatives, validate=validate,
-                index_associated=index_associated,include=include,
+                index_associated=index_associated, include=include,
                 absolute_paths=absolute_paths, derivatives=None, config=None,
                 sources=self, **kwargs)
 
@@ -211,8 +232,8 @@ def check_for_description(dir):
                 'PipelineDescription', {}).get('Name', None)
             if pipeline_name is None:
                 raise ValueError("Every valid BIDS-derivatives dataset must "
-                                "have a PipelineDescription.Name field set "
-                                "inside dataset_description.json.")
+                                 "have a PipelineDescription.Name field set "
+                                 "inside dataset_description.json.")
             if pipeline_name in self.derivatives:
                 raise ValueError("Pipeline name '%s' has already been added "
                                  "to this BIDSLayout. Every added pipeline "
@@ -223,13 +244,15 @@ def check_for_description(dir):
             self.derivatives[pipeline_name] = BIDSLayout(deriv, **kwargs)
 
             # Propagate derivative entities into top-level dynamic getters
-            deriv_entities = set(ent.name
-                                 for ent in self.derivatives[pipeline_name].entities.values())
+            deriv_entities = set(
+                ent.name
+                for ent in self.derivatives[pipeline_name].entities.values())
             for deriv_ent in deriv_entities - local_entities:
                 local_entities.add(deriv_ent)
                 getter = 'get_' + inflect.engine().plural(deriv_ent)
                 if not hasattr(self, getter):
-                    func = partial(self.get, target=deriv_ent, return_type='id')
+                    func = partial(
+                        self.get, target=deriv_ent, return_type='id')
                     setattr(self, getter, func)
 
     def to_df(self, **kwargs):
@@ -302,9 +325,9 @@ def _get_nearest_helper(self, path, extension, suffix=None, **kwargs):
                 )
             suffix = f.entities['suffix']
 
-        tmp = self.get_nearest(path, extensions=extension, all_=True,
-                               suffix=suffix, ignore_strict_entities=['suffix'],
-                               **kwargs)
+        tmp = self.get_nearest(
+            path, extensions=extension, all_=True, suffix=suffix,
+            ignore_strict_entities=['suffix'], **kwargs)
 
         if len(tmp):
             return tmp
@@ -349,10 +372,11 @@ def get(self, return_type='object', target=None, extensions=None,
                 only files that match the first two subjects.
 
         Returns:
-            A list of BIDSFile (default) or other (see return_type for details) objects.
+            A list of BIDSFile (default) or other objects
+            (see return_type for details).
         """
 
-        if derivatives == True:
+        if derivatives is True:
             derivatives = list(self.derivatives.keys())
         elif derivatives:
             derivatives = listify(derivatives)
@@ -445,7 +469,6 @@ def get_metadata(self, path, include_entities=False, **kwargs):
         results.update(self.metadata_index.file_index[path])
         return results
 
-
     def get_bvec(self, path, **kwargs):
         """Get bvec file for passed path."""
         tmp = self._get_nearest_helper(path, 'bvec', suffix='dwi', **kwargs)[0]
@@ -601,7 +624,7 @@ def index_file(self, f, overwrite=False):
         if f.path in self.file_index and not overwrite:
             return
 
-        if 'suffix' not in f.entities: # Skip files without suffixes
+        if 'suffix' not in f.entities:  # Skip files without suffixes
             return
 
         md = self._get_metadata(f.path)


Test patch: