Repository: ClimateImpactLab/impactlab-tools

Issue:
Add custom class for handling configuration dictionaries
Idea from Rising to have `implactlab_tools.utils.files.get_allargv_config()` return a specialized dictionary for run configuration data. This new feature is also an onboarding project for @brews.

The rules for these configs for the projection system are (paraphrasing):
> 
> 1. A configuration file may contain multiple levels of configuration: dictionaries within other dictionaries, and so on.
> 
> 2. Different parts of the code are passed different parts of the configuration. This has two use cases: (1) because they are meant to use a specific sub-level of the configuration (e.g., a function may just want the sub-dictionary that contains weather variable definitions), or (2) when different parts of the config. initialize different parts of the calculation, so the same function is called with different parts at different times. The config. dictionary is usually included in a function call as an optional `config` argument.
> 
> 3. Dictionary keys are "inherited" into all lower levels. So, when we access a sub-level of the dictionary, what we actually use as the configuration dictionary at that level contains all of the keys of that sub-level, plus any keys of the parent level(s) that were not over-written by the sub-level. The code that currently creates this combined config. is merge(parent, child) in https://bitbucket.org/ClimateImpactLab/impact-calculations/src/master/interpret/configs.py.
> 
> 4. Key access should be case-insensitive and underscore-vs.-dash insensitive. This is not fully implemented in the current system.

Specifically looking for these features in the new class:

> 1. It looks just like a dict object, although I think only the __getitem__, __setitem__, and __iter__ methods need to be defined.
> 
> 2. It will always be initialized by a dictionary.
> 
> 3. It silently implements (3) and (4) above. I suggest that (3) is implemented by defining a parent attribute for each ConfigurationDictionary, which is accessed if the key is not found in the present dictionary. This will be useful for the next item.
> 
> 4. Whenever a key is accessed, the inspect.stack() is stored, associated with that key. Only the most recent is needed.
> 
> 5. A merge method, like the one from the code above, which can be used to merge two different configurations. Probably this can be implemented using by just making them both ConfigurationDictionarys, and setting the parent attribute on one of them. Make sure that a merge does not cause all keys to be accessed in the sense used in 4.
> 
> 6. A method on the object, "assert_completelyused" would assert that every key in the config. tree has been accessed.

Also need to write documentation and tests.

Discussion:

Patch:
diff --git a/docs/impactlab_tools.utils.rst b/docs/impactlab_tools.utils.rst
index fbd5a12..d51a332 100644
--- a/docs/impactlab_tools.utils.rst
+++ b/docs/impactlab_tools.utils.rst
@@ -20,6 +20,16 @@ impactlab_tools.utils.files module
     :undoc-members:
     :show-inheritance:
 
+
+impactlab_tools.utils.configdict module
+---------------------------------------
+
+.. automodule:: impactlab_tools.utils.configdict
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
 impactlab_tools.utils.versions module
 -------------------------------------
 
diff --git a/impactlab_tools/utils/configdict.py b/impactlab_tools/utils/configdict.py
new file mode 100644
index 0000000..cb9e5ec
--- /dev/null
+++ b/impactlab_tools/utils/configdict.py
@@ -0,0 +1,257 @@
+"""Class for representing tool configuration files
+"""
+import inspect
+# For python2 support:
+try:
+    from collections import UserDict
+    import collections.abc as collections_abc
+except ImportError:
+    from UserDict import UserDict
+    import collections as collections_abc
+
+
+def gather_configtree(d):
+    """Chains nested-dicts into a connected tree of ConfigDict(s)
+
+    Parameters
+    ----------
+    d : dict
+        Cast to :py:class:`ConfigDict`. Nested dicts within are also
+        recursively cast and assigned parents, reflecting their nested
+        structure.
+
+    Returns
+    -------
+    out : ConfigDict
+
+    Examples
+    --------
+    .. code-block:: python
+
+        >>> nest = {'a': 1, 'b': {'a': 2}, 'c': 3, 'd-4': 4, 'e_5': 5, 'F': 6}
+        >>> tree = gather_configtree(nest)
+        >>> tree['b']['a']
+        2
+
+    Returns the value for "a" in the *nested* dictionary "b". However, if we
+    request a key that is not available in this nested "b" dictionary, it
+    will search through all parents.
+
+        >>> tree['b']['d-4']
+        4
+
+    A `KeyError` is only thrown if the search has been exhausted with no
+    matching keys found.
+    """
+    out = ConfigDict(d)
+    for k, v in out.data.items():
+        # Replace nested maps with new ConfigDicts
+        if isinstance(v, collections_abc.MutableMapping):
+            out.data[k] = gather_configtree(v)
+            out.data[k].parent = out
+    return out
+
+
+class ConfigDict(UserDict, object):
+    """Chain-able dictionary to hold projection configurations.
+
+    A ConfigDict is a dictionary-like interface to a chainmap/linked list.
+    Nested dicts can be access like a traditional dictionary but it searches
+    parent dictionaries for keys:values not found. All string keys normalized,
+    by transforming all characters to lowercase, and all underscores to
+    hyphens.
+
+    Attributes
+    ----------
+    parent : ConfigDict or None
+        Parent ConfigDict object to query for keys if not in ``self.data``.
+    key_access_stack : dict
+        Dictionary with values giving the :py:func:`inspect.stack()` from the
+        most recent time a key was retrieved (via ``self.__getitem__()``).
+    data : dict
+        The 'local' dictionary, not in parents.
+
+    See Also
+    --------
+    gather_configtree : Chains nested-dicts into a connected tree of
+        ConfigDict(s)
+
+    Examples
+    --------
+    .. code-block:: python
+
+        >>> d = {'a': 1, 'b': {'a': 2}, 'c': 3, 'd-4': 4, 'e_5': 5, 'F': 6}
+        >>> cd = ConfigDict(d)
+        >>> cd['b']
+        {'a': 2}
+
+        'F' key is now lowercase.
+
+        >>> cd['f']
+        6
+
+        '_' is now '-'
+
+        >>> cd['e-5']
+        5
+
+        Keys that have been accessed.
+
+        >>> cd.key_access_stack.keys() # doctest: +SKIP
+        dict_keys(['b', 'f', 'e-5'])
+    """
+    def __init__(self, *args, **kwargs):
+        super(ConfigDict, self).__init__(*args, **kwargs)
+        self.parent = None
+
+        for k, v in self.data.items():
+            # Normalize string keys, only when needed
+            if isinstance(k, str) and (k.isupper() or '_' in k):
+                new_key = self._normalize_key(k)
+                self.data[new_key] = self.pop(k)
+
+        self.key_access_stack = dict()
+
+    def __getitem__(self, key):
+        key = self._normalize_key(key)
+        out = super(ConfigDict, self).__getitem__(key)
+
+        # We don't want to store in key_access_stack if __missing__() was used
+        if key in self.data.keys():
+            self.key_access_stack[key] = inspect.stack()
+        return out
+
+    def __missing__(self, key):
+        if self.parent is None:
+            raise KeyError
+        return self.parent[key]
+
+    def __setitem__(self, key, value):
+        # Note we're not changing `value.parents` if `value` is ConfigDict.
+        key = self._normalize_key(key)
+        super(ConfigDict, self).__setitem__(key, value)
+
+    @staticmethod
+    def _normalize_key(key):
+        """If `key` is str, make lowercase and replace underscores with hyphens
+        """
+        if isinstance(key, str):
+            return key.lower().replace('_', '-')
+
+    def accessed_all_keys(self, search='local'):
+        """Were all the keys used in the config tree?
+
+        Parameters
+        ----------
+        search : {'local', 'parents', 'children'}
+            What should the search cover? Options are:
+
+            ``"local"``
+                Only check whether keys were used locally (in `self`).
+
+            ``"parents"``
+                Recursively check keys in parents, moving up the tree, after
+                checking local keys.
+
+            ``"children"``
+                Recursively check keys in children, moving down the tree, after
+                checking local keys.
+
+        Returns
+        -------
+        bool
+
+        Examples
+        --------
+        .. code-block:: python
+
+            >>> d = {'a': 1, 'b': {'a': 2}, 'c': 3, 'd-4': 4, 'e_5': 5, 'F': 6}
+            >>> root_config = gather_configtree(d)
+            >>> child_config = root_config['b']
+            >>> child_config['a']
+            2
+
+            We can check whether all the keys in `child_config` have been
+            accessed.
+
+            >>> child_config.accessed_all_keys()
+            True
+
+            Same but also checking that all keys up the tree in parents have
+            been used.
+
+            >>> child_config.accessed_all_keys('parents')
+            False
+
+            Several keys in root_config were not accessed, so False is
+            returned.
+
+            Can also check key use locally and down the tree in nested, child
+            ConfigDict instances.
+
+            >>> root_config.accessed_all_keys('children')
+            False
+
+            ...which is still False in this case -- all keys in nested
+            child_config have been used, but not all of the local keys in
+            root_config have been used.
+
+        """
+        search = str(search)
+        search_options = ('local', 'parents', 'children')
+        if search not in search_options:
+            raise ValueError('`search` must be in {}'.format(search_options))
+
+        local_access = set(self.key_access_stack.keys())
+        local_keys = set(self.data.keys())
+        all_used = local_access == local_keys
+
+        # Using a "fail fast" strategy...
+
+        if all_used is False:
+            return False
+
+        if search == 'parents':
+            # Recursively check parents keys, if any haven't been used,
+            # immediately return False.
+            if self.parent is not None:
+                parent_used = self.parent.accessed_all_keys(search=search)
+                if parent_used is False:
+                    return False
+        elif search == 'children':
+            # Recursively check children keys, if any haven't been used,
+            # immediately return False.
+            for k, v in self.data.items():
+                # Assuming its faster to ask for forgiveness than to check
+                # with `isinstance()` or `hasattr()q...
+                try:
+                    child_used = v.accessed_all_keys(search=search)
+                    if child_used is False:
+                        return False
+                except AttributeError:
+                    continue
+
+        return True
+
+    def merge(self, x, xparent=False):
+        """Merge, returning new copy
+
+        Parameters
+        ----------
+        x : ConfigDict or dict
+        xparent : bool, optional
+            Attach ``x.parent`` to ``out.parent``? If False, attaches
+            ``self.parent``. Only works if `x` is :py:class:`ConfigDict`.
+
+        Return
+        ------
+        out : ConfigDict
+            Merged ConfigDict, using copied values from ``self``.
+        """
+        out = self.copy()
+        out.update(x)
+
+        if xparent is True:
+            out.parent = x.parent
+
+        return out
diff --git a/impactlab_tools/utils/files.py b/impactlab_tools/utils/files.py
index 5611335..837bd4c 100644
--- a/impactlab_tools/utils/files.py
+++ b/impactlab_tools/utils/files.py
@@ -8,6 +8,7 @@ import sys
 import os
 import yaml
 
+
 default_server_config_path = "../server.yml"
 server_config = None
 
@@ -26,6 +27,7 @@ def sharedpath(subpath):
 
     return os.path.join(server_config['shareddir'], subpath)
 
+
 def configpath(path):
     """Return an configured absolute path.  If the path is absolute, it
     will be left alone; otherwise, it is assumed to be a subpath of the
@@ -51,6 +53,7 @@ def use_config(config):
 
     server_config = config
 
+
 def get_file_config(filepath):
     """Load a configuration file from a given path."""
 
@@ -58,6 +61,7 @@ def get_file_config(filepath):
         config = yaml.load(fp)
         return config
 
+
 def get_argv_config(index=1):
     """
     Load a configuration file specified as the `index` argv argument.
@@ -70,6 +74,7 @@ def get_argv_config(index=1):
         config = yaml.load(fp)
         return config
 
+
 def get_allargv_config():
     """
     Load a configuration from the command line, merging all arguments into a single configuration dictionary.
@@ -104,5 +109,6 @@ def get_allargv_config():
 
     return config
 
+
 if __name__ == '__main__':
     print(configpath('testing'))
diff --git a/whatsnew.rst b/whatsnew.rst
index 820029a..72145fd 100644
--- a/whatsnew.rst
+++ b/whatsnew.rst
@@ -8,6 +8,13 @@ v0.4.0
 
  - fix bug causing assets to be missing from installed package (:issue:`187`)
 
+ - Add new :py:class:`impactlab_tools.utils.configdict.ConfigDict` class and convenience function :py:func:`impactlab_tools.utils.configdict.gather_configtree` (:issue:`434`)
+
+v0.3.1 (March 19, 2018)
+-----------------------
+
+ - "stability and performance improvements"
+
 v0.3.0 (March 17, 2018)
 -----------------------
 


Test patch:
diff --git a/tests/utils/test_configdict.py b/tests/utils/test_configdict.py
new file mode 100644
index 0000000..1d299f0
--- /dev/null
+++ b/tests/utils/test_configdict.py
@@ -0,0 +1,182 @@
+import pytest
+
+from impactlab_tools.utils.configdict import ConfigDict, gather_configtree
+
+
+@pytest.fixture
+def simple_nested_tree():
+    return {'a': 1, 'b': {'a': 2}, 'c': 3, 'd-4': 4, 'e_5': 5, 'F': 6}
+
+
+def test_configdict_climbs_tree(simple_nested_tree):
+    conf = gather_configtree(simple_nested_tree)
+    assert conf['b']['c'] == 3
+
+
+def test_configdict_prefer_child(simple_nested_tree):
+    conf = gather_configtree(simple_nested_tree)['b']
+    assert conf['a'] == 2
+
+
+def test_configdict_selfreference(simple_nested_tree):
+    conf = gather_configtree(simple_nested_tree)
+    conf['b'] = conf
+    assert conf['b'] == conf
+
+
+def test_configdict_caseinsensitive_keys(simple_nested_tree):
+    conf = gather_configtree(simple_nested_tree)
+    assert conf['A'] == 1
+    assert conf['f'] == 6
+
+    conf['Z'] = 'foobar'
+    assert conf['z'] == 'foobar'
+
+
+def test_configdict_normalize(simple_nested_tree):
+    """Normalize underscores and hyphens in keys
+    """
+    conf = ConfigDict(simple_nested_tree)
+    assert conf['d_4'] == 4
+    assert conf['e-5'] == 5
+
+    conf['y-7'] = 'foobar'
+    assert conf['y_7'] == 'foobar'
+
+    conf['z-9'] = 'foobar'
+    assert conf['z_9'] == 'foobar'
+
+
+def test_configdict_throws_keyerror(simple_nested_tree):
+    conf = ConfigDict(simple_nested_tree)
+    with pytest.raises(KeyError):
+        conf['foobar']
+
+
+def test_configdict_merge(simple_nested_tree):
+    conf1 = ConfigDict(simple_nested_tree)
+    conf2 = ConfigDict({'foo': 0})
+
+    internal_goal = simple_nested_tree.copy()
+    internal_goal['foo'] = 0
+
+    conf_merge = conf1.merge(conf2)
+
+    assert conf_merge['foo'] == 0
+    for k, v in simple_nested_tree.items():
+        assert conf_merge[k] == v
+    assert conf_merge.parent == conf1.parent
+
+
+def test_configdict_merge_parentswap(simple_nested_tree):
+    conf1 = gather_configtree(simple_nested_tree)['b']
+
+    nested_tree_mod = simple_nested_tree.copy()
+    nested_tree_mod['a'] = 9
+    conf2 = gather_configtree(nested_tree_mod)['b']
+
+    conf_merge = conf1.merge(conf2, xparent=True)
+
+    assert conf_merge.data == conf2.data
+    assert conf_merge.parent == conf2.parent
+
+
+def test_configdict_key_access_stack(simple_nested_tree):
+    """Test ConfigDict adds values to ``self.key_access_stack`` on query
+    """
+    conf = ConfigDict(simple_nested_tree)
+    assert conf.key_access_stack == {}
+    conf['a']
+    assert 'a' in list(conf.key_access_stack.keys())
+
+
+def test_configdict_key_access_stack_nested(simple_nested_tree):
+    """Test ConfigDict.key_access_stack sores keys in appropriate configdict
+    """
+    conf = gather_configtree(simple_nested_tree)
+    nested = conf['b']
+    nested['a']
+    nested['f']
+
+    assert list(nested.key_access_stack.keys()) == ['a']
+
+    top_keys = list(conf.key_access_stack.keys())
+    top_keys.sort()
+    assert top_keys == ['b', 'f']
+
+
+def test_configdict_accessed_all_keys_local(simple_nested_tree):
+    root_conf = gather_configtree(simple_nested_tree)
+    child_conf = root_conf['b']
+
+    root_conf['a']
+    root_conf['c']
+    root_conf['d-4']
+    root_conf['e-5']
+
+    assert root_conf.accessed_all_keys(search='local') is False
+
+    root_conf['f']
+
+    assert root_conf.accessed_all_keys(search='local') is True
+
+    assert child_conf.accessed_all_keys(search='local') is False
+
+
+def test_configdict_accessed_all_keys_local(simple_nested_tree):
+    kwargs = {'search': 'local'}
+    root_conf = gather_configtree(simple_nested_tree)
+    child_conf = root_conf['b']
+
+    root_conf['a']
+    root_conf['c']
+    root_conf['d-4']
+    root_conf['e-5']
+
+    assert root_conf.accessed_all_keys(**kwargs) is False
+
+    root_conf['f']
+
+    assert root_conf.accessed_all_keys(**kwargs) is True
+
+    assert child_conf.accessed_all_keys(**kwargs) is False
+
+
+def test_configdict_accessed_all_keys_children(simple_nested_tree):
+    kwargs = {'search': 'children'}
+
+    root_conf = gather_configtree(simple_nested_tree)
+
+    child_conf = root_conf['b']
+    root_conf['a']
+    root_conf['c']
+    root_conf['d-4']
+    root_conf['e-5']
+    root_conf['f']
+
+    assert root_conf.accessed_all_keys(**kwargs) is False
+
+    child_conf['a']
+
+    assert root_conf.accessed_all_keys(**kwargs) is True
+    assert child_conf.accessed_all_keys(**kwargs) is True
+
+
+def test_configdict_accessed_all_keys_parents(simple_nested_tree):
+    kwargs = {'search': 'parents'}
+
+    root_conf = gather_configtree(simple_nested_tree)
+
+    child_conf = root_conf['b']
+    root_conf['a']
+    root_conf['c']
+    root_conf['d-4']
+    root_conf['e-5']
+    child_conf['a']
+
+    assert child_conf.accessed_all_keys(**kwargs) is False
+
+    root_conf['f']
+
+    assert child_conf.accessed_all_keys(**kwargs) is True
+    assert root_conf.accessed_all_keys(**kwargs) is True