Repository: dask/dask

Issue:
Key normalization when reading config breaks logging configuration of Distributed
I wanted to configure logging using [configuration dictionary schema](
https://docs.python.org/2/library/logging.config.html#logging-config-dictschema) as it is documented in [the Dask documentation](http://distributed.dask.org/en/latest/configuration.html#logging). However, I ran into a road block. The configuration dictionary schema uses mappings to specify keyword arguments for the constructors of filters, log handlers, etc. As such, the keys of these must be valid Python identifiers and in many cases the expected arguments have underscores in their names. But, Dask replaces all underscores by hyphens when reading the configuration files. In result, logging configuration is broken the keys no longer map the expected values of the objects to be instantiated and, actually, aren't even valid Python identifiers anymore.

A somewhat minimal Dask configuration file to reproduce the issue is as follows:
```
logging:
    version: 1
    disable_existing_loggers: False
    root:
        level: INFO
        handlers:
            - gelf
    handlers:
        gelf:
            level: INFO
            class: pygelf.GelfUdpHandler
            host: localhost
            port: 31337
            include_extra_fields: True
            debug: True
            static_fields:
                _something: 11
```
In case you are wondering, host and port are obviously just dummy values. But, as we won't get as far as to send any packages this won't matter.

The example requires the following packages:
```
dask[distributed]
pygelf
```

Assuming the config is in `dask.config`, the scheduler will fail as follows:
```
$ DASK_CONFIG=dask.config dask-scheduler
Traceback (most recent call last):
  File "<redacted>/lib/python3.6/logging/config.py", line 558, in configure
    handler = self.configure_handler(handlers[name])
  File "<redacted>/lib/python3.6/logging/config.py", line 729, in configure_handler
    kwargs = dict([(k, config[k]) for k in config if valid_ident(k)])
  File "<redacted>/lib/python3.6/logging/config.py", line 729, in <listcomp>
    kwargs = dict([(k, config[k]) for k in config if valid_ident(k)])
  File "<redacted>/lib/python3.6/logging/config.py", line 277, in valid_ident
    raise ValueError('Not a valid Python identifier: %r' % s)
ValueError: Not a valid Python identifier: 'include-extra-fields'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<redacted>/bin/dask-scheduler", line 7, in <module>
    from distributed.cli.dask_scheduler import go
  File "<redacted>/lib/python3.6/site-packages/distributed/__init__.py", line 3, in <module>
    from . import config
  File "<redacted>/lib/python3.6/site-packages/distributed/config.py", line 136, in <module>
    initialize_logging(dask.config.config)
  File "<redacted>/lib/python3.6/site-packages/distributed/config.py", line 131, in initialize_logging
    _initialize_logging_new_style(config)
  File "<redacted>/lib/python3.6/site-packages/distributed/config.py", line 110, in _initialize_logging_new_style
    logging.config.dictConfig(config.get('logging'))
  File "<redacted>/lib/python3.6/logging/config.py", line 795, in dictConfig
    dictConfigClass(config).configure()
  File "<redacted>/lib/python3.6/logging/config.py", line 566, in configure
    '%r: %s' % (name, e))
ValueError: Unable to configure handler 'gelf': Not a valid Python identifier: 'include-extra-fields'
```

When using legacy Python the same issue occurs, so the traceback looks somewhat different.

Sorry if this issue would have better been filed with Distributed. But I as it is an both are involved I assumed this was the proper place.


Discussion:
cc @jrbourbeau @djhoese 
@jrbourbeau @mrocklin So I've thinking about this a little bit, what if we change the default "normalized" keys to be internally underscored instead of hyphenated. This flips the problem from not being able to represent underscored keys to not being able to represent hyphenated keys. Are there any cases where hyphenated keys internal to the config are needed?

Matt and I kind of talked about this in the related #4143 PR: https://github.com/dask/dask/pull/4143#issuecomment-433538506
I implemented your suggestion to reverse the normalized form in #4422. I still think we should only normalize declared keys (we could infer these from the default config files shipped with each library), but that's a more expansive change that could be done later.
I've had a similar issue. The key normalization is causing a bug that disables existing loggers if calling `logging.config.dictConfig(dask.config.config.get('logging'))` , e.g. https://github.com/dask/distributed/blob/master/distributed/config.py#L110

Basically, setting `disable_existing_loggers: False` in dask config also does not work because the normalization will change it to `disable-existing-loggers`, which cannot be recognized by `logging.config.dictConfig` and results in existing logger being disabled.

Not sure if I followed the resolution of this issue, but it seems to me the '_' => '-' normalization still exists. Does anyone know if it can be removed for logging config? If not, what's the best approach to make disable_existing_loggers works?

More details related to "disable_existing_loggers" attribute see: https://python.readthedocs.io/en/stable/library/logging.config.html#dictionary-schema-details

Resolving patch:
diff --git a/dask/config.py b/dask/config.py
--- a/dask/config.py
+++ b/dask/config.py
@@ -9,7 +9,7 @@
 except ImportError:
     yaml = None
 
-from .compatibility import makedirs, builtins, Mapping, string_types
+from .compatibility import makedirs, builtins, Mapping
 
 
 no_default = '__no_default__'
@@ -38,6 +38,29 @@
 defaults = []
 
 
+def canonical_name(k, config):
+    """Return the canonical name for a key.
+
+    Handles user choice of '-' or '_' conventions by standardizing on whichever
+    version was set first. If a key already exists in either hyphen or
+    underscore form, the existing version is the canonical name. If neither
+    version exists the original key is used as is.
+    """
+    try:
+        if k in config:
+            return k
+    except TypeError:
+        # config is not a mapping, return the same name as provided
+        return k
+
+    altk = k.replace('_', '-') if '_' in k else k.replace('-', '_')
+
+    if altk in config:
+        return altk
+
+    return k
+
+
 def update(old, new, priority='new'):
     """ Update a nested dictionary with values from another
 
@@ -68,11 +91,10 @@ def update(old, new, priority='new'):
     dask.config.merge
     """
     for k, v in new.items():
-        if k not in old and isinstance(v, Mapping):
-            old[k] = {}
+        k = canonical_name(k, old)
 
         if isinstance(v, Mapping):
-            if old[k] is None:
+            if k not in old or old[k] is None:
                 old[k] = {}
             update(old[k], v, priority=priority)
         else:
@@ -104,38 +126,6 @@ def merge(*dicts):
     return result
 
 
-def normalize_key(key):
-    """ Replaces underscores with hyphens in string keys
-
-    Parameters
-    ----------
-    key : string, int, or float
-        Key to assign.
-    """
-    if isinstance(key, string_types):
-        key = key.replace('_', '-')
-    return key
-
-
-def normalize_nested_keys(config):
-    """ Replaces underscores with hyphens for keys for a nested Mapping
-
-    Examples
-    --------
-    >>> a = {'x': 1, 'y_1': {'a_2': 2}}
-    >>> normalize_nested_keys(a)
-    {'x': 1, 'y-1': {'a-2': 2}}
-    """
-    config_norm = {}
-    for key, value in config.items():
-        if isinstance(value, Mapping):
-            value = normalize_nested_keys(value)
-        key_norm = normalize_key(key)
-        config_norm[key_norm] = value
-
-    return config_norm
-
-
 def collect_yaml(paths=paths):
     """ Collect configuration from yaml files
 
@@ -166,7 +156,6 @@ def collect_yaml(paths=paths):
         try:
             with open(path) as f:
                 data = yaml.safe_load(f.read()) or {}
-                data = normalize_nested_keys(data)
                 configs.append(data)
         except (OSError, IOError):
             # Ignore permission errors
@@ -184,7 +173,6 @@ def collect_env(env=None):
 
     -  Lower-cases the key text
     -  Treats ``__`` (double-underscore) as nested access
-    -  Replaces ``_`` (underscore) with a hyphen.
     -  Calls ``ast.literal_eval`` on the value
     """
     if env is None:
@@ -193,7 +181,6 @@ def collect_env(env=None):
     for name, value in env.items():
         if name.startswith('DASK_'):
             varname = name[5:].lower().replace('__', '.')
-            varname = normalize_key(varname)
             try:
                 d[varname] = ast.literal_eval(value)
             except (SyntaxError, ValueError):
@@ -329,7 +316,7 @@ def _assign(cls, keys, value, d, old=None, path=[]):
         path: List[str]
             Used internally to hold the path of old values
         """
-        key = normalize_key(keys[0])
+        key = canonical_name(keys[0], d)
         if len(keys) == 1:
             if old is not None:
                 path_key = tuple(path + [key])
@@ -436,7 +423,7 @@ def get(key, default=no_default, config=config):
     keys = key.split('.')
     result = config
     for k in keys:
-        k = normalize_key(k)
+        k = canonical_name(k, result)
         try:
             result = result[k]
         except (TypeError, IndexError, KeyError):
@@ -452,8 +439,8 @@ def rename(aliases, config=config):
 
     This helps migrate older configuration versions over time
     """
-    old = list()
-    new = dict()
+    old = []
+    new = {}
     for o, n in aliases.items():
         value = get(o, None, config=config)
         if value is not None:
diff --git a/dask/context.py b/dask/context.py
--- a/dask/context.py
+++ b/dask/context.py
@@ -65,7 +65,7 @@ def globalmethod(default=None, key=None, falsey=None):
 class GlobalMethod(object):
     def __init__(self, default, key, falsey=None):
         self._default = default
-        self._key = config.normalize_key(key)
+        self._key = key
         self._falsey = falsey
 
     def __get__(self, instance, owner=None):


Test patch:
diff --git a/dask/tests/test_config.py b/dask/tests/test_config.py
--- a/dask/tests/test_config.py
+++ b/dask/tests/test_config.py
@@ -11,10 +11,22 @@
 from dask.config import (update, merge, collect, collect_yaml, collect_env,
                          get, ensure_file, set, config, rename,
                          update_defaults, refresh, expand_environment_variables,
-                         normalize_key, normalize_nested_keys)
+                         canonical_name)
+
 from dask.utils import tmpfile
 
 
+def test_canonical_name():
+    c = {'foo-bar': 1,
+         'fizz_buzz': 2}
+    assert canonical_name('foo-bar', c) == 'foo-bar'
+    assert canonical_name('foo_bar', c) == 'foo-bar'
+    assert canonical_name('fizz-buzz', c) == 'fizz_buzz'
+    assert canonical_name('fizz_buzz', c) == 'fizz_buzz'
+    assert canonical_name('new-key', c) == 'new-key'
+    assert canonical_name('new_key', c) == 'new_key'
+
+
 def test_update():
     a = {'x': 1, 'y': {'a': 1}}
     b = {'x': 2, 'z': 3, 'y': OrderedDict({'b': 2})}
@@ -134,7 +146,7 @@ def test_env():
            }
 
     expected = {
-        'a-b': 123,
+        'a_b': 123,
         'c': True,
         'd': 'hello',
         'e': {'x': 123, 'y': 456},
@@ -142,7 +154,8 @@ def test_env():
         'g': '/not/parsable/as/literal',
     }
 
-    assert collect_env(env) == expected
+    res = collect_env(env)
+    assert res == expected
 
 
 def test_collect():
@@ -201,7 +214,7 @@ def test_ensure_file(tmpdir):
     ensure_file(source=source, destination=dest, comment=False)
 
     with open(destination) as f:
-        result = yaml.load(f)
+        result = yaml.safe_load(f)
     assert result == a
 
     # don't overwrite old config files
@@ -211,7 +224,7 @@ def test_ensure_file(tmpdir):
     ensure_file(source=source, destination=dest, comment=False)
 
     with open(destination) as f:
-        result = yaml.load(f)
+        result = yaml.safe_load(f)
     assert result == a
 
     os.remove(destination)
@@ -224,7 +237,7 @@ def test_ensure_file(tmpdir):
     assert '123' in text
 
     with open(destination) as f:
-        result = yaml.load(f)
+        result = yaml.safe_load(f)
     assert not result
 
 
@@ -343,37 +356,32 @@ def test_expand_environment_variables(inp, out):
         del os.environ['FOO']
 
 
-@pytest.mark.parametrize('inp,out', [
-    ('custom_key', 'custom-key'),
-    ('custom-key', 'custom-key'),
-    (1, 1),
-    (2.3, 2.3)
-])
-def test_normalize_key(inp, out):
-    assert normalize_key(inp) == out
-
-
-def test_normalize_nested_keys():
-    config = {'key_1': 1,
-              'key_2': {'nested_key_1': 2},
-              'key_3': 3
-              }
-    expected = {'key-1': 1,
-                'key-2': {'nested-key-1': 2},
-                'key-3': 3
-                }
-    assert normalize_nested_keys(config) == expected
-
-
-def test_env_var_normalization(monkeypatch):
+def test_env_var_canonical_name(monkeypatch):
     value = 3
-    monkeypatch.setenv('DASK_A_B', value)
+    monkeypatch.setenv('DASK_A_B', str(value))
     d = {}
     dask.config.refresh(config=d)
     assert get('a_b', config=d) == value
     assert get('a-b', config=d) == value
 
 
+def test_get_set_canonical_name():
+    c = {'x-y': {'a_b': 123}}
+
+    keys = ['x_y.a_b', 'x-y.a-b', 'x_y.a-b']
+    for k in keys:
+        assert dask.config.get(k, config=c) == 123
+
+    with dask.config.set({'x_y': {'a-b': 456}}, config=c):
+        for k in keys:
+            assert dask.config.get(k, config=c) == 456
+
+    # No change to new keys in sub dicts
+    with dask.config.set({'x_y': {'a-b': {'c_d': 1}, 'e-f': 2}}, config=c):
+        assert dask.config.get('x_y.a-b', config=c) == {'c_d': 1}
+        assert dask.config.get('x_y.e_f', config=c) == 2
+
+
 @pytest.mark.parametrize('key', ['custom_key', 'custom-key'])
 def test_get_set_roundtrip(key):
     value = 123