Repository: dask/dask

Issue:
Use environment variable to override root dask config location
Dask's new configuration file structure looks in `/etc/dask` for a system wide configuration. I'd like to suggest this be overridable via a environment variable. I've included the relevant lines below.

https://github.com/dask/dask/blob/69fc200c9d6524820a75b1f7713af44848c522a6/dask/config.py#L18-L29

In particular, I'd like to see line 19 changed to something like:

```Python
    os.getenv('DASK_ROOT_CONFIG', '/etc/dask')
```

This would allow system admins to point to a common config file without mandating it be placed in `/etc/...`.

I'll note that there is already a `DASK_CONFIG` environment variable. This is however appended to the list of dask configs so it would difficult to use for global/system wide configurations.

Btw, my particular use case is for providing a system wide configuration for dask-jobqueue on a large HPC system. 


Discussion:
Any thoughts on this issue? I'd be happy to issue a PR if there is agreement on how to proceed. 
Thoughts @mrocklin?
My apologies for the lack of response here @jhamman .  I've been swamped.  

I don't have any particular objection to this if we can get a couple other people to weigh in.  Maybe @jacobtomlinson, @lesteve, and @jcrist for variety across deployment scenarios?
No strong thoughts. If I were to rewrite those lines, I'd cut back the number of potential directories that configurations are loaded from. As is the user gets a mix of files from potentially a few spots. This seems potentially error prone, as one user might store `kubernetes.yaml` in one of those directories, only to have it be unexpectedly overridden by configuration in a different directory. 

I'd prefer to have a single root config directory, and a single user config directory. The location of each could be set by an environment variable, and if not set a list of potential directories will be searched, stopping when a path exists. I don't see the need for more than 2 configuration locations (one admin one user), and cutting back the number of locations that may be loaded from seems easier to reason about.

Something like (untested):

```python
def get_config_path(envvar, search_paths):
    if envvar in os.environ:
        # if explicitly specified, only use that
        return os.environ[envvar]
    else:
        # Find the first directory that has dask configurations, and use that exclusively.
        for path in search_paths:
            if os.path.exists(path):
                return path

    # No path found
    return None

# find the root configuration directory, either from an environment
# variable, or the first existing directory in a search list
root_config_path = get_config_path('DASK_ROOT_CONFIG',
                                   [os.path.join('etc', 'dask'),
                                    os.path.join(sys.prefix, 'etc', 'dask')])

# find the user configuration directory, either from an environment
# variable, or the first existing directory in a search list
user_config_path = get_config_path('DASK_CONFIG',
                                   [os.path.join(os.path.expanduser('~'), '.config', 'dask'), 
                                    os.path.join(os.path.expanduser('~'), '.dask')])

paths = []
if root_config_path is not None:
    paths.append(root_config_path)

if user_config_path is not None:
    paths.append(user_config_path)

# continue on with code as written...
```
I think I agree with @jcrist's comments.

Personally I'm a big fan of [storing config in the environment](https://12factor.net/config). So I tend to use the environment variable version of this stuff. However when using config files locally I prefer to have a single system config dir and then a user specific config dir. Generally `appdirs` helps achieve this.
> I'd prefer to have a single root config directory, and a single user config directory

Reasons for the following:

-  '/etc/dask': system administrators
-  os.path.join(sys.prefix, 'etc', 'dask'): same, but at the self-contained environment level
-  os.path.join(os.path.expanduser('~'), '.config', 'dask'): current recommended user config directory
-  os.path.join(os.path.expanduser('~'), '.dask'): backwards compatibility with previous recommended user config directory


In general I agree with the sentiment of reducing config locations, but pragmatically I have yet to run into a case where conflicts have confused anyone.  I've seen active use cases for all of them except for `sys.prefix/etc/dask`
It sounds like in general people are ok with the name `DASK_ROOT_CONFIG`.  It seems like there is an active question about whether this should replace `/etc/dask` or work in addition to it
I'm not saying we should remove support for any of them, just that I doubt people have a need for more than one root/user config location (and the possibility of using more than one root/user config location seems dangerous). If I see a `/etc/dask` config location exists, I'd expect all admin config to be in there. If admin config is also loaded from `os.path.join(sys.prefix, 'etc', 'dask')` that'd be surprising to me. Same goes for user config - if I set `DASK_CONFIG` to somewhere else, I wouldn't expect files in `~/.config/dask` or `~/.dask` to be loaded.

>  It seems like there is an active question about whether this should replace `/etc/dask` or work in addition to it.

My vote is for override, not additional, following the logic above.
> '/etc/dask': system administrators

I think you understand the motivation for this issue already but for others, our system administrators are not willing to put a dask config in /etc/. They are willing however to set a environment variable in the default user environment.

Resolving patch:
diff --git a/dask/config.py b/dask/config.py
--- a/dask/config.py
+++ b/dask/config.py
@@ -16,7 +16,7 @@
 
 
 paths = [
-    '/etc/dask',
+    os.getenv('DASK_ROOT_CONFIG', '/etc/dask'),
     os.path.join(sys.prefix, 'etc', 'dask'),
     os.path.join(os.path.expanduser('~'), '.config', 'dask'),
     os.path.join(os.path.expanduser('~'), '.dask')


Test patch: