Repository: rochacbruno/dynaconf (python)

Issue:
[RFC] Allow dynaconf cli to use a customized settings object
I have a customized settings object:

```python
from dynaconf import LazySettings

settings = LazySettings(
    GLOBAL_ENV_FOR_DYNACONF='MYPROJECT',
    MERGE_ENABLED_FOR_DYNACONF=True,
)
```

When I run `dynaconf` cli tool, it knows nothing about my customization. How can I tell `dynaconf` cli tool to use my customized version of settings?

Pull request description:
Add INSTANCE_FOR_DYNACONF and --instance
These parameters allows dynaconf to use different `LazySettings` instance
if project uses one.

Also did some other fixes along the way:

- Added `catch_exceptions=False` to `CliRunner.invoke` in order to
  prevent click from swallowing errors silently. This uncovered other
  errors in init and validate cli commands.

- Removed module level code execution from cli module. Module level code
  execution makes it really difficult to test code. Now cli does not
  rely on global state and can be tested properly.

- Removed a code snipper from `LazySettings` which modified global
  default_settings values. This means, that each `LazySettings`
  constructor call has side effects.

- `dynaconf validate` command tests were useless because they didn't
  test anything and I found, that `dynaconf validate` command don't even
  work and raises ValidationError if there are any validation errors.
  Changed that to cli friendly error message.

Fixes #108.

Patch:
diff --git a/HISTORY b/HISTORY
index 24d9922..fd979d9 100644
--- a/HISTORY
+++ b/HISTORY
@@ -3,10 +3,14 @@ Changelog
 
 These are all the changes in Dynaconf since the first public release.
 
-1.1.1 
+1.1.1
 -----
 
 - Unreleased
+- Added `INSTANCE_FOR_DYNACONF` and `--instance` parameters. These parameters can tell cli tools to use custom LazySettings instance. Fixes #108.
+- Fixed `dynaconf validate` bug, when `--path` value does not end with `.toml` exception was rised.
+- Fixed `dynaconf validate` validation error handling, now instead of raising ValidationError an error message is printed and if there are errors "Validateion success!" is not printed.
+
 
 1.1.0
 -----
@@ -15,7 +19,7 @@ These are all the changes in Dynaconf since the first public release.
 - Support for dot notation to access nested dictionaries like `settings['KEY.OTHER.NESTED.DEEPER']` PR #93
 - Support dotted notation for validators PR #98
 - Fixed a bug in SETTINGS_MODULE cleanup when `.setenv` method was called PR #97
-- Added Python 3.7 to test matrix PR #99 
+- Added Python 3.7 to test matrix PR #99
 
 1.0.6
 -----
@@ -31,20 +35,20 @@ These are all the changes in Dynaconf since the first public release.
 1.0.4
 -----
 
-- Fix Django definition of GLOBAL_ENV 
+- Fix Django definition of GLOBAL_ENV
 - Fix in __getitem__ method
 
 1.0.3
 -----
 
-- Excluded example and tests from realease dist 
+- Excluded example and tests from realease dist
 - removed root logger configuration
 
 1.0.2
 -----
 - Improved docs
 - Fix error when some loader is not installed
-- Fix loader import error 
+- Fix loader import error
 - Improved logs
 
 1.0.1
@@ -55,9 +59,9 @@ These are all the changes in Dynaconf since the first public release.
 
 1.0.0
 -----
-- Breaking changes 
+- Breaking changes
 - `namespace` renamed to `env`
-- added concept of DEFAULT and GLOBAL envs 
+- added concept of DEFAULT and GLOBAL envs
 - `env` will default to DEVELOPMENT until changed by envvar `ENV_FOR_DYNACONF`
 
 0.7.6
@@ -98,13 +102,13 @@ These are all the changes in Dynaconf since the first public release.
 
 0.6.0
 -----
-- Drop Python 2.x from Travis CI (must work but will not be tested) 
+- Drop Python 2.x from Travis CI (must work but will not be tested)
 - Fix Flask requirement #41
 
 0.5.6
 -----
-- Added support for .ini and .json files 
-- parse conf is now recursive to parse dict inner data 
+- Added support for .ini and .json files
+- parse conf is now recursive to parse dict inner data
 - Cloned find_file function from dotenv
 
 0.5.5
diff --git a/docs/guides/cli.md b/docs/guides/cli.md
index f57f359..b579ca4 100644
--- a/docs/guides/cli.md
+++ b/docs/guides/cli.md
@@ -15,6 +15,7 @@ Options:
   --version  Show dynaconf version
   --docs     Open documentation in browser
   --banner   Show awesome banner
+  -i, --instance TEXT  Custom instance of LazySettings
   --help     Show this message and exit.
 
 Commands:
diff --git a/docs/guides/configuration.md b/docs/guides/configuration.md
index 2b477fe..a1ed4d7 100644
--- a/docs/guides/configuration.md
+++ b/docs/guides/configuration.md
@@ -49,6 +49,8 @@ Dynaconf can be configured through variables suffixed with `_FOR_DYNACONF` those
 +-----------------+---------+--------------------------------------------------+-------------------------+--------------------------------------------------------------+
 | VAULT_VERIFY    | bool    | vault should verify                              | None                    | VAULT_VERIFY_FOR_DYNACONF=true                               |
 +-----------------+---------+--------------------------------------------------+-------------------------+--------------------------------------------------------------+
+| INSTANCE        | str     | custom instance of LazySettings                  | None                    | INSTANCE_FOR_DYNACONF=myapp.settings                         |
++-----------------+---------+--------------------------------------------------+-------------------------+--------------------------------------------------------------+
 
 ```
 
diff --git a/dynaconf/base.py b/dynaconf/base.py
index 00e125c..556f2b6 100644
--- a/dynaconf/base.py
+++ b/dynaconf/base.py
@@ -83,8 +83,6 @@ class LazySettings(LazyObject):
         """
         compat_kwargs(kwargs)
         self._kwargs = kwargs
-        for k, v in kwargs.items():
-            setattr(default_settings, k.upper(), v)
         super(LazySettings, self).__init__()
 
     def __getattr__(self, name):
@@ -113,7 +111,8 @@ class LazySettings(LazyObject):
 
     def _setup(self):
         """Initial setup, run once."""
-        environment_variable = default_settings.ENVVAR_FOR_DYNACONF
+        environment_variable = self._kwargs.get(
+            'ENVVAR_FOR_DYNACONF', default_settings.ENVVAR_FOR_DYNACONF)
         settings_module = os.environ.get(environment_variable)
         self._wrapped = Settings(
             settings_module=settings_module, **self._kwargs
@@ -514,7 +513,22 @@ class Settings(object):
         for key in keys:
             self.unset(key)
 
-    def set(self, key, value, loader_identifier=None, tomlfy=False):
+    def _dotted_set(self, dotted_key, value, tomlfy=False, **kwargs):
+
+        data = DynaBox(default_box=True)
+        tree = data
+        split_keys = dotted_key.split('.')
+
+        for k in split_keys[:-1]:
+            tree = tree.setdefault(k, {})
+
+        value = parse_conf_data(value, tomlfy=tomlfy)
+        tree[split_keys[-1]] = value
+
+        self.update(data=data, **kwargs)
+
+    def set(self, key, value, loader_identifier=None, tomlfy=False,
+            dotted_lookup=True):
         """Set a value storing references for the loader
 
         :param key: The key to store
@@ -522,6 +536,14 @@ class Settings(object):
         :param loader_identifier: Optional loader name e.g: toml, yaml etc.
         :param tomlfy: Bool define if value is parsed by toml (defaults False)
         """
+
+        if '.' in key and dotted_lookup is True:
+            return self._dotted_set(
+                key,
+                value,
+                loader_identifier=loader_identifier,
+                tomlfy=tomlfy)
+
         value = parse_conf_data(value, tomlfy=tomlfy)
         key = key.strip().upper()
 
diff --git a/dynaconf/cli.py b/dynaconf/cli.py
index 163743d..2202be0 100644
--- a/dynaconf/cli.py
+++ b/dynaconf/cli.py
@@ -9,44 +9,75 @@ import webbrowser
 from pathlib import Path
 from dynaconf import default_settings
 from dynaconf import constants
-from dynaconf.validator import Validator
+from dynaconf import LazySettings
+from dynaconf.validator import Validator, ValidationError
 from dynaconf.utils.parse_conf import parse_conf_data
 from dotenv import cli as dotenv_cli
 from contextlib import suppress
 
+CWD = Path.cwd()
+ENVS = ['default', 'development', 'staging', 'testing', 'production', 'global']
+EXTS = ['ini', 'toml', 'yaml', 'json', 'py', 'env']
+WRITERS = ['ini', 'toml', 'yaml', 'json', 'py', 'redis', 'vault', 'env']
 
-flask_app = None
-django_app = None
+ENC = default_settings.ENCODING_FOR_DYNACONF
 
-if 'FLASK_APP' in os.environ:  # pragma: no cover
-    with suppress(ImportError, click.UsageError):
-        from flask.cli import ScriptInfo
-        flask_app = ScriptInfo().load_app()
-        settings = flask_app.config
-        click.echo(click.style('Flask app detected', fg='white', bg='black'))
 
+def set_settings(instance=None):
+    """Pick correct settings instance and set it to a global variable."""
 
-if 'DJANGO_SETTINGS_MODULE' in os.environ:  # pragma: no cover
-    sys.path.insert(0, os.path.abspath('.'))
-    with suppress(Exception):
-        import dynaconf.contrib.django_dynaconf  # noqa
-        from django.conf import settings as django_settings
-        django_settings.configure()
-        settings = django_settings
-        django_app = True
-        click.echo(click.style('Django app detected', fg='white', bg='black'))
+    global settings
 
+    settings = None
 
-if not django_app and not flask_app:
-    from dynaconf import settings
+    if instance:
+        settings = import_settings(instance)
 
+    elif 'INSTANCE_FOR_DYNACONF' in os.environ:
+        settings = import_settings(os.environ['INSTANCE_FOR_DYNACONF'])
 
-CWD = Path.cwd()
-ENVS = ['default', 'development', 'staging', 'testing', 'production', 'global']
-EXTS = ['ini', 'toml', 'yaml', 'json', 'py', 'env']
-WRITERS = ['ini', 'toml', 'yaml', 'json', 'py', 'redis', 'vault', 'env']
+    elif 'FLASK_APP' in os.environ:  # pragma: no cover
+        with suppress(ImportError, click.UsageError):
+            from flask.cli import ScriptInfo
+            flask_app = ScriptInfo().load_app()
+            settings = flask_app.config
+            click.echo(click.style(
+                'Flask app detected', fg='white', bg='black'))
+
+    elif 'DJANGO_SETTINGS_MODULE' in os.environ:  # pragma: no cover
+        sys.path.insert(0, os.path.abspath('.'))
+        with suppress(Exception):
+            import dynaconf.contrib.django_dynaconf  # noqa
+            from django.conf import settings as django_settings
+            django_settings.configure()
+            settings = django_settings
+            click.echo(click.style(
+                'Django app detected', fg='white', bg='black'))
+
+    if settings is None:
+        settings = LazySettings()
 
-ENC = default_settings.ENCODING_FOR_DYNACONF
+
+def import_settings(dotted_path):
+    """Import settings instance from python dotted path.
+
+    Last item in dotted path must be settings instace.
+
+    Example: import_settings('path.to.settings')
+    """
+    if '.' in dotted_path:
+        module, name = dotted_path.rsplit('.', 1)
+    else:
+        raise click.UsageError(
+            "invalid path to settings instance: {}".format(dotted_path))
+    try:
+        module = importlib.import_module(module)
+    except ImportError as e:
+        raise click.UsageError(e)
+    try:
+        return getattr(module, name)
+    except AttributeError as e:
+        raise click.UsageError(e)
 
 
 def split_vars(_vars):
@@ -86,6 +117,7 @@ def show_banner(ctx, param, value):
     """Shows dynaconf awesome banner"""
     if not value or ctx.resilient_parsing:
         return
+    set_settings()
     click.echo(settings.dynaconf_banner)
     click.echo('Learn more at: http://github.com/rochacbruno/dynaconf')
     ctx.exit()
@@ -98,10 +130,13 @@ def show_banner(ctx, param, value):
               is_eager=True, help="Open documentation in browser")
 @click.option('--banner', is_flag=True, callback=show_banner,
               expose_value=False, is_eager=True, help="Show awesome banner")
-def main():
+@click.option('--instance', '-i', default=None,
+              help="Custom instance of LazySettings")
+def main(instance):
     """Dynaconf - Command Line Interface\n
     Documentation: http://dynaconf.readthedocs.io/
     """
+    set_settings(instance)
 
 
 @main.command()
@@ -401,10 +436,12 @@ def validate(path):  # pragma: no cover
     # for each section register the validator for specific env
     # call validate
 
+    path = Path(path)
+
     if not str(path).endswith('.toml'):
         path = path / "dynaconf_validators.toml"
 
-    if not Path(path).exists():  # pragma: no cover  # noqa
+    if not path.exists():  # pragma: no cover  # noqa
         click.echo(click.style(
             "{} not found".format(path), fg="white", bg="red"
         ))
@@ -412,6 +449,7 @@ def validate(path):  # pragma: no cover
 
     validation_data = toml.load(open(str(path)))
 
+    success = True
     for env, name_data in validation_data.items():
         for name, data in name_data.items():
             if not isinstance(data, dict):  # pragma: no cover
@@ -419,15 +457,21 @@ def validate(path):  # pragma: no cover
                     "Invalid rule for parameter '{}'".format(name),
                     fg="white", bg="yellow"
                 ))
-            else:  # pragma: no cover
+            else:
                 data.setdefault('env', env)
                 click.echo(click.style(
                     "Validating '{}' with '{}'".format(name, data),
                     fg="white", bg="blue"
                 ))
-                Validator(name, **data).validate(settings)
-
-    # pragma: no cover
-    click.echo(click.style(
-        "Validation success!", fg="white", bg="green"
-    ))
+                try:
+                    Validator(name, **data).validate(settings)
+                except ValidationError as e:
+                    click.echo(click.style(
+                        "Error: {}".format(e), fg="white", bg="red"
+                    ))
+                    success = False
+
+    if success:
+        click.echo(click.style(
+            "Validation success!", fg="white", bg="green"
+        ))
diff --git a/dynaconf/default_settings.py b/dynaconf/default_settings.py
index d62ced6..0c2bc9c 100644
--- a/dynaconf/default_settings.py
+++ b/dynaconf/default_settings.py
@@ -148,3 +148,8 @@ INI = get('INI', None)
 DOTENV_PATH_FOR_DYNACONF = get('DOTENV_PATH_FOR_DYNACONF', None)
 DOTENV_VERBOSE_FOR_DYNACONF = get('DOTENV_VERBOSE_FOR_DYNACONF', False)
 DOTENV_OVERRIDE_FOR_DYNACONF = get('DOTENV_OVERRIDE_FOR_DYNACONF', False)
+
+# Currently this is only used by cli. INSTANCE_FOR_DYNACONF specifies python
+# dotted path to custom LazySettings instance. Last dotted path item should be
+# instance of LazySettings.
+INSTANCE_FOR_DYNACONF = get('INSTANCE_FOR_DYNACONF', None)
diff --git a/example/full_example.py b/example/full_example.py
index b601350..4578f22 100644
--- a/example/full_example.py
+++ b/example/full_example.py
@@ -58,3 +58,6 @@ print("\nDotted path lookup value :")
 print(settings('ADICT.KEY'))
 
 print(settings.WORKS)
+
+print("\nDotted path set value :")
+print(settings.set('ONE.TWO', 'value'))


Test patch:
diff --git a/tests/test_base.py b/tests/test_base.py
index a0ede2f..2453394 100644
--- a/tests/test_base.py
+++ b/tests/test_base.py
@@ -201,7 +201,8 @@ def test_dotted_traversal_access(settings):
             'PASSWORD': 'secret',
             'SSL': {'CONTEXT': 'SECURE'},
             'DOTTED.KEY': True
-        }
+        },
+        dotted_lookup=False
     )
     assert settings.get('PARAMS') == {
         'PASSWORD': 'secret',
@@ -237,7 +238,7 @@ def test_dotted_traversal_access(settings):
     assert settings('PARAMS.DOTTED.KEY') is None
     assert settings('PARAMS').get('DOTTED.KEY') is True
 
-    settings.set('DOTTED.KEY', True)
+    settings.set('DOTTED.KEY', True, dotted_lookup=False)
     assert settings('DOTTED.KEY', dotted_lookup=False) is True
 
     settings.set(
@@ -256,3 +257,47 @@ def test_dotted_traversal_access(settings):
     assert settings('NESTED_1.nested_2.nested_3.nested_4') is True
     # First key is always transformed to upper()
     assert settings('nested_1.nested_2.nested_3.nested_4') is True
+
+
+def test_dotted_set(settings):
+
+    settings.set('nested_1.nested_2.nested_3.nested_4', 'secret')
+
+    assert settings.NESTED_1.NESTED_2.NESTED_3.NESTED_4 == 'secret'
+    assert settings.NESTED_1.NESTED_2.NESTED_3.to_dict() == {'nested_4': 'secret'}
+    assert settings.NESTED_1.NESTED_2.to_dict() == {
+        'nested_3': {
+            'nested_4': 'secret'
+        }
+    }
+
+    assert settings.get('nested_1').to_dict() == {
+        'nested_2': {
+            'nested_3': {
+                'nested_4': 'secret'
+            }
+        }
+    }
+
+    with pytest.raises(KeyError):
+        settings.NESTED_1.NESTED_5
+
+
+def test_dotted_set_with_merge(settings):
+    settings.set("MERGE_ENABLED_FOR_DYNACONF", True)
+
+    settings.set("MERGE.KEY", {
+        "items": [{"name": "item 1"}, {"name": "item 2"}]
+    })
+    settings.set("MERGE.KEY", {
+        "items": [{"name": "item 3"}, {"name": "item 4"}]
+    })
+
+    assert settings.MERGE.KEY == {
+        'items': [
+            {'name': 'item 1'},
+            {'name': 'item 2'},
+            {'name': 'item 3'},
+            {'name': 'item 4'},
+        ]
+    }
diff --git a/tests/test_cli.py b/tests/test_cli.py
index ff64286..d1ec6f1 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -3,7 +3,7 @@ import os
 import pytest
 from pathlib import Path
 from click.testing import CliRunner
-from dynaconf import default_settings
+from dynaconf import LazySettings, default_settings
 from dynaconf.cli import main, EXTS, WRITERS, ENVS, read_file_in_root_directory
 from dotenv import cli as dotenv_cli
 
@@ -11,8 +11,8 @@ from dotenv import cli as dotenv_cli
 runner = CliRunner()
 
 
-def run(cmd):
-    result = runner.invoke(main, cmd)
+def run(cmd, env=None):
+    result = runner.invoke(main, cmd, env=env, catch_exceptions=False)
     # assert result.exit_code == 0
     return result.output
 
@@ -87,9 +87,9 @@ def test_init_with_path(fileformat, tmpdir):
             [
                 'init',
                 '--format={}'.format(fileformat),
-                '--path={}'.format(str(path)),
+                '--path={}'.format(str(tmpdir)),
                 '-y'
-            ]
+            ],
         )
 
     sets = Path(str(path))
@@ -147,6 +147,61 @@ def test_list_with_env(env):
     assert "Working in {} environment".format(env) in result
 
 
+settings = LazySettings(OPTION_FOR_TESTS=True)
+
+
+def test_list_with_instance():
+    result = run(
+        [
+            '-i', 'tests.test_cli.settings',
+            'list',
+        ]
+    )
+    assert "OPTION_FOR_TESTS: True" in result
+
+
+def test_list_with_instance_from_env():
+    result = run(
+        [
+            'list',
+        ],
+        {
+            'INSTANCE_FOR_DYNACONF': 'tests.test_cli.settings',
+        }
+    )
+    assert "OPTION_FOR_TESTS: True" in result
+
+
+def test_instance_attribute_error():
+    result = run(
+        [
+            '-i', 'tests.test_cli.idontexist',
+            'list',
+        ]
+    )
+    assert "has no attribute 'idontexist'" in result
+
+
+def test_instance_import_error():
+    result = run(
+        [
+            '-i', 'idontexist.settings',
+            'list',
+        ]
+    )
+    assert "Error: No module named 'idontexist'" in result
+
+
+def test_instance_pypath_error():
+    result = run(
+        [
+            '-i', 'idontexist',
+            'list',
+        ]
+    )
+    assert "Error: invalid path to settings instance: idontexist" in result
+
+
 def test_list_with_key():
     result = run(
         [
@@ -292,21 +347,31 @@ def test_validate(tmpdir):
     toml_invalid = tmpdir.mkdir('invalid').join('settings.toml')
     toml_invalid.write(TOML_INVALID)
 
-    os.chdir(str(Path(str(toml_valid)).parent))
     result = run(
         [
             'validate',
             '-p',
             str(validation_file)
-        ]
+        ],
+        {
+            'SETTINGS_MODULE_FOR_DYNACONF': str(toml_valid),
+        }
     )
+    assert "Validation success!" in result
 
-    os.chdir(str(Path(str(toml_invalid)).parent))
     result = run(
         [
             'validate',
             '-p',
             str(Path(str(validation_file)).parent)
-        ]
+        ],
+        {
+            'SETTINGS_MODULE_FOR_DYNACONF': str(toml_invalid),
+        }
     )
-    print(result)
+    assert "age must lte 30 but it is 35 in env default" in result
+    assert "project must eq hello_world but it is This is not hello_world " \
+           "in env production" in result
+    assert "host must is_not_in ['test.com'] but it is test.com in env " \
+           "production" in result
+    assert "Validation success!" not in result