Repository: rochacbruno/dynaconf (python)

Issue:
RFC: Template Substitutions - dynaconf-render-template plugin
Require: #146 

## Allow template substitutions

There are 2 types of variables that would be useful to have access within dynaconf settings

- Environment Variables `${HOME}`, `${PWD}`
- Dynaconf variables `settings.get('value')`, `settings.ROOT_PATH_FOR_DYNACONF`

When setting a new variable it would be nice to have access to that vars like using special marker `$D{}` or `@D{}` (to be defined).

To avoid overhead, this feature must be enabled via plugin.

```python
pip install dynaconf-template-render
```

```yaml
default:
  databases:
    default:
      name: foo_database
      driver: sqlite3
      file: $D{ROOT_PATH_FOR_DYNACONF}/foo.db
   conf_location: $D{HOME}/.config/foo/  
```

Then after executing the loaders (the bottom of `.execute_loaders` method) there will be a `plugin_hook` which will be called having `obj` on context, then plugin can make changes to `obj` (in place)

Pull request description:
Fix #145 allow string formatting using envvars
## Template substitutions 

Dynaconf has 2 tokens to enable string substitutions `@format` and `@jinja`.

### @format token

Dynaconf allows template substitutions for strings values, by using the `@format` token prefix and including placeholders accepted by Python's `str.format` method Dynaconf will call
it lazily upon access time.

The call will be like:

```py
"<YOURVALUE>".format(env=os.environ, this=dynaconf.settings)
```

So in your string you can refer to environment variables via `env` object, and also to variables defined int the settings object itself via `this` reference. It is lazily evaluated on access it will use the final value for a settings regardless the order of load.

Example:

```bash
export PROGRAM_NAME=calculator
```

settings.toml

```toml
[default]
DB_NAME = "mydb.db"

[development]
DB_PATH = "@format {env[HOME]}/{this.current_env}/{env[PROGRAM_NAME]}/{this.DB_NAME}"
```

- `{env[HOME]}` is the same as `os.environ["HOME"]` or `$HOME` in the shell.
- `{this.current_env}` is the same as `settings.current_env`
- `{env[PROGRAM_NAME]}` is the same as `os.environ["PROGRAM_NAME"]` or `$PROGRAM_NAME` in the shell.
- `{this.DB_NAME}` is the same as `settins.DB_NAME` or `settings["DB_NAME"]`

so in your `program`

```py
from dynaconf import settings

settings.DB_PATH == '~/development/calculator/mydb.db'
```

### @jinja token

If `jinja2` package is installed then dynaconf will also allow the use jinja to render string values.

Example:

```bash
export PROGRAM_NAME=calculator
```

settings.toml

```toml
[default]
DB_NAME = "mydb.db"

[development]
DB_PATH = "@jinja {{env.HOME}}/{{this.current_env | lower}}/{{env['PROGRAM_NAME']}}/{{this.DB_NAME}}"
```

so in your `program`

```py
from dynaconf import settings

settings.DB_PATH == '~/development/calculator/mydb.db'
```

The main difference is that Jinja allows some Python expressions to be avaluated such as `{% for, if, while %}` and also supports calling methods and has lots of filters like `| lower`.

Patch:
diff --git a/Makefile b/Makefile
index 62c0e41..2880947 100644
--- a/Makefile
+++ b/Makefile
@@ -14,6 +14,7 @@ test_examples:
 	cd example/;pwd;python compat.py
 	cd example/app;pwd;python app.py
 	cd example/dunder;pwd;python app.py
+	cd example/format;pwd;python app.py
 	cd example/app_with_dotenv;pwd;python app.py
 	cd example/merge_enabled;pwd;python app.py
 	cd example/new_merge;pwd;python app.py
diff --git a/docs/guides/environment_variables.md b/docs/guides/environment_variables.md
index 5c92425..46675b7 100644
--- a/docs/guides/environment_variables.md
+++ b/docs/guides/environment_variables.md
@@ -54,6 +54,14 @@ DYNACONF_NONE='@none None'
 
 # toml syntax does not allow mixed type arrays so use @json
 DYNACONF_ARRAY='@json [42, 3.14, "hello", true, ["otherarray"], {"foo": "bar"}]'
+
+# Lazily formatted string can access env vars and settings variables.
+
+# using str.format
+DYNACONF_DATABASE_PATH="@format {env[HOME]}/.config/databases/{this.DB_NAME}"
+
+# using jinja2
+DYNACONF_DATABASE_PATH="@jinja {{env.HOME}}/.config/databases/{{this.DB_NAME}}"
 ```
 
 > **NOTE**: Older versions of Dynaconf used the `@casting` prefixes for env vars like `export DYNACONF_INTEGER='@int 123'` still works but this casting is deprecated in favor of using TOML syntax described above. To disable the `@casting` do `export AUTO_CAST_FOR_DYNACONF=false`
diff --git a/docs/guides/usage.md b/docs/guides/usage.md
index 6cb3890..8cc02dd 100644
--- a/docs/guides/usage.md
+++ b/docs/guides/usage.md
@@ -501,7 +501,85 @@ settings.load_file(path="/path/to/file.toml")  # list or `;/,` separated allowed
 
 > **NOTE**: programmatically loaded file is not persisted, once `env` is changed via `setenv|ugin_env`, or a `reload` or `configure` is invoked it will be cleaned, to persist it needs to go to `INCLUDES_FOR_DYNACONF` variable or you need to load it programmatically again.
 
-## Merging existing values
+## Template substitutions
+
+Dynaconf has 2 tokens to enable string substitutions `@format` and `@jinja`.
+
+### @format token
+
+Dynaconf allows template substitutions for strings values, by using the `@format` token prefix and including placeholders accepted by Python's `str.format` method Dynaconf will call
+it lazily upon access time.
+
+The call will be like:
+
+```py
+"<YOURVALUE>".format(env=os.environ, this=dynaconf.settings)
+```
+
+So in your string you can refer to environment variables via `env` object, and also to variables defined int the settings object itself via `this` reference. It is lazily evaluated on access it will use the final value for a settings regardless the order of load.
+
+Example:
+
+```bash
+export PROGRAM_NAME=calculator
+```
+
+settings.toml
+
+```toml
+[default]
+DB_NAME = "mydb.db"
+
+[development]
+DB_PATH = "@format {env[HOME]}/{this.current_env}/{env[PROGRAM_NAME]}/{this.DB_NAME}"
+```
+
+- `{env[HOME]}` is the same as `os.environ["HOME"]` or `$HOME` in the shell.
+- `{this.current_env}` is the same as `settings.current_env`
+- `{env[PROGRAM_NAME]}` is the same as `os.environ["PROGRAM_NAME"]` or `$PROGRAM_NAME` in the shell.
+- `{this.DB_NAME}` is the same as `settins.DB_NAME` or `settings["DB_NAME"]`
+
+so in your `program`
+
+```py
+from dynaconf import settings
+
+settings.DB_PATH == '~/DEVELOPMENT/calculator/mydb.db'
+```
+
+### @jinja token
+
+If `jinja2` package is installed then dynaconf will also allow the use jinja to render string values.
+
+Example:
+
+```bash
+export PROGRAM_NAME=calculator
+```
+
+settings.toml
+
+```toml
+[default]
+DB_NAME = "mydb.db"
+
+[development]
+DB_PATH = "@jinja {{env.HOME}}/{{this.current_env | lower}}/{{env['PROGRAM_NAME']}}/{{this.DB_NAME}}"
+```
+
+so in your `program`
+
+```py
+from dynaconf import settings
+
+settings.DB_PATH == '~/development/calculator/mydb.db'
+```
+
+The main difference is that Jinja allows some Python expressions to be avaluated such as `{% for, if, while %}` and also supports calling methods and has lots of filters like `| lower`.
+
+Jinja supports its built-in filters listed in [Builtin Filters Page](http://jinja.palletsprojects.com/en/master/templates/#builtin-filters) and Dynaconf includes aditional filters for `os.path` module: `abspath`. `realpath`, `relpath`, `basename` and `dirname` and usage is like: `VALUE = "@jinja {{this.FOO | abspath}}"`
+
+## Merging existing data structures
 
 If your settings has existing variables of types `list` ot `dict` and you want to `merge` instead of `override` then 
 the `dynaconf_merge` and `dynaconf_merge_unique` stanzas can mark that variable as a candidate for merging.
diff --git a/dynaconf/base.py b/dynaconf/base.py
index db7fef5..b6a96d1 100644
--- a/dynaconf/base.py
+++ b/dynaconf/base.py
@@ -25,6 +25,7 @@ from dynaconf.utils.files import find_file
 from dynaconf.utils.functional import empty
 from dynaconf.utils.functional import LazyObject
 from dynaconf.utils.parse_conf import converters
+from dynaconf.utils.parse_conf import evaluate_lazy_format
 from dynaconf.utils.parse_conf import parse_conf_data
 from dynaconf.utils.parse_conf import true_values
 from dynaconf.validator import ValidatorList
@@ -90,6 +91,7 @@ class LazySettings(LazyObject):
         self._kwargs = kwargs
         super(LazySettings, self).__init__()
 
+    @evaluate_lazy_format
     def __getattr__(self, name):
         """Allow getting keys from self.store using dot notation"""
         if self._wrapped is empty:
@@ -276,6 +278,7 @@ class Settings(object):
             ".".join(keys), default=default, parent=result, **kwargs
         )
 
+    @evaluate_lazy_format
     def get(
         self,
         key,
diff --git a/dynaconf/utils/parse_conf.py b/dynaconf/utils/parse_conf.py
index d9fc4c5..d33673f 100644
--- a/dynaconf/utils/parse_conf.py
+++ b/dynaconf/utils/parse_conf.py
@@ -1,11 +1,21 @@
 import json
 import os
+from functools import wraps
 
 import toml
 from box import BoxKeyError
 
 from dynaconf.utils.boxing import DynaBox
 
+try:
+    from jinja2 import Environment
+
+    jinja_env = Environment()
+    for p_method in ("abspath", "realpath", "relpath", "dirname", "basename"):
+        jinja_env.filters[p_method] = getattr(os.path, p_method)
+except ImportError:  # pragma: no cover
+    jinja_env = None
+
 true_values = ("t", "true", "enabled", "1", "on", "yes", "True")
 false_values = ("f", "false", "disabled", "0", "off", "no", "False", "")
 
@@ -42,13 +52,55 @@ class Merge(MetaValue):
     dynaconf_merge = True
 
 
+class LazyFormat:
+    """Holds data to format lazily."""
+
+    dynaconf_lazy_format = True
+
+    def __init__(self, value, formatter=str.format):
+        self.value = value
+        self.formatter = formatter
+
+    @property
+    def context(self):
+        """Builds a context for formatting."""
+        return {"env": os.environ, "this": self.settings}
+
+    def __call__(self, settings):
+        """LazyValue triggers format lazily."""
+        self.settings = settings
+        return self.formatter(self.value, **self.context)
+
+
+def evaluate_lazy_format(f):
+    """Marks a method on Settings instance to
+    lazily evaluate LazyFormat objects upon access."""
+
+    @wraps(f)
+    def evaluate(settings, *args, **kwargs):
+        value = f(settings, *args, **kwargs)
+        if getattr(value, "dynaconf_lazy_format", None):
+            return value(settings)
+        return value
+
+    return evaluate
+
+
+def jinja_formatter(value, **context):
+    if jinja_env is None:  # pragma: no cover
+        raise ImportError(
+            "jinja2 must be installed to enable '@jinja' settings in dynaconf"
+        )
+    return jinja_env.from_string(value).render(**context)
+
+
 converters = {
     "@int": int,
     "@float": float,
-    "@bool": (
-        lambda value: True if str(value).lower() in true_values else False
-    ),
+    "@bool": lambda value: str(value).lower() in true_values,
     "@json": json.loads,
+    "@format": lambda value: LazyFormat(value),
+    "@jinja": lambda value: LazyFormat(value, formatter=jinja_formatter),
     # Meta Values to trigger pre assignment actions
     "@reset": lambda value: Reset(value),
     "@del": lambda value: Del(value),
diff --git a/example/format/.env b/example/format/.env
new file mode 100644
index 0000000..bf7bb51
--- /dev/null
+++ b/example/format/.env
@@ -0,0 +1,1 @@
+USERNAME='robert_plant'
diff --git a/example/format/app.py b/example/format/app.py
new file mode 100644
index 0000000..234dd3d
--- /dev/null
+++ b/example/format/app.py
@@ -0,0 +1,18 @@
+import os
+
+from dynaconf import settings
+
+assert settings.get("USERNAME") is None
+assert settings.DATABASE_NAME == "my_database.db"
+assert os.environ["USERNAME"] == "robert_plant"
+
+DB_PATH = "/home/robert_plant/databases/my_database.db"
+assert settings.DATABASE_PATH == DB_PATH, settings.DATABASE_PATH
+print(settings.DATABASE_PATH)
+
+
+DB_PATH_JINJA = "/home/robert_plant/development/my_database.db"
+assert (
+    settings.DATABASE_PATH_JINJA == DB_PATH_JINJA
+), settings.DATABASE_PATH_JINJA
+print(settings.DATABASE_PATH_JINJA)
diff --git a/example/format/settings.toml b/example/format/settings.toml
new file mode 100644
index 0000000..4b90106
--- /dev/null
+++ b/example/format/settings.toml
@@ -0,0 +1,6 @@
+[default]
+database_name = 'my_database.db'
+
+[development]
+database_path = '@format /home/{env[USERNAME]}/databases/{this[database_name]}'
+database_path_jinja = '@jinja /home/{{env.USERNAME}}/{{this.current_env | lower}}/{{this["database_name"] }}'


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 1abc3cd..a5e28a5 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -12,6 +12,9 @@ from dynaconf.utils import trimmed_split
 from dynaconf.utils import upperfy
 from dynaconf.utils.files import find_file
 from dynaconf.utils.files import get_local_filename
+from dynaconf.utils.parse_conf import evaluate_lazy_format
+from dynaconf.utils.parse_conf import jinja_formatter
+from dynaconf.utils.parse_conf import LazyFormat
 from dynaconf.utils.parse_conf import parse_conf_data
 from dynaconf.utils.parse_conf import unparse_conf_data
 
@@ -242,3 +245,59 @@ def test_upperfy():
     )
     assert upperfy("foo_bar") == "FOO_BAR"
     assert upperfy("foo_BAR") == "FOO_BAR"
+
+
+def test_lazy_format_class():
+    value = LazyFormat("{this[FOO]}/bar")
+    settings = {"FOO": "foo"}
+    assert value(settings) == "foo/bar"
+
+
+def test_evaluate_lazy_format_decorator():
+    class Settings:
+        FOO = "foo"
+
+        @evaluate_lazy_format
+        def get(self):
+            return parse_conf_data("@format {this.FOO}/bar")
+
+    settings = Settings()
+    assert settings.get() == "foo/bar"
+
+
+def test_lazy_format_on_settings(settings):
+    os.environ["ENV_THING"] = "LazyFormat"
+    settings.set("set_1", "really")
+    settings.set("lazy", "@format {env[ENV_THING]}/{this[set_1]}/{this.SET_2}")
+    settings.set("set_2", "works")
+
+    assert settings.LAZY == settings.get("lazy") == "LazyFormat/really/works"
+
+
+def test_lazy_format_class_jinja():
+    value = LazyFormat("{{this['FOO']}}/bar", formatter=jinja_formatter)
+    settings = {"FOO": "foo"}
+    assert value(settings) == "foo/bar"
+
+
+def test_evaluate_lazy_format_decorator_jinja():
+    class Settings:
+        FOO = "foo"
+
+        @evaluate_lazy_format
+        def get(self):
+            return parse_conf_data("@jinja {{this.FOO}}/bar")
+
+    settings = Settings()
+    assert settings.get() == "foo/bar"
+
+
+def test_lazy_format_on_settings_jinja(settings):
+    os.environ["ENV_THING"] = "LazyFormat"
+    settings.set("set_1", "really")
+    settings.set(
+        "lazy", "@jinja {{env.ENV_THING}}/{{this['set_1']}}/{{this.SET_2}}"
+    )
+    settings.set("set_2", "works")
+
+    assert settings.LAZY == settings.get("lazy") == "LazyFormat/really/works"