Repository: dynaconf/dynaconf
Pull request #1106 (2024-06-18 19:19:01)

feat: dynaconf list with `--json` will print valid JSON to stdout
Problem: `dynaconf list` is formatted and cannot be processed by JSON tools in terminal.

stdout can't be piped to a file `> settings.json` or processed by tools such as `jq` 

Current output on `$ cd dynaconf/tests_functional/full_example`

```console
$ dynaconf -i app.settings list                        
Working in development environment 
LOAD_DOTENV<bool> True
DEFAULT_SETTINGS_PATHS<list> ['settings.py',
 'settings.toml',
 'settings.tml',
 'settings.yaml',
 'settings.yml',
 'settings.ini',
 'settings.conf',
 'settings.properties',
 'settings.json',
 '.secrets.py',
 '.secrets.toml',
 '.secrets.tml',
 '.secrets.yaml',
 '.secrets.yml',
 '.secrets.ini',
 '.secrets.conf',
 '.secrets.properties',
 '.secrets.json']
WORKS<str> 'full_example'
HOSTNAME<str> 'host.com'
PORT<int> 5000
VALUE<float> 42.1
ALIST<list> ['item1', 'item2', 'item3']
ADICT<dict> {'key': 'value'}
DEBUG<bool> True
ONE<dict> {'TWO': 'value'}

```

## Whith this new feature

JSON output

```console
$ dynaconf -i app.settings list --json
{"ADICT": {"key": "value"}, "ALIST": ["item1", "item2", "item3"], "DEBUG": true, "DEFAULT_SETTINGS_PATHS": ["settings.py", "settings.toml", "settings.tml", "settings.yaml", "settings.yml", "settings.ini", "settings.conf", "settings.properties", "settings.json", ".secrets.py", ".secrets.toml", ".secrets.tml", ".secrets.yaml", ".secrets.yml", ".secrets.ini", ".secrets.conf", ".secrets.properties", ".secrets.json"], "HOSTNAME": "host.com", "LOAD_DOTENV": true, "ONE": {"TWO": "value"}, "PORT": 5000, "VALUE": 42.1, "WORKS": "full_example"}%      
```

Processing output

```console 
$ dynaconf -i app.settings list --json | jq
{
  "ADICT": {
    "key": "value"
  },
  "ALIST": [
    "item1",
    "item2",
    "item3"
  ],
  "DEBUG": true,
  "DEFAULT_SETTINGS_PATHS": [
    "settings.py",
    "settings.toml",
    "settings.tml",
    "settings.yaml",
    "settings.yml",
    "settings.ini",
    "settings.conf",
    "settings.properties",
    "settings.json",
    ".secrets.py",
    ".secrets.toml",
    ".secrets.tml",
    ".secrets.yaml",
    ".secrets.yml",
    ".secrets.ini",
    ".secrets.conf",
    ".secrets.properties",
    ".secrets.json"
  ],
  "HOSTNAME": "host.com",
  "LOAD_DOTENV": true,
  "ONE": {
    "TWO": "value"
  },
  "PORT": 5000,
  "VALUE": 42.1,
  "WORKS": "full_example"
}

```

Alternative to `get` command

```console
$ dynaconf -i app.settings list --json | jq ".HOSTNAME"
"host.com"

```

github-actions[bot]: ## Test Results (Python 3.10)
486 tests  +8   486 [:heavy_check_mark:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "passed tests") +8   36s [:stopwatch:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "duration of all tests") ±0s
    1 suites ±0       0 [:zzz:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "skipped / disabled tests") ±0 
    1 files   ±0       0 [:x:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "failed tests") ±0 

Results for commit 9ba3049b. ± Comparison against base commit 096cab13.

[test-results]:data:application/gzip;base64,H4sIAJrecWYC/13Myw6CMBCF4VchXbuoMpbiy5jOlCYTgZpeVsZ3t6Ap4PL7T3JewvE4RHFrzqdGxMypwuZgEvu5sFXFZUnLBrrqHjPRf3rwsyRZgzM8HsIQgg+/EvK8fS44Xn7L9rh6d7h6/0d+mjgViB5NK6FHqWxnUaE25FpEdAAkEVSHdAV50eL9ATXRFqsEAQAA

github-actions[bot]: ## Test Results (Python 3.8)
486 tests  +8   486 [:heavy_check_mark:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "passed tests") +8   36s [:stopwatch:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "duration of all tests") -1s
    1 suites ±0       0 [:zzz:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "skipped / disabled tests") ±0 
    1 files   ±0       0 [:x:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "failed tests") ±0 

Results for commit 9ba3049b. ± Comparison against base commit 096cab13.

[test-results]:data:application/gzip;base64,H4sIAJ7ecWYC/13Myw6CMBCF4VchXbuoMpbiy5jOlCYTgZpeVsZ3t6Ap4PL7T3JewvE4RHFrzqdGxMypwuZgEvu5sFXFZUnLBrrqHjPRf3rwsyRZgzM8HsIQgg+/EvK8fS44Xn7L9rh6d7h6/0d+mjgViB5NK6FHqWxnUaE25FpEdAAkEVSHdAV50eL9ATXRFqsEAQAA

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1106?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf) Report
Attention: Patch coverage is `97.05882%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 98.60%. Comparing base [(`096cab1`)](https://app.codecov.io/gh/dynaconf/dynaconf/commit/096cab13ca9630f832a8dc80d5615bc162b669d8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf) to head [(`9ba3049`)](https://app.codecov.io/gh/dynaconf/dynaconf/commit/9ba3049b06d7db6b8acf3bbbf44c0b467bc54028?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf).

| [Files](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1106?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf) | Patch % | Lines |
|---|---|---|
| [dynaconf/cli.py](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1106?src=pr&el=tree&filepath=dynaconf%2Fcli.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf#diff-ZHluYWNvbmYvY2xpLnB5) | 94.44% | [1 Missing :warning: ](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1106?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master    #1106      +/-   ##
==========================================
- Coverage   98.63%   98.60%   -0.03%     
==========================================
  Files          23       23              
  Lines        2271     2297      +26     
==========================================
+ Hits         2240     2265      +25     
- Misses         31       32       +1     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/dynaconf/dynaconf/pull/1106?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dynaconf).

github-actions[bot]: ## Test Results (Python 3.9)
486 tests  +8   484 [:heavy_check_mark:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "passed tests") +6   40s [:stopwatch:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "duration of all tests") +5s
    1 suites ±0       0 [:zzz:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "skipped / disabled tests") ±0 
    1 files   ±0       2 [:x:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "failed tests") +2 

For more details on these failures, see [this check](https://github.com/dynaconf/dynaconf/runs/26388537819).

Results for commit 867a8232. ± Comparison against base commit 16537593.

[test-results]:data:application/gzip;base64,H4sIAPjncWYC/1WMzQ7CIBAGX6Xh7KHFFTa+TEMpJMS2GH5Oje/uFhXqcebbnZ1Zt5jI7t1w6VjMLlWYc1DJ+Y0QemJa0rEBih+NMWtdFDT1cE9S9WO0yi0keBUmBB++JyFvrXnAf/JjWrHwKVj43NN+XV0iYCikQn7lAiROaEBPYPWNT4NSM0IvNXnklrPXG4c5QDcEAQAA

github-actions[bot]: ## Test Results (Python 3.12)
486 tests  +8   486 [:heavy_check_mark:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "passed tests") +8   48s [:stopwatch:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "duration of all tests") -1s
    1 suites ±0       0 [:zzz:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "skipped / disabled tests") ±0 
    1 files   ±0       0 [:x:](https://github.com/EnricoMi/publish-unit-test-result-action/blob/v1.20/README.md#the-symbols "failed tests") ±0 

Results for commit 867a8232. ± Comparison against base commit 16537593.

[test-results]:data:application/gzip;base64,H4sIAAXocWYC/13Myw7CIBCF4VdpWLtoEenElzEwhWRiWwyXVeO7O6L24vL7T3IW4Wl0SVyb7tSIVCivGEo0mcLMVMDmJacK/dMtFcT/dKcHp3YN3tB4CC7GEL8llnn7fON4+SnbY/XusHr/h2GaKDME6N6APEuterDgFFrl8SJtZ8wAqu2RO0gvxfMF7V+6AgQBAAA=


Patch:
diff --git a/docs/cli.md b/docs/cli.md
index 8cdcefe..a17c5bf 100644
--- a/docs/cli.md
+++ b/docs/cli.md
@@ -212,10 +212,21 @@ Options:
   -l, --loader TEXT  a loader identifier to filter e.g: toml|yaml
   -a, --all          show dynaconf internal settings?
   -o, --output FILE  Filepath to write the listed values as json
+  -j, --json         Prints out a valid JSON string
   --output-flat      Output file is flat (do not include [env] name)
   --help             Show this message and exit.
 ```
 
+#### Printing out the settings as JSON
+
+```console
+$ dynaconf list --json
+{"a_valid_raw": "json_output"}
+```
+
+The above output is printed to stdout and can be piped to a file or processed
+using tools such as `jq`.
+
 #### Exporting current environment as a file
 
 ```bash
diff --git a/dynaconf/cli.py b/dynaconf/cli.py
index 108464d..c8b7529 100644
--- a/dynaconf/cli.py
+++ b/dynaconf/cli.py
@@ -7,6 +7,7 @@ import pprint
 import sys
 import warnings
 import webbrowser
+from contextlib import redirect_stdout
 from contextlib import suppress
 from pathlib import Path
 from typing import TYPE_CHECKING
@@ -17,6 +18,7 @@ from dynaconf import LazySettings
 from dynaconf import loaders
 from dynaconf import settings as legacy_settings
 from dynaconf.loaders.py_loader import get_module
+from dynaconf.utils import prepare_json
 from dynaconf.utils import upperfy
 from dynaconf.utils.files import read_file
 from dynaconf.utils.functional import empty
@@ -131,7 +133,8 @@ def import_settings(dotted_path):
             f"invalid path to settings instance: {dotted_path}"
         )
     try:
-        module = importlib.import_module(module)
+        with redirect_stdout(None):
+            module = importlib.import_module(module)
     except ImportError as e:
         raise click.UsageError(e)
     except FileNotFoundError:
@@ -485,7 +488,7 @@ def get(key, default, env, unparse):
         result = unparse_conf_data(result)
 
     if isinstance(result, (dict, list, tuple)):
-        result = json.dumps(result, sort_keys=True)
+        result = json.dumps(prepare_json(result), sort_keys=True)
 
     click.echo(result, nl=False)
 
@@ -530,7 +533,24 @@ def get(key, default, env, unparse):
     default=False,
     help="Output file is flat (do not include [env] name)",
 )
-def _list(env, key, more, loader, _all=False, output=None, flat=False):
+@click.option(
+    "--json",
+    "_json",
+    "-j",
+    is_flag=True,
+    default=False,
+    help="Prints out data serialized as JSON",
+)
+def _list(
+    env,
+    key,
+    more,
+    loader,
+    _all=False,
+    output=None,
+    flat=False,
+    _json=False,
+):
     """
     Lists user defined settings or all (including internal configs).
 
@@ -552,14 +572,15 @@ def _list(env, key, more, loader, _all=False, output=None, flat=False):
     if cur_env == "main":
         flat = True
 
-    click.echo(
-        click.style(
-            f"Working in {cur_env} environment ",
-            bold=True,
-            bg="bright_blue",
-            fg="bright_white",
+    if not _json:
+        click.echo(
+            click.style(
+                f"Working in {cur_env} environment ",
+                bold=True,
+                bg="bright_blue",
+                fg="bright_white",
+            )
         )
-    )
 
     if not loader:
         data = settings.as_dict(env=env, internal=_all)
@@ -585,14 +606,18 @@ def _list(env, key, more, loader, _all=False, output=None, flat=False):
         return f"{key}{data_type} {value}"
 
     if not key:
-        datalines = "\n".join(
-            format_setting(k, v)
-            for k, v in data.items()
-            if k not in data.get("RENAMED_VARS", [])
-        )
-        (click.echo_via_pager if more else click.echo)(datalines)
+        if not _json:
+            datalines = "\n".join(
+                format_setting(k, v)
+                for k, v in data.items()
+                if k not in data.get("RENAMED_VARS", [])
+            )
+            (click.echo_via_pager if more else click.echo)(datalines)
         if output:
-            loaders.write(output, data, env=not flat and cur_env)
+            loaders.write(output, prepare_json(data), env=not flat and cur_env)
+        if _json:
+            json_data = json.dumps(prepare_json(data), sort_keys=True)
+            click.echo(json_data, nl=False)
     else:
         key = upperfy(key)
 
@@ -605,9 +630,14 @@ def _list(env, key, more, loader, _all=False, output=None, flat=False):
             click.secho("Key not found", bg="red", fg="white", err=True)
             return
 
-        click.echo(format_setting(key, value))
+        if not _json:
+            click.echo(format_setting(key, value))
         if output:
-            loaders.write(output, {key: value}, env=not flat and cur_env)
+            loaders.write(
+                output, prepare_json({key: value}), env=not flat and cur_env
+            )
+        if _json:
+            click.echo(json.dumps(prepare_json({key: value})), nl=True)
 
     if env:
         settings.setenv()
diff --git a/dynaconf/utils/__init__.py b/dynaconf/utils/__init__.py
index 0456cf4..1a1632e 100644
--- a/dynaconf/utils/__init__.py
+++ b/dynaconf/utils/__init__.py
@@ -5,6 +5,7 @@ import warnings
 from collections import defaultdict
 from collections.abc import Iterator
 from json import JSONDecoder
+from pathlib import Path
 from typing import Any
 from typing import Literal
 from typing import TYPE_CHECKING
@@ -537,3 +538,23 @@ def find_the_correct_casing(key: str, data: dict[str, Any]) -> str | None:
         if k.replace(" ", "_").lower() == key.lower():
             return k
     return None
+
+
+def prepare_json(data: Any) -> Any:
+    """Takes a data dict and transforms unserializable values to str for JSON.
+    {1: PosixPath("/foo")} -> {"1": "/foo"}
+    """
+    unserializable_types = (Path,)
+    if isinstance(data, dict):
+        return_data = {}
+        for key, val in data.items():
+            value = str(val) if isinstance(val, unserializable_types) else val
+            return_data[str(key)] = value
+        return return_data
+    elif isinstance(data, (list, tuple)):
+        return_data = []
+        for val in data:
+            value = str(val) if isinstance(val, unserializable_types) else val
+            return_data.append(value)
+        return return_data
+    return data


Test patch:
diff --git a/tests/config.py b/tests/config.py
index 33d1291..502e079 100644
--- a/tests/config.py
+++ b/tests/config.py
@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from pathlib import Path
+
 from dynaconf import Dynaconf
 
 settingsenv = Dynaconf(environments=True)
@@ -7,4 +9,5 @@ settings = Dynaconf(
     TEST_KEY="test_value",
     ANOTHER_KEY="another_value",
     DATA={"KEY": "value", "OTHERKEY": "other value"},
+    A_PATH=Path("foo"),
 )
diff --git a/tests/test_cli.py b/tests/test_cli.py
index c679ae5..c359547 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -231,6 +231,17 @@ def test_list_export_json(testdir):
     assert json.loads(read_file("sets.json"))["TEST_KEY"] == "test_value"
 
 
+def test_list_prints_json(testdir):
+    """Ensure output of list --json is JSON compatible"""
+    result = run(
+        ["-i", "tests.config.settings", "list", "--json"],
+        env={"ROOT_PATH_FOR_DYNACONF": testdir},
+    )
+    data = json.loads(result)
+    assert data["TEST_KEY"] == "test_value"
+    assert data["A_PATH"] == "foo"
+
+
 def test_list_with_all(testdir):
     """Test list command with --all includes interval vars"""
     result = run(
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 57b8ad6..ca7743a 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -22,6 +22,7 @@ from dynaconf.utils import isnamedtupleinstance
 from dynaconf.utils import Missing
 from dynaconf.utils import missing
 from dynaconf.utils import object_merge
+from dynaconf.utils import prepare_json
 from dynaconf.utils import trimmed_split
 from dynaconf.utils import upperfy
 from dynaconf.utils.files import find_file
@@ -541,3 +542,20 @@ def test_get_converter_error_when_converting(settings):
 
     with pytest.raises(DynaconfFormatError):
         settings.BLA
+
+
+@pytest.mark.parametrize(
+    "input_expected",
+    [
+        ({"path": Path("foo")}, {"path": "foo"}),
+        ({1: Path("foo")}, {"1": "foo"}),
+        ({True: Path("foo")}, {"True": "foo"}),
+        ([Path("one"), Path("two"), 1, True], ["one", "two", 1, True]),
+        (1, 1),
+        (True, True),
+        ("test", "test"),
+    ],
+)
+def test_prepare_json(input_expected):
+    data, expected = input_expected
+    assert prepare_json(data) == expected