Repository: dynaconf/dynaconf

Issue:
[bug] `dynaconf validate ` not working with `dynaconf_validators.toml`
### Discussed in https://github.com/dynaconf/dynaconf/discussions/868

<div type='discussions-op-text'>

<sup>Originally posted by **Holzhauer** February 14, 2023</sup>

> I attempt to validate the type of a variable in `dynaconf_validators.toml`. However, if I code it as
> 
> `startyear = {is_type_of=int}`
> 
> I get a `dynaconf.vendor.tomllib.TOMLDecodeError: Invalid value` for the character after "=". If I set the type in quotes like
> 
> `startyear = {is_type_of="int"}`
> 
> I get a `TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union`.
> What is the correct syntax here?

</div>

---

I'll describe the bug from the initial discussion but I think there is more room to improvement in the implementation of the `dynaconf validate`

### Describe the bug

When trying to cast a value to `int` using the cli tool `dynaconf validate` and the file `dynaconf_validators.toml`, there program raises a `TypeError`. Also, a wrong use of TOML format using unquoted `int` raises an unhandled `tomllib` specific error.

### To Reproduce

1) Download [gist sample](https://gist.github.com/pedro-psb/d26a78d2d3742b1086d935ba8e9bd109), setup a virtualenv, pip install `dyanconf`  and execute `dynaconf -i config.settings validate`. It should raise a TypeError.

Shortcut:

```bash
# get code
git clone https://gist.github.com/d26a78d2d3742b1086d935ba8e9bd109.git issue-879 && cd issue-879

# review and execute
cat run.sh
chmod +x run.sh
./run.sh
```

2) Using unquoted `int` raises some unhandled `tomllib` errors.

### Expected behavior

1) Validate the settings without any problem
2) Handle error to suggest it should probably be quoted.

### Environment

- OS: Ubuntu 22.04
- Python Version: 3.11.2
- Dynaconf Version: 3.1.12

---
### Additional comments

I've found that the cli command `validate` has some other bugs and could be better tested. Eg, if in this code sample the `environments` setting was false, and there were no `[default]` in the `.toml`, the output would be:

```bash
Invalid rule for parameter 'is_type_of'
Validation success!
```

I'd like to propose to write some more unit test for this command and try to fix these inconsistencies. Should I create another issue for that?

[edit: add gist sample code]

[bug] `dynaconf validate ` not working with `dynaconf_validators.toml`
### Discussed in https://github.com/dynaconf/dynaconf/discussions/868

<div type='discussions-op-text'>

<sup>Originally posted by **Holzhauer** February 14, 2023</sup>

> I attempt to validate the type of a variable in `dynaconf_validators.toml`. However, if I code it as
> 
> `startyear = {is_type_of=int}`
> 
> I get a `dynaconf.vendor.tomllib.TOMLDecodeError: Invalid value` for the character after "=". If I set the type in quotes like
> 
> `startyear = {is_type_of="int"}`
> 
> I get a `TypeError: isinstance() arg 2 must be a type, a tuple of types, or a union`.
> What is the correct syntax here?

</div>

---

I'll describe the bug from the initial discussion but I think there is more room to improvement in the implementation of the `dynaconf validate`

### Describe the bug

When trying to cast a value to `int` using the cli tool `dynaconf validate` and the file `dynaconf_validators.toml`, there program raises a `TypeError`. Also, a wrong use of TOML format using unquoted `int` raises an unhandled `tomllib` specific error.

### To Reproduce

1) Download [gist sample](https://gist.github.com/pedro-psb/d26a78d2d3742b1086d935ba8e9bd109), setup a virtualenv, pip install `dyanconf`  and execute `dynaconf -i config.settings validate`. It should raise a TypeError.

Shortcut:

```bash
# get code
git clone https://gist.github.com/d26a78d2d3742b1086d935ba8e9bd109.git issue-879 && cd issue-879

# review and execute
cat run.sh
chmod +x run.sh
./run.sh
```

2) Using unquoted `int` raises some unhandled `tomllib` errors.

### Expected behavior

1) Validate the settings without any problem
2) Handle error to suggest it should probably be quoted.

### Environment

- OS: Ubuntu 22.04
- Python Version: 3.11.2
- Dynaconf Version: 3.1.12

---
### Additional comments

I've found that the cli command `validate` has some other bugs and could be better tested. Eg, if in this code sample the `environments` setting was false, and there were no `[default]` in the `.toml`, the output would be:

```bash
Invalid rule for parameter 'is_type_of'
Validation success!
```

I'd like to propose to write some more unit test for this command and try to fix these inconsistencies. Should I create another issue for that?

[edit: add gist sample code]



Discussion:
The expected way for this to work given the toml parser would be

```toml
startyear = {is_type_of="@int"}
```

I am not sure this is supported, but must be
> The expected way for this to work given the toml parser would be
> 
> ```toml
> startyear = {is_type_of="@int"}
> ```
> 
> I am not sure this is supported, but must be

This still throws the same error
## Problem analysis

I had a look at the code and `cli.py` doesn't use the custom parser which deals with `"@int"` converters in the `dynaconf_validators.toml` file, it directly parses it with tomllib or toml. [source](https://github.com/dynaconf/dynaconf/blob/46d3ab4e256212305fb3103847d56344dc339b46/dynaconf/cli.py#LL726-L735C10)

After parsing, here is what happens in `cli.py`: ([source](https://github.com/dynaconf/dynaconf/blob/46d3ab4e256212305fb3103847d56344dc339b46/dynaconf/cli.py#LL757-L763C36))

```diff
# this is inside a loop through `dynaconf_validators.toml` in-memory structure where:
# name -> "startyear"
# data -> dict = { 'is_type_of' = '@int' }
try:
-    Validator(name, **data).validate(settings)
+    Validator("startyear", is_type_of="@int").validate(settings) 
except ValidationError as e:
    click.echo(
        click.style(f"Error: {e}", fg="white", bg="red")
    )
    success = False
```

Having a look at  `Validator` usage, it should receive an `int` class (a type class) so, for this to work, a converter would need to be implemented:

```python
# data -> dict = { 'is_type_of' = '@int' }
data = convert_to_type_class(data)
# data -> dict = { 'is_type_of' = int }
Validator(name, **data).validate(settings)
```
## Sample solution

I've actually tested a simple implementation using a convert map, and it indeed works. It is not able to force types, but maybe it shouldn't (for consistence).

Here is the outcomes after implementing the converter:

```bash
# for all cases
# - dynaconf_validators.toml -> startyear = {is_type_of="@int"}

# case 1: toml-is-int
# - settings.toml > startyear = 12
$ dynaconf -i config.settings validate
> Validation success!

# case 2: toml-is-str
# - settings.toml > startyear = "12"
$ dynaconf -i config.settings validate
> Validating 'startyear' with '{'is_type_of': '@int', 'env': 'default'}'
> Error: startyear must is_type_of <class 'int'> but it is 12 in env default
Validation error!

# case 3: toml-is-converter-to-int
# - settings.toml > startyear = "@int 12"
$ dynaconf -i config.settings validate
> Validation success!
```

## Conclusion

First, I thought the types inside `dynaconf_validators.toml` should actually be used as `"int"` (not `"@int"`), so it is clear that it will not try casting types and will strictly validate what is loaded from `settings.toml`. It seems the expected behavior for validating and is easy enough to fix.

But maybe there should be a "@type int" marker. From a design perspective, this really seems more appropriate, as it would avoid introducing unnecessary special case handling by leveraging the existing casting system. In this case, the parsing section of `cli.py:validate` would need to be re-implemented to use the same parsing process as in loading settings, which already can process markers. It would take a little more work, but seems the right way.

The expected way for this to work given the toml parser would be

```toml
startyear = {is_type_of="@int"}
```

I am not sure this is supported, but must be
> The expected way for this to work given the toml parser would be
> 
> ```toml
> startyear = {is_type_of="@int"}
> ```
> 
> I am not sure this is supported, but must be

This still throws the same error
## Problem analysis

I had a look at the code and `cli.py` doesn't use the custom parser which deals with `"@int"` converters in the `dynaconf_validators.toml` file, it directly parses it with tomllib or toml. [source](https://github.com/dynaconf/dynaconf/blob/46d3ab4e256212305fb3103847d56344dc339b46/dynaconf/cli.py#LL726-L735C10)

After parsing, here is what happens in `cli.py`: ([source](https://github.com/dynaconf/dynaconf/blob/46d3ab4e256212305fb3103847d56344dc339b46/dynaconf/cli.py#LL757-L763C36))

```diff
# this is inside a loop through `dynaconf_validators.toml` in-memory structure where:
# name -> "startyear"
# data -> dict = { 'is_type_of' = '@int' }
try:
-    Validator(name, **data).validate(settings)
+    Validator("startyear", is_type_of="@int").validate(settings) 
except ValidationError as e:
    click.echo(
        click.style(f"Error: {e}", fg="white", bg="red")
    )
    success = False
```

Having a look at  `Validator` usage, it should receive an `int` class (a type class) so, for this to work, a converter would need to be implemented:

```python
# data -> dict = { 'is_type_of' = '@int' }
data = convert_to_type_class(data)
# data -> dict = { 'is_type_of' = int }
Validator(name, **data).validate(settings)
```
## Sample solution

I've actually tested a simple implementation using a convert map, and it indeed works. It is not able to force types, but maybe it shouldn't (for consistence).

Here is the outcomes after implementing the converter:

```bash
# for all cases
# - dynaconf_validators.toml -> startyear = {is_type_of="@int"}

# case 1: toml-is-int
# - settings.toml > startyear = 12
$ dynaconf -i config.settings validate
> Validation success!

# case 2: toml-is-str
# - settings.toml > startyear = "12"
$ dynaconf -i config.settings validate
> Validating 'startyear' with '{'is_type_of': '@int', 'env': 'default'}'
> Error: startyear must is_type_of <class 'int'> but it is 12 in env default
Validation error!

# case 3: toml-is-converter-to-int
# - settings.toml > startyear = "@int 12"
$ dynaconf -i config.settings validate
> Validation success!
```

## Conclusion

First, I thought the types inside `dynaconf_validators.toml` should actually be used as `"int"` (not `"@int"`), so it is clear that it will not try casting types and will strictly validate what is loaded from `settings.toml`. It seems the expected behavior for validating and is easy enough to fix.

But maybe there should be a "@type int" marker. From a design perspective, this really seems more appropriate, as it would avoid introducing unnecessary special case handling by leveraging the existing casting system. In this case, the parsing section of `cli.py:validate` would need to be re-implemented to use the same parsing process as in loading settings, which already can process markers. It would take a little more work, but seems the right way.


Resolving patch:
diff --git a/dynaconf/cli.py b/dynaconf/cli.py
--- a/dynaconf/cli.py
+++ b/dynaconf/cli.py
@@ -12,6 +12,7 @@
 
 from dynaconf import constants
 from dynaconf import default_settings
+from dynaconf import Dynaconf
 from dynaconf import LazySettings
 from dynaconf import loaders
 from dynaconf import settings as legacy_settings
@@ -713,7 +714,6 @@ def validate(path):  # pragma: no cover
     # reads the 'dynaconf_validators.toml' from path
     # for each section register the validator for specific env
     # call validate
-
     path = Path(path)
 
     if not str(path).endswith(".toml"):
@@ -723,6 +723,7 @@ def validate(path):  # pragma: no cover
         click.echo(click.style(f"{path} not found", fg="white", bg="red"))
         sys.exit(1)
 
+    # parse validator file
     try:  # try tomlib first
         validation_data = tomllib.load(open(str(path), "rb"))
     except UnicodeDecodeError:  # fallback to legacy toml (TBR in 4.0.0)
@@ -733,6 +734,20 @@ def validate(path):  # pragma: no cover
         validation_data = toml.load(
             open(str(path), encoding=default_settings.ENCODING_FOR_DYNACONF),
         )
+    except tomllib.TOMLDecodeError as e:
+        click.echo(
+            click.style(
+                f"Error parsing TOML: {e}. Maybe it should be quoted.",
+                fg="white",
+                bg="red",
+            )
+        )
+        sys.exit(1)
+
+    # guarantee there is an environment
+    validation_data = {k.lower(): v for k, v in validation_data.items()}
+    if not validation_data.get("default"):
+        validation_data = {"default": validation_data}
 
     success = True
     for env, name_data in validation_data.items():
@@ -740,7 +755,8 @@ def validate(path):  # pragma: no cover
             if not isinstance(data, dict):  # pragma: no cover
                 click.echo(
                     click.style(
-                        f"Invalid rule for parameter '{name}'",
+                        f"Invalid rule for parameter '{name}'"
+                        "(this will be skipped)",
                         fg="white",
                         bg="yellow",
                     )
diff --git a/dynaconf/validator.py b/dynaconf/validator.py
--- a/dynaconf/validator.py
+++ b/dynaconf/validator.py
@@ -282,13 +282,12 @@ def _validate_items(
             if self.must_exist in (False, None) and value is empty:
                 continue
 
-            if self.cast:
-                # value or default value already set
-                # by settings.setdefault above
-                # however we need to cast it
-                # so we call .set again
-                value = self.cast(settings.get(name))
-                settings.set(name, value)
+            # value or default value already set
+            # by settings.setdefault above
+            # however we need to cast it
+            # so we call .set again
+            value = self.cast(settings.get(name))
+            settings.set(name, value)
 
             # is there a callable condition?
             if self.condition is not None:
@@ -304,7 +303,28 @@ def _validate_items(
             # operations
             for op_name, op_value in self.operations.items():
                 op_function = getattr(validator_conditions, op_name)
-                if not op_function(value, op_value):
+                op_succeeded = False
+
+                # 'is_type_of' special error handling - related to #879
+                if op_name == "is_type_of":
+                    # auto transform quoted types
+                    if isinstance(op_value, str):
+                        op_value = __builtins__.get(  # type: ignore
+                            op_value, op_value
+                        )
+
+                    # invalid type (not in __builtins__) may raise TypeError
+                    try:
+                        op_succeeded = op_function(value, op_value)
+                    except TypeError:
+                        raise ValidationError(
+                            f"Invalid type '{op_value}' for condition "
+                            "'is_type_of'. Should provide a valid type"
+                        )
+                else:
+                    op_succeeded = op_function(value, op_value)
+
+                if not op_succeeded:
                     _message = self.messages["operations"].format(
                         name=name,
                         operation=op_function.__name__,


Test patch:
diff --git a/tests/test_validators.py b/tests/test_validators.py
--- a/tests/test_validators.py
+++ b/tests/test_validators.py
@@ -875,3 +875,16 @@ def test_ensure_cast_works_for_non_default_values(tmpdir):
     settings.validators.validate()
 
     assert isinstance(settings.offset, int), type(settings.offset)
+
+
+def test_is_type_of__raises__with_type_error():
+    """
+    When invalid type is given to is_type_of,
+    should raise ValidationError
+    """
+    settings = Dynaconf(
+        validators=[Validator("ENV_FOR_DYNACONF", is_type_of="invalid")]
+    )
+
+    with pytest.raises(ValidationError):
+        settings.validators.validate()
diff --git a/tests_functional/issues/879_cli_validate_isinstance/Makefile b/tests_functional/issues/879_cli_validate_isinstance/Makefile
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/Makefile
@@ -0,0 +1,4 @@
+.phony: test
+
+test:
+	pytest -v test.py
diff --git a/tests_functional/issues/879_cli_validate_isinstance/config.py b/tests_functional/issues/879_cli_validate_isinstance/config.py
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/config.py
@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from dynaconf import Dynaconf
+
+# this settings should output validate only`
+settings_noenvs = Dynaconf(settings_file="settings-no-envs.toml")
+settings_envs = Dynaconf(environments=True, settings_file="settings-envs.toml")
+
+# this settings should output the proper validation error
+settings_noenvs_invalid = Dynaconf(
+    settings_file="settings-no-envs-invalid.toml"
+)
+settings_envs_invalid = Dynaconf(
+    environments=True, settings_file="settings-envs-invalid.toml"
+)
diff --git a/tests_functional/issues/879_cli_validate_isinstance/pyrightconfig.json b/tests_functional/issues/879_cli_validate_isinstance/pyrightconfig.json
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/pyrightconfig.json
@@ -0,0 +1,4 @@
+{
+  "venvPath": "/home/pedro-psb/dev/projects/dynaconf-issues/879_cli_validate_isinstace",
+  "venv": "venv"
+}
diff --git a/tests_functional/issues/879_cli_validate_isinstance/settings-envs-invalid.toml b/tests_functional/issues/879_cli_validate_isinstance/settings-envs-invalid.toml
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/settings-envs-invalid.toml
@@ -0,0 +1,2 @@
+[default]
+should_be_number="foo"
diff --git a/tests_functional/issues/879_cli_validate_isinstance/settings-envs.toml b/tests_functional/issues/879_cli_validate_isinstance/settings-envs.toml
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/settings-envs.toml
@@ -0,0 +1,2 @@
+[default]
+should_be_number=12
diff --git a/tests_functional/issues/879_cli_validate_isinstance/settings-no-envs-invalid.toml b/tests_functional/issues/879_cli_validate_isinstance/settings-no-envs-invalid.toml
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/settings-no-envs-invalid.toml
@@ -0,0 +1 @@
+should_be_number="foo"
diff --git a/tests_functional/issues/879_cli_validate_isinstance/settings-no-envs.toml b/tests_functional/issues/879_cli_validate_isinstance/settings-no-envs.toml
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/settings-no-envs.toml
@@ -0,0 +1 @@
+should_be_number=12
diff --git a/tests_functional/issues/879_cli_validate_isinstance/test.py b/tests_functional/issues/879_cli_validate_isinstance/test.py
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/test.py
@@ -0,0 +1,89 @@
+from __future__ import annotations
+
+import subprocess
+
+import pytest
+
+
+@pytest.mark.parametrize(
+    "config_option, validator_file",
+    (
+        (
+            pytest.param(
+                "config.settings_envs",
+                "validators_envs.toml",
+                id="with-envs-true",
+            ),
+            pytest.param(
+                "config.settings_noenvs",
+                "validators_noenvs.toml",
+                id="with-envs-false",
+            ),
+        )
+    ),
+)
+def test_validates_correctly(config_option, validator_file):
+    cmd = ("dynaconf", "-i", config_option, "validate", "-p", validator_file)
+    result = subprocess.run(cmd, text=True, capture_output=True)
+    stdout = result.stdout.lower()
+    assert "validation success" in stdout
+    assert "invalid rule for parameter" not in stdout
+
+
+@pytest.mark.parametrize(
+    "config_option, validator_file",
+    (
+        (
+            pytest.param(
+                "config.settings_envs_invalid",
+                "validators_envs.toml",
+                id="with-envs-true",
+            ),
+            pytest.param(
+                "config.settings_noenvs_invalid",
+                "validators_noenvs.toml",
+                id="with-envs-false",
+            ),
+        )
+    ),
+)
+def test_invalidates_correctly(config_option, validator_file):
+    cmd = ("dynaconf", "-i", config_option, "validate", "-p", validator_file)
+    result = subprocess.run(cmd, text=True, capture_output=True)
+    stdout = result.stdout.lower()
+    assert "error" in stdout
+    assert "validation success" not in stdout
+
+
+@pytest.mark.parametrize(
+    "validator_file, error_msg",
+    (
+        (
+            pytest.param(
+                "validators_invalid_toml.toml",
+                "error parsing toml",
+                id="invalid-toml-token",
+            ),
+            pytest.param(
+                "validators_invalid_type.toml",
+                "error: invalid type",
+                id="invalid-builtin-type",
+            ),
+        )
+    ),
+)
+def test_wrong_validator_rule_gives_error_msg_correctly(
+    validator_file, error_msg
+):
+    cmd = (
+        "dynaconf",
+        "-i",
+        "config.settings_noenvs",
+        "validate",
+        "-p",
+        validator_file,
+    )
+    result = subprocess.run(cmd, text=True, capture_output=True)
+    stdout = result.stdout.lower()
+    assert error_msg in stdout
+    assert "validation success" not in stdout
diff --git a/tests_functional/issues/879_cli_validate_isinstance/validators_envs.toml b/tests_functional/issues/879_cli_validate_isinstance/validators_envs.toml
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/validators_envs.toml
@@ -0,0 +1,2 @@
+[default]
+should_be_number={is_type_of="int"}
diff --git a/tests_functional/issues/879_cli_validate_isinstance/validators_invalid_toml.toml b/tests_functional/issues/879_cli_validate_isinstance/validators_invalid_toml.toml
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/validators_invalid_toml.toml
@@ -0,0 +1 @@
+should_be_number={is_type_of=int}
diff --git a/tests_functional/issues/879_cli_validate_isinstance/validators_invalid_type.toml b/tests_functional/issues/879_cli_validate_isinstance/validators_invalid_type.toml
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/validators_invalid_type.toml
@@ -0,0 +1 @@
+should_be_number={is_type_of="foo"}
diff --git a/tests_functional/issues/879_cli_validate_isinstance/validators_noenvs.toml b/tests_functional/issues/879_cli_validate_isinstance/validators_noenvs.toml
new file mode 100644
--- /dev/null
+++ b/tests_functional/issues/879_cli_validate_isinstance/validators_noenvs.toml
@@ -0,0 +1 @@
+should_be_number={is_type_of="int"}