Repository: omni-us/jsonargparse

Issue:
Failure to check choice matches for options when using a configuration yaml or json file.
<!-- If you like this project, please ⭐ star it https://github.com/omni-us/jsonargparse/ -->

## 🐛 Bug report

<!-- A clear and concise description of the bug. -->
Using a config file (yaml or json) skips the choice validation step for any other argument.

### To reproduce

```python
import jsonargparse
import yaml

parser = jsonargparse.ArgumentParser()
parser.add_argument('--arg1', type=str, choices=["choice1", "choice2"])
parser.add_argument("--config", action=jsonargparse.ActionConfigFile)
config = yaml.safe_dump(
    {
        "arg1": "i_wont_choose_from_choices",
    }
)
result = parser.parse_args([f"--config={config}"])
print("Skips choice validation and wrong config is parsed:")
print(parser.dump(result))

print("\n----------The above one parses wrongly. The expected behaviour follows---------------------\n\n")

print("But pass the arg1 as a argument override and it immediately fails. As its supposed to - the tests make sure of this.")
result = parser.parse_args([f"--arg1=i_definitely_wont_choose_from_choices"])
print(parser.dump(result))
```

### Expected behavior

<!-- Describe how the behavior or output of the reproduction snippet should be. -->
The same script above shows the expected behavior as well. But what should happen is parser should throw an error stating that the given argument is not valid as it is not within the set of options given.

### Environment

<!-- Fill in the list below. -->

- jsonargparse version (e.g., 4.8.0): 4.25.0
- Python version (e.g., 3.9): 3.8.18
- How jsonargparse was installed (e.g. `pip install jsonargparse[all]`): as in example
- OS (e.g., Linux): Ubuntu 22.04 LTS


My initial suspicions are that this has to do with how config file arguments are parsed. 
This line (calling a function from argparse which further down the line calls the choice validation function in argparse):
https://github.com/omni-us/jsonargparse/blob/5fc835701e52e864c497806b92ce217f9eb56ed5/jsonargparse/_core.py#L259

is on a high level responsible for choice validation. But, in the current code, the config argument parsing doesn't fill out the other fields before the validation work is handed over to argparse. So whatever arguments come from the config file are not at all choice checked. The current code does have choice checking but that is only for subcommands and not for '--arg' (not sure what to call them but the ones with dashes or just arguments). Hence, any choice arguments provided via a config file go under the radar and can potentially wreak havoc down the line.


Comments:

Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 2bfec93..6550d69 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -27,6 +27,8 @@ Fixed
 - Failures with subcommands and default_config_files when keys are repeated
   (`#160 <https://github.com/omni-us/jsonargparse/issues/160>`__).
 - Key related errors printing messages within single quotes.
+- Choices not validated when value comes from config file (`#404
+  <https://github.com/omni-us/jsonargparse/issues/404>`__).
 
 Changed
 ^^^^^^^
diff --git a/jsonargparse/_core.py b/jsonargparse/_core.py
index 1857bc3..d8056df 100644
--- a/jsonargparse/_core.py
+++ b/jsonargparse/_core.py
@@ -1330,11 +1330,12 @@ class ArgumentParser(ParserDeprecations, ActionsContainer, ArgumentLinking, argp
         """
         if value is None and lenient_check.get():
             return value
-        if action.choices is not None and isinstance(action, _ActionSubCommands):
+        is_subcommand = isinstance(action, _ActionSubCommands)
+        if is_subcommand and action.choices:
             leaf_key = split_key_leaf(key)[-1]
             if leaf_key == action.dest:
                 return value
-            subparser = action._name_parser_map[leaf_key]
+            subparser = action._name_parser_map[leaf_key]  # type: ignore
             subparser.check_config(value)
         elif isinstance(action, _ActionConfigLoad):
             if isinstance(value, str):
@@ -1351,6 +1352,8 @@ class ArgumentParser(ParserDeprecations, ActionsContainer, ArgumentLinking, argp
                         value[k] = action.type(v)
             except (TypeError, ValueError) as ex:
                 raise TypeError(f'Parser key "{key}": {ex}') from ex
+        if not is_subcommand and action.choices and value not in action.choices:
+            raise TypeError(f'Parser key "{key}": {value!r} not among choices {action.choices}')
         return value
 
     ## Properties ##


Test patch:
diff --git a/jsonargparse_tests/test_core.py b/jsonargparse_tests/test_core.py
index 314b386..07cd4ac 100644
--- a/jsonargparse_tests/test_core.py
+++ b/jsonargparse_tests/test_core.py
@@ -139,13 +139,23 @@ def test_parse_args_positional_config(parser):
 
 def test_parse_args_choices(parser):
     parser.add_argument("--ch1", choices="ABC")
-    parser.add_argument("--ch2", choices=["v1", "v2"])
+    parser.add_argument("--ch2", type=str, choices=["v1", "v2"])
     cfg = parser.parse_args(["--ch1", "C", "--ch2", "v1"])
     assert cfg.as_dict() == {"ch1": "C", "ch2": "v1"}
     pytest.raises(ArgumentError, lambda: parser.parse_args(["--ch1", "D"]))
     pytest.raises(ArgumentError, lambda: parser.parse_args(["--ch2", "v0"]))
 
 
+def test_parse_args_choices_config(parser):
+    parser.add_argument("--cfg", action=ActionConfigFile)
+    parser.add_argument("--ch1", choices="ABC")
+    parser.add_argument("--ch2", type=str, choices=["v1", "v2"])
+    assert parser.parse_args(["--cfg=ch1: B"]).ch1 == "B"
+    assert parser.parse_args(["--cfg=ch2: v2"]).ch2 == "v2"
+    pytest.raises(ArgumentError, lambda: parser.parse_args(["--cfg=ch1: D"]))
+    pytest.raises(ArgumentError, lambda: parser.parse_args(["--cfg=ch2: v0"]))
+
+
 def test_parse_object_simple(parser):
     parser.add_argument("--op", type=int)
     assert parser.parse_object({"op": 1}) == Namespace(op=1)