Repository: omni-us/jsonargparse

Issue:
Failure to check choice matches for options when using a configuration yaml or json file.
<!-- If you like this project, please ⭐ star it https://github.com/omni-us/jsonargparse/ -->

## 🐛 Bug report

<!-- A clear and concise description of the bug. -->
Using a config file (yaml or json) skips the choice validation step for any other argument.

### To reproduce

```python
import jsonargparse
import yaml

parser = jsonargparse.ArgumentParser()
parser.add_argument('--arg1', type=str, choices=["choice1", "choice2"])
parser.add_argument("--config", action=jsonargparse.ActionConfigFile)
config = yaml.safe_dump(
    {
        "arg1": "i_wont_choose_from_choices",
    }
)
result = parser.parse_args([f"--config={config}"])
print("Skips choice validation and wrong config is parsed:")
print(parser.dump(result))

print("\n----------The above one parses wrongly. The expected behaviour follows---------------------\n\n")

print("But pass the arg1 as a argument override and it immediately fails. As its supposed to - the tests make sure of this.")
result = parser.parse_args([f"--arg1=i_definitely_wont_choose_from_choices"])
print(parser.dump(result))
```

### Expected behavior

<!-- Describe how the behavior or output of the reproduction snippet should be. -->
The same script above shows the expected behavior as well. But what should happen is parser should throw an error stating that the given argument is not valid as it is not within the set of options given.

### Environment

<!-- Fill in the list below. -->

- jsonargparse version (e.g., 4.8.0): 4.25.0
- Python version (e.g., 3.9): 3.8.18
- How jsonargparse was installed (e.g. `pip install jsonargparse[all]`): as in example
- OS (e.g., Linux): Ubuntu 22.04 LTS


My initial suspicions are that this has to do with how config file arguments are parsed. 
This line (calling a function from argparse which further down the line calls the choice validation function in argparse):
https://github.com/omni-us/jsonargparse/blob/5fc835701e52e864c497806b92ce217f9eb56ed5/jsonargparse/_core.py#L259

is on a high level responsible for choice validation. But, in the current code, the config argument parsing doesn't fill out the other fields before the validation work is handed over to argparse. So whatever arguments come from the config file are not at all choice checked. The current code does have choice checking but that is only for subcommands and not for '--arg' (not sure what to call them but the ones with dashes or just arguments). Hence, any choice arguments provided via a config file go under the radar and can potentially wreak havoc down the line.


Discussion:
sonarcloud[bot]: Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=omni-us_jsonargparse&pullRequest=405)

[![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=BUG)  
[![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=VULNERABILITY)  
[![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=SECURITY_HOTSPOT)  
[![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=omni-us_jsonargparse&pullRequest=405&resolved=false&types=CODE_SMELL)

[![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=omni-us_jsonargparse&pullRequest=405) No Coverage information  
[![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=omni-us_jsonargparse&pullRequest=405&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=omni-us_jsonargparse&pullRequest=405&metric=new_duplicated_lines_density&view=list)


codecov[bot]: ## [Codecov](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) Report
All modified lines are covered by tests :white_check_mark:
> Comparison is base [(`5fc8357`)](https://app.codecov.io/gh/omni-us/jsonargparse/commit/5fc835701e52e864c497806b92ce217f9eb56ed5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) 100.00% compared to head [(`7fcfa2c`)](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) 100.00%.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff            @@
##              main      #405   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           21        21           
  Lines         5879      5882    +3     
=========================================
+ Hits          5879      5882    +3     
```

| [Flag](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | Coverage Δ | |
|---|---|---|
| [py3.10](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `?` | |
| [py3.10_all](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `98.70% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.10_pydantic1](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `48.29% <60.00%> (+<0.01%)` | :arrow_up: |
| [py3.10_pydantic2](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `48.02% <60.00%> (+<0.01%)` | :arrow_up: |
| [py3.10_types](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `98.70% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.11](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `88.15% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.11_all](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `98.70% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.11_types](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `98.70% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.7](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `88.74% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.7_all](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `99.48% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.7_types](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `99.46% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.8](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `88.89% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.8_all](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `99.50% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.8_types](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `99.48% <100.00%> (+<0.01%)` | :arrow_up: |
| [py3.9](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `?` | |
| [py3.9_all](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `?` | |
| [py3.9_types](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | `?` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Files](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us) | Coverage Δ | |
|---|---|---|
| [jsonargparse/\_core.py](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us#diff-anNvbmFyZ3BhcnNlL19jb3JlLnB5) | `100.00% <100.00%> (ø)` | |


</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/omni-us/jsonargparse/pull/405?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=omni-us).


Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 2bfec93..6550d69 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -27,6 +27,8 @@ Fixed
 - Failures with subcommands and default_config_files when keys are repeated
   (`#160 <https://github.com/omni-us/jsonargparse/issues/160>`__).
 - Key related errors printing messages within single quotes.
+- Choices not validated when value comes from config file (`#404
+  <https://github.com/omni-us/jsonargparse/issues/404>`__).
 
 Changed
 ^^^^^^^
diff --git a/jsonargparse/_core.py b/jsonargparse/_core.py
index 1857bc3..d8056df 100644
--- a/jsonargparse/_core.py
+++ b/jsonargparse/_core.py
@@ -1330,11 +1330,12 @@ class ArgumentParser(ParserDeprecations, ActionsContainer, ArgumentLinking, argp
         """
         if value is None and lenient_check.get():
             return value
-        if action.choices is not None and isinstance(action, _ActionSubCommands):
+        is_subcommand = isinstance(action, _ActionSubCommands)
+        if is_subcommand and action.choices:
             leaf_key = split_key_leaf(key)[-1]
             if leaf_key == action.dest:
                 return value
-            subparser = action._name_parser_map[leaf_key]
+            subparser = action._name_parser_map[leaf_key]  # type: ignore
             subparser.check_config(value)
         elif isinstance(action, _ActionConfigLoad):
             if isinstance(value, str):
@@ -1351,6 +1352,8 @@ class ArgumentParser(ParserDeprecations, ActionsContainer, ArgumentLinking, argp
                         value[k] = action.type(v)
             except (TypeError, ValueError) as ex:
                 raise TypeError(f'Parser key "{key}": {ex}') from ex
+        if not is_subcommand and action.choices and value not in action.choices:
+            raise TypeError(f'Parser key "{key}": {value!r} not among choices {action.choices}')
         return value
 
     ## Properties ##


Test patch:
diff --git a/jsonargparse_tests/test_core.py b/jsonargparse_tests/test_core.py
index 314b386..07cd4ac 100644
--- a/jsonargparse_tests/test_core.py
+++ b/jsonargparse_tests/test_core.py
@@ -139,13 +139,23 @@ def test_parse_args_positional_config(parser):
 
 def test_parse_args_choices(parser):
     parser.add_argument("--ch1", choices="ABC")
-    parser.add_argument("--ch2", choices=["v1", "v2"])
+    parser.add_argument("--ch2", type=str, choices=["v1", "v2"])
     cfg = parser.parse_args(["--ch1", "C", "--ch2", "v1"])
     assert cfg.as_dict() == {"ch1": "C", "ch2": "v1"}
     pytest.raises(ArgumentError, lambda: parser.parse_args(["--ch1", "D"]))
     pytest.raises(ArgumentError, lambda: parser.parse_args(["--ch2", "v0"]))
 
 
+def test_parse_args_choices_config(parser):
+    parser.add_argument("--cfg", action=ActionConfigFile)
+    parser.add_argument("--ch1", choices="ABC")
+    parser.add_argument("--ch2", type=str, choices=["v1", "v2"])
+    assert parser.parse_args(["--cfg=ch1: B"]).ch1 == "B"
+    assert parser.parse_args(["--cfg=ch2: v2"]).ch2 == "v2"
+    pytest.raises(ArgumentError, lambda: parser.parse_args(["--cfg=ch1: D"]))
+    pytest.raises(ArgumentError, lambda: parser.parse_args(["--cfg=ch2: v0"]))
+
+
 def test_parse_object_simple(parser):
     parser.add_argument("--op", type=int)
     assert parser.parse_object({"op": 1}) == Namespace(op=1)