Repository: omni-us/jsonargparse

Issue:
``discard_init_args_on_class_path_change`` does not handle nested ``dict`` subclass specs in some contexts
## 🐛 Bug report
Firstly, thanks again for all your work maintaining this remarkably valuable package!

While a ``subclass_spec`` can be either a ``dict`` or ``Namespace`` 
https://github.com/omni-us/jsonargparse/blob/ac790ba09bcc920cab1acf712422152b931d337a/jsonargparse/typehints.py#L790-L795
when entering ``ActionTypeHint.discard_init_args_on_class_path_change``:
https://github.com/omni-us/jsonargparse/blob/ac790ba09bcc920cab1acf712422152b931d337a/jsonargparse/typehints.py#L295-L310


``jsonargparse.typehints.discard_init_args_on_class_path_change`` below currently assumes any previous value for a nested subclass will already have been cast as a ``Namespace``: 
https://github.com/omni-us/jsonargparse/blob/ac790ba09bcc920cab1acf712422152b931d337a/jsonargparse/typehints.py#L934-L942

with ``L941`` resulting in:
```
argparse.ArgumentError: Parser key "some_class":
  Problem with given class_path '__main__.CustomModule':
    - 'dict' object has no attribute 'init_args'
```
Right now, nested generic ``dict`` configurations like the repro case I include below don't trigger the ``subclass_spec_as_namespace`` ``Namespace`` wrapping that ``jsonargparse.typehints.discard_init_args_on_class_path_change`` requires when removing overridden subclass spec configuration in this context.

With the below minor patch to ``jsonargparse.typehints.discard_init_args_on_class_path_change``, this use case can be accommodated (I haven't had a chance to run the full jsongarparse test suite but can't immediately think of any problem with this additional check/transformation):

```python
def discard_init_args_on_class_path_change(parser_or_action, prev_val, value):
    if prev_val and 'init_args' in prev_val and prev_val['class_path'] != value['class_path']:
        parser = parser_or_action
        if isinstance(parser_or_action, ActionTypeHint):
            sub_add_kwargs = getattr(parser_or_action, 'sub_add_kwargs', {})
            parser = ActionTypeHint.get_class_parser(value['class_path'], sub_add_kwargs)
        del_args = {}
        if isinstance(prev_val, dict):
            prev_val = subclass_spec_as_namespace(prev_val)
        for key, val in list(prev_val.init_args.__dict__.items()):
            action = _find_action(parser, key)
```
### To reproduce
```python
from typing import Any, Dict

import yaml
import lightning.pytorch as pl
from jsonargparse import ArgumentParser


override_config = yaml.safe_dump(
    {
      "class_path": "CustomModule",
      "init_args": {
        "some_custom_init": {
          "class_path": "torch.optim.lr_scheduler.StepLR",
          "init_args": {"step_size": 1, "gamma": 0.7},
        }
      }
    }
  )

default_config = yaml.safe_dump(
  {
    "class_path": "CustomModule",
    "init_args": {
      "some_custom_init": {
        "class_path": "torch.optim.lr_scheduler.CosineAnnealingWarmRestarts",
        "init_args": {"T_0": 1, "T_mult": 2, "eta_min": 1.0e-07}
      }
    }
  }
)


class CustomModule(pl.LightningModule):
    def __init__(self, some_custom_init: Dict[str, Any],):
        super().__init__()


parser = ArgumentParser()
parser.add_argument("--some_class", type=pl.LightningModule)
result = parser.parse_args(["--some_class", default_config, "--some_class", override_config])
print(f"expected_result= {result}")
```
Current behavior:
```
argparse.ArgumentError: Parser key "some_class":
  Problem with given class_path '__main__.CustomModule':
    - 'dict' object has no attribute 'init_args'
```

Expected result, generated with additional handling discussed above:
```
expected_result= Namespace(some_class=Namespace(class_path='__main__.CustomModule', init_args=Namespace(some_custom_init={'class_path': 'torch.optim.lr_scheduler.StepLR', 'init_args': {'gamma': 0.7, 'step_size': 1}})))
```
### Environment

- jsonargparse version (e.g., 4.8.0): 4.20.0
- Python version (e.g., 3.9): 3.10
- How jsonargparse was installed (e.g. `pip install jsonargparse[all]`): pip install "jsonargparse[signatures]==4.20.0"
- OS (e.g., Linux): Ubuntu 20.04


Comments:

Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index 4faf1b4..1aafaf5 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -12,11 +12,14 @@ The semantic versioning only considers the public API as described in
 paths are considered internals and can change in minor and patch releases.
 
 
-v4.20.1 (2023-02-??)
+v4.20.1 (2023-03-??)
 --------------------
 
 Fixed
 ^^^^^
+- Allow ``discard_init_args_on_class_path_change`` to handle more nested contexts `#247
+  <https://github.com/omni-us/jsonargparse/issues/247>`__.
+
 - Dump not working for partial callable with return instance
   `pytorch-lightning#15340 (comment)
   <https://github.com/Lightning-AI/lightning/issues/15340#issuecomment-1439203008>`__.
diff --git a/jsonargparse/typehints.py b/jsonargparse/typehints.py
index fc11db1..7f67ae1 100644
--- a/jsonargparse/typehints.py
+++ b/jsonargparse/typehints.py
@@ -939,6 +939,7 @@ def discard_init_args_on_class_path_change(parser_or_action, prev_val, value):
             sub_add_kwargs = getattr(parser_or_action, 'sub_add_kwargs', {})
             parser = ActionTypeHint.get_class_parser(value['class_path'], sub_add_kwargs)
         del_args = {}
+        prev_val = subclass_spec_as_namespace(prev_val)
         for key, val in list(prev_val.init_args.__dict__.items()):
             action = _find_action(parser, key)
             if action:


Test patch:
diff --git a/jsonargparse_tests/test_signatures.py b/jsonargparse_tests/test_signatures.py
index 463be12..3c85032 100755
--- a/jsonargparse_tests/test_signatures.py
+++ b/jsonargparse_tests/test_signatures.py
@@ -1427,6 +1427,46 @@ class SignaturesConfigTests(TempDirTestCase):
                     self.assertIsInstance(init.main.sub, Sub2)
                     self.assertTrue(any("discarding init_args: {'s1': 'x'}" in o for o in log.output))
 
+    def test_config_nested_dict_discard_init_args(self):
+        class Base:
+            def __init__(self, b: float = 0.5):
+                pass
+
+        class Sub1(Base):
+            def __init__(self, s1: int = 3, **kwargs):
+                super().__init__(**kwargs)
+
+        class Sub2(Base):
+            def __init__(self, s2: int = 4, **kwargs):
+                super().__init__(**kwargs)
+
+        class Main:
+            def __init__(self, sub: Optional[Dict] = None) -> None:
+                self.sub = sub
+
+        configs, subconfigs, config_paths = {}, {}, {}
+        with mock_module(Base, Sub1, Sub2, Main) as module:
+            parser = ArgumentParser(exit_on_error=False, logger={'level': 'DEBUG'})
+            parser.add_argument('--config', action=ActionConfigFile)
+            parser.add_subclass_arguments(Main, 'main')
+            parser.set_defaults(main=lazy_instance(Main))
+            for c in [1, 2]:
+                subconfigs[c] = {
+                    'sub': {
+                        'class_path': f'{module}.Sub{c}',
+                        'init_args': {f's{c}': c},
+                    }
+                }
+                configs[c] = {'main': {'class_path': f'{module}.Main','init_args': subconfigs[c],}}
+                config_paths[c] = Path(f'config{c}.yaml')
+                config_paths[c].write_text(yaml.safe_dump(configs[c]))
+
+            with self.assertLogs(logger=parser.logger, level='DEBUG') as log:
+                cfg = parser.parse_args([f'--config={config_paths[1]}', f'--config={config_paths[2]}'])
+            init = parser.instantiate_classes(cfg)
+            self.assertIsInstance(init.main, Main)
+            self.assertTrue(init.main.sub['init_args']['s2'], 2)
+            self.assertTrue(any("discarding init_args: {'s1': 1}" in o for o in log.output))
 
 @dataclasses.dataclass(frozen=True)
 class MyDataClassA: