Repository: bwindsor/typed-config

Issue:
Post configuration read transformations
I have a use case that has a very narrow set of supported values. Additionally, some configuration items may be inferred based on other values. Specifically, I'm looking at integrating [scrapy-redis](https://github.com/rmax/scrapy-redis) into a project and maintain the ability to swap between schedulers. 

```
@section('my_config')
class Scrapy(Config):
    scheduler = key(cast=str)
    scheduler_persist = key(cast=bool)
    scheduler_dupefilter_class = key(required=False, cast=str)
```

In the context of scrapy-redis, I would like to support a use case that allows just overwriting the scheduler, then automatically set scheduler_dupefilter_class to the right string if it isn't set.  

The cleanest way I can think to handle this today would be to use a custom `ConfigSource` and do an initial application config read, add the custom `ConfigSource` with read config, and then read again.

```
class ScrapyTransformationConfigSource(ConfigSource):
    def __init__(self, scrapy_config: Scrapy):
        # override based on currently set config

config = Scrapy(sources=[
    EnvironmentConfigSource(prefix="EXAMPLE")
])
config.read()
config.add_source(ScrapyTransformationConfigSource(config.scrapy))
config.read()
```

This isn't an ideal workflow, and doesn't scale great if this needs to be done for multiple configs. A built in post config read transformation would make this much more seamless.


Discussion:

Patch:
diff --git a/README.md b/README.md
index 45aec6b..45935a4 100644
--- a/README.md
+++ b/README.md
@@ -171,6 +171,9 @@ In this example we have three ways of casting:
 3. Defining a custom function. Your function should take one string input and return one output of any type. To get type hint, just make sure your function has type annotations.
 4. Using a lambda expression. The type inference may or may not work depending on your expression, so if it doesn't just write it as a function with type annotations.
 
+### Validation
+You can validate what has been supplied by providing a custom `cast` function to a `key`, which validates the configuration value in addition to parsing it.
+
 ### Extending configuration using shared ConfigProvider
 
 Multiple application modules may use different configuration schemes while sharing the same configuration source. Analogously, various `Config` classes may provide different view of the same configuration data, sharing the same `ConfigProvider`.
@@ -274,6 +277,51 @@ print(extended_config.app_extension.api_key)
 print(extended_config.database.host)
 ```
 
+### Configuration variables which depend on other configuration variables
+Sometimes you may wish to set the value of some configuration variables based on others. You may also wish to validate some variables, for example allowed values may be different depending on the value of another config variable. For this you can add a `post_read_hook`.
+
+The default implementation of `post_read_hook` returns an empty `dict`. You can override this by implementing your own `post_read_hook` method. It should receive only `self` as an input, and return a `dict`. This `dict` should be a simple mapping from config keys to values. For hierarchical configurations, you can nest the dictionaries. If you provide a `post_read_hook` in both a parent and a child class which both make changes to the same keys (don't do this) then the values returned by the child method will overwrite those by the parent.
+
+This hook is called whenever you call the `read` method. If you use lazy loading and skip calling the `read` method, you cannot use this hook.
+```python
+# my_app/config.py
+from typedconfig import Config, key, group_key, section
+from typedconfig.source import EnvironmentConfigSource
+
+@section('child')
+class ChildConfig(Config):
+    http_port_plus_one = key(cast=int, required=False)
+
+@section('app')
+class AppConfig(Config):
+    use_https = key(cast=bool)
+    http_port = key(key_name='port', cast=int, required=False)
+    child = group_key(ChildConfig)
+    
+    def post_read_hook(self) -> dict:
+        config_updates = dict()
+        # If the port has not been provided, set it based on the value of use_https
+        if self.http_port is None:
+            config_updates.update(http_port=443 if self.use_https else 80)
+        else:
+            # Modify child config
+            config_updates.update(child=dict(http_port_plus_one=self.http_port + 1))
+            
+        # Validate that the port number has a sensible value
+        # It is recommended to do validation inside the cast method for individual keys, however for dependent keys it can be useful here
+        if self.http_port is not None:
+            if self.use_https:
+                assert self.http_port in [443, 444, 445]
+            else:
+                assert self.http_port in [80, 81, 82]
+
+        return config_updates
+            
+config = AppConfig()
+config.add_source(EnvironmentConfigSource())
+config.read()
+```
+
 ## Configuration Sources
 Configuration sources are how your main `Config` class knows where to get its data from. These are totally extensible so that you can read in your configuration from wherever you like - from a database, from S3, anywhere that you can write code for.
 
diff --git a/typedconfig/config.py b/typedconfig/config.py
index 3fbee9e..1f155f3 100644
--- a/typedconfig/config.py
+++ b/typedconfig/config.py
@@ -1,9 +1,9 @@
 # This future import allows methods of Config to use Config as their return type
 import typing
-from typing import TypeVar, List, Optional, Callable, Type, Union
+from itertools import chain
+from typing import TypeVar, List, Optional, Callable, Type, Union, Tuple, Any
 from typedconfig.provider import ConfigProvider
 from typedconfig.source import ConfigSource
-from itertools import dropwhile, islice, chain
 import logging
 import inspect
 
@@ -58,39 +58,25 @@ def key(section_name: str = None,
         value: the parsed config value
         """
 
-        if section_name is None:
-            resolved_section_name = self._section_name
-            if resolved_section_name is None:
-                raise ValueError(
-                    "Section name was not specified by the key function or the section class decorator.")
-        else:
-            resolved_section_name = section_name
-
-        if _mutable_state['key_name'] is None:
-            def base_dict_items(cls):
-                base = cls
-                while True:
-                    yield base.__dict__.items()
-                    base = base.__base__
-                    if base is None:
-                        break
-            resolved_key_name = list(islice(dropwhile(
-                lambda x: x[1] is not getter, chain.from_iterable(base_dict_items(self.__class__))), 1))[0][0]
-            _mutable_state['key_name'] = resolved_key_name.upper()
+        resolved_section_name = self._resolve_section_name(section_name)
+
+        resolved_key_name = _mutable_state['key_name']
+        if resolved_key_name is None:
+            resolved_key_name = self._resolve_key_name(getter)
+            _mutable_state['key_name'] = resolved_key_name
 
         # If value is cached, just use the cached value
-        cached_value = self._provider.get_from_cache(resolved_section_name, _mutable_state['key_name'])
+        cached_value = self._provider.get_from_cache(resolved_section_name, resolved_key_name)
         if cached_value is not None:
             return cached_value
 
-        value = self._provider.get_key(resolved_section_name, _mutable_state['key_name'])
+        value = self._provider.get_key(resolved_section_name, resolved_key_name)
 
         # If we still haven't found a config value and this parameter is required,
         # raise an exception, otherwise use the default
         if value is None:
             if required:
-                raise KeyError("Config parameter {0}.{1} not found".format(resolved_section_name,
-                                                                           _mutable_state['key_name']))
+                raise KeyError("Config parameter {0}.{1} not found".format(resolved_section_name, resolved_key_name))
             else:
                 value = default
 
@@ -100,11 +86,13 @@ def key(section_name: str = None,
 
         # Cache this for next time if still not none
         if value is not None:
-            self._provider.add_to_cache(resolved_section_name, _mutable_state['key_name'], value)
+            self._provider.add_to_cache(resolved_section_name, resolved_key_name, value)
 
         return value
 
     setattr(getter.fget, Config._config_key_registration_string, True)
+    setattr(getter.fget, Config._config_key_key_name_string, key_name.upper() if key_name is not None else None)
+    setattr(getter.fget, Config._config_key_section_name_string, section_name)
     return getter
 
 
@@ -152,6 +140,8 @@ class Config:
     """
     _composed_config_registration_string = '__composed_config__'
     _config_key_registration_string = '__config_key__'
+    _config_key_key_name_string = '__config_key_key_name__'
+    _config_key_section_name_string = '__config_key_section_name__'
 
     def __init__(self, section_name=None, sources: List[ConfigSource] = None,
                  provider: Optional[ConfigProvider] = None):
@@ -189,10 +179,31 @@ class Config:
         -------
         A list of strings giving the names of the registered properties/methods
         """
-        all_properties = inspect.getmembers(self.__class__, predicate=lambda x: self.is_member_registered(
-            x, Config._config_key_registration_string))
+        all_properties = self._get_registered_properties_with_values()
         return [f[0] for f in all_properties]
 
+    def _get_registered_properties_with_values(self) -> List[Tuple[str, Any]]:
+        return inspect.getmembers(self.__class__, predicate=lambda x: self.is_member_registered(
+            x, Config._config_key_registration_string))
+
+    def _resolve_section_name(self, key_section_name: Optional[str]) -> str:
+        if key_section_name is not None:
+            return key_section_name
+
+        if self._section_name is None:
+            raise ValueError(
+                "Section name was not specified by the key function or the section class decorator.")
+        return self._section_name
+
+    def _resolve_key_name(self, property_object: property) -> str:
+        key_key_name = getattr(property_object.fget, self._config_key_key_name_string)
+        if key_key_name is not None:
+            return key_key_name
+
+        members = inspect.getmembers(self.__class__, lambda x: x is property_object)
+        assert len(members) == 1
+        return members[0][0].upper()
+
     @staticmethod
     def is_member_registered(member, reg_string: str):
         if isinstance(member, property):
@@ -229,6 +240,39 @@ class Config:
         for f in registered_properties:
             getattr(self, f)
 
+        self._post_read(self.post_read_hook())
+
+    def post_read_hook(self) -> dict:
+        """
+        This method can be overridden to modify config values after read() is called.
+        Returns
+        -------
+        A dict of key-value pairs containing new configuration values for key() items in this Config class
+        """
+        return dict()
+
+    def _post_read(self, updated_values: dict):
+        registered_properties = set(self.get_registered_properties())
+
+        for k, v in updated_values.items():
+            if isinstance(v, dict):
+                property_object = getattr(self.__class__, k)
+                if not self.is_member_registered(property_object, self._composed_config_registration_string):
+                    raise KeyError(f"{k} is not a valid typed config group_key() of {self.__class__.__name__}")
+                child_config = getattr(self, k)
+                child_config._post_read(v)
+            else:
+                if k not in registered_properties:
+                    raise KeyError(f"{k} is not a valid attribute of {self.__class__.__name__}")
+
+                property_object: property = getattr(self.__class__, k)
+                if not self.is_member_registered(property_object, self._config_key_registration_string):
+                    raise KeyError(f"{k} is not a valid typed config key() object of {self.__class__.__name__}")
+
+                section_name = self._resolve_section_name(getattr(property_object.fget, self._config_key_section_name_string))
+                key_name = self._resolve_key_name(property_object)
+                self._provider.add_to_cache(section_name, key_name, v)
+
     def clear_cache(self):
         """
         Config values are cached the first time they are requested. This means that if, for example, config values are


Test patch:
diff --git a/test/test_configuration.py b/test/test_configuration.py
index 4663753..2c9e205 100644
--- a/test/test_configuration.py
+++ b/test/test_configuration.py
@@ -231,6 +231,23 @@ def test_key_name_inference():
     assert 'def' == c.prop2
 
 
+def test_key_name_inference_multi_level():
+    class SampleConfigBase(Config):
+        prop1 = key(section_name='s')
+
+    class SampleConfig(SampleConfigBase):
+        prop2 = key(section_name='s')
+
+    c = SampleConfig()
+
+    c.add_source(DictConfigSource({'s': dict(
+        PROP1='abc',
+        PROP2='def',
+    )}))
+    assert 'abc' == c.prop1
+    assert 'def' == c.prop2
+
+
 def test_least_verbose_config():
     @section('X')
     class SampleConfig(Config):
@@ -387,6 +404,109 @@ def test_set_sources():
     assert config.config_sources[1] is new_sources[1]
 
 
+def test_property_is_read_only():
+    config = GrandchildConfig()
+    with pytest.raises(AttributeError):
+        config.prop1 = 'a'
+
+
+def test_post_read_hook():
+    @section('s')
+    class SampleConfig(Config):
+        prop1 = key(cast=str)
+        prop2 = key(cast=str, required=False)
+
+        def post_read_hook(self) -> dict:
+            return dict(prop2='x' + self.prop1)
+
+    config_source = DictConfigSource({
+        's': {
+            'prop1': 'a',
+        }
+    })
+    config = SampleConfig(sources=[config_source])
+    config.read()
+
+    assert config.prop1 == 'a'
+    assert config.prop2 == 'xa'
+
+
+def test_post_read_hook_different_key_name():
+    @section('s')
+    class SampleConfig(Config):
+        prop1 = key('s', 'key1', cast=str)
+        prop2 = key('s', 'key2', cast=str, required=False)
+
+        def post_read_hook(self) -> dict:
+            return dict(prop2='x' + self.prop1)
+
+    config_source = DictConfigSource({
+        's': {
+            'key1': 'a',
+        }
+    })
+    config = SampleConfig(sources=[config_source])
+    config.read()
+
+    assert config.prop1 == 'a'
+    assert config.prop2 == 'xa'
+
+
+def test_post_read_hook_modify_child():
+    class SampleChildConfig(Config):
+        prop3 = key('s', 'key3', cast=str)
+
+    class SampleConfig(Config):
+        prop3 = group_key(SampleChildConfig)
+
+        def post_read_hook(self) -> dict:
+            return dict(
+                prop3=dict(
+                    prop3='new_value'
+                )
+            )
+
+    config_source = DictConfigSource({
+        's': {
+            'key3': 'b',
+        }
+    })
+    config = SampleConfig(sources=[config_source])
+    config.read()
+
+    assert config.prop3.prop3 == 'new_value'
+
+
+def test_post_read_hook_child_takes_priority():
+    class SampleChildConfig(Config):
+        prop3 = key('s', 'key3', cast=str)
+
+        def post_read_hook(self) -> dict:
+            return dict(
+                prop3='child_new_value'
+            )
+
+    class SampleConfig(Config):
+        prop3 = group_key(SampleChildConfig)
+
+        def post_read_hook(self) -> dict:
+            return dict(
+                prop3=dict(
+                    prop3='new_value'
+                )
+            )
+
+    config_source = DictConfigSource({
+        's': {
+            'key3': 'b',
+        }
+    })
+    config = SampleConfig(sources=[config_source])
+    config.read()
+
+    assert config.prop3.prop3 == 'child_new_value'
+
+
 def test_config_repr():
     class SampleChildConfig(Config):
         a = key(section_name='test', cast=str, required=False, default='A')