Repository: kedro-org/kedro

Issue #1363: Make config glob paths configurable by the user
Created: 2022-03-23T10:57:25Z
Description

Is your feature request related to a problem? A clear and concise description of what the problem is: "I'm always frustrated when ..."

Fix #1210
Today catalog config is loaded via the following paths (this also applies the same to parameters):
conf_catalog = conf_loader.get('catalog*', 'catalog*/**')

  
    
      kedro/kedro/framework/context/context.py
    
    
         Line 356
      in
      e78990c
    
  
  
    

        
          
           conf_catalog = self.config_loader.get("catalog*", "catalog*/**", "**/catalog*") 
        
    
  


These are hardcoded in the library and the user of a project cannot change without getting creative. I think there would be a lot of user value (1) making these configurable (2) being transparent with how this works.
Context

Why is this change important to you? How would you use it? How can it benefit other users?


It feels restrictive to stop users controlling this when so much of the framework is configurable
We have evidence of users want to do _catalog suffixes (as opposed to prefixes) which aren't supported by the default pattern

Possible Implementation
(Optional) Suggest an idea for implementing the addition or change.

In 0.18.x it may make sense to expose in settings.py

Duplicate issue #1210: Allow multiple globals_patterns on TemplatedConfigLoader
Created: 2022-02-04T02:43:17Z
Description
It would be great if we could pass a list of patterns in the constructor: 
  
    
      kedro/kedro/config/templated_config.py
    
    
         Line 99
      in
      e78990c
    
  
  
    

        
          
           globals_pattern: Optional[str] = None, 
        
    
  


Context
We do have our global parameters into different subfolders with different names, why not allow multiple patterns here?
Possible Implementation
Alter this constructor to this: 
  
    
      kedro/kedro/config/templated_config.py
    
    
         Line 95
      in
      e78990c
    
  
  
    

        
          
           def __init__( 
        
    
  


To this:
def __init__(
        self,
        conf_paths: Union[str, Iterable[str]],
        *,
        globals_pattern: Optional[str] = None,
        globals_dict: Optional[Dict[str, Any]] = None,
        globals_patterns: Optional[List[str]] = None,
):
    """Instantiate a ``TemplatedConfigLoader``.
    Args:
        conf_paths: Non-empty path or list of paths to configuration
            directories.
        globals_pattern: Optional keyword-only argument specifying a glob
            pattern. Files that match the pattern will be loaded as a
            formatting dictionary.
        globals_dict: Optional keyword-only argument specifying a formatting
            dictionary. This dictionary will get merged with the globals dictionary
            obtained from the globals_pattern. In case of duplicate keys, the
            ``globals_dict`` keys take precedence.
        globals_patterns: Optional list of keyword-only arguments specifying multiple glob
            patterns. Files that match the pattern will be loaded as a
            formatting dictionary.
    """

    super().__init__(conf_paths)

    patterns = deepcopy(globals_patterns) or []

    if globals_pattern:
        patterns.append(globals_pattern)

    self._arg_dict = super().get(*globals_patterns) if globals_patterns else {}
    globals_dict = deepcopy(globals_dict) or {}
    self._arg_dict = {**self._arg_dict, **globals_dict}
Possible Alternatives
(Optional) Describe any alternative solutions or features you've considered.

Closing comment (2022-04-25T13:13:26Z): I think this is now tracked by #1363.