Package: genconfig
GenConfigManage config using folder structure, currently support Json and Yaml.InstallationExample usageProject overviewUsageParametersInstallationpipinstallgenconfigExample usageArrange the config files into folders, sub-folders, and files.sample-config/config-mix/
├── basic.yml
├── function
│   ├── function1.json
│   └── function2.yml
├── param.yml
└── pipeline.json

1 directory, 5 filesRun thegenconfiggenconfigsample-config/config-mix-oconfig.jsonRetrieve a single merged config{
    "name": "config-01",
    "training": true,
    "parameters": {
        "num_nodes": 200,
        "num_samples": 100,
        "max_time": 40
    },
    "function": {
        "function1": {
            "name": "transform",
            "param": "col1"
        },
        "function2": {
            "name": "load",
            "param": "col"
        }
    },
    "pipeline": [
        {
            "name": "extraction",
            "function": "etl.extraction"
        },
        {
            "name": "training",
            "function": "model.training"
        },
        {
            "name": "evaluation",
            "function": "model.evaluation"
        },
        {
            "name": "deployment",
            "function": "cloud.deploy"
        }
    ]
}Project overviewThe dream of a mature project is to cover most of the essential functions, and
the interactions users would require is to modify the configuration files
based on specific needs. For example, a user running model training for a new
region A does not need to write any new codes, instead simply modifying the
configuration files.However, as the project grows, so does the configuration file. There are various
settings region A would require but not all of them need constant updates.
Therefore, having a programmatic way to update the configuration files and to
display them in an easy-to-understand format is crucial.I propose to present the configuration files in a folder structure. The root
folder represents the first level keys in the configs, and the subfolders
represent nested keys.Furthermore, we should allow both json and yaml config files. Json is easy to
create while yaml allows comments and variables creation. Both Json and yaml
have their advantages.The API documentation is available athttps://lingjie00.github.io/genconfig/The GitHub repo is athttps://github.com/lingjie00/genconfigUsageConvert a folder structure into a single json config filegenconfig--path=folder_path--output="config.json"ParametersignoredUser can choose to ignore some keys and not expand into sub-folders# supports regex matchinggenconfigconfig_path-oconfig.json--ignore"config1.json""config2.yaml""debug.*json"keepDifferent from ignored, instead read only files with matching file name# supports regex matchinggenconfigconfig_path-oconfig.json--keep"only_this_config.yaml"verbosethe level of logging to display# refer to python logging for acceptable levelsgenconfigconfig_path-oconfig.json--verbose"DEBUG"appenduseful for manual replace/update values# input has to be in json formatgenconfigconfig_path-oconfig.json--append"{'name': 'updated_name'}"readUseful when there is a mixture of config file types but only want to read
a specific file typegenconfigconfig_path-oconfig.json--read"json"folderuseful to ignore folder names as key, then user can arrange config files
in any subfolders without worrying about nested keysgenconfigconfig_path-oconfig.json--folderFalse