Package: onion-config
onion_configonion_configis a python package that allows for easy configuration management. It allows for loading and validating configuration data from environment variables and config files in JSON and YAML formats.Pydanticbased custom config package for python projects.FeaturesMain configbased onPydantic schema-https://pypi.org/project/pydanticLoadenvironment variables-https://pypi.org/project/python-dotenvLoad frommultipleconfigs directoriesLoad configs fromYAMLandJSONfilesUpdate the default config with additional configurations (extra_dirdirectory)Pre-load hookfunction to modify config data before loading and validationValidate configvalues withPydantic validatorsConfig asdictionaryorPydantic model(with type hints)Pre-definedbase config schema for common config (BaseConfig)Basefor custom config loader (ConfigLoader)Installation1. PrerequisitesPython (>= v3.8)PyPi (>= v23)2. Install onion-config packageChoose one of the following methods to install the package[A ~ F]:A.[RECOMMENDED] Install fromPyPi# Install or upgrade package:# Pydantic-v1:pipinstall-Uonion-config[pydantic-v1]# Pydantic-v2:pipinstall-Uonion-config[pydantic-settings]B.Install latest version fromGitHub# Install package by git:# Pydantic-v1:pipinstallgit+https://github.com/bybatkhuu/module.python-config.git[pydantic-v1]# Pydantic-v2:pipinstallgit+https://github.com/bybatkhuu/module.python-config.git[pydantic-settings]C.Install frompre-built releasefilesDownload.whlor.tar.gzfile fromreleases-https://github.com/bybatkhuu/module.python-config/releasesInstall with pip:# Pydantic-v1:# Install from .whl file:pipinstall./onion_config-[VERSION]-py3-none-any.whl[pydantic-v1]# Or install from .tar.gz file:pipinstall./onion_config-[VERSION].tar.gz[pydantic-v1]# Pydantic-v2:# Install from .whl file:pipinstall./onion_config-[VERSION]-py3-none-any.whl[pydantic-settings]# Or install from .tar.gz file:pipinstall./onion_config-[VERSION].tar.gz[pydantic-settings]D.Install fromsource codeby building package# Clone repository by git:gitclonehttps://github.com/bybatkhuu/module.python-config.gitonion_configcd./onion_config# Install python build tool:pipinstall-Upipbuild# Build python package:python-mbuild_VERSION=$(./scripts/get-version.sh)# Pydantic-v1:# Install from .whl file:pipinstall./dist/onion_config-${_VERSION}-py3-none-any.whl[pydantic-v1]# Or install from .tar.gz file:pipinstall./dist/onion_config-${_VERSION}.tar.gz[pydantic-v1]# Pydantic-v2:# Install from .whl file:pipinstall./dist/onion_config-${_VERSION}-py3-none-any.whl[pydantic-settings]# Or install from .tar.gz file:pipinstall./dist/onion_config-${_VERSION}.tar.gz[pydantic-settings]E.Install with pip editabledevelopment mode(from source code)# Clone repository by git:gitclonehttps://github.com/bybatkhuu/module.python-config.gitonion_configcd./onion_config# Install with editable development mode:# Pydantic-v1:pipinstall-e.[pydantic-v1]# Pydantic-v2:pipinstall-e.[pydantic-settings]F.Manually add toPYTHONPATH(not recommended)# Clone repository by git:gitclonehttps://github.com/bybatkhuu/module.python-config.gitonion_configcd./onion_config# Install python dependencies:# Pydantic-v1:pipinstall-r./requirements.txt# Pydantic-v2:pipinstall-r./requirements.pydantic-v2.txt# Add current path to PYTHONPATH:exportPYTHONPATH="${PWD}:${PYTHONPATH}"Usage/ExamplesTo useonion_config, import theConfigLoaderclass from the package:fromonion_configimportConfigLoader,BaseConfigYou can create an instance ofConfigLoaderwithauto_loadflag. This will automatically load configuration data from environment variables and config files located in the default directory ('./configs'). The configuration data can then be accessed via theconfigproperty of theConfigLoaderinstance:config:BaseConfig=ConfigLoader(auto_load=True).configSimple.envENV=productionconfigs/1.base.yml:env:testapp:name:"MyApp"version:"0.0.1"nested:key:"value"configs/2.extra.yml:app:name:"NewApp"nested:some:"value"description:"Descriptionofmyapp."another_val:extra:1main.pyimportpprintfromloguruimportloggertry:importpydantic_settings_has_pydantic_settings=TrueexceptImportError:_has_pydantic_settings=Falsefromonion_configimportConfigLoader,BaseConfigclassConfigSchema(BaseConfig):env:str="local"try:config:ConfigSchema=ConfigLoader(config_schema=ConfigSchema).load()exceptException:logger.exception("Failed to load config:")exit(2)if__name__=="__main__":logger.info(f"All:{config}")logger.info(f"App name:{config.app['name']}")if_has_pydantic_settings:# Pydantic-v2:logger.info(f"Config:\n{pprint.pformat(config.model_dump())}\n")else:# Pydantic-v1:logger.info(f"Config:\n{pprint.pformat(config.dict())}\n")Run theexamples/simple:cd./examples/simple

python./main.pyOutput:2023-09-01 00:00:00.000 | INFO     | __main__:<module>:29 - All: env='production' another_val={'extra': 1} app={'name': 'New App', 'version': '0.0.1', 'nested': {'key': 'value', 'some': 'value'}, 'description': 'Description of my app.'}
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:30 - App name: New App
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:35 - Config:
{'another_val': {'extra': 1},
 'app': {'description': 'Description of my app.',
         'name': 'New App',
         'nested': {'key': 'value', 'some': 'value'},
         'version': '0.0.1'},
 'env': 'production'}Advanced.env.base:ENV=developmentDEBUG=trueAPP_NAME="Old App"ONION_CONFIG_EXTRA_DIR="extra_configs".env.prod:ENV=productionAPP_NAME="New App"APP_SECRET="my_secret"configs/config.yml:env:localapp:name:"MyApp"port:9000bind_host:"0.0.0.0"version:"0.0.1"ignore_val:"Ignoreme"logger:output:"file"configs/logger.json:{"logger":{"level":"info","output":"stdout"}}configs_2/config.yml:extra:config:key1:1configs_2/config_2.yml:extra:config:key2:2extra_configs/extra.json:{"extra":{"type":"json"}}schema.py:fromenumimportEnumfromtypingimportUnionimportpydanticfrompydanticimportField,SecretStr_has_pydantic_settings=Falseif"2.0.0"<=pydantic.__version__:try:frompydantic_settingsimportSettingsConfigDict_has_pydantic_settings=TrueexceptImportError:passfromonion_configimportBaseConfig# Environments as Enum:classEnvEnum(str,Enum):LOCAL="local"DEVELOPMENT="development"TEST="test"DEMO="demo"STAGING="staging"PRODUCTION="production"# App config schema:classAppConfig(BaseConfig):name:str=Field("App",min_length=2,max_length=32)bind_host:str=Field("localhost",min_length=2,max_length=128)port:int=Field(8000,ge=80,lt=65536)secret:SecretStr=Field(...,min_length=8,max_length=64)version:str=Field(...,min_length=5,max_length=16)description:Union[str,None]=Field(None,min_length=4,max_length=64)if_has_pydantic_settings:# Pydantic-v2:model_config=SettingsConfigDict(extra="ignore",env_prefix="APP_")else:# Pydantic-v1:classConfig:extra="ignore"env_prefix="APP_"# Main config schema:classConfigSchema(BaseConfig):env:EnvEnum=Field(EnvEnum.LOCAL)debug:bool=Field(False)app:AppConfig=Field(...)config.py:fromloguruimportloggerfromonion_configimportConfigLoaderfromschemaimportConfigSchema# Pre-load function to modify config data before loading and validation:def_pre_load_hook(config_data:dict)->dict:config_data["app"]["port"]="80"config_data["extra_val"]="Something extra!"returnconfig_dataconfig=Nonetry:_config_loader=ConfigLoader(config_schema=ConfigSchema,configs_dirs=["configs","configs_2","/not_exists/path/configs_3"],env_file_paths=[".env",".env.base",".env.prod"],pre_load_hook=_pre_load_hook,config_data={"base":"start_value"},warn_mode="ALWAYS",)# Main config object:config:ConfigSchema=_config_loader.load()exceptException:logger.exception("Failed to load config:")exit(2)main.py:importpprintfromloguruimportloggertry:importpydantic_settings_has_pydantic_settings=TrueexceptImportError:_has_pydantic_settings=Falsefromconfigimportconfigif__name__=="__main__":logger.info(f"All:{config}")logger.info(f"ENV:{config.env}")logger.info(f"DEBUG:{config.debug}")logger.info(f"Extra:{config.extra_val}")logger.info(f"Logger:{config.logger}")logger.info(f"App:{config.app}")logger.info(f"Secret: '{config.app.secret.get_secret_value()}'\n")if_has_pydantic_settings:# Pydantic-v2:logger.info(f"Config:\n{pprint.pformat(config.model_dump())}\n")else:# Pydantic-v1:logger.info(f"Config:\n{pprint.pformat(config.dict())}\n")try:# This will raise ValidationErrorconfig.app.port=8443exceptExceptionase:logger.error(f"{e}\n")Run theexamples/advanced:cd./examples/advanced

python./main.pyOutput:2023-09-01 00:00:00.000 | INFO     | onion_config._base:load:143 - Loading all configs...
2023-09-01 00:00:00.000 | WARNING  | onion_config._base:_load_dotenv_file:201 - '/home/user/workspaces/projects/onion_config/examples/advanced/.env' file is not exist!
2023-09-01 00:00:00.000 | WARNING  | onion_config._base:_load_configs_dir:257 - '/not_exists/path/configs_3' directory is not exist!
2023-09-01 00:00:00.000 | SUCCESS  | onion_config._base:load:171 - Successfully loaded all configs!
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:19 - All: env=<EnvEnum.PRODUCTION: 'production'> debug=True app=AppConfig(name='New App', bind_host='0.0.0.0', port=80, secret=SecretStr('**********'), version='0.0.1', description=None) extra={'config': {'key1': 1, 'key2': 2}, 'type': 'json'} extra_val='Something extra!' logger={'output': 'stdout', 'level': 'info'} base='start_value'
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:20 - ENV: production
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:21 - DEBUG: True
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:22 - Extra: Something extra!
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:23 - Logger: {'output': 'stdout', 'level': 'info'}
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:24 - App: name='New App' bind_host='0.0.0.0' port=80 secret=SecretStr('**********') version='0.0.1' description=None
2023-09-01 00:00:00.000 | INFO     | __main__:<module>:25 - Secret: 'my_secret'

2023-09-01 00:00:00.000 | INFO     | __main__:<module>:30 - Config:
{'app': {'bind_host': '0.0.0.0',
         'description': None,
         'name': 'New App',
         'port': 80,
         'secret': SecretStr('**********'),
         'version': '0.0.1'},
 'base': 'start_value',
 'debug': True,
 'env': <EnvEnum.PRODUCTION: 'production'>,
 'extra': {'config': {'key1': 1, 'key2': 2}, 'type': 'json'},
 'extra_val': 'Something extra!',
 'logger': {'level': 'info', 'output': 'stdout'}}

2023-09-01 00:00:00.000 | ERROR    | __main__:<module>:36 - "AppConfig" is immutable and does not support item assignmentRunning TestsTo run tests, run the following command:# Pydantic-v1:pipinstall-r./requirements.txt# Pydantic-v2:pipinstall-r./requirements.pydantic-v2.txt# Install python test dependencies:pipinstall-r./requirements.test.txt# Run tests:python-mpytest-vFAQWhat is the order of loading config?Load order:Load all dotenv files fromenv_file_pathsinto environment variables.Check if required environment variables exist or not.Load all config files fromconfigs_dirsintoconfig_data.Load extra config files fromextra_dirintoconfig_data.Executepre_load_hookmethod to modifyconfig_data.Initconfig_schemawithconfig_datainto finalconfig.Environment VariablesYou can use the following environment variables inside.env.examplefile:ONION_CONFIG_EXTRA_DIR="./extra_configs"DocumentationdocsscriptsReferenceshttps://docs.pydantic.devhttps://github.com/pydantic/pydantichttps://docs.pydantic.dev/latest/usage/pydantic_settingshttps://github.com/pydantic/pydantic-settingshttps://saurabh-kumar.com/python-dotenvhttps://github.com/theskumar/python-dotenvhttps://packaging.python.org/tutorials/packaging-projects