Package: confgetti
Tool for configuration and variables management.Fetch variables for your application easily fromConsul KV➡️config*.json➡️environment.via simple python method/s!ContentInstallation and QuickStartInstall with pipGet a single variableGet multiple variablesOverride module variablesThe ProblemKing of n00bs waySlightly less n00b wayThe SolutionLogic flowConsul settingsThrough environment variablesUpon initializationAPIShorthand methodsget_variablesload_and_validate_configConfgettiget_variableget_variablesDemosDeveloper NotesReleasing new versionInstallation and QuickStartInstallConfgettiwithpip:pip install confgettiGet a single variable:# my_app/config.pyfromConfgettiimportget_variablecgtti=Confgetti({'host':'consul_instance_host'})my_variable=cgtti.get_variable('MY_VARIABLE')Get multiple variables:# my_app/config.pyfromConfgettiimportget_variablecgtti=Confgetti({'host':'consul_instance_host'})my_variables_dict=cgtti.get_variables(keys=['MY_VARIABLE','YOUR_VARIABLE','OUR_VARIABLE'])Override module variables:ENV CONSUL_HOST=consul_instance_hostvariables underMY_APPnamespace in consul# my_app/config.pyfromvoluptuousimportSchema,Coercefromconfgettiimportload_and_validate_configmy_variable=Noneyour_variable=None_schema=Schema({"my_variable":str,"your_variable":Coerce(int)})load_and_validate_config(__name__,'MY_APP',_schema)# my_app/some_logic.pyfrom.configimportmy_variable,your_variableprint(my_variable)# should be a string and not Noneprint(your_variable# should be an integer and not NoneThe ProblemModern web app development and deployment often consider isolated app environments that are easily manageble and quickly deployed with software asVMs orDocker.As your app gets bigger and needs more and moresettingsvariables declared in your configuration modules or classes, management of those variables becomes frustrating, especially for those who manage the production state of the app or multiple apps and do not care about actual application code.Imagine a simple web application that uses a database for data storage, cache mechanism, and AWS S3 Bucket static file storage. Oh yes, and our app is Dockerized.To run that app successfully usually you need to pass configuration variables to methods/drivers that are communicating with those services. So at least, you'll need:DatabaseDatabase nameHostUsernamePasswordPort?CacheHostIndex?Username?Password?S3 BucketAWS secret access keyAWS access key idBucket nameSo for just3external services, we could end with up to12different settings variables that are crucial for the successful running of our simple web app.How do you deal with a problem like ann00b?1. King of n00bs wayYou could always leave those variable hardcoded to your configuration module, but perhaps, what if suddenly you need to switch to different AWS user and use the different bucket?We even do not touch application code or logic actually but we need to:push those changes to our repository (wait for a Merge request, eh?)build a new Docker imagedeploy a new imageWhole deployment process just because  of those3simple variables. Not to mention that there are some sensitive data in those12variables, so storing them inside codebase isNEVERa good idea andALWAYSsecurity issue.2. Slightly less n00b wayThe most common way of variables management, especially in the Docker world, is to assign necessary variables into the container environment via Docker runner and then get them from application code by checking the value of some agreed and known environment key name.If we put aside the security problem of such an approach (yes, environment variables could be readable by a malicious user), there is still one more common and frustrating problem: A bunch of sensitive variables that need to be correctly passed each time as our Docker container is restarted or redeployed.Each time you need to pass those12variables todockercommand, and even withdocker-composeyou still need to declare those variables in thedocker-compose.ymlfile which returns us to previous"King of n00bs" way.Do not forget, we are dealing with just one simple web app. Imagine the size of the problem on some cluster of web apps. Your DevOps(in most cases you) will hate you.The SolutionHere comesConfgettito save a day! 🎉🎉🎉ConfgettiusesConsulkey/value storage for setting and getting your variables.
If you have running consul instance andMY_VARIABLEexists in its KV, you can get it simple as that:fromConfgettiimportget_variablecgtti=Confgetti({'host':'consul_instance_host'})my_variable=cgtti.get_variable('MY_VARIABLE')Maybe you still want to store some or all variables into environment?No problem!Confgettican get a variable from your environment also.So now we set environment variableMY_VARIABLEwith some custom value.How to get a variable from the environment?With the sameget_variablemethod used in the example above.No need for extra setup, custom code or monkey patching and it is because ofConfgettiefficient logic flow.Logic flowConfgettitries to fetch a variable from two different sources in order, overriding the previous source result.
When you ask for a variable withget_variable, the lookup is made in the following order:Consul⬇️environment⬇️AppSo if you haveMY_VARIABLEkey stored in consul and in the environment,Confgettiwill return the value
stored in the environment (if you do not tellConfgettiotherwise.).Confgettidoes not punish you if you do not have the Consul server running, it will still return value from the environment variable!Slightlyhigh-levelfunctionload_and_validate_config, that is used for fetching multiple variables at once and overriding declared module variables, will try to get variable from one extra
source, the local JSON configuration file in the following order:Consul⬇️config.json⬇️environment⬇️AppWith the sameoverridelogic.Consul settingsConfgettiuses apython-consulpackage for communication with Consul's KV store.Default connection settings are:host:consulport:8500scheme:httpConnection settings can be configured in2ways:Through environment variablesAvailable environment variables:CONSUL_HOST - default: 'consul'
CONSUL_PORT - default: 8500
CONSUL_SCHEME - default: 'http'
CONSUL_TOKEN - default: None
CONSUL_DC - default: NoneExampleYou have running consul instance onmy_host, port7500, and on securedhttps,
all you need to set the following environment variables:CONSUL_HOST=my_host
CONSUL_PORT=7500
CONSUL_SCHEME=httpsAnd you do not have to pass any configuration dictionary when initializing Confgetti, because
it will read settings from the environment.# my_app/config.pyfromConfgettiimportget_variablecgtti=Confgetti()my_variable=cgtti.get_variable('MY_VARIABLE')!!!ALERT:This is only way of configuration forload_and_validate_configshorthand use.Upon initializationWhen initializingConfgettiinstance you can pass a dictionary withConsnulconnection settings.Default configuration dictionaryconsul_settings={'host':'consul','port':8500,'scheme':'http','token':None,'dc':None}ExampleYou have running consul instance onmy_host, port7500, and on securedhttps:# my_app/config.pyfromConfgettiimportget_variablecgtti=Confgetti({'host':'my_host''port':7500,'scheme':'https'})my_variable=cgtti.get_variable('MY_VARIABLE')APIShorthand methods!!!ALERT:If you are using shorthand functions, make sure that you have providedConsulconnection settings via environment variables!confgetti.get_variables(path, keys, use_env=True, use_consul=True)This is shorthand function forconfgetti.Confgetti.get_variables.
Used for fetching multiple variables at once fromConsulor environment. Returns dictionary for fetched variables.Arguments:path(optional) - Namespace of variable location insideConsulKV storage. By default asNone, it looks to root of KV for the variable.keys- list of keys under which variables are defined. This can be a plainlistof names, ordictwhere the key is the key name of variable and value type of value that should be returned. By default, variables are returned as astring. Available types:str,int,bool,float,dict. For example:my_variables_dict=cgtti.get_variables(keys=['MY_VARIABLE','YOUR_VARIABLE','OUR_VARIABLE'])ormy_variables_dict=cgtti.get_variables(keys={'MY_VARIABLE':str,'YOUR_VARIABLE':bool,'OUR_VARIABLE':int})use_env(optional) - shouldConfgettilook to environment or no?use_consul(optional) - shouldConfgettilook toConsulor no?Example:fromconfgettiimportget_variablesconvert_dict={"my_variable":str,"your_variable":int,"my_bool":bool,"my_env_variable":str}variables=get_variables(path='AWESOMEAPP',keys=convert_dict)confgetti.load_and_validate_config(config_module_name, env_var, schema=None, keys=None, uppercase=False)Used for overriding current module variables. Usually it is used withvoluptuous.Schemaasschemaargument for validation, but can be used without it, or with some custom method.Arguments:config_module_name- Usually__name__variable from current configuration fileenv_var- Prefix of variables in the environment, the namespace of variables location inCONSULand key under whichJSONconfiguration path is stored in the environment. See the example for better understanding.schema(optional) - Pass custom method here that should returndictof variables which will be glued to module later. Usually used withvoluptuous.Schema.keys(optional) - variable names list or dict with the desired type. If you do not pass this, and you passvoluptuous.Schemaunderschemaargument, the method will return variables declared dict passed toSchemainstance.uppercase(optional) - By default, variables aregluedto the module in lowercase. If this is passed asTrue, variables will begluedin uppercase.Example:variables must be defined underMY_APPnamespace in consul*ifconfiguration.jsonis used, environment variableMY_APPmust be set with a path to the file# my_app/config.pyfromvoluptuousimportSchema,Coercefromconfgettiimportload_and_validate_configmy_variable=Noneyour_variable=None_schema=Schema({"my_variable":str,"your_variable":Coerce(int)})load_and_validate_config(__name__,'MY_APP',_schema)# my_app/some_logic.pyfrom.configimportmy_variable,your_variableprint(my_variable)# should be a string and not Noneprint(your_variable# should be an integer and not Noneconfgetti.Confgetti(consul_config=None, prepare_consul=True)Confgetti intialization accepts two optional arguments, both refering to communication
with Consul.Arguments:consul_config(optional) - if dictionary with connection settings is passed client for communication with consul is initialized wit these settings.Example:fromconfgettiimportConfgetticgtti=Confgetti({'host':'localhost'})prepare_consul(optional) - ifFalseis passed, connection to Consul instance is not prepared, and Confgetti will only seek for variables in environment.Example:fromconfgettiimportConfgetticgtti=Confgetti(prepare_consul=False)cggti.get_variable('my_variable')# only environment lookupconfgetti.Confgetti.get_variable(key, path=None, fallback=None, convert_to=None, use_env=True, use_consul=True)Used for fetching single variable fromConsulor environment. Returns single variable valueArguments:key- key under which variable is definedpath(optional) - Namespace of variable location insikeConsulKV storage. By default asNoneit looks to root of KV for variable.fallback(optional) - what is returned if variable is not foundconvert_to(optional) - should variable be converted to certain type? If type is passed,Confgettitries to convert variable to passed type. By default, variable is returend asstring. Available types:str,int,bool,float,dictuse_env(optional) - shouldConfgettilook to environment or no?use_consul(optional) - shouldConfgettilook toConsulor no?Example:fromconfgettiimportConfgetticgtti=Confgetti()my_variable=cgtti.get_variable('my_variable')confgetti.Confgetti.get_variables(path=None, keys=None, use_env=True, use_consul=True)This is internal method that is used forget_variablesshorthand. Arguments and logic is exactly the same.Example:fromconfgettiimportConfgetticgtti=Confgetti()my_variable=cgtti.get_variables(['my_variable','your_variable'])DemosCheckdemosfolder for example usages as simple python scripts.Developer NotesReleasing new versionThis repository has automatic deployment configured via CI runner.Steps for automatic deployment toStyria's PyPI server:UpdateCHANGELOG.mdwith future version release notesCommit and push changelog tomasterbranchFrom the root of repository runbumpversion patchPush changes that command in the previous step has made to the repoWait forCI/CDpipelinedeploystep is finished, done!