Package: pyaml-env
Python YAML configuration with environment variables parsingTL;DRA very small library that parses a yaml configuration file and it resolves the environment variables,
so that no secrets are kept in text.Installpipinstallpyaml-envHow to use:Basic Usage: Environment variable parsingThis yaml file:database:name:test_dbusername:!ENV${DB_USER}password:!ENV${DB_PASS}url:!ENV'http://${DB_BASE_URL}:${DB_PORT}'given that we've set these:export$DB_USER=super_secret_userexport$DB_PASS=extra_super_secret_passwordexport$DB_BASE_URL=localhostexport$DB_PORT=5432becomes this:frompyaml_envimportparse_configconfig=parse_config('path/to/config.yaml')print(config)# outputs the following, with the environment variables resolved{'database':{'name':'test_db','username':'super_secret_user','password':'extra_super_secret_password','url':'http://localhost:5432',}}Attribute Access usingBaseConfigWhich can also become this:frompyaml_envimportparse_config,BaseConfigconfig=BaseConfig(parse_config('path/to/config.yaml'))# you can then access the config properties as atrributes# I'll explain why this might be useful in a bit.print(config.database.url)Default Values with:You can also set default values for when the environment variables are not set for some reason,
using thedefault_sepkwarg (which is:by default) like this:databse:name:test_dbusername:!ENV${DB_USER:paws}password:!ENV${DB_PASS:meaw2}url:!ENV'http://${DB_BASE_URL:straight_to_production}:${DB_PORT}'And if no environment variables are found then we get:frompyaml_envimportparse_configconfig=parse_config('path/to/config.yaml')print(config){'database':{'name':'test_db','username':'paws','password':'meaw2','url':'http://straight_to_production:N/A',}}NOTE 0: Special characters like*,{etc. are not currently supported as separators. Let me know if you'd like them handled also.NOTE 1: If you settagtoNone, then, the current behavior is that environment variables in all places in the yaml will be resolved (if set).Datatype parsing with yaml's tag:yaml.org,2002:# because this is not allowed:# data1: !TAG !!float ${ENV_TAG2:27017}# use tag:yaml.org,2002:datatype to convert value:test_data='''data0: !TAG ${ENV_TAG1}data1: !TAG tag:yaml.org,2002:float ${ENV_TAG2:27017}data2: !!float 1024data3: !TAG ${ENV_TAG2:some_value}data4: !TAG tag:yaml.org,2002:bool ${ENV_TAG2:false}'''Will become:os.environ['ENV_TAG1']="1024"config=parse_config(data=test_data,tag='!TAG')print(config){'data0':'1024','data1':27017.0,'data2':1024.0,'data3':'some_value','data4':False}reference in yaml codeIf nothing matches:N/Aasdefault_value:If no defaults are found and no environment variables, thedefault_value(which isN/Aby default)  is used:{'database':{'name':'test_db','username':'N/A','password':'N/A','url':'http://N/A:N/A',}}Which, of course, means something went wrong and we need to set the correct environment variables.
If you want this process to fail if adefault valueis not found, you can set theraise_if_naflag toTrue.
For example:test1:data0:!TEST${ENV_TAG1:has_default}/somethingelse/${ENV_TAG2:also_has_default}data1:!TEST${ENV_TAG2}will raise aValueErrorbecausedata1:  !TEST ${ENV_TAG2}there is no default value forENV_TAG2in this line.Using a different loader:The default yaml loader isyaml.SafeLoader. If you need to work with serialized Python objects,
you can specify a different loader.So given a class:classOtherLoadTest:def__init__(self):self.data0='it works!'self.data1='this works too!'Which has become a yaml output like the following usingyaml.dump(OtherLoadTest()):!!python/object:__main__.OtherLoadTestdata0:it works!data1:this works too!You can useparse_configto load the object like this:importyamlfrompyaml_envimportparse_configother_load_test=parse_config(path='path/to/config.yaml',loader=yaml.UnsafeLoader)print(other_load_test)<__main__.OtherLoadTestobjectat0x7fc38ccd5470>Long story: Load a YAML configuration file and resolve any environment variablesIf you’ve worked with Python projects, you’ve probably have stumbled across the many ways to provide configuration. I am not going to go through all the ways here, but a few of them are:using .ini filesusing a python classusing .env filesusing JSON or XML filesusing a yaml fileAnd so on. I’ve put some useful links about the different ways below, in case you are interested in digging deeper.My preference is working with yaml configuration because I usually find very handy and easy to use and I really like that yaml files are also used in e.g. docker-compose configuration so it is something most are familiar with.For yaml parsing I use thePyYAMLPython library.In this article we’ll talk about the yaml file case and more specifically what you can do toavoid keeping your secrets, e.g. passwords, hosts, usernames etc, directly on it.Let’s say we have a very simple example of a yaml file configuration:database:
 name: database_name
 user: me
 password: very_secret_and_complex
 host: localhost
 port: 5432

ws:
 user: username
 password: very_secret_and_complex_too
 host: localhostWhen you come to a point where you need to deploy your project, it is not really safe to have passwords and sensitive data in a plain text configuration file lying around on your production server. That’s where**environment variables**come in handy. So the goal here is to be able to easily replace the very_secret_and_complex password with input from an environment variable, e.g. DB_PASS, so that this variable only exists when you set it and run your program instead of it being hardcoded somewhere.For PyYAML to be able to resolve environment variables, we need three main things:A regex pattern for the environment variable identification e.g. pattern = re.compile(‘.?${(\w+)}.?’)A tag that will signify that there’s an environment variable (or more) to be parsed, e.g. !ENV.And a function that the loader will use to resolve the environment variablesdefconstructor_env_variables(loader,node):"""Extracts the environment variable from the node's value:param yaml.Loader loader: the yaml loader:param node: the current node in the yaml:return: the parsed string that contains the value of the environmentvariable"""value=loader.construct_scalar(node)match=pattern.findall(value)ifmatch:full_value=valueforginmatch:full_value=full_value.replace(f'${{{g}}}',os.environ.get(g,g))returnfull_valuereturnvalueExample of a YAML configuration with environment variables:database:
 name: database_name
 user: !ENV ${DB_USER}
 password: !ENV ${DB_PASS}
 host: !ENV ${DB_HOST}
 port: 5432

ws:
 user: !ENV ${WS_USER}
 password: !ENV ${WS_PASS}
 host: !ENV ‘[https://${CURR_ENV}.ws.com.local'](https://${CURR_ENV}.ws.com.local')This can also workwith more than one environment variablesdeclared in the same line for the same configuration parameter like this:ws:
 user: !ENV ${WS_USER}
 password: !ENV ${WS_PASS}
 host: !ENV '[https://${CURR_ENV}.ws.com.](https://${CURR_ENV}.ws.com.local')[${MODE}](https://${CURR_ENV}.ws.com.local')'  # multiple env varAnd how to use this:First set the environment variables. For example, for the DB_PASS :export DB_PASS=very_secret_and_complexOr even better, so that the password is not echoed in the terminal:read -s ‘Database password: ‘ db_pass
export DB_PASS=$db_pass# To run this:# export DB_PASS=very_secret_and_complex# python use_env_variables_in_config_example.py -c /path/to/yaml# do stuff with conf, e.g. access the database password like this: conf['database']['DB_PASS']if__name__=='__main__':parser=argparse.ArgumentParser(description='My awesome script')parser.add_argument("-c","--conf",action="store",dest="conf_file",help="Path to config file")args=parser.parse_args()conf=parse_config(path=args.conf_file)Then you can run the above script:pythonuse_env_variables_in_config_example.py-c/path/to/yamlAnd in your code, do stuff with conf, e.g. access the database password like this:conf['database']['DB_PASS']I hope this was helpful. Any thoughts, questions, corrections and suggestions are very welcome :)Useful linksThe Many Faces and Files of Python Configs*As we cling harder and harder to Dockerfiles, Kubernetes, or any modern preconfigured app environment, our dependency…*hackersandslackers.com4 Ways to manage the configuration in Python*I’m not a native speaker. Sorry for my english. Please understand.*hackernoon.comPython configuration files*A common need when writing an application is loading and saving configuration values in a human-readable text format…*www.devdungeon.comConfiguration files in Python*Most interesting programs need some kind of configuration: Content Management Systems like WordPress blogs, WikiMedia…*martin-thoma.com