Package: conff
Simple config parser with evaluator library.Why Another Config Parser Module?This project inspired of the necessity complex config in a project. By means complex:ReusabilityImport values from fileReference values from other objectSecureEncrypt/decrypt sensitive valuesFlexibleMake logical expression to derive valuesCombine withjinja2template basedPowerfulAdd custom functions in PythonLink name data from PythonFeedback and DiscussionCome to Gitter channel to discuss, pass any feedbacks and suggestions. If you like to be contributor, please do let me know.Important NotesParsing Orderconff will only parse and resolve variable/names top to bottom order. Please ensure you arrange your configuration
in the same manner, there is no auto-dependencies resolver to handle complex and advanced names currently.dict vs collections.OrderedDictIn Python 3.5, the dict data type has inconsistent ordering, it isSTRONGLYrecommended to useOrderedDictif
you manually parse object. If you load from YAML file, the library already handled it. The reason of order is important,
this due to simplification and assumption of order execution. The library will parse the values from top to bottom as
per order in the key-value dictionary.Install[sudo]pipinstallconffBasic UsageTo get very basic parsing:Simple parseimportconffp=conff.Parser()r=p.parse({'math':'1 + 3'})assertr=={'math':4}Load YAML fileimportconffp=conff.Parser()r=p.load('path_of_file.yml')Template based configUsingjinja2to craft more powerful config.importconffp=conff.Parser()r=p.parse('F.template("{{ 1 + 2 }}")')assertr==3ExamplesMore advances examples:Parse with simple expressionimportconffp=conff.Parser()r=p.parse('1 + 2')assertr==3Parse objectimportconffp=conff.Parser()r=p.parse({"math":"1 + 2"})assertr=={'math':3}Ignore expression (declare it as string)importconffp=conff.Parser()r=conff.parse('"1 + 2"')assertr=='1 + 2'Parse error behavioursimportconffp=conff.Parser()r=p.parse({'math':'1 / 0'})# Exception raised# ZeroDivisionError: division by zeroimport filesimportconff## y1.yml# shared_conf: 1## y2.yml# conf: F.inc('y1.yml')p=conff.Parser()r=p.load('y2.yml')assertr=={'conf':{'shared_conf':1}}Parse with functionsimportconffdeffn_add(a,b):returna+bp=conff.Parser(fns={'add':fn_add})r=p.parse('F.add(1, 2)')assertr==3Parse with namesimportconffp=conff.Parser(names={'a':1,'b':2})r=conff.parse('a + b')assertr==3Parse with extendsimportconffdata={'t1':{'a':'a'},'t2':{'F.extend':'R.t1','b':'b'}}p=conff.Parser()r=p.parse(data)assertr=={'t1':{'a':'a'},'t2':{'a':'a','b':'b'}}Parse with updatesimportconffdata={'t1':{'a':'a'},'t2':{'b':'b','F.update':{'c':'c'},}}p=conff.Parser()r=p.parse(data)assertr=={'t1':{'a':'a'},'t2':{'b':'b','c':'c'}}Parse with extends and updatesimportconffdata={'t1':{'a':'a'},'t2':{'F.extend':'R.t1','b':'b','F.update':{'a':'A','c':'c'},}}p=conff.Parser()r=p.parse(data)assertr=={'t1':{'a':'a'},'t2':{'a':'A','b':'b','c':'c'}}Create a list of valuesThis creates a list of floats, similar to numpy.linspaceimportconffdata={'t2':'F.linspace(0, 10, 5)'}p=conff.Parser()r=p.parse(data)assertr=={'t2':[0.0,2.5,5.0,7.5,10.0]}This also creates a list of floats, but behaves like numpy.arange (although
slightly different in that it is inclusive of the endpoint).importconffdata={'t2':'F.arange(0, 10, 2)'}p=conff.Parser()r=p.parse(data)assertr=={'t2':[0,2,4,6,8,10]}Parse with for eachOne can mimic the logic of a for loop with the following exampleimportconffdata={'t1':2,'F.foreach':{'values':'F.linspace(0, 10, 2)',# You have access to loop.index, loop.value, and loop.length# within the template, as well as all the usual names'template':{'"test%i"%loop.index':'R.t1*loop.value','length':'loop.length'}}}p=conff.Parser()r=p.parse(data)assertr=={'length':3,'t1':2,'test0':0.0,'test1':10.0,'test2':20.0}EncryptionThis section to help you to quickly generate encryption key, initial encrypt values and test to decrypt the value.importconff# generate key, save it somewhere safenames={'R':{'_':{'etype':'fernet'}}}etype=conff.generate_key(names)()# or justekey=conff.generate_key()('fernet')# encrypt data# BIG WARNING: this should be retrieved somewhere secured for example in ~/.secret# below just for example purposesekey='FOb7DBRftamqsyRFIaP01q57ZLZZV6MVB2xg1Cg_E7g='names={'R':{'_':{'etype':'fernet','ekey':ekey}}}# gAAAAABbBBhOJDMoQSbF9jfNgt97FwyflQEZRxv2L2buv6YD_Jiq8XNrxv8VqFis__J7YlpZQA07nDvzYwMU562Mlm978uP9BQf6M9Priy3btidL6Pm406w=encrypted_value=conff.encrypt(names)('ACCESSSECRETPLAIN1234')# decrypt dataekey='FOb7DBRftamqsyRFIaP01q57ZLZZV6MVB2xg1Cg_E7g='names={'R':{'_':{'etype':'fernet','ekey':ekey}}}encrypted_value='gAAAAABbBBhOJDMoQSbF9jfNgt97FwyflQEZRxv2L2buv6YD_Jiq8XNrxv8VqFis__J7YlpZQA07nDvzYwMU562Mlm978uP9BQf6M9Priy3btidL6Pm406w='conff.decrypt(names)(encrypted_value)Real World ExamplesAll the example below located indata directory.
Imagine you start an important project, your code need to analyse image/videos which involves workflow
with set of tasks with AWS Rekognition. The steps will be more/less like this:Read images/videos from a specific folder, if images goes to (2), if videos goes to (3).Analyse the images with AWS API, then goes (4)Analyse the videos with AWS API, then goes (4)Write the result back to JSON file, finishedThe configuration required:Read images/videos (where is the folder)Analyse images (AWS API credential and max resolution for image)Analyse videos (AWS API credential and max resolution for video)Write results (where is the result should be written)1. Without conff libraryFile:data/sample_config_01.ymlWhere it is all started, if we require to store the configuration as per normally, it should be like this.job:read_image:# R01root_path:/data/project/images_and_videos/analyse_image:# R02api_cred:region_name:ap-southeast-2aws_access_key_id:ACCESSKEY1234# R03aws_secret_access_key:ACCESSSECRETPLAIN1234max_res:[1024,768]analyse_video:# R04api_cred:region_name:ap-southeast-2aws_access_key_id:ACCESSKEY1234aws_secret_access_key:ACCESSSECRETPLAIN1234max_res:[800,600]write_result:# R05output_path:/data/project/result.jsonimportyamlwithopen('data/sample_config_01.yml')asstream:r1=yaml.safe_load(stream)Notes:R01: The subpath of “/data/project” is repeated between R01 and R05R02: api_cred is repeatedly defined with R04R03: the secret is plain visible, if this stored in GIT, it is pure disaster2. Fix the repeatFile:data/sample_config_02.ymlRepeating values/configuration is bad, this could potentially cause human mistake if changes made is not
consistently applied in all occurences.# this can be any name, as long as not reserved in Pythonshared:project_path:/data/projectaws_cred:region_name:ap-southeast-2aws_access_key_id:ACCESSKEY1234# F03aws_secret_access_key:F.decrypt('gAAAAABbBBhOJDMoQSbF9jfNgt97FwyflQEZRxv2L2buv6YD_Jiq8XNrxv8VqFis__J7YlpZQA07nDvzYwMU562Mlm978uP9BQf6M9Priy3btidL6Pm406w=')job:read_image:# F01root_path:R.shared.project_path + '/images_and_videos/'analyse_image:# F02api_cred:R.shared.aws_credmax_res:[1024,768]analyse_video:# F04api_cred:R.shared.aws_credmax_res:[800,600]write_result:# F05output_path:R.shared.project_path + '/result.json'importconff# ekey is the secured encryption key# WARNING: this is just demonstration purposesekey='FOb7DBRftamqsyRFIaP01q57ZLZZV6MVB2xg1Cg_E7g='r2=conff.load(fs_path='data/sample_config_02.yml',params={'ekey':ekey})Notes:F01: it is safe if the prefix ‘/data/project’ need to be changed, it will automatically changed for F05F02: no more duplicated config with F04F03: it is secured to save this to GIT, as long as the encryption key is stored securely somewhere in server such
as ~/.secret3. Optimise to the extremeFile:data/sample_config_03.ymlThis is just demonstration purposes to see the full capabilities of this library.# this can be any name, as long as not reserved in Pythonshared:project_path:/data/projectanalyse_image_video:api_cred:region_name:ap-southeast-2aws_access_key_id:ACCESSKEY1234aws_secret_access_key:F.decrypt('gAAAAABbBBhOJDMoQSbF9jfNgt97FwyflQEZRxv2L2buv6YD_Jiq8XNrxv8VqFis__J7YlpZQA07nDvzYwMU562Mlm978uP9BQf6M9Priy3btidL6Pm406w=')max_res:[1024,768]job:read_image:root_path:R.shared.project_path + '/images_and_videos/'analyse_image:R.shared.analyse_image_videoanalyse_video:F.extend:R.shared.analyse_image_videoF.update:max_res:[800,600]write_result:output_path:R.shared.project_path + '/result.json'For completeness, ensuring data is consistent and correct between sample_config_01.yml, sample_config_02.yml
and sample_config_03.yml.# nose2 conff.test.ConffTestCase.test_samplefs_path='data/sample_config_01.yml'withopen(fs_path)asstream:r1=yaml.safe_load(stream)fs_path='data/sample_config_02.yml'ekey='FOb7DBRftamqsyRFIaP01q57ZLZZV6MVB2xg1Cg_E7g='r2=conff.load(fs_path=fs_path,params={'ekey':ekey})fs_path='data/sample_config_03.yml'r3=conff.load(fs_path=fs_path,params={'ekey':ekey})self.assertDictEqual(r1['job'],r2['job'],'Mismatch value')self.assertDictEqual(r2['job'],r3['job'],'Mismatch value')TestTo test this project:# default testnose2# test with coveragenose2--with-coverage# test specificnose2conff.test.ConffTestCase.test_sampleTODO[X] Setup basic necessity[X] Stop procrastinating[X] Project registration in pypi[X] Create unit tests[X] Setup travis[X] Setup coveralls[ ] Add more support onPython versions[ ] 2.7[ ] 3.4[X] 3.5[X] 3.6[ ] FeaturesWish List Features now moved towiki page.[ ] Improve docs[ ] Add more code comments and visibilities[ ] Make github layout code into two left -> right[X] Put more examples[ ] Setup readthedocs[ ] Add code conduct, issue template into git project.[ ] Add information that conff currently accept YML and it not limited, it can take any objectsOther Open SourceThis project uses other awesome projects:cryptographyjinja2munchsimpleevalyamlWho uses conff?Please send a PR to keep the list growing, if you may please add your handle and company.