Package: himl
himlA hierarchical config using yaml in Python.Latest version is: 0.15.1DescriptionA python module which allows you to merge hierarchical config files using YAML syntax. It offers deep merge, variable interpolation and secrets retrieval from secrets managers.It is ideal if you want to structure your hierarchy in such a way that you avoid duplication. You can define a structure for your configuration using a hierarchy such environment/project/cluster/app. It is up to you what layers you want to use in this hierarchy. The tool will read all yaml files starting from the root (where default values would be) all the way to the leaf (where most specific values would be, which will take precedence).Idea came from puppet's hiera.Table of ContentshimlDescriptionTable of ContentsInstallationUsingpipUsingdockerimageFrom SourceExamplesUsing the python moduleUsing the cliFeaturesInterpolationInterpolating simple valuesInterpolating wholedictDeep mergeSecrets retrievalAWS SSMAWS S3VaultMerge with Terraform remote stateMerge with env variableshiml config mergerExtra merger featuresInstallationUsingpippipinstallhimlUsingdockerimagedockerrunghcr.io/adobe/himl:latesthiml-config-merger--helpSee all docker tags at:https://github.com/adobe/himl/pkgs/container/himl/versionsFrom Sourcegit clone https://github.com/adobe/himl
cd himl
sudo python install -e .ExamplesUsing the python moduleThis will merge simple/default.yaml with simple/production/env.yamlfromhimlimportConfigProcessorconfig_processor=ConfigProcessor()path="examples/simple/production"filters=()# can choose to output only specific keysexclude_keys=()# can choose to remove specific keysoutput_format="yaml"# yaml/jsonconfig_processor.process(path=path,filters=filters,exclude_keys=exclude_keys,output_format=output_format,print_data=True)The above example will mergesimple/default.yamlwithsimple/production/env.yaml:$ tree examples/simple
examples/simple
├── default.yaml
└── production
    └── env.yamlThe example also showcases deep merging of lists and maps.examples/simple/default.yaml---env:defaultdeep:key1:v1key2:v2deep_list:-item1-item2examples/simple/production/env.yaml---env:proddeep:key3:v3deep_list:-item3Result:env:proddeep:key1:v1key2:v2key3:v3deep_list:-item1-item2-item3Using the cliA cli tool calledhimlis automatically installed viapip. You can use it to parse a tree of yamls and it will either output the combined configuration at standard output or write it to a file.usage:himl[-h][--output-fileOUTPUT_FILE][--formatOUTPUT_FORMAT][--filterFILTER][--excludeEXCLUDE][--skip-interpolation-validation][--skip-interpolation-resolving][--enclosing-keyENCLOSING_KEY][--cwdCWD][--list-merge-strategy{append,override,prepend,append_unique}]pathhimlexamples/complex/env=dev/region=us-east-1/cluster=cluster2Based on the configuration tree from theexamples/complexfolder, the output of the above command will be the following:cluster:
  description: 'This is cluster: cluster2. It is using c3.2xlarge instance type.'
  name: cluster2
  node_type: c3.2xlarge
region:
  location: us-east-1
env: devWhere the examples folder looks something like this:$ tree examples/complex
examples/complex
├── default.yaml
├── env=dev
│   ├── env.yaml
│   ├── region=us-east-1
│   │   ├── cluster=cluster1
│   │   │   └── cluster.yaml
│   │   ├── cluster=cluster2
│   │   │   └── cluster.yaml
│   │   └── region.yaml
│   └── region=us-west-2
│       ├── cluster=cluster1
│       │   └── cluster.yaml
│       └── region.yaml
└── env=prod
    ├── env.yaml
    └── region=eu-west-2
        ├── cluster=ireland1
        │   └── cluster.yaml
        └── region.yamlFeaturesInterpolationIn order to avoid repetition, we wanted to make it possible to define a value once and reuse it in other parts of the yaml config.
Unlike yaml anchors, these interpolations work across multiple files.Interpolating simple valuesdata/default.yaml:allowed_roles:-"arn:aws:iam::{{account.id}}:role/myrole"data/dev/env.yaml:account:
  id: "123456"Interpolating wholedictprojects:webapp1:tagging:Owner:"WebServiceTeam"Environment:"dev"CostCenter:"123"data-store:Owner:"BackendTeam"Environment:"dev"CostCenter:"455"# this will copy the whole projects.webapp1.tagging dict to this keytagging:"{{projects.webapp1.tagging}}"# or even a double interpolationtagging:"{{projects.{{project.name}}.tagging}}"Deep mergeIt's possible to have the same key (eg. a dict/list) in multiple files and combine them using a deep merge.
See an examplehere.Secrets retrievalAWS SSMpassphrase:"{{ssm.path(/key/coming/from/aws/secrets/store/manager).aws_profile(myprofile)}}"AWS S3my_value:"{{s3.bucket(my-bucket).path(path/to/file.txt).base64encode(true).aws_profile(myprofile)}}"VaultUse vault cli to authenticate, fallback method via LDAP.Retrieve only one key value from a secret, the path tail is used as key:my_value:"{{vault.key(/path/from/vault/key)}}"Retrieve all key/value pairs from a vault path:my_dict:"{{vault.path(/path/from/vault)}}"Generate a token for a policy:my_token:"{{vault.token_policy(my_vault_policy)}}"Merge withTerraform remote state### Terraform remote states ###remote_states:-name:cluster_compositiontype:terraformaws_profile:"my_aws_profile"s3_bucket:"my_terraform_bucket"s3_key:"mycluster.tfstate"endpoint:"{{outputs.cluster_composition.output.value.redis_endpoint}}"Merge with env variableskubeconfig_location:"{{env(KUBECONFIG)}}"himl-config-mergerThehiml-config-mergerscript, contains logic of merging a hierarchical config directory and creating the end result YAML files.himl-config-mergerexamples/complex--output-dirmerged_output--levelsenvregioncluster--leaf-directoriesclusterINFO:__main__:Found input config directory: examples/complex/env=prod/region=eu-west-2/cluster=ireland1
INFO:__main__:Storing generated config to: merged_output/prod/eu-west-2/ireland1.yaml
INFO:__main__:Found input config directory: examples/complex/env=dev/region=us-west-2/cluster=cluster1
INFO:__main__:Storing generated config to: merged_output/dev/us-west-2/cluster1.yaml
INFO:__main__:Found input config directory: examples/complex/env=dev/region=us-east-1/cluster=cluster1
INFO:__main__:Storing generated config to: merged_output/dev/us-east-1/cluster1.yaml
INFO:__main__:Found input config directory: examples/complex/env=dev/region=us-east-1/cluster=cluster2
INFO:__main__:Storing generated config to: merged_output/dev/us-east-1/cluster2.yamlInput example:> tree examples/complex
examples/complex
├── default.yaml
├── env=dev
│   ├── env.yaml
│   ├── region=us-east-1
│   │   ├── cluster=cluster1
│   │   │   └── cluster.yaml
│   │   ├── cluster=cluster2
│   │   │   └── cluster.yaml
│   │   └── region.yaml
│   └── region=us-west-2
│       ├── cluster=cluster1
│       │   └── cluster.yaml
│       └── region.yaml
└── env=prod
    ├── env.yaml
    └── region=eu-west-2
        ├── cluster=ireland1
        │   └── cluster.yaml
        └── region.yamlOutput:merged_output
├── dev
│   ├── us-east-1
│   │   ├── cluster1.yaml
│   │   └── cluster2.yaml
│   └── us-west-2
│       └── cluster1.yaml
└── prod
    └── eu-west-2
        └── ireland1.yamlLeveraging HIML, the config-merger script loads the configs tree structure and deep-merges all keys from all YAML files found from a root path to an edge. For each leaf directory, a file will be created under--output-dir.Under each level, there is a mandatory "level key" that is used by config-merger for computing the end result. This key should be present in one of the files under each level. (eg. env.yaml under env).Output filteringSome configs that are specified in the higher levels of the directory tree might not be needed in the end (leaf) result. For this reason, the config-merger script can apply a set of filter rules that are specified via the--filter-rules-keyparameter. This property must be present in the config and contains rules for removing root level keys from the output. The filter is applied if the selector object matches a subset of the output keys and will keep the keys specified in thevalueslist or the keys that match theregexpattern.# intermediate config after hierarchical mergeenv:devcluster:cluster1region:us-east-1key1:persistedkey2:droppedkeep_1:persistedtags:cost_center:123_filters:-selector:env:"dev"keys:values:-key1regex:"keep_.*"-selector:cluster:regex:"cluster1"keys:values:-tagsBuild the output with filtering:himl-config-mergerexamples/filters--output-dirmerged_output--levelsenvregioncluster--leaf-directoriescluster--filter-rules-key_filters# output after filteringenv:devcluster:cluster1region:us-east-1key1:persistedkeep_1:persistedtags:cost_center:123Filtering limitationsRule selectors and keys filtering only works at the root level of the config. It is not possible to filter nested keys.Extra merger featuresApart from the standard features found in thePyYamllibrary, thehiml-config-mergercomponent also implements a custom YAML tag called!include.Example:VA7:!includeconfigs/env=int/region=va7/kafka-brokers.yaml regionBrokers.VA7This will replace the value after interpolation with the value of the regionBrokers.VA7 found under the configs/env=int/region=va7/kafka-brokers.yaml path.Custom merge strategyAn optional parametertype_strategiescan be passed into ConfigProcessor to define custom merging behavior. It could be custom functions that fit your needs.
Your function should take the arguments of (config, path, base, nxt) and return the merged result.Example:fromhimlimportConfigProcessordefstrategy_merge_override(config,path,base,nxt):"""merge list of dicts. if objects have same id, nxt replaces base.""""""if remove flag is present in nxt item, remove base and not add nxt"""result=deepcopy(base)fornxtoinnxt:forbaseoinresult:# if list is not a list of dicts, bail out and let the next strategy to executeifnotisinstance(baseo,dict)ornotisinstance(nxto,dict):returnSTRATEGY_ENDif'id'inbaseoand'id'innxtoandbaseo['id']==nxto['id']:result.remove(baseo)#same id, remove previous itemif'remove'notinnxto:result.append(nxto)returnresultconfig_processor=ConfigProcessor()path="examples/simple/production"filters=()# can choose to output only specific keysexclude_keys=()# can choose to remove specific keysoutput_format="yaml"# yaml/jsonconfig_processor.process(path=path,filters=filters,exclude_keys=exclude_keys,output_format=output_format,print_data=True,type_strategies=[(list,[strategy_merge_override,'append']),(dict,["merge"])]))