Is Python the Perfect JSON/YAML Templating Engine?
By: Anthony Hawkins
Tags: Templating, Yaml, Software Development, DevOps, Json

Python, I Choose You!

Python has been around for a long, long, long time. It has a great community, it’s readable, it’s powerful and it can happily dump JSON/YAML all day long. It is a language and thus it has EVERYTHING you could ever want to generate some YAML/JSON. Read vars from env? check. Concat two strings, two lists, merge two maps? Check, Check, Check. Let’s try it out. Just as a note, this is going to use python 3.9 because it introduces the new | dictionary merge operator.

A Concourse Pipeline Experiment

For my full-time job, I work with a lot of very large, complex, YAML configurations for our concourse pipelines which deploy cloud foundry. These pipelines can easily get into the hundreds and thousands of lines of YAML. We use a combination of spruce and some bash to template or merge our configs. They aren’t the most approachable configurations and any updates requires a lot of find-and-replace type workflows.

A typical pipeline which we operate is built around provisioning orgs, spaces, and users into Cloud Foundry using this tool, cf-mgmt.

It produces a pipeline like this.

I cut out about a dozen jobs which are similar but not quite the same which this pipeline also includes. All in all, the complete YAML config is about 500 lines of code. Yes, you can use YAML’s built in tricks to reference anchors and point to them etc. But I found many people aren’t aware that YAML can do that, and getting into pointers, anchors, and references are kinda wacky to use and sometimes produce unexpected results. I omitted such features for this example.

Each one of these jobs have the following subtle differences, noted in side comments.

Infrastructure as Actual Code

The magic I envisioned which I implemented was if some-config.py is executed I expect a some-config.yml file to be spit out right next to it. BUT, you can just spit it out to std.out, it’s up to you, you have all of Python to make that choice. I chose a file for now as it was easier to develop this example.

I accomplished that by starting to write a generate(config) function to do this, and it uses a bit of Python magic to figure out the original calling python script and use that as the base for the yaml file name.

I stuck this within a module called configer so I could use this and other helper functions in this config and in the future.

My cloud foundry org-management-pipeline as code so far:

python org-management-pipeline.py and I now have an org-management-pipeline.yml which now contains the basic structure of a concourse-pipeline:

resources: []

jobs: []

Dealing with Environment Variables

Next, let’s add a single resource.

Oh, I need to dip into the environment for some variables, no problem. os.getenv() to the rescue. And look at that, let’s just concat the return value to a string. Easy. But even os.getenv() can be a little simpler on the eyes, so let’s wrap it in a helper.

Note, I’m returning an empty string but you can choose to return None and determine another angle for dealing with the scenario when None values are concated to a String.

Sourcing Variables from External Files

When working with configurations which slightly change from environment to environment, or just reference some common set of data needed in many configs, it’s common to source in some sort of external inputs to pluck out these variables.

One thing I never liked about overlay and merge tools is that I can never reason where the data will come from or if any data will come at all. The merge tools pretty much smoosh it all together and you are wondering where certain values come from.

Looking to the Zen of Python…

Explicit is better than implicit.

So let’s explicitly declare all the source files our configuration will need.

What’s going on here? Well we’re in Python! A complete OOP language, so I simply created a small class which takes in all file paths you want to source from and maps them to a convenience label to use for referencing them.

Using an object makes sense here because if we reference multiple values from the same file, we only want to do the File IO once and just store the contents for future lookups.

Using a bit of eval() magic to turn dot paths into bracket paths.

The other issue to solve for is we don’t want to simply lookup values in a dictionary as you could run into all sorts of errors like KeyNotFound and IndexOutOfRang etc. and you want know what went wrong without a big stack trace, so let’s just write a grab method for the Sources class.

Using it within the config:

The org-management-pipeline.yml generated so far:

If we typo a lookup for grab we now get a nicer message.

Error- meta.docker_host does not exist in file: vars.yml

Lets do something similar for referencing the contents of an entire file, like a cert CA.

and use it as such:

Looping

Our resources are looking good, now moving into jobs we saw from earlier that concourse pipelines can be pretty verbose and often repeat similar, yet subtlety different. Lets configure a single job

which produces:

The org-management pipeline essentially uses the same job 20 times, with the only difference being the CF_MGMT_COMMAND parameter passed into the task.

Instead of declaring 20 blocks of yaml we’re going to just see how we can use this one block as a base to meet all the nuances of the 20 jobs.

Let’s start with introducing a loop to give us 20 jobs, one for each command. We can easily do this using Python’s in-line list for loop pattern.

[ item + "-foo" for item in ["x", "y", "z"] will give us ["x-foo", "y-foo", "z-foo"]

outputs:

You can also do things like this to combine looping and non-looping blocks using list concatenation.

What if you need an if?

So far everything seen is pretty straight forward. We define some configs using dictionaries and lists and if we need to inject a value we call out to a function and use the return value as that value. The looping also plays well in-line to the configuration so far. But the trickiest bit is dealing with conditional elements or blocks within the configuration.

Templating engines usually handle this fairly well like such

But in Python we can’t really do something that easily. For example, this will absolutely not work.

kudos to Gist syntax highlighting for figuring this one out…

However, we can use functions to facilitate similar behavior. Here I’ve made a class called Include with one static method called when() . This simply implements the if statement for us so we can use it within data structures. Classes with static methods are great ways to easily name-space methods and make things readable.

To render whether a key should exist at all within a dictionary we can use the new Python 3.9 syntax for merging two dictionaries. {dict_a} | {dict_b} and do something like this.

In Python earlier than 3.9, merging dictionaries is as follows: {**dict1, **dict2}

This will give us the following because the Include.when() will return an empty dictionary unless otherwise specified.

foo: bar

biz: baz

We can also toggle values in-line within a dictionary if the key will always exist but it’s value is conditional.

Used in this way, the Include.when() takes a conditional, and the value to return if true and the value to return as the else.

lastly, the Include.when() method can be used in-line within a list like such.

The prune_empty() helper just removes all empty values from a list and is just a wrapper around filter() .

The above, combined with the prune_empty() will give us the following:

foo: bar

biz: baz

stuff:

- a

- c

- d

Applying Conditionals within the Pipeline Configuration

Now that we have a method to implement conditional blocks and values, we can account for all the differences between the 20 jobs for this pipeline. Here is what the entire job config will look like now.

Here is a snippet of the output. Notice the differences in passed and the use of the time-trigger .

Custom Functions and Classes

The above will fully generate this pipeline configuration, and it turns the 500–600 lines of YAML into about 155 lines!

Looking at it though, too much of a good thing can often be bad and the constant chaining of Include.when() can be a little hard to rationalize when chained so many times in a row. Another way of approaching this configuration is creating custom methods and classes which can help us organize similar types of configuration blocks.

Here is a simple class with a static method which organizes task blocks.

applied:

This will still need 20 Jobs, but a lot less lines. In addition we can also pass in parameters into functions which isn’t possible in YAML anchors and pointers.

We can make this even slimmer by defining a custom class to handle the get resource blocks required in each plan as those are repetitive as well.

helpers like this are especially helpful when all your gets have a tags key as well!

all together:

This solution is a best of both worlds approach. It forgoes the looping so you can see each job as it would be declared within a normal config file, but the common blocks have been implemented using helper classes and methods to keep things as concise as possible. Overall, this solution would implement all 20 jobs in about 280 lines of code, including the class definitions.

Wrapping Up

This entire example implementation of generic-code (not including custom classes specific to concourse) is about ~100 lines of code once it’s cleaned up, polished and refined (it’s not right now). You can find this little configer.py module below along with the final configs and YAML, but first let’s conclude this article.

After trying this experiment out it dawned on me that we’re going down a really bad trend with our obsession to find the right templating and merging tools. They’re often sold as a declarative solution, but it’s far from that. At some point you have to ask yourself if it’s easier to define data structures within a programing language or a programing language within data structures?

I also want to touch on a few points: