Terraform And PagerDuty — Before You Try
By: Kamil Świechowski
Tags: Infrastructure As Code, Programming, Monitoring, Terraform, DevOps

Terraform ideas

Terraform And PagerDuty — Before You Try

Terraform is well known for his vast choice of providers. Among these, we can find PagerDuty integration. If you are having a lot of services, users and teams inside PagerDuty you should consider making such infrastructure managed as IaC.

As you spend more and more time writing HCL code you are going to find a few things which you may tolerate or can be completely unacceptable. I do not want to say that terraform PagerDuty provider is bad. Actually, I use it and recommend it. But let’s look at what I discovered in my first attempts.

No support for EventRules inside Services

Taken from https://community.pagerduty.com/forum/t/automation-of-service-event-rules/1507/2

You do not have the ability to deploy EventRules inside Services with terraform. This feature is actually not supported at all in API. Why this can be important to you? In my case, I wanted to manage RuleSets, Rules inside RuleSets, and EventRules inside Services in terraform.

RuleSet with rules should have only routing based on services, not alerts (only metadata from which app/component alert comes). Should ask only one question: Where alert happened?

EventRules inside Services should set event behavior/status based on alert description and name: What alert happened?

This setup is important if you are running many components in your cluster and each has it’s own service so you can be able to peek at PagerDuty and immediately tell what is wrong with your cluster. And because you divide rules you do not have to deal with one huge ruleset filled with hundreds of rules.

Rules order randomness

Rules in PagerDuty are executed one after another until they meet expected condition. Then they add additional metadata and route event to destined service. In terraform rule position is achieved currently by field position (did not expect that, did you?).

Example of rule resource definitionBefore position rules were applied in order using depends_on meta-argument (back when resource was named EventRule).



Lets say after one week you decide to change order. Then you have to update again every depends_on — which is troublesome and ineffective. Of course you can find some workaround and use your shiny scripting skills but that is still just workaround.

So position field is like a blessing. But even in resources, you have to update numbers every time order changes. You can make such process easier using variables and index function:

Now to change rules order I need to only switch strings order in a list in variables.tf .

At this moment everything should go off the bat. After terraform apply:

Apply complete! Resources: 0 added, 2 changed, 0 destroyed.

Looks promising. And then lets run terraform plan to be sure:

Plan: 0 to add, 2 to change, 0 to destroy.

Did it just lie to me? Rules in which position was changed are still in previous positions. So what I do? Run apply again. And after the second run, rules changed place. But I had experienced situation a few days ago when after 7th run changes were correctly applied.

The problem with such a bug is reproducibility. I tried many times never achieving expected behaviour. Then after a few days on workshop with team bug suddenly occurred. Moreover, when tried to deploy all rules at once into empty ruleset, terraform could not set them in the correct order.

For me, it is quite a sore in the eye to set up rules in terraform. I am also always thinking about the worst-case scenario when someone would apply changes and forget to check if an order is correct. You can design some pipeline to apply changes until an order is correct but that may not work — once I had to manually move rules in PagerDuty GUI after 10th terraform run.

Less manual work — but writing module can be hard

As many IaC solutions this also gives us less clicking-job. One command and the whole infrastructure is up. In my PagerDuty I decided to keep such terraform repo structure:

My PagerDuty terraform structure

The first question you should ask is why I did not write a module to cover file redundancy. The answer is simple — writing module in PagerDuty is not worth time in my case. There are few differences in monitoring between our clusters we want to keep.

Let’s say we want a module that will deploy all mentioned resources. Most of the time you still have to pass detailed information for rules and with a module, this can be tricky. I also bounced off my team unwillingness to use hard to read modules (you have to maintain them also). They prefer simplicity. Worth mentioning is main.tf where we keep provider info and terraform data sources — references to teams, users, schedules, priorities, and escalation policies.

You can try now

Photo by Basil Samuel Lade on Unsplash

There are two small inconveniences in setting PagerDuty with terraform. Especially deploying whole PagerDuty. Also being unable to use EventRule resource makes terraform experience lacking. I do not know if they going to do something about rules order because it is not such a common problem. For now we can have hopes to get EventRules API.

If you are planning on using terraform for your PagerDuty I can recommend you this solution but keep these few things in mind. Your requirements can be higher than mine and knowing the mentioned limitations can save you unnecessary frustration someday.