Getting to know Terraform data sources
By: Pat Conheady
Tags: Iac, Infrastructure As Code, Infrastructure, Terraform

Photo by Leo Rivas on Unsplash

Getting to know Terraform data sources

A powerful way of bringing information into your Terraform configuration, but one which must be used with care Pat Conheady Jan 29·5 min read

A data source in Terraform is a lot like a resource, but it only implements the “read” operation. So what does that actually mean? Let’s take a walk.

It’s like a query

Well, that was a short walk. A data source is a query, a means of getting data from the outside world and making it available to your Terraform configuration, perhaps to use in an input attribute of a resource or a submodule.

Take for example a Terraform configuration like the following (borrowed from the doco):

data "azurerm_role_definition" "example" {

name = "Contributor"

}



output "contributor_role_definition_id" {

value = data.azurerm_role_definition.example.id

}

This can be thought of as equivalent to the following SQL query:

SELECT id AS contributor_role_definition_id

FROM azurerm_role_definitions

WHERE name = "Contributor";

In this example, we’re sending the data from the data source directly to an output, but you could also send it (for example) into a role assignment resource:

resource "azurerm_role_assignment" "example" {

scope = data.azurerm_subscription.primary.id

role_definition_id = data.azurerm_role_definition.example.id

principal_id = data.azurerm_client_config.example.object_id

}

/* Other data sources omitted... borrowing them from the doco is left as an exercise for the reader... */

When is the query run?

In a simple case, it doesn’t matter exactly when the query is run. You say terraform plan and there’s the data. Then when you say terraform apply it’s there again. But what if the underlying facts change between plan and apply ? Or what if obtaining the data was expensive or time-consuming in some way?

As with all things Terraform, the documentation contains some simple answers:

A data source is evaluated at plan time unless it can’t be because one of its inputs will not be known until apply time.

time. It is evaluated every time terraform is run. Therefore running terraform plan and then terraform apply will run the data source’s read operation twice.

These are simple rules and it seems quite obvious that this is what you want to do every time. However, there is a degree of subtlety required here. The best way to understand it is with examples, such as this GitHub issue on the azurerm provider:

To illustrate, suppose you are trying to create an Azure Function App and export its host keys (used for authenticating HTTP requests to functions). This does not work:

resource "azurerm_function_app" "something" {

...

} data "azurerm_function_app_host_keys" "example" {

name = azurerm_function_app.something.name

resource_group_name = azurerm_function_app.something.resource_group_name

} output "function_host_key" {

value = data.azurerm_function_app_host_keys.example.default_function_key

}

If you run this configuration, Terraform follows its rules: it checks whether it has sufficient information to evaluate the azurerm_function_app_host_keys data source. However, during the plan phase when this is occurring, the Function App has not yet been deployed. How can Terraform get its host keys if it does not exist?

The wrinkle is that Terraform has all the inputs for the data source, namely the Function App’s name and resource group name. This framework does not, however, allow Terraform to notice or recover from the fact that the underlying Function App described by that information does not yet exist. Terraform throws an error and does not complete the plan phase. The Function App will never be deployed, and nobody will ever get any host keys.

The solution is straightforward:

data "azurerm_function_app_host_keys" "example" {

name = azurerm_function_app.something.name

resource_group_name = azurerm_function_app.something.resource_group_name



depends_on = [

azurerm_function_app. something

]

}

In this version of the data source, the depends_on field forces evaluation of the data source to wait until after the Function App exists, during the apply phase. Many Terraform users cringe at this solution because referring to an output attribute of a resource is supposed to create an implicit dependency relationship. But, that’s not sufficient to control the order in which a data source is evaluated. You need an explicit depends_on field.

Photo by Eugene Zaycev on Unsplash

Data sources with side effects

I said earlier that data sources are like queries, and that they only implement a “read” operation. However, there is no real limit on what providers can do when a data source is evaluated.

For example, the archive_file data source creates a zip file on the local filesystem. Within the Terraform configuration, this gives you access to the checksum of the resulting zip file through the data source’s attributes, and use can also be made of the file itself.

Data sources vs locals

When getting used to Terraform terminology, it’s easy to mix up data sources, locals and variables. Here’s how I keep them separated:

“data sources” bring in data from the outside world via providers, such as the azuread provider for talking to Azure Active Directory;

provider for talking to Azure Active Directory; “variables” (or at least what Terraform specifically calls “variables” — I call them “input variables”) bring in data from the parent module or from the process invoking terraform ;

; “locals” hold intermediate data which may come from data sources or variables, or from internal sources such as expressions (calculations) or the output attributes of resources or submodules.

Data sources vs importing a resource

Importing an existing resource is similar to pointing a data source at it. In both cases, you get information about the resource made available inside your Terraform configuration.

However, the key difference is that if you import a resource then you now manage it. That means:

If you terraform destroy your configuration, that resource will be destroyed.

your configuration, that resource will be destroyed. If you modify or destroy the resource through some other means, your Terraform configuration will try to put it back how it was next time apply runs.

So, if you are thinking of importing an existing resource — perhaps a static piece of infrastructure that was configured manually— just so that you can access its attributes within your Terraform configuration, then instead use a data source. Only import the resource if you intend to bring that resource under the control of this particular Terraform configuration.

Conclusion

We have gone for a bit of a meander together, taking a look at some features and behaviours of data sources in Terraform. Data sources are a useful tool, but one that will get you into trouble if you make assumptions about its behaviour. While data sources can be tricky to test, the issues touched on in this article should give you an indication of where to look for gotchas in any given case. Careful consideration of the behaviour of your data source against your precise needs will yield a good design and a robust solution.