Repository: kubernetes/k8s.io
Pull request #7776: Add terraform for capz monitoring cluster
Author: login: willie-yao
Labels: name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: size/M
description: Denotes a PR that changes 30-99 lines, ignoring generated files.
name: area/provider/azure
description: Issues or PRs related to azure provider
name: sig/cluster-lifecycle
description: Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: area/infra
description: Infrastructure management, infrastructure design, code in infra/
name: sig/k8s-infra
description: Categorizes an issue or PR as relevant to SIG K8s Infra.
name: area/infra/azure
description: Issues or PRs related to Kubernetes Azure infrastructure
Created: 2025-02-10T23:40:39Z | Merged: 2025-04-04T22:34:37Z

Adds terraform needed to recreate the capz monitoring cluster used to monitor/export metrics from CAPZ e2e tests.


Diff:
diff --git a/infra/azure/terraform/capz/capz-monitoring/main.tf b/infra/azure/terraform/capz/capz-monitoring/main.tf
new file mode 100644
index 00000000000..5d9e8520ae0
--- /dev/null
+++ b/infra/azure/terraform/capz/capz-monitoring/main.tf
@@ -0,0 +1,79 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+variable "resource_group_name" {
+  type = string
+}
+
+variable "location" {
+  type = string
+}
+
+variable "subscription_id" {
+  type = string
+}
+
+# Create the "capz-monitoring" resource group
+resource "azurerm_resource_group" "capz-monitoring" {
+  location = var.location
+  name     = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = timestamp()
+  }
+}
+
+resource "azurerm_user_assigned_identity" "capz_monitoring_user_identity" {
+  name                = "capz-monitoring-user-identity"
+  location            = azurerm_resource_group.capz-monitoring.location
+  resource_group_name = azurerm_resource_group.capz-monitoring.name
+}
+
+resource "azurerm_role_assignment" "monitoring_reader" {
+  principal_id         = azurerm_user_assigned_identity.capz_monitoring_user_identity.principal_id
+  role_definition_name = "Monitoring Reader"
+  scope                = "/subscriptions/${var.subscription_id}"
+  depends_on = [ azurerm_user_assigned_identity.capz_monitoring_user_identity ]
+}
+
+resource "azurerm_kubernetes_cluster" "capz-monitoring" {
+  dns_prefix            = var.resource_group_name
+  location              = var.location
+  name                  = var.resource_group_name
+  resource_group_name   = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = timestamp()
+  }
+  depends_on = [
+    azurerm_resource_group.capz-monitoring,
+    azurerm_user_assigned_identity.capz_monitoring_user_identity,
+  ]
+  kubelet_identity {
+    user_assigned_identity_id = azurerm_user_assigned_identity.capz_monitoring_user_identity.id
+  }
+  identity {
+    type                     = "UserAssigned"
+    identity_ids = [
+      azurerm_user_assigned_identity.capz_monitoring_user_identity.id
+    ]
+  }
+  default_node_pool {
+    name       = "default"
+    node_count = 1
+    vm_size    = "Standard_Ds2_v2"
+  }
+}
diff --git a/infra/azure/terraform/capz/main.tf b/infra/azure/terraform/capz/main.tf
index b94b887c834..7e964485719 100644
--- a/infra/azure/terraform/capz/main.tf
+++ b/infra/azure/terraform/capz/main.tf
@@ -133,3 +133,11 @@ module "cluster_api_gallery" {
   location            = var.location
   depends_on          = module.role_assignments
 }
+
+# Import CAPZ monitoring module
+module "capz_monitoring" {
+  source              = "./capz-monitoring"
+  resource_group_name = var.resource_group_name
+  location            = var.location
+  subscription_id     = data.azurerm_client_config.current.subscription_id
+}


Reviews:
user: login: mboersma
body: /lgtm

> Is it okay to hard-code the name and location

I'm not sure actually, I followed the `var.location` convention when I made changes, but I'm not a terraform expert. I doubt it's a problem in practice to use `eastus`, but maybe @jsturtevant knows?
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-02-12T16:24:15Z
user: login: jsturtevant
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T17:23:25Z
user: login: willie-yao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T21:49:45Z
user: login: jsturtevant
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T21:52:46Z
user: login: willie-yao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T21:59:16Z
user: login: jsturtevant
body: LGTM, minor nit on timestamp but otherwise looks good
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T13:28:55Z
user: login: mboersma
body: /lgtm
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T18:03:09Z
user: login: jsturtevant
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T18:16:34Z
user: login: marosset
body: LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T20:14:35Z
user: login: mboersma
body: /lgtm
/approve
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T22:28:47Z

Review comments:
diff_hunk: @@ -0,0 +1,89 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+variable "resource_group_name" {
+  type = string
+}
+
+variable "location" {
+  type = string
+}
+
+variable "subscription_id" {
+  type = string
+}
+
+# Create the "capz-monitoring" resource group
+resource "azurerm_resource_group" "capz-monitoring" {
+  location = var.location
+  name     = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = "2024-10-24T00:00:00Z"
+  }
+}
+
+resource "azurerm_resource_group" "MC_capz-monitoring_capz-monitoring_eastus" {
path: infra/azure/terraform/capz/capz-monitoring/main.tf
user: login: jsturtevant
body: I am surprised that this resource group needs to be captured in the terraform.  This is a RG that is managed by AKS and shouldn't be managed by terraform AFAIK.

https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#node_resource_group-1

Says confirms that for me, in that we could provide a name but otherwise don't need to specify it.

created_at: 2025-03-05T17:23:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+variable "resource_group_name" {
+  type = string
+}
+
+variable "location" {
+  type = string
+}
+
+variable "subscription_id" {
+  type = string
+}
+
+# Create the "capz-monitoring" resource group
+resource "azurerm_resource_group" "capz-monitoring" {
+  location = var.location
+  name     = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = "2024-10-24T00:00:00Z"
+  }
+}
+
+resource "azurerm_resource_group" "MC_capz-monitoring_capz-monitoring_eastus" {
path: infra/azure/terraform/capz/capz-monitoring/main.tf
user: login: willie-yao
body: Ah that makes sense. I included it because I noticed that this resource group has been getting deleted on the CNCF sub periodically while I'm testing even though I have the do not delete tag set on it. Does this need to be in terraform if I don't want it to be deleted, or is that being caused by something else?
created_at: 2025-03-05T21:49:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+variable "resource_group_name" {
+  type = string
+}
+
+variable "location" {
+  type = string
+}
+
+variable "subscription_id" {
+  type = string
+}
+
+# Create the "capz-monitoring" resource group
+resource "azurerm_resource_group" "capz-monitoring" {
+  location = var.location
+  name     = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = "2024-10-24T00:00:00Z"
+  }
+}
+
+resource "azurerm_resource_group" "MC_capz-monitoring_capz-monitoring_eastus" {
path: infra/azure/terraform/capz/capz-monitoring/main.tf
user: login: jsturtevant
body: I believe that if you put the tags do-not-delete on the AKS resource group and AKS cluster, it will carry those through when it creates managed RG.
created_at: 2025-03-05T21:52:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,89 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+variable "resource_group_name" {
+  type = string
+}
+
+variable "location" {
+  type = string
+}
+
+variable "subscription_id" {
+  type = string
+}
+
+# Create the "capz-monitoring" resource group
+resource "azurerm_resource_group" "capz-monitoring" {
+  location = var.location
+  name     = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = "2024-10-24T00:00:00Z"
+  }
+}
+
+resource "azurerm_resource_group" "MC_capz-monitoring_capz-monitoring_eastus" {
path: infra/azure/terraform/capz/capz-monitoring/main.tf
user: login: willie-yao
body: Hmm okay, next time it gets deleted I might need some help figuring out what is causing it. Thanks!
created_at: 2025-03-05T21:59:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,79 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+variable "resource_group_name" {
+  type = string
+}
+
+variable "location" {
+  type = string
+}
+
+variable "subscription_id" {
+  type = string
+}
+
+# Create the "capz-monitoring" resource group
+resource "azurerm_resource_group" "capz-monitoring" {
+  location = var.location
+  name     = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = "2024-10-24T00:00:00Z"
path: infra/azure/terraform/capz/capz-monitoring/main.tf
user: login: jsturtevant
body: maybe we could use `timestamp()` here (https://developer.hashicorp.com/terraform/language/functions/timestamp). rg-cleanup uses rf3339 so `timestamp()` should work here https://github.com/Azure/rg-cleanup/blob/1f19a91e03d7c78f0ce1634f1e022a6e8c2713d3/main.go#L31
created_at: 2025-04-01T13:27:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,79 @@
+/*
+Copyright 2024 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+variable "resource_group_name" {
+  type = string
+}
+
+variable "location" {
+  type = string
+}
+
+variable "subscription_id" {
+  type = string
+}
+
+# Create the "capz-monitoring" resource group
+resource "azurerm_resource_group" "capz-monitoring" {
+  location = var.location
+  name     = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = "2024-10-24T00:00:00Z"
+  }
+}
+
+resource "azurerm_user_assigned_identity" "capz_monitoring_user_identity" {
+  name                = "capz-monitoring-user-identity"
+  location            = azurerm_resource_group.capz-monitoring.location
+  resource_group_name = azurerm_resource_group.capz-monitoring.name
+}
+
+resource "azurerm_role_assignment" "monitoring_reader" {
+  principal_id         = azurerm_user_assigned_identity.capz_monitoring_user_identity.principal_id
+  role_definition_name = "Monitoring Reader"
+  scope                = "/subscriptions/${var.subscription_id}"
+  depends_on = [ azurerm_user_assigned_identity.capz_monitoring_user_identity ]
+}
+
+resource "azurerm_kubernetes_cluster" "capz-monitoring" {
+  dns_prefix            = var.resource_group_name
+  location              = var.location
+  name                  = var.resource_group_name
+  resource_group_name   = var.resource_group_name
+  tags = {
+    DO-NOT-DELETE     = "contact capz"
+    creationTimestamp = "2024-10-24T00:00:00Z"
path: infra/azure/terraform/capz/capz-monitoring/main.tf
user: login: jsturtevant
body: same here for `timestamp`
created_at: 2025-04-01T13:27:49Z
author_association: CONTRIBUTOR

PR comments:
user: login: willie-yao
created_at: 2025-02-10T23:42:17Z
body: /hold for reviews
/cc @jsturtevant @nojnhuh @ritikaguptams @mboersma 

My first go with terraform stuff. I just modeled the file off of my existing capz-monitoring cluster. Is it okay to hard-code the name and location since I don't think it's currently set to the default? If not, I can re-create the cluster in the correct location
author_association: CONTRIBUTOR
user: login: jsturtevant
created_at: 2025-03-05T17:26:43Z
body: > My first go with terraform stuff. I just modeled the file off of my existing capz-monitoring cluster. Is it okay to hard-code the name and location since I don't think it's currently set to the default? If not, I can re-create the cluster in the correct location

I think you've done the right thing with the location/name. Those are variables, you can specify them with variables to use them with pre-created clusters.  The part you don't need is the `node_resource_group` which shouldn't be managed by Terraform, if I understand the docs correct: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#example-usage
author_association: CONTRIBUTOR
user: login: willie-yao
created_at: 2025-04-03T22:25:48Z
body: /hold
whoops forgot to squash
author_association: CONTRIBUTOR
user: login: willie-yao
created_at: 2025-04-03T22:26:59Z
body: /hold cancel
author_association: CONTRIBUTOR
user: login: willie-yao
created_at: 2025-04-04T21:12:49Z
body: Can I get one more approval here? Sorry I had to squash @jsturtevant @mboersma @marosset 
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-04-04T22:28:55Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/k8s.io/pull/7776#pullrequestreview-2740767416" title="Approved">jsturtevant</a>*, *<a href="https://github.com/kubernetes/k8s.io/pull/7776#pullrequestreview-2741031188" title="Approved">marosset</a>*, *<a href="https://github.com/kubernetes/k8s.io/pull/7776#pullrequestreview-2744207932" title="Approved">mboersma</a>*, *<a href="https://github.com/kubernetes/k8s.io/pull/7776#" title="Author self-approved">willie-yao</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fk8s.io).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[infra/azure/terraform/capz/OWNERS](https://github.com/kubernetes/k8s.io/blob/main/infra/azure/terraform/capz/OWNERS)~~ [mboersma,willie-yao]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR