Repository: kubernetes-sigs/cluster-api-provider-azure
Pull request #5509: Add flag to disable bootstrap extension and improve error message
Author: login: willie-yao
Labels: name: kind/feature
description: Categorizes issue or PR as related to a new feature.
name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: release-note
description: Denotes a PR that will be considered when it comes time to generate release notes.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
Created: 2025-03-24T22:48:55Z | Merged: 2025-04-22T17:46:58Z

 <!-- If this is your first PR, welcome! Please make sure you read the [contributing guidelines](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/CONTRIBUTING.md). -->

 <!-- Please label this pull request according to what type of issue you are addressing (see ../CONTRIBUTING.md) -->
**What type of PR is this?**
/kind feature

<!--
Add one of the following kinds:
/kind feature
/kind bug
/kind api-change
/kind cleanup
/kind deprecation
/kind design
/kind documentation
/kind failing-test
/kind flake
-->

**What this PR does / why we need it**:
This PR adds a more descriptive error message for the CAPZ Bootstrapping Extension so that users can better understand what went wrong when the extension fails to install. It also adds a new field `DisableVMBootstrapExtension` to disable only the bootstrap extension and not every extension like `DisableExtensionOperations`

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes #5482 

**Special notes for your reviewer**:
<!-- Refer to https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/docs/book/src/developers/releasing.md#release-support for more information about which changes are eligible for backport -->


**TODOs**:
<!-- Put an "X" character inside the brackets of each completed task. Some may be optional depending on the PR. -->

- [ ] squashed commits
- [ ] includes documentation
- [ ] adds unit tests
- [ ] cherry-pick candidate

**Release note**:
<!--  Write your release note:
1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string "action required".
2. If no release note is required, just write "NONE".
-->
```release-note
Add flag to disable bootstrap extension and improve error message
```


Linked issue #5482: Make CAPZ bootstrapping extension optional and add more helpful error message
/kind cleanup

**What needs cleanup:**
Since the CAPZ bootstrap extension is just a wrapper on the custom script extension, we can replace it to avoid confusion in error messages, as well as making it optional and disabled by default. The extension was used to check if cloud init failed and can mask the cloud init failures.



Linked issue comments: user: login: willie-yao
created_at: 2025-03-12T17:20:56Z
body: /priority important-soon
author_association: CONTRIBUTOR
user: login: willie-yao
created_at: 2025-03-12T17:21:12Z
body: /assign
author_association: CONTRIBUTOR
user: login: willie-yao
created_at: 2025-04-17T16:21:00Z
body: /retitle Make CAPZ bootstrapping extension optional and add more helpful error message
author_association: CONTRIBUTOR
user: login: nawazkh
created_at: 2025-04-17T16:22:13Z
body: We increased the default kubeadm logging verbosity in our CAPZ templates in the release https://github.com/kubernetes-sigs/cluster-api-provider-azure/releases/tag/v1.19.0. 
For anyone who are watching this issue, please use CAPZ release v1.19.x and higher in your deployments for more verbose logs on kubeadm failures.
author_association: MEMBER
user: login: zylxjtu
created_at: 2025-04-17T21:10:50Z
body: do we have any plans to replace cloudbase-init?
author_association: CONTRIBUTOR
user: login: jackfrancis
created_at: 2025-04-17T21:18:28Z
body: @jsturtevant do we have a path to a cloudbase-init replacement (to answer @zylxjtu's question)
author_association: CONTRIBUTOR

Diff:
diff --git a/api/v1beta1/azuremachine_types.go b/api/v1beta1/azuremachine_types.go
index 04a8a70c7c6..090f24edace 100644
--- a/api/v1beta1/azuremachine_types.go
+++ b/api/v1beta1/azuremachine_types.go
@@ -137,6 +137,11 @@ type AzureMachineSpec struct {
 	// +optional
 	DisableExtensionOperations *bool `json:"disableExtensionOperations,omitempty"`
 
+	// DisableVMBootstrapExtension specifies whether the VM bootstrap extension should be disabled on the virtual machine.
+	// Use this setting if you want to disable only the bootstrapping extension and not all extensions.
+	// +optional
+	DisableVMBootstrapExtension *bool `json:"disableVMBootstrapExtension,omitempty"`
+
 	// VMExtensions specifies a list of extensions to be added to the virtual machine.
 	// +optional
 	VMExtensions []VMExtension `json:"vmExtensions,omitempty"`
diff --git a/api/v1beta1/azuremachine_webhook.go b/api/v1beta1/azuremachine_webhook.go
index d3f892da778..246d160da24 100644
--- a/api/v1beta1/azuremachine_webhook.go
+++ b/api/v1beta1/azuremachine_webhook.go
@@ -221,6 +221,13 @@ func (mw *azureMachineWebhook) ValidateUpdate(_ context.Context, oldObj, newObj
 		allErrs = append(allErrs, err)
 	}
 
+	if err := webhookutils.ValidateImmutable(
+		field.NewPath("spec", "disableVMBootstrapExtension"),
+		old.Spec.DisableVMBootstrapExtension,
+		m.Spec.DisableVMBootstrapExtension); err != nil {
+		allErrs = append(allErrs, err)
+	}
+
 	if len(allErrs) == 0 {
 		return nil, nil
 	}
diff --git a/api/v1beta1/zz_generated.deepcopy.go b/api/v1beta1/zz_generated.deepcopy.go
index b523662a8d0..f27a8bf70aa 100644
--- a/api/v1beta1/zz_generated.deepcopy.go
+++ b/api/v1beta1/zz_generated.deepcopy.go
@@ -938,6 +938,11 @@ func (in *AzureMachineSpec) DeepCopyInto(out *AzureMachineSpec) {
 		*out = new(bool)
 		**out = **in
 	}
+	if in.DisableVMBootstrapExtension != nil {
+		in, out := &in.DisableVMBootstrapExtension, &out.DisableVMBootstrapExtension
+		*out = new(bool)
+		**out = **in
+	}
 	if in.VMExtensions != nil {
 		in, out := &in.VMExtensions, &out.VMExtensions
 		*out = make([]VMExtension, len(*in))
diff --git a/azure/defaults.go b/azure/defaults.go
index 0f86619c7d1..02e5508fa5c 100644
--- a/azure/defaults.go
+++ b/azure/defaults.go
@@ -111,7 +111,7 @@ const (
 
 var (
 	// LinuxBootstrapExtensionCommand is the command the VM bootstrap extension will execute to verify Linux nodes bootstrap completes successfully.
-	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
+	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then echo 'Error joining node to cluster: kubeadm init or join failed. To debug, check the cloud-init, kubelet, or other bootstrap logs: https://capz.sigs.k8s.io/self-managed/troubleshooting.html#checking-cloud-init-logs-ubuntu'; exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
 	// WindowsBootstrapExtensionCommand is the command the VM bootstrap extension will execute to verify Windows nodes bootstrap completes successfully.
 	WindowsBootstrapExtensionCommand = fmt.Sprintf("powershell.exe -Command \"for ($i = 0; $i -lt %d; $i++) {if (Test-Path '%s') {exit 0} else {Start-Sleep -Seconds %d}} exit -2\"",
 		bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionSleep)
diff --git a/azure/scope/machine.go b/azure/scope/machine.go
index d6a96964f36..67635826d8b 100644
--- a/azure/scope/machine.go
+++ b/azure/scope/machine.go
@@ -161,29 +161,30 @@ func (m *MachineScope) InitMachineCache(ctx context.Context) error {
 // VMSpec returns the VM spec.
 func (m *MachineScope) VMSpec() azure.ResourceSpecGetter {
 	spec := &virtualmachines.VMSpec{
-		Name:                       m.Name(),
-		Location:                   m.Location(),
-		ExtendedLocation:           m.ExtendedLocation(),
-		ResourceGroup:              m.NodeResourceGroup(),
-		ClusterName:                m.ClusterName(),
-		Role:                       m.Role(),
-		NICIDs:                     m.NICIDs(),
-		SSHKeyData:                 m.AzureMachine.Spec.SSHPublicKey,
-		Size:                       m.AzureMachine.Spec.VMSize,
-		OSDisk:                     m.AzureMachine.Spec.OSDisk,
-		DataDisks:                  m.AzureMachine.Spec.DataDisks,
-		AvailabilitySetID:          m.AvailabilitySetID(),
-		Zone:                       m.AvailabilityZone(),
-		Identity:                   m.AzureMachine.Spec.Identity,
-		UserAssignedIdentities:     m.AzureMachine.Spec.UserAssignedIdentities,
-		SpotVMOptions:              m.AzureMachine.Spec.SpotVMOptions,
-		SecurityProfile:            m.AzureMachine.Spec.SecurityProfile,
-		DiagnosticsProfile:         m.AzureMachine.Spec.Diagnostics,
-		DisableExtensionOperations: ptr.Deref(m.AzureMachine.Spec.DisableExtensionOperations, false),
-		AdditionalTags:             m.AdditionalTags(),
-		AdditionalCapabilities:     m.AzureMachine.Spec.AdditionalCapabilities,
-		CapacityReservationGroupID: m.GetCapacityReservationGroupID(),
-		ProviderID:                 m.ProviderID(),
+		Name:                        m.Name(),
+		Location:                    m.Location(),
+		ExtendedLocation:            m.ExtendedLocation(),
+		ResourceGroup:               m.NodeResourceGroup(),
+		ClusterName:                 m.ClusterName(),
+		Role:                        m.Role(),
+		NICIDs:                      m.NICIDs(),
+		SSHKeyData:                  m.AzureMachine.Spec.SSHPublicKey,
+		Size:                        m.AzureMachine.Spec.VMSize,
+		OSDisk:                      m.AzureMachine.Spec.OSDisk,
+		DataDisks:                   m.AzureMachine.Spec.DataDisks,
+		AvailabilitySetID:           m.AvailabilitySetID(),
+		Zone:                        m.AvailabilityZone(),
+		Identity:                    m.AzureMachine.Spec.Identity,
+		UserAssignedIdentities:      m.AzureMachine.Spec.UserAssignedIdentities,
+		SpotVMOptions:               m.AzureMachine.Spec.SpotVMOptions,
+		SecurityProfile:             m.AzureMachine.Spec.SecurityProfile,
+		DiagnosticsProfile:          m.AzureMachine.Spec.Diagnostics,
+		DisableExtensionOperations:  ptr.Deref(m.AzureMachine.Spec.DisableExtensionOperations, false),
+		DisableVMBootstrapExtension: ptr.Deref(m.AzureMachine.Spec.DisableVMBootstrapExtension, false),
+		AdditionalTags:              m.AdditionalTags(),
+		AdditionalCapabilities:      m.AzureMachine.Spec.AdditionalCapabilities,
+		CapacityReservationGroupID:  m.GetCapacityReservationGroupID(),
+		ProviderID:                  m.ProviderID(),
 	}
 	if m.cache != nil {
 		spec.SKU = m.cache.VMSKU
@@ -400,15 +401,17 @@ func (m *MachineScope) VMExtensionSpecs() []azure.ResourceSpecGetter {
 		})
 	}
 
-	cpuArchitectureType, _ := m.cache.VMSKU.GetCapability(resourceskus.CPUArchitectureType)
-	bootstrapExtensionSpec := azure.GetBootstrappingVMExtension(m.AzureMachine.Spec.OSDisk.OSType, m.CloudEnvironment(), m.Name(), cpuArchitectureType)
+	if !ptr.Deref(m.AzureMachine.Spec.DisableVMBootstrapExtension, false) {
+		cpuArchitectureType, _ := m.cache.VMSKU.GetCapability(resourceskus.CPUArchitectureType)
+		bootstrapExtensionSpec := azure.GetBootstrappingVMExtension(m.AzureMachine.Spec.OSDisk.OSType, m.CloudEnvironment(), m.Name(), cpuArchitectureType)
 
-	if bootstrapExtensionSpec != nil {
-		extensionSpecs = append(extensionSpecs, &vmextensions.VMExtensionSpec{
-			ExtensionSpec: *bootstrapExtensionSpec,
-			ResourceGroup: m.NodeResourceGroup(),
-			Location:      m.Location(),
-		})
+		if bootstrapExtensionSpec != nil {
+			extensionSpecs = append(extensionSpecs, &vmextensions.VMExtensionSpec{
+				ExtensionSpec: *bootstrapExtensionSpec,
+				ResourceGroup: m.NodeResourceGroup(),
+				Location:      m.Location(),
+			})
+		}
 	}
 
 	return extensionSpecs
diff --git a/azure/scope/machine_test.go b/azure/scope/machine_test.go
index cfbb44b3588..30c4127ca1b 100644
--- a/azure/scope/machine_test.go
+++ b/azure/scope/machine_test.go
@@ -917,6 +917,74 @@ func TestMachineScope_VMExtensionSpecs(t *testing.T) {
 				},
 			},
 		},
+		{
+			name: "If a custom VM extension is specified and bootstrap extension is disabled, it returns only the custom VM extension",
+			machineScope: MachineScope{
+				Machine: &clusterv1.Machine{},
+				AzureMachine: &infrav1.AzureMachine{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: "machine-name",
+					},
+					Spec: infrav1.AzureMachineSpec{
+						OSDisk: infrav1.OSDisk{
+							OSType: "Linux",
+						},
+						DisableVMBootstrapExtension: ptr.To(true),
+						VMExtensions: []infrav1.VMExtension{
+							{
+								Name:      "custom-vm-extension",
+								Publisher: "Microsoft.Azure.Extensions",
+								Version:   "2.0",
+								Settings: map[string]string{
+									"timestamp": "1234567890",
+								},
+								ProtectedSettings: map[string]string{
+									"commandToExecute": "echo hello world",
+								},
+							},
+						},
+					},
+				},
+				ClusterScoper: &ClusterScope{
+					AzureClients: AzureClients{
+						EnvironmentSettings: auth.EnvironmentSettings{
+							Environment: azureautorest.Environment{
+								Name: azureautorest.PublicCloud.Name,
+							},
+						},
+					},
+					AzureCluster: &infrav1.AzureCluster{
+						Spec: infrav1.AzureClusterSpec{
+							ResourceGroup: "my-rg",
+							AzureClusterClassSpec: infrav1.AzureClusterClassSpec{
+								Location: "westus",
+							},
+						},
+					},
+				},
+				cache: &MachineCache{
+					VMSKU: resourceskus.SKU{},
+				},
+			},
+			want: []azure.ResourceSpecGetter{
+				&vmextensions.VMExtensionSpec{
+					ExtensionSpec: azure.ExtensionSpec{
+						Name:      "custom-vm-extension",
+						VMName:    "machine-name",
+						Publisher: "Microsoft.Azure.Extensions",
+						Version:   "2.0",
+						Settings: map[string]string{
+							"timestamp": "1234567890",
+						},
+						ProtectedSettings: map[string]string{
+							"commandToExecute": "echo hello world",
+						},
+					},
+					ResourceGroup: "my-rg",
+					Location:      "westus",
+				},
+			},
+		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
diff --git a/azure/services/virtualmachines/spec.go b/azure/services/virtualmachines/spec.go
index 7f626366edc..70ccced92f8 100644
--- a/azure/services/virtualmachines/spec.go
+++ b/azure/services/virtualmachines/spec.go
@@ -34,32 +34,33 @@ import (
 
 // VMSpec defines the specification for a Virtual Machine.
 type VMSpec struct {
-	Name                       string
-	ResourceGroup              string
-	Location                   string
-	ExtendedLocation           *infrav1.ExtendedLocationSpec
-	ClusterName                string
-	Role                       string
-	NICIDs                     []string
-	SSHKeyData                 string
-	Size                       string
-	AvailabilitySetID          string
-	Zone                       string
-	Identity                   infrav1.VMIdentity
-	OSDisk                     infrav1.OSDisk
-	DataDisks                  []infrav1.DataDisk
-	UserAssignedIdentities     []infrav1.UserAssignedIdentity
-	SpotVMOptions              *infrav1.SpotVMOptions
-	SecurityProfile            *infrav1.SecurityProfile
-	AdditionalTags             infrav1.Tags
-	AdditionalCapabilities     *infrav1.AdditionalCapabilities
-	DiagnosticsProfile         *infrav1.Diagnostics
-	DisableExtensionOperations bool
-	CapacityReservationGroupID string
-	SKU                        resourceskus.SKU
-	Image                      *infrav1.Image
-	BootstrapData              string
-	ProviderID                 string
+	Name                        string
+	ResourceGroup               string
+	Location                    string
+	ExtendedLocation            *infrav1.ExtendedLocationSpec
+	ClusterName                 string
+	Role                        string
+	NICIDs                      []string
+	SSHKeyData                  string
+	Size                        string
+	AvailabilitySetID           string
+	Zone                        string
+	Identity                    infrav1.VMIdentity
+	OSDisk                      infrav1.OSDisk
+	DataDisks                   []infrav1.DataDisk
+	UserAssignedIdentities      []infrav1.UserAssignedIdentity
+	SpotVMOptions               *infrav1.SpotVMOptions
+	SecurityProfile             *infrav1.SecurityProfile
+	AdditionalTags              infrav1.Tags
+	AdditionalCapabilities      *infrav1.AdditionalCapabilities
+	DiagnosticsProfile          *infrav1.Diagnostics
+	DisableExtensionOperations  bool
+	DisableVMBootstrapExtension bool
+	CapacityReservationGroupID  string
+	SKU                         resourceskus.SKU
+	Image                       *infrav1.Image
+	BootstrapData               string
+	ProviderID                  string
 }
 
 // ResourceName returns the name of the virtual machine.
diff --git a/config/crd/bases/infrastructure.cluster.x-k8s.io_azuremachines.yaml b/config/crd/bases/infrastructure.cluster.x-k8s.io_azuremachines.yaml
index 088d7358d5c..df1d1983160 100644
--- a/config/crd/bases/infrastructure.cluster.x-k8s.io_azuremachines.yaml
+++ b/config/crd/bases/infrastructure.cluster.x-k8s.io_azuremachines.yaml
@@ -248,6 +248,11 @@ spec:
                   Use this setting only if VMExtensions are not supported by your image, as it disables CAPZ bootstrapping extension used for detecting Kubernetes bootstrap failure.
                   This may only be set to True when no extensions are configured on the virtual machine.
                 type: boolean
+              disableVMBootstrapExtension:
+                description: |-
+                  DisableVMBootstrapExtension specifies whether the VM bootstrap extension should be disabled on the virtual machine.
+                  Use this setting if you want to disable only the bootstrapping extension and not all extensions.
+                type: boolean
               dnsServers:
                 description: DNSServers adds a list of DNS Server IP addresses to
                   the VM NICs.
diff --git a/config/crd/bases/infrastructure.cluster.x-k8s.io_azuremachinetemplates.yaml b/config/crd/bases/infrastructure.cluster.x-k8s.io_azuremachinetemplates.yaml
index 54d3ef25afc..1f8e95256b6 100644
--- a/config/crd/bases/infrastructure.cluster.x-k8s.io_azuremachinetemplates.yaml
+++ b/config/crd/bases/infrastructure.cluster.x-k8s.io_azuremachinetemplates.yaml
@@ -259,6 +259,11 @@ spec:
                           Use this setting only if VMExtensions are not supported by your image, as it disables CAPZ bootstrapping extension used for detecting Kubernetes bootstrap failure.
                           This may only be set to True when no extensions are configured on the virtual machine.
                         type: boolean
+                      disableVMBootstrapExtension:
+                        description: |-
+                          DisableVMBootstrapExtension specifies whether the VM bootstrap extension should be disabled on the virtual machine.
+                          Use this setting if you want to disable only the bootstrapping extension and not all extensions.
+                        type: boolean
                       dnsServers:
                         description: DNSServers adds a list of DNS Server IP addresses
                           to the VM NICs.


Reviews:
user: login: jackfrancis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T22:55:58Z
user: login: nawazkh
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T19:51:11Z
user: login: willie-yao
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:18:28Z
user: login: mboersma
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T13:08:51Z
user: login: mboersma
body: /lgtm
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T18:56:22Z
user: login: mboersma
body: /lgtm
/approve
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T17:19:13Z

Review comments:
diff_hunk: @@ -112,7 +112,7 @@ const (
 
 var (
 	// LinuxBootstrapExtensionCommand is the command the VM bootstrap extension will execute to verify Linux nodes bootstrap completes successfully.
-	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
+	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then echo 'Error joining node to cluster: kubeadm init failed. To debug, check the cloud-init, kubelet, or other bootstrap logs: https://capz.sigs.k8s.io/self-managed/troubleshooting.html?highlight=kubeadmcontrolplane#checking-cloud-init-logs-ubuntu.'; exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
path: azure/defaults.go
user: login: jackfrancis
body: we might want to say "kubeadm init or join", as this error could occur on either the first control plane node (init) or any other node that joins the cluster afterwards
created_at: 2025-03-24T22:55:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -112,7 +112,7 @@ const (
 
 var (
 	// LinuxBootstrapExtensionCommand is the command the VM bootstrap extension will execute to verify Linux nodes bootstrap completes successfully.
-	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
+	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then echo 'Error joining node to cluster: kubeadm init or join failed. To debug, check the cloud-init, kubelet, or other bootstrap logs: https://capz.sigs.k8s.io/self-managed/troubleshooting.html?highlight=kubeadmcontrolplane#checking-cloud-init-logs-ubuntu.'; exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
path: azure/defaults.go
user: login: nawazkh
body: I am not so sure of the helpfulness of this message. 
- Unless the verbosity of kubeadm init/join is increased by `--v=<any number greater than 5>` users do not have a way to debug the issue. 
- Where else do we expect to see this message ? Is it available on the AzureMachine.Status ? 

Suggestions:
- Should we be renaming `#checking-cloud-init-logs-ubuntu` to `checking-cloud-init-logs-linux` since Linux is more generic ? 
- Can we paraphrase the error message from `Error joining node to cluster: kubeadm init or join failed......` to `Error: kubeadm init or join failed. Refer: https://capz.sigs.k8s.io/self-managed/troubleshooting.html?highlight=kubeadmcontrolplane#checking-cloud-init-logs-ubuntu for more guidance on debugging this.`

created_at: 2025-03-25T19:51:11Z
author_association: MEMBER
diff_hunk: @@ -112,7 +112,7 @@ const (
 
 var (
 	// LinuxBootstrapExtensionCommand is the command the VM bootstrap extension will execute to verify Linux nodes bootstrap completes successfully.
-	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
+	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then echo 'Error joining node to cluster: kubeadm init or join failed. To debug, check the cloud-init, kubelet, or other bootstrap logs: https://capz.sigs.k8s.io/self-managed/troubleshooting.html?highlight=kubeadmcontrolplane#checking-cloud-init-logs-ubuntu.'; exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
path: azure/defaults.go
user: login: willie-yao
body: Great suggestions! Gonna work on modifying it now...

> Unless the verbosity of kubeadm init/join is increased by --v=<any number greater than 5> users do not have a way to debug the issue.

Is that true? If so, is there a way to change that from CAPZ? I've debugged bootstrap failures previously with these methods.

> Where else do we expect to see this message ? Is it available on the AzureMachine.Status ?

I think so but I'm not 100% sure. It should show up in the CAPZ controller logs for sure
created_at: 2025-03-26T16:18:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -111,7 +111,7 @@ const (
 
 var (
 	// LinuxBootstrapExtensionCommand is the command the VM bootstrap extension will execute to verify Linux nodes bootstrap completes successfully.
-	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
+	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then echo 'Error joining node to cluster: kubeadm init failed. To debug, check the cloud-init, kubelet, or other bootstrap logs: https://capz.sigs.k8s.io/self-managed/troubleshooting.html?highlight=kubeadmcontrolplane#checking-cloud-init-logs-ubuntu.'; exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
path: azure/defaults.go
user: login: mboersma
body: ```suggestion
	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then echo 'Error joining node to cluster: kubeadm init or join failed. To debug, check the cloud-init, kubelet, or other bootstrap logs: https://capz.sigs.k8s.io/self-managed/troubleshooting.html#checking-cloud-init-logs-ubuntu'; exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
```
created_at: 2025-04-18T13:04:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -111,7 +111,7 @@ const (
 
 var (
 	// LinuxBootstrapExtensionCommand is the command the VM bootstrap extension will execute to verify Linux nodes bootstrap completes successfully.
-	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
+	LinuxBootstrapExtensionCommand = fmt.Sprintf("for i in $(seq 1 %d); do test -f %s && break; if [ $i -eq %d ]; then echo 'Error joining node to cluster: kubeadm init failed. To debug, check the cloud-init, kubelet, or other bootstrap logs: https://capz.sigs.k8s.io/self-managed/troubleshooting.html?highlight=kubeadmcontrolplane#checking-cloud-init-logs-ubuntu.'; exit 1; else sleep %d; fi; done", bootstrapExtensionRetries, bootstrapSentinelFile, bootstrapExtensionRetries, bootstrapExtensionSleep)
path: azure/defaults.go
user: login: mboersma
body: The `highlight=` URL param doesn't add anything, and it's easier to copy-and-paste without a trailing dot (`.`).
created_at: 2025-04-18T13:06:27Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov[bot]
created_at: 2025-03-24T22:54:13Z
body: ## [Codecov](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/pull/5509?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs) Report
Attention: Patch coverage is `32.50000%` with `27 lines` in your changes missing coverage. Please review.
> Project coverage is 52.87%. Comparing base [(`bc33778`)](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/commit/bc3377848f774a62ca09fa10538c96aa5bfc9cb2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs) to head [(`b0f3689`)](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/commit/b0f3689a6615e62226752effc0c4a33b4d0d1330?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs).
> Report is 8 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/pull/5509?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs) | Patch % | Lines |
|---|---|---|
| [azure/scope/machine.go](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/pull/5509?src=pr&el=tree&filepath=azure%2Fscope%2Fmachine.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs#diff-YXp1cmUvc2NvcGUvbWFjaGluZS5nbw==) | 29.41% | [24 Missing :warning: ](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/pull/5509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs) |
| [api/v1beta1/azuremachine\_webhook.go](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/pull/5509?src=pr&el=tree&filepath=api%2Fv1beta1%2Fazuremachine_webhook.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs#diff-YXBpL3YxYmV0YTEvYXp1cmVtYWNoaW5lX3dlYmhvb2suZ28=) | 50.00% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/pull/5509?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #5509      +/-   ##
==========================================
- Coverage   52.87%   52.87%   -0.01%     
==========================================
  Files         272      272              
  Lines       29473    29481       +8     
==========================================
+ Hits        15583    15587       +4     
- Misses      13083    13086       +3     
- Partials      807      808       +1     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/kubernetes-sigs/cluster-api-provider-azure/pull/5509?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kubernetes-sigs).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: willie-yao
created_at: 2025-03-25T18:45:25Z
body: /retest
author_association: CONTRIBUTOR
user: login: willie-yao
created_at: 2025-03-28T22:52:57Z
body: As per discussion with @jackfrancis, I'll work on potentially changing the script to search the cloud init logs automatically for any kind of error, and displaying that to the user.
author_association: CONTRIBUTOR
user: login: willie-yao
created_at: 2025-04-17T23:47:02Z
body: /assign @nawazkh @mboersma 
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-04-18T18:56:27Z
body: LGTM label has been added.  <details>Git tree hash: 8e183ab5a3cdde10510a60e336ce6d482e8e8f5c</details>
author_association: CONTRIBUTOR
user: login: nawazkh
created_at: 2025-04-18T19:10:47Z
body: /lgtm 
~/approve~ <- Removed it for a question 💭 

author_association: MEMBER
user: login: nawazkh
created_at: 2025-04-18T19:14:33Z
body: Can we have an e2e test that could check for disabled/enabled CAPZ Linux Bootstrap Extension ? 
We can pursue that e2e test via another PR if it is too big of an ask, but for our confidence, it be nice to have :) 
What are your thoughts @willie-yao  ? 
author_association: MEMBER
user: login: willie-yao
created_at: 2025-04-18T19:29:21Z
body: @nawazkh I'm not sure if adding an E2E test for this would be too much. We'll have to create an entirely new cluster with its own template to check it, and verifying the list of extensions in the unit test should be enough imo. It should be an easy implementation, but I'm just not sure if it's what we should do. wdyt @mboersma @jackfrancis 
author_association: CONTRIBUTOR
user: login: nawazkh
created_at: 2025-04-18T21:17:53Z
body: > @nawazkh I'm not sure if adding an E2E test for this would be too much. We'll have to create an entirely new cluster with its own template to check it, and verifying the list of extensions in the unit test should be enough imo. It should be an easy implementation, but I'm just not sure if it's what we should do. wdyt @mboersma @jackfrancis

Thank you for sharing the context, I align with your thoughts. Just wanted a source of truth to show that this works. 
Since you already tested this in your dev time, I am ok with skipping an e2e test for this scenario.
Besides, having a dedicated e2e would be an overkill. 
We could probably tweak an existing e2e for this if need be :) 

/lgtm
author_association: MEMBER
user: login: nawazkh
created_at: 2025-04-18T21:18:29Z
body: I will wait for @mboersma or @jackfrancis to weigh in since you pinged them. Otherwise, ready to be shipped from my end!
author_association: MEMBER
user: login: k8s-ci-robot
created_at: 2025-04-21T17:19:20Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cluster-api-provider-azure/pull/5509#pullrequestreview-2781814682" title="Approved">mboersma</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcluster-api-provider-azure).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/main/OWNERS)~~ [mboersma]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: nawazkh
created_at: 2025-04-22T16:28:21Z
body: Approved yesterday and still hasn't merged ? Strange. 
author_association: MEMBER
user: login: nawazkh
created_at: 2025-04-22T17:46:48Z
body: Raised a question on tide regarding this in https://kubernetes.slack.com/archives/C7J9RP96G/p1745342990569219 
I am manually merging this PR for now. 
author_association: MEMBER