Change:
zone := c.makeZone(c.Location, zoneID) klog.V(4).Infof("Got zone %q for Azure disk %q", zone, diskName) <del> labels := map[string]string{ v1.LabelZoneRegion:        c.Location, v1.LabelZoneFailureDomain: zone, } </del> <ins> labels[v1.LabelZoneFailureDomain] = zone </ins> return labels, nil }

Discussion:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Overview A persistent volume claim / storage class configured in a certain way can cause all the Kube API servers on an ARO cluster to crashloop. We’ve seen this issue on a production cluster (ICM ). In development we have proven one vector by which the cluster could have got into this state but it doesn’t seem that likely. We wonder if there are other vectors. We don’t currently have a good way to rescue a cluster that gets into this state because our service accesses are predicated on API server availability. Request We would like help: agree and expedite an appropriate fix in order to protect all other production clusters. suggestions on how we might fix the broken production cluster. Causes certain cases upon PV create, the persistentvolume admission controller attempts to retrieve the managed disk object via the Azure API, but in ARO the relevant client object is a nil pointer. The pointer is dereferenced and the kube API server panics. PVC controller repeatedly attempts to create the PV object, which DoSes the kube API server. client object in (1) is nil because of ARO’s cloud provider authentication architecture, which differs from that of OCP. More information below. Technical detail For a panic stack trace and reproducer, see Creation of an Azure Disk persistent volume is attempted The persistentvolume admission controller runs, (l persistentVolumeLabel) Admit(), in In our possible reproducer, the admission controller does not return early in (l persistentVolumeLabel) findVolumeLabels(), because the zone labels are not set.

A cloud provider is (incorrectly) instantiated via: getAzurePVLabeler(), cloudprovider.GetCloudProvider(), NewCloud(), (az Cloud) InitializeCloudFromConfig, We see the following loglines: Azure cloud provider is starting without credentials Azure cloudprovider using try backoff: retries=%d, exponent=%f, duration=%d, jitter=%f We believe that an early return occurs at , before c.DisksClient is initialised, because servicePrincipalToken is nil Later, a nil pointer dereference of c.DisksClient occurs in (c Cloud) GetAzureDiskLabels, OCP 4.3 cloud provider authentication architecture ================================================== The installer lays down an Azure managed identity (MSI) resource, grants it Contributor access to cluster resource group and to any pre-existing virtual network (VNET). In the context of the Kube API server, the persistentvolume admission controller uses MSI to authenticate to Azure for its API calls. ARO 4.3 cloud provider authentication architecture ================================================== ARO can’t follow OCP 4.3’s architecture because of the following chicken-and-egg problem: As an Azure service, ARO is not permitted to add role assignments to the customer’s pre-existing virtual network - the customer must explicitly do this. No cluster managed identity can exist before the customer runs (by definition), so there is nothing for the customer to grant the role assignment to. As a result, ARO does not use a managed identity at all, and instead uses a user-created cluster service principal. ARO carries a patch to the installer which creates a kube-config/azure-cloud-provider Secret during cluster install. The controller manager merges the cloud config with this Secret at start-up time (see ), however this code path does not run in the API server. In ARO, any calls to the Azure API from the API server are probably broken. Prior to GA, ARO wrote the cloud config secret raw into the cloud config. Reverting to this behaviour would work around this issue, but is clearly suboptimal. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.

): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure
/assign