Change:
What happened: In a StatefulSet where each Pod has its own persistent volume provisioned by , it occasionally takes up to 30 min to detach and re-attach those disks to their node during a rollout. The following Pod events repeat: A subset of the controller-manager's logs (full logs on demand): State of the disk few minutes before the attach operation succeeds, returned by the Azure API: In this particular VMSS, each instance has at most 2 disks attached. In busy clusters, a node can have up to 5 disks attached. The operation eventually succeeds, ~20m later in this case: What you expected to happen: Disks take no longer than 3 min to re-attach (the typical time it takes in our customers' environments). How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brStorage class <details<summarydefault-</summary</details</brAnything else we need to know?: Environment: Kubernetes version: v1.12.7 Cloud provider: Azure (via AKS-Engine) OS (e.g: ): Ubuntu 16.04.6 LTS Kernel: 4.15.0-1042-azure Install tools: Network plugin and version: Others: observed in westeurope and eastus regions paging /sig azure
can you also provide following info on master node: How many disks are on one VM? We are experiencing slow disk attach/detach issue when disk num is large:
I just the info about disks to the original message: between 2 and 5 per VMSS instance. The command you posted returns nothing, but if I remove "vm" from the last grep I get: Feel free to catch up on Slack if you want to have a real-time chat.
/assign paste some info here: the 2 disk attach/detach operations costs s totally, that's reason why it's stuck in so long time. BTW, there is metrics calculation issue here: , but anyway the two operations cost too long time in vmss API call level.
this seems to be an extreme case. In other environments where the issue occurs, API calls cost between 15 and 60 sec on average.
API calls cost between 15 and 60 sec on average and still takes up 30min to re-attach? could it repro easily?
Only in selected clusters unfortunately (about 5%), but when it happens it happens quite consistently. I haven't been able to reproduce outside of clusters belonging to live customers, which makes the issue a bit tricky to troubleshoot at the moment. I'm sitting in front of a cluster where it's currently happening. A Pod has been reporting for 53 min. Yesterday I observed it in the same cluster for 2 hours. We eventually deleted the Pod to unblock the situation (which doesn't always work btw). The following Pod events repeat over and over again: The disk has and just like in my first message. The controller-manager metrics also don't tell us much (14h uptime):
could you provide detailed steps about how to repro this issue? I tried to drain statefulset pods from one node to another on k8s v1.12.7, cannot repro.
The correct way to get disk attach/detach metrics on k8s v1.12.7 is like this The rolling update would cause k8s to detach disk first and then attach disk after, so I will still get following error in my testing env, but finally it could recover:
This one is a patched 1.10, with your retry fix. I have some metrics from an affected 1.12 cluster, but like I said, nothing very useful: We only have one customer on 1.12 so I'm waiting for the next incident to get back to you on a proper 1.12 setup. My understanding of what happens in most (if not all) cases is: Imagine this simplified setup 2 nodes 3 Pods belonging to a Deployment "commerce" 3 Pods belonging to a Deployment "api" 3 Pods with Azure Disks belonging to a StatefulSet "solr" We update both Deployments and the StatefulSet simultaneously (we have an operator that actually does exactly this). Both and terminate before . This time and are both scheduled on . is still Terminating. When is finally down, is already packed, so the scheduler has no other choice but to move it to . And then we are in that mode where the disk is detached, but never re-attached. It shouldn't be any different from a drain, but at least it can maybe help you understand our typical repro scenario. We recently optimized scheduling to avoid that schenario where stateful Pods must be relocated, but other events can cause relocation like nodes failures, autoscaling, ...
here is the conclusion about what we found now: Periodically k8s cluster failed to attach a disk to a VMSS instance, that disk is actually in “unattached” state, it's caused by vmss instance in unhealthy state, although by azure cli command , the instance shows healthy. workaround: run to update the VMSS instance manually, it could recover in a minute or so We are asking VMSS team to find the root cause of this issue. In brief, if you find k8s could not attach an state disk to a VMSS node, it's probably due to the vmss instance issue, run to update the VMSS instance manually could solve this issue.

Discussion:
return false } <ins> // processHTTPRetryResponse : return true means stop retry, false means continue retry </ins> func (az *Cloud) processHTTPRetryResponse(service *v1.Service, reason string, resp *http.Response, err error) (bool, error) { <del> if resp != nil && isSuccessHTTPResponse(resp) { </del> <ins> if err == nil && resp != nil && isSuccessHTTPResponse(resp) { </ins> // HTTP 2xx suggests a successful response return true, nil }

} func (az *Cloud) processHTTPResponse(service *v1.Service, reason string, resp *http.Response, err error) error { <del> if isSuccessHTTPResponse(resp) { </del> <ins> if err == nil && isSuccessHTTPResponse(resp) { </ins> // HTTP 2xx suggests a successful response return nil }

stop: true, }, { <ins> code: http.StatusOK, err:  fmt.Errorf("some error"), stop: false, }, { </ins> code: 399, stop: true, },