Code change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: after granting sp access to azure ACR , pull image from ACR would fail, and after wait about 15-30min, pull image would success. Root cause is that needs to be refreshed when doing , otherwise it will always return empty registry list. Related issue: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.7 - v1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign I will implement that function.
Thanks -- can we add a milestone label?
+1. We should fix this within v1.12 release and cherry pick to old releases. /priority critical-urgent /milestone v1.12
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I'm not sure if this is related, but I'm seeing nodes (after they've been working fine for a few days) randomly start to fail pulling images with the error . Edit: I see this as the culprit: , after which all image pulls start to fail.
this is not related, original issue is that after granting sp access to azure acr, pull image still fails with . After my repro and investigation(really spent a lot of time...), I found the issue lies here, would return empty even after granting sp access to azure acr, and after around 15min, it returns correct loginServer list: Looks like it's a SDK or acr list with specific sp issue, workaround is wait around 15min or more time, it could recover by itself. Paste my investigation logs here, I grant sp access to acr in around , you could see it returns empty registry list first and then after a few more minutes, it returns the correct list with loginServer:
BTW, in v1.12, acr list() uses new API, I will check whether in v1.12 would have same issue. Acutally I have already implemented LazyProvide func(check here: ), while it won't fix the original issue.
Update: After a few more tries, I found it's related to this code snippet, before , we need to generate a new ServicePrincipalToken every time: I will move this code snippet into Provide() func, and send out a PR! The failure recover time is within 2min now.

Related discussion:
text: ctx, cancel := getContextWithCancel() defer cancel() <del> glog.V(4).Infof("listing registries") result, err := a.registryClient.List(ctx) if err != nil { glog.Errorf("Failed to list registries: %v", err) return cfg } for ix := range result { loginServer := getLoginServer(result[ix]) var cred *credentialprovider.DockerConfigEntry </del> <ins> if a.config.UseManagedIdentityExtension { glog.V(4).Infof("listing registries") result, err := a.registryClient.List(ctx) if err != nil { glog.Errorf("Failed to list registries: %v", err) return cfg } </ins> <del> if a.config.UseManagedIdentityExtension { cred, err = getACRDockerEntryFromARMToken(a, loginServer) </del> <ins> for ix := range result { loginServer := getLoginServer(result[ix]) glog.V(2).Infof("loginServer: %s", loginServer) cred, err := getACRDockerEntryFromARMToken(a, loginServer) </ins> if err != nil { continue } <del> } else { cred = &credentialprovider.DockerConfigEntry{ </del> <ins> cfg[loginServer] = *cred } } else { // Add our entry for each of the supported container registry URLs for _, url := range containerRegistryUrls { cred := &credentialprovider.DockerConfigEntry{ </ins> Username: a.config.AADClientID, Password: a.config.AADClientSecret, Email:    dummyRegistryEmail, } <ins> cfg[url] = *cred </ins> } <del> cfg[loginServer] = *cred </del> } return cfg }