Code change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: after granting sp access to azure ACR , pull image from ACR would fail, and after wait about 15-30min, pull image would success. Root cause is that needs to be refreshed when doing , otherwise it will always return empty registry list. Related issue: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.7 - v1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign I will implement that function.
Thanks -- can we add a milestone label?
+1. We should fix this within v1.12 release and cherry pick to old releases. /priority critical-urgent /milestone v1.12
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I'm not sure if this is related, but I'm seeing nodes (after they've been working fine for a few days) randomly start to fail pulling images with the error . Edit: I see this as the culprit: , after which all image pulls start to fail.
this is not related, original issue is that after granting sp access to azure acr, pull image still fails with . After my repro and investigation(really spent a lot of time...), I found the issue lies here, would return empty even after granting sp access to azure acr, and after around 15min, it returns correct loginServer list: Looks like it's a SDK or acr list with specific sp issue, workaround is wait around 15min or more time, it could recover by itself. Paste my investigation logs here, I grant sp access to acr in around , you could see it returns empty registry list first and then after a few more minutes, it returns the correct list with loginServer:
BTW, in v1.12, acr list() uses new API, I will check whether in v1.12 would have same issue. Acutally I have already implemented LazyProvide func(check here: ), while it won't fix the original issue.
Update: After a few more tries, I found it's related to this code snippet, before , we need to generate a new ServicePrincipalToken every time: I will move this code snippet into Provide() func, and send out a PR! The failure recover time is within 2min now.

Related discussion:
text: { Name: to.StringPtr("foo"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("foo-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.io"), </ins> }, }, { Name: to.StringPtr("bar"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("bar-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.cn"), </ins> }, }, { Name: to.StringPtr("baz"), RegistryProperties: &containerregistry.RegistryProperties{ <del> LoginServer: to.StringPtr("baz-microsoft.azurecr.io"), </del> <ins> LoginServer: to.StringPtr("*.azurecr.de"), }, }, { Name: to.StringPtr("bus"), RegistryProperties: &containerregistry.RegistryProperties{ LoginServer: to.StringPtr("*.azurecr.us"), </ins> }, }, }