Change:
if len(items) == 0 { lruItemObj, ok := l.liveLookupCache.Get(a.GetNamespace()) if !ok || lruItemObj.(liveLookupEntry).expiry.Before(time.Now()) { <del> // TODO: If there are multiple operations at the same time and cache has just expired, // this may cause multiple List operations being issued at the same time. // If there is already in-flight List() for a given namespace, we should wait until // it is finished and cache is updated instead of doing the same, also to avoid // throttling - see #22422 for details. liveList, err := l.client.CoreV1().LimitRanges(a.GetNamespace()).List(context.TODO(), metav1.ListOptions{}) </del> <ins> // Fixed: #22422 // use singleflight to alleviate simultaneous calls to lruItemObj, err, _ = l.group.Do(a.GetNamespace(), func() (interface{}, error) { liveList, err := l.client.CoreV1().LimitRanges(a.GetNamespace()).List(context.TODO(), metav1.ListOptions{}) if err != nil { return nil, admission.NewForbidden(a, err) } newEntry := liveLookupEntry{expiry: time.Now().Add(l.liveTTL)} for i := range liveList.Items { newEntry.items = append(newEntry.items, &liveList.Items[i]) } l.liveLookupCache.Add(a.GetNamespace(), newEntry) return newEntry, nil }) </ins> if err != nil { <del> return nil, admission.NewForbidden(a, err) } newEntry := liveLookupEntry{expiry: time.Now().Add(l.liveTTL)} for i := range liveList.Items { newEntry.items = append(newEntry.items, &liveList.Items[i]) </del> <ins> return nil, err </ins> } <del> l.liveLookupCache.Add(a.GetNamespace(), newEntry) lruItemObj = newEntry </del> } lruEntry := lruItemObj.(liveLookupEntry)

Discussion:
Recently we observed high latencies on POST pods on our scalability tests. For more context see: It appeared to be throttling related, e.g.: As an example look at resource quota admission controller. In there was some fallback to use lru cache in case of on objects. However, not what happens if we are sending multiple POST pods at the same time to apiserver. What happens in Admit() if there are no results in Indexer, we list ResourceQuotas. But if there are multiple calls at the same time, there can be multiple LISTs before the cache is updated: But this can cause throttling of these requests. What we should do is: if there is List() in flight, we should wait until it is finished and lookupCache is updated.
- thanks for the analysis. - fyi, this is an interesting side-effect of the LRU cache pattern, not sure if we have adopted it in other places as well, but good to keep in mind.
To clarify, it was adopted in and admission plug-ins in separate PRs.
- it took me some time, but was useful experience. Thanks for assigning yourself.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details