Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have about 8000 pdbs in our cluster. When we call several eviction requests at the same time, etcd will meet a performance spike. What you expected to happen: eviction request read pdbs from apiserver cache How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/sig scalability /cc throttling evictions at server-side may be a better fix i presume
We use eviction to auto migrate pods. If throttling eviction requests, it will make migration latency too long. I am not sure if list without rv by designed (get pdb consistently) or not (just forget to add rv).
What does eviction do (I can't remember from the top of my head)? Would you be able to quickly point me to the code?
Create create a eviction will list all pdbs, make sure eviction the pod won't break any pdb. If check passed, it will create the eviction, otherwise it will return 429 to reject.
Thanks - so this is what is problematic: So what you're saying is that you have 8000 PDBs in a single namespace? [because we only list pdbs from pod`s namespace]. I'm we should list from etcd, because otherwise we risk using old version of pdb ( are allowed to evict only 1 pod, and we just did it, but that wasn't yet propagated to apiserver cache and we would evict another pod). My personal guess would be that the problem may not really be number of PDB, but amount of concurrent requests touching the same pdb - because that would be resulting in conflicts and retrying them in: I'm not sure we can do much about it (at least I don't see any immediate low-hanging fruit).
Thanks a lot for your reply. How about retrying conflict update pdb, instead of re-listing all pdbs?
How about retrying conflict update pdb, instead of re-listing all pdbs? Hmm - I'm not sure I understand. Ohh, but I know what else we can do (maybe that's what you had on your mind). In case of conflict, instead of relisting all PDBs, we should simply GET the current version of that PDB (we already know that there is exactly one which we are supposed to use). That would reduce the cost of retries significantly. - would you be able to take a look into it and prepare a PR? I'm happy to review it.
- That's what I mean. I will take a look and send a PR in a few days.
- great! Please ping me on slack if I won't be responding to it.
alternatively, will pagination probably help ease the pressure? it'll skip the cache tho but relaxes the "spike"..
Does pagination mean range key prefix with limit?

Discussion:
} var rtStatus *metav1.Status var pdbName string <del> err = retry.RetryOnConflict(EvictionsRetry, func() error { </del> <ins> err = func() error { </ins> pdbs, err := r.getPodDisruptionBudgets(ctx, pod) if err != nil { return err

Code:    500, } return nil <del> } else if len(pdbs) == 1 { pdb := pdbs[0] pdbName = pdb.Name </del> <ins> } if len(pdbs) == 0 { return nil } pdb := &pdbs[0] pdbName = pdb.Name refresh := false err = retry.RetryOnConflict(EvictionsRetry, func() error { if refresh { pdb, err = r.podDisruptionBudgetClient.PodDisruptionBudgets(pod.Namespace).Get(pdbName, metav1.GetOptions{}) if err != nil { return err } } </ins> // Try to verify-and-decrement // If it was false already, or if it becomes false during the course of our retries, // raise an error marked as a 429. <del> if err := r.checkAndDecrement(pod.Namespace, pod.Name, pdb, dryrun.IsDryRun(deletionOptions.DryRun)); err != nil { </del> <ins> if err = r.checkAndDecrement(pod.Namespace, pod.Name, *pdb, dryrun.IsDryRun(deletionOptions.DryRun)); err != nil { refresh = true </ins> return err } <del> } return nil }) </del> <ins> return nil }) return err }() </ins> if err == wait.ErrWaitTimeout { err = errors.NewTimeoutError(fmt.Sprintf("couldn't update PodDisruptionBudget %q due to conflicts", pdbName), 10) }