Change:
What happened: When I try to test after implementing the 1st approach according to , I found pod disruption budgets cannot handle concurrent requests. The error message is: It is an Optimistic Lock error message. It means PDB can't handle concurrent requests. How to reproduce it (as minimally and precisely as possible): 1) Create a deployment with 6 replicas and schedule them to the same node, say node A. You can leave only one node scheduling enabled, and then create the deployment. 2) Create the PDB that match the deployment's label 3) Uncordon another node, say node B 4) Drain node A using concurrent POST eviction requests. Ref: I'm not sure if this behavior is OK for PDB. I think I can make work by waiting for a while if current pod label(s) match previous pod label(s).
Oh, that's interesting. I think the problem is that this function doesn't retry if UpdateStatus() fails due to version conflict. I think the solution is to modify the caller, i.e. this function so that instead of just it instead does something like This will loop re-doing checkAndDecrement() and getPodDisruptionBudgets() until checkAndDecrement() succeeds, or fails with a non-conflict error. Does that make sense? cc/ for sanity check
Or use PATCH?
Yes, PATCH should work also. I haven't written code that uses PATCH, so I can't advise exactly the code to write here (there are examples by grepping for patch, but I'm not sure exactly what is the right way to do it from inside of API server code, which is where we are here). can you do a PR to implement this, either using the approach I described above or using PATCH? I think this is important to get into 1.5.
I think PATCH may NOT be safe here. For example: 1) current pdb 2) creating eviction A: get 3) creating eviction B: get 4) creating eviction A: PATCH pdb 5) creating eviction B: PATCH pdb 6) pdb's . It's wrong. Retry loop make more sense to me.
Good point! You're right. Retry loop should be OK.
I can help to create a PR using retry loop.
Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
Done.
who is actually working on this issue? Could one of you set yourselves as an assignee for this issue? Thanks! cc:
retry logic with PR needs reviews/lgtm

Discussion:
"//pkg/api/validation:go_default_library", "//pkg/apis/policy:go_default_library", "//pkg/client/clientset_generated/internalclientset/typed/policy/internalversion:go_default_library", <ins> "//pkg/client/retry:go_default_library", </ins> "//pkg/kubelet/client:go_default_library", "//pkg/labels:go_default_library", "//pkg/registry/cachesize:go_default_library",

"//pkg/registry/generic/registry:go_default_library", "//pkg/runtime:go_default_library", "//pkg/storage:go_default_library", <ins> "//pkg/util/wait:go_default_library", </ins> ], )

"k8s.io/kubernetes/pkg/api/unversioned" "k8s.io/kubernetes/pkg/apis/policy" policyclient "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/policy/internalversion" <ins> "k8s.io/kubernetes/pkg/client/retry" </ins> "k8s.io/kubernetes/pkg/labels" "k8s.io/kubernetes/pkg/registry/generic/registry" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> ) const (

MaxDisruptedPodSize = 2000 ) <ins> // EvictionsRetry is the retry for a conflict where multiple clients // are making changes to the same resource. var EvictionsRetry = wait.Backoff{ Steps:    20, Duration: 500 * time.Millisecond, Factor:   1.0, Jitter:   0.1, } </ins> func newEvictionStorage(store *registry.Store, podDisruptionBudgetClient policyclient.PodDisruptionBudgetsGetter) *EvictionREST { return &EvictionREST{store: store, podDisruptionBudgetClient: podDisruptionBudgetClient} }

return nil, err } pod := obj.(*api.Pod) <del> pdbs, err := r.getPodDisruptionBudgets(ctx, pod) if err != nil { return nil, err } if len(pdbs) > 1 { return &unversioned.Status{ Status:  unversioned.StatusFailure, Message: "This pod has more than one PodDisruptionBudget, which the eviction subresource does not support.", Code:    500, }, nil } else if len(pdbs) == 1 { pdb := pdbs[0] // Try to verify-and-decrement // If it was false already, or if it becomes false during the course of our retries, // raise an error marked as a 429. ok, err := r.checkAndDecrement(pod.Namespace, pod.Name, pdb) </del> <ins> var rtStatus *unversioned.Status err = retry.RetryOnConflict(EvictionsRetry, func() error { pdbs, err := r.getPodDisruptionBudgets(ctx, pod) </ins> if err != nil { <del> return nil, err </del> <ins> return err </ins> } <del> if !ok { return &unversioned.Status{ Status: unversioned.StatusFailure, // TODO(mml): Include some more details about why the eviction is disallowed. // Ideally any such text is generated by the DisruptionController (offline). Message: "Cannot evict pod as it would violate the pod's disruption budget.", Code:    429, // TODO(mml): Add a Retry-After header.  Once there are time-based // budgets, we can sometimes compute a sensible suggested value.  But // even without that, we can give a suggestion (10 minutes?) that // prevents well-behaved clients from hammering us. }, nil </del> <ins> if len(pdbs) > 1 { rtStatus = &unversioned.Status{ Status:  unversioned.StatusFailure, Message: "This pod has more than one PodDisruptionBudget, which the eviction subresource does not support.", Code:    500, } return nil } else if len(pdbs) == 1 { pdb := pdbs[0] // Try to verify-and-decrement // If it was false already, or if it becomes false during the course of our retries, // raise an error marked as a 429. ok, err := r.checkAndDecrement(pod.Namespace, pod.Name, pdb) if err != nil { return err } if !ok { rtStatus = &unversioned.Status{ Status: unversioned.StatusFailure, // TODO(mml): Include some more details about why the eviction is disallowed. // Ideally any such text is generated by the DisruptionController (offline). Message: "Cannot evict pod as it would violate the pod's disruption budget.", Code:    429, // TODO(mml): Add a Retry-After header.  Once there are time-based // budgets, we can sometimes compute a sensible suggested value.  But // even without that, we can give a suggestion (10 minutes?) that // prevents well-behaved clients from hammering us. } } </ins> } <ins> return nil }) if err != nil { return nil, err } if rtStatus != nil { return rtStatus, nil </ins> } // At this point there was either no PDB or we succeded in decrementing