Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set. Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?

Related discussion:
text: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "testing" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/apis/policy" "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/fake" ) func TestEviction(t *testing.T) { testcases := []struct { name     string pdbs     []runtime.Object pod      *api.Pod eviction *policy.Eviction expectError   bool expectDeleted bool }{ { name:          "no pdbs, unscheduled pod, nil delete options, deletes immediately", pdbs:          nil, pod:           validNewPod(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}}, expectDeleted: true, }, { name:          "no pdbs, scheduled pod, nil delete options, deletes gracefully", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}}, expectDeleted: false, // not deleted immediately because of graceful deletion }, { name:          "no pdbs, scheduled pod, empty delete options, deletes gracefully", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: &metav1.DeleteOptions{}}, expectDeleted: false, // not deleted immediately because of graceful deletion }, { name:          "no pdbs, scheduled pod, graceless delete options, deletes immediately", pdbs:          nil, pod:           func() *api.Pod { pod := validNewPod(); pod.Spec.NodeName = "foo"; return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, { name: "matching pdbs with no disruptions allowed", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"a": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 0}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:    &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectError: true, }, { name: "matching pdbs with disruptions allowed", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"a": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 1}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, { name: "non-matching pdbs", pdbs: []runtime.Object{&policy.PodDisruptionBudget{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, Spec:       policy.PodDisruptionBudgetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"b": "true"}}}, Status:     policy.PodDisruptionBudgetStatus{PodDisruptionsAllowed: 0}, }}, pod: func() *api.Pod { pod := validNewPod() pod.Labels = map[string]string{"a": "true"} pod.Spec.NodeName = "foo" return pod }(), eviction:      &policy.Eviction{ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "default"}, DeleteOptions: metav1.NewDeleteOptions(0)}, expectDeleted: true, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { testContext := genericapirequest.WithNamespace(genericapirequest.NewContext(), metav1.NamespaceDefault) storage, _, _, server := newStorage(t) defer server.Terminate(t) defer storage.Store.DestroyFunc() if tc.pod != nil { if _, err := storage.Create(testContext, tc.pod, nil, &metav1.CreateOptions{}); err != nil { t.Error(err) } } client := fake.NewSimpleClientset(tc.pdbs...) evictionRest := newEvictionStorage(storage.Store, client.Policy()) _, err := evictionRest.Create(testContext, tc.eviction, nil, &metav1.CreateOptions{}) if (err != nil) != tc.expectError { t.Errorf("expected error=%v, got %v", tc.expectError, err) return } if tc.expectError { return } if tc.pod != nil { existingPod, err := storage.Get(testContext, tc.pod.Name, &metav1.GetOptions{}) if tc.expectDeleted { if !apierrors.IsNotFound(err) { t.Errorf("expected to be deleted, lookup returned %#v", existingPod) } return } else if apierrors.IsNotFound(err) { t.Errorf("expected graceful deletion, got %v", err) return } if err != nil { t.Errorf("%#v", err) return } if existingPod.(*api.Pod).DeletionTimestamp == nil { t.Errorf("expected gracefully deleted pod with deletionTimestamp set, got %#v", existingPod) } } }) } } </ins>