Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Some k8s client tried to evict a pod by setting negative value for deletionGracePeriodSeconds. But all eventual delete calls (and force-delete calls) made by kubelet, pod-garbage-collector, etc were not able to delete that pod, because apiserver doesnt allow deletionGracePeriodSeconds to be increased once it is set. So force-delete (which set it to 0) doesn't take effect when it's already negative. Not even kubectl is able to call delete with negative grace-period. But curl with an even more negative grace-period value worked. What you expected to happen: force-delete works. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I think the issue is in the code below As the comment suggest, in phase 1, the was set to a negative value (meaning was in the past), and in phase 2, has to be set even lower, otherwise, it will be considered as pending graceful deletion and not performing the immediate delete. I think there're two possible fixes. not allowing setting in the past ( should be non-negative). In that case, we could convert to 0 if negative. that could be set in the past. In that case, in phase 2, we treat negative value of the same as 0, and allow immediate delete. Thoughts? Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/cc /triage accepted
No component in the system should ever treat the timestamp of DeletionTimestamp as authoritative. In Kube, components are required to calculate the time period from when they observe the event in their local clock (i.e. 30s from now), not use the deletion timestamp. A negative value of deletion grace period should be rejected by the apiserver as an invalid value (bad request / invalid field).
attempts to take a missing or positive deletion grace period negative should be rejected, I agree. if a negative value is already persisted, I'd be inclined to allow a negative value in an update and clip to 0
As I mentioned in the PR, eviction is passing negatives through to the store: In kubectl, we specify the default value for GracePeriod is -1. In the case of delete, we simply don't set the value in the deleteOptions we send to the API. In the case of eviction, we send the -1 value, and that gets passed along to the store.
I don't see the -1 getting sent to the eviction API. It looks like values < 0 result in a nil value being sent to the API: I would not expect the API to accept a negative value as-is.
Looks like you are correct. Do the deletionOptions go through their own validation or through some kind of validation for the evict API? The original bug report mentions 'evict' though it's not clear if the eviction API was used or not.
ValidateDeleteOptions is called inside pod deletion, but did not check that the field was non-negative. I don't see any eviction-specific validation.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/lifecycle rotten
Im investigating the and . Surprisingly see you here...

Discussion:
// 1.  spec.containers[*].image // 2.  spec.initContainers[*].image // 3.  spec.activeDeadlineSeconds <ins> // 4.  spec.terminationGracePeriodSeconds </ins> containerErrs, stop := ValidateContainerUpdates(newPod.Spec.Containers, oldPod.Spec.Containers, specPath.Child("containers")) allErrs = append(allErrs, containerErrs...)

// tolerations are checked before the deep copy, so munge those too mungedPodSpec.Tolerations = oldPod.Spec.Tolerations // +k8s:verify-mutation:reason=clone <ins> // Relax validation of immutable fields to allow it to be set to 1 if it was previously negative. if oldPod.Spec.TerminationGracePeriodSeconds != nil && *oldPod.Spec.TerminationGracePeriodSeconds < 0 && mungedPodSpec.TerminationGracePeriodSeconds != nil && *mungedPodSpec.TerminationGracePeriodSeconds == 1 { mungedPodSpec.TerminationGracePeriodSeconds = oldPod.Spec.TerminationGracePeriodSeconds // +k8s:verify-mutation:reason=clone } </ins> if !apiequality.Semantic.DeepEqual(mungedPodSpec, oldPod.Spec) { // This diff isn't perfect, but it's a helluva lot better an "I'm not going to tell you what the difference is". //TODO: Pinpoint the specific field that causes the invalid error after we have strategic merge diff specDiff := diff.ObjectDiff(mungedPodSpec, oldPod.Spec) <del> allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)n%v", specDiff))) </del> <ins> allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)n%v", specDiff))) </ins> } return allErrs

"spec: Forbidden: pod updates", "removed priority class name", }, <ins> { core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ TerminationGracePeriodSeconds: utilpointer.Int64Ptr(1), }, }, core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ TerminationGracePeriodSeconds: utilpointer.Int64Ptr(-1), }, }, "", "update termination grace period seconds", }, { core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ TerminationGracePeriodSeconds: utilpointer.Int64Ptr(0), }, }, core.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "foo", }, Spec: core.PodSpec{ TerminationGracePeriodSeconds: utilpointer.Int64Ptr(-1), }, }, "spec: Forbidden: pod updates", "update termination grace period seconds not 1", }, </ins> } for _, test := range tests {

if pod.Status.Phase == api.PodFailed || pod.Status.Phase == api.PodSucceeded { period = 0 } <ins> if period < 0 { period = 1 } </ins> // ensure the options and the pod are in sync options.GracePeriodSeconds = &period return true

api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/kubelet/client" <ins> utilpointer "k8s.io/utils/pointer" </ins> // ensure types are installed _ "k8s.io/kubernetes/pkg/apis/core/install"

func TestCheckGracefulDelete(t *testing.T) { defaultGracePeriod := int64(30) tcs := []struct { <del> in          *api.Pod gracePeriod int64 </del> <ins> name              string pod               *api.Pod deleteGracePeriod *int64 gracePeriod       int64 </ins> }{ { <del> in: &api.Pod{ </del> <ins> name: "in pending phase with has node name", pod: &api.Pod{ </ins> Spec:   api.PodSpec{NodeName: "something"}, Status: api.PodStatus{Phase: api.PodPending}, }, <del> gracePeriod: defaultGracePeriod, </del> <ins> deleteGracePeriod: &defaultGracePeriod, gracePeriod:       defaultGracePeriod, </ins> }, { <del> in: &api.Pod{ </del> <ins> name: "in failed phase with has node name", pod: &api.Pod{ </ins> Spec:   api.PodSpec{NodeName: "something"}, Status: api.PodStatus{Phase: api.PodFailed}, }, <del> gracePeriod: 0, </del> <ins> deleteGracePeriod: &defaultGracePeriod, gracePeriod:       0, </ins> }, { <del> in: &api.Pod{ </del> <ins> name: "in failed phase", pod: &api.Pod{ </ins> Spec:   api.PodSpec{}, Status: api.PodStatus{Phase: api.PodPending}, }, <del> gracePeriod: 0, </del> <ins> deleteGracePeriod: &defaultGracePeriod, gracePeriod:       0, </ins> }, { <del> in: &api.Pod{ </del> <ins> name: "in succeeded phase", pod: &api.Pod{ </ins> Spec:   api.PodSpec{}, Status: api.PodStatus{Phase: api.PodSucceeded}, }, <del> gracePeriod: 0, </del> <ins> deleteGracePeriod: &defaultGracePeriod, gracePeriod:       0, </ins> }, { <del> in: &api.Pod{ </del> <ins> name: "no phase", pod: &api.Pod{ </ins> Spec:   api.PodSpec{}, Status: api.PodStatus{}, }, <del> gracePeriod: 0, </del> <ins> deleteGracePeriod: &defaultGracePeriod, gracePeriod:       0, }, { name: "has negative grace period", pod: &api.Pod{ Spec: api.PodSpec{ NodeName:                      "something", TerminationGracePeriodSeconds: utilpointer.Int64(-1), }, Status: api.PodStatus{}, }, gracePeriod: 1, </ins> }, } for _, tc := range tcs { <del> out := &metav1.DeleteOptions{GracePeriodSeconds: &defaultGracePeriod} Strategy.CheckGracefulDelete(genericapirequest.NewContext(), tc.in, out) if out.GracePeriodSeconds == nil { t.Errorf("out grace period was nil but supposed to be %v", tc.gracePeriod) } if *(out.GracePeriodSeconds) != tc.gracePeriod { t.Errorf("out grace period was %v but was expected to be %v", *out, tc.gracePeriod) } </del> <ins> t.Run(tc.name, func(t *testing.T) { out := &metav1.DeleteOptions{} if tc.deleteGracePeriod != nil { out.GracePeriodSeconds = utilpointer.Int64(*tc.deleteGracePeriod) } Strategy.CheckGracefulDelete(genericapirequest.NewContext(), tc.pod, out) if out.GracePeriodSeconds == nil { t.Errorf("out grace period was nil but supposed to be %v", tc.gracePeriod) } if *(out.GracePeriodSeconds) != tc.gracePeriod { t.Errorf("out grace period was %v but was expected to be %v", *out, tc.gracePeriod) } }) </ins> } }

"k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apiserver/pkg/admission" <ins> utilpointer "k8s.io/utils/pointer" </ins> ) // RESTDeleteStrategy defines deletion behavior on an object that follows Kubernetes

return false, false, errors.NewConflict(schema.GroupResource{Group: gvk.Group, Resource: gvk.Kind}, objectMeta.GetName(), fmt.Errorf("the ResourceVersion in the precondition (%s) does not match the ResourceVersion in record (%s). The object might have been modified", *options.Preconditions.ResourceVersion, objectMeta.GetResourceVersion())) } } <ins> // Negative values will be treated as the value `1s` on the delete path. if gracePeriodSeconds := options.GracePeriodSeconds; gracePeriodSeconds != nil && *gracePeriodSeconds < 0 { options.GracePeriodSeconds = utilpointer.Int64(1) } if deletionGracePeriodSeconds := objectMeta.GetDeletionGracePeriodSeconds(); deletionGracePeriodSeconds != nil && *deletionGracePeriodSeconds < 0 { objectMeta.SetDeletionGracePeriodSeconds(utilpointer.Int64(1)) } </ins> gracefulStrategy, ok := strategy.(RESTGracefulDeleteStrategy) if !ok { // If we're not deleting gracefully there's no point in updating Generation, as we won't update

<ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package rest import ( "context" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" utilpointer "k8s.io/utils/pointer" ) type mockStrategy struct { RESTDeleteStrategy RESTGracefulDeleteStrategy } func (m mockStrategy) ObjectKinds(obj runtime.Object) ([]schema.GroupVersionKind, bool, error) { gvk := obj.GetObjectKind().GroupVersionKind() if len(gvk.Kind) == 0 { return nil, false, runtime.NewMissingKindErr("object has no kind field ") } if len(gvk.Version) == 0 { return nil, false, runtime.NewMissingVersionErr("object has no apiVersion field") } return []schema.GroupVersionKind{obj.GetObjectKind().GroupVersionKind()}, false, nil } func TestBeforeDelete(t *testing.T) { type args struct { strategy RESTDeleteStrategy ctx      context.Context pod      *v1.Pod options  *metav1.DeleteOptions } makePod := func(deletionGracePeriodSeconds int64) *v1.Pod { return &v1.Pod{ TypeMeta: metav1.TypeMeta{APIVersion: "v1", Kind: "Pod"}, ObjectMeta: metav1.ObjectMeta{ DeletionTimestamp:          &metav1.Time{}, DeletionGracePeriodSeconds: &deletionGracePeriodSeconds, }, } } makeOption := func(gracePeriodSeconds int64) *metav1.DeleteOptions { return &metav1.DeleteOptions{ GracePeriodSeconds: &gracePeriodSeconds, } } tests := []struct { name                           string args                           args wantGraceful                   bool wantGracefulPending            bool wantGracePeriodSeconds         *int64 wantDeletionGracePeriodSeconds *int64 wantErr                        bool }{ { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=-1", args: args{ pod:     makePod(-1), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=0", args: args{ pod:     makePod(-1), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=1", args: args{ pod:     makePod(-1), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=-1, GracePeriodSeconds=2", args: args{ pod:     makePod(-1), options: makeOption(2), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=-1", args: args{ pod:     makePod(0), options: makeOption(-1), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=0", args: args{ pod:     makePod(0), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=1", args: args{ pod:     makePod(0), options: makeOption(1), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=0, GracePeriodSeconds=2", args: args{ pod:     makePod(0), options: makeOption(2), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=-1", args: args{ pod:     makePod(1), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=0", args: args{ pod:     makePod(1), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=1", args: args{ pod:     makePod(1), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=1, GracePeriodSeconds=2", args: args{ pod:     makePod(1), options: makeOption(2), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=-1", args: args{ pod:     makePod(2), options: makeOption(-1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=0", args: args{ pod:     makePod(2), options: makeOption(0), }, // want 0 wantDeletionGracePeriodSeconds: utilpointer.Int64(0), wantGracePeriodSeconds:         utilpointer.Int64(0), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=1", args: args{ pod:     makePod(2), options: makeOption(1), }, // want 1 wantDeletionGracePeriodSeconds: utilpointer.Int64(1), wantGracePeriodSeconds:         utilpointer.Int64(1), wantGraceful:                   true, wantGracefulPending:            false, }, { name: "when DeletionGracePeriodSeconds=2, GracePeriodSeconds=2", args: args{ pod:     makePod(2), options: makeOption(2), }, // want 2 wantDeletionGracePeriodSeconds: utilpointer.Int64(2), wantGracePeriodSeconds:         utilpointer.Int64(2), wantGraceful:                   false, wantGracefulPending:            true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if tt.args.strategy == nil { tt.args.strategy = mockStrategy{} } if tt.args.ctx == nil { tt.args.ctx = context.Background() } gotGraceful, gotGracefulPending, err := BeforeDelete(tt.args.strategy, tt.args.ctx, tt.args.pod, tt.args.options) if (err != nil) != tt.wantErr { t.Errorf("BeforeDelete() error = %v, wantErr %v", err, tt.wantErr) return } if gotGraceful != tt.wantGraceful { t.Errorf("BeforeDelete() gotGraceful = %v, want %v", gotGraceful, tt.wantGraceful) } if gotGracefulPending != tt.wantGracefulPending { t.Errorf("BeforeDelete() gotGracefulPending = %v, want %v", gotGracefulPending, tt.wantGracefulPending) } if gotGracefulPending != tt.wantGracefulPending { t.Errorf("BeforeDelete() gotGracefulPending = %v, want %v", gotGracefulPending, tt.wantGracefulPending) } if !utilpointer.Int64Equal(tt.args.pod.DeletionGracePeriodSeconds, tt.wantDeletionGracePeriodSeconds) { t.Errorf("metadata.DeletionGracePeriodSeconds = %v, want %v", utilpointer.Int64Deref(tt.args.pod.DeletionGracePeriodSeconds, 0), utilpointer.Int64Deref(tt.wantDeletionGracePeriodSeconds, 0)) } if !utilpointer.Int64Equal(tt.args.options.GracePeriodSeconds, tt.wantGracePeriodSeconds) { t.Errorf("options.GracePeriodSeconds = %v, want %v", utilpointer.Int64Deref(tt.args.options.GracePeriodSeconds, 0), utilpointer.Int64Deref(tt.wantGracePeriodSeconds, 0)) } }) } } </ins>