Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We enabled AlwaysPullImages admission in an existing cluster, and found annotations/labels/finalizers of existing Pods whose imagePullPolicy configured as could not be updated. AlwaysPullImages admission mutated containerSpec by . And this change later got denied by Pod validation. It also caused terminating Pods with finalizers stuck forever until we removed AlwaysPullImages admission. What you expected to happen: Existing Pods are not affected by AlwaysPullImages admission controller. How to reproduce it (as minimally and precisely as possible): the kube-apiserver with AlwaysPullImages admission controller disabled a Pod with imagePullPolicy configured as in its containers AlwaysPullImages admission controller annotation to Pod Anything else we need to know?: If this is confirmed a bug, I can submit a PR to fix this. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
seems to be a bug. /assign
/remove-sig api-machinery /sig node auth
try to fix it in When AlwaysPullImages is disabled, you cannot update imagePullPolicy of pod. The error message is the same as yours. imagePullPolicy is immutable. When AlwaysPullImages is enabled, old pod(with imagePullPolicy=ifNotPresent) update will change imagePullPolicy as well. Workaround: recreated pod -_-
Thanks for the fixes. I still have regarding the PR. IMHO, it's OK that imagePullPolicy is not updatable. Why don't we just ignore Update in the admission, but choose to update spec instead? Any concerns?
This is another fix option. I'm not sure which is better. The pod will be still ifNotPresent after updating, even if the AlwaysPullImages is enabled. Which is expected behavior?
In my case, I would expect the admission not to touch existing Pods. Just want to make sure if there's any concerns.
I would recommend ignoring pod updates in AlwaysPullImages if the update does not change the set of image names referenced by containers in the pod. Changing images inline in a pod update is unusual, and that change would allow unrelated updates (to annotations, labels, etc) to proceed
Agree.
is updated .
This issue is fixed by

Discussion:
"//pkg/apis/core/pods:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/errors:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/validation/field:go_default_library", "//staging/src/k8s.io/apiserver/pkg/admission:go_default_library", <ins> "//vendor/k8s.io/klog/v2:go_default_library", </ins> ], )

apierrors "k8s.io/apimachinery/pkg/api/errors" utilerrors "k8s.io/apimachinery/pkg/util/errors" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/apiserver/pkg/admission" <ins> "k8s.io/klog/v2" </ins> api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/apis/core/pods" )

return nil } <ins> // check if it's update and it doesn't change the images referenced by the pod spec func isUpdateWithNoNewImages(attributes admission.Attributes) bool { if attributes.GetOperation() != admission.Update { return false } pod, ok := attributes.GetObject().(*api.Pod) if !ok { klog.Warningf("Resource was marked with kind Pod but pod was unable to be converted.") return false } oldPod, ok := attributes.GetOldObject().(*api.Pod) if !ok { klog.Warningf("Resource was marked with kind Pod but old pod was unable to be converted.") return false } oldImages := sets.NewString() pods.VisitContainersWithPath(&oldPod.Spec, field.NewPath("spec"), func(c *api.Container, _ *field.Path) bool { oldImages.Insert(c.Image) return true }) hasNewImage := false pods.VisitContainersWithPath(&pod.Spec, field.NewPath("spec"), func(c *api.Container, _ *field.Path) bool { if !oldImages.Has(c.Image) { hasNewImage = true } return !hasNewImage }) return !hasNewImage } </ins> func shouldIgnore(attributes admission.Attributes) bool { // Ignore all calls to subresources or resources other than pods. if len(attributes.GetSubresource()) != 0 || attributes.GetResource().GroupResource() != api.Resource("pods") { return true } <ins> if isUpdateWithNoNewImages(attributes) { return true } </ins> return false }

t.Errorf("%s: image pull policy was changed to %s", tc.name, a) } } <ins> } // TestUpdatePod ensures that this admission controller is a no-op for update pod if no // images were changed in the new pod spec. func TestUpdatePod(t *testing.T) { namespace := "testnamespace" name := "testname" oldPod := &api.Pod{ ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace}, Spec: api.PodSpec{ Containers: []api.Container{ {Name: "ctr2", Image: "image", ImagePullPolicy: api.PullIfNotPresent}, }, }, } // only add new annotation pod := &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: namespace, Annotations: map[string]string{ "test": "test", }, }, Spec: api.PodSpec{ Containers: []api.Container{ {Name: "ctr2", Image: "image", ImagePullPolicy: api.PullIfNotPresent}, }, }, } // add new label and change image podWithNewImage := &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: namespace, Annotations: map[string]string{ "test": "test", }, }, Spec: api.PodSpec{ Containers: []api.Container{ {Name: "ctr2", Image: "image2", ImagePullPolicy: api.PullIfNotPresent}, }, }, } tests := []struct { name         string kind         string resource     string subresource  string object       runtime.Object oldObject    runtime.Object expectError  bool expectIgnore bool }{ { name:         "update IfNotPresent pod annotations", kind:         "Pod", resource:     "pods", subresource:  "finalizers", object:       pod, oldObject:    oldPod, expectIgnore: true, }, { name:        "update IfNotPresent pod image", kind:        "Pod", resource:    "pods", subresource: "finalizers", object:      podWithNewImage, oldObject:   oldPod, }, } for _, tc := range tests { handler := admissiontesting.WithReinvocationTesting(t, &AlwaysPullImages{}) err := handler.Admit(context.TODO(), admission.NewAttributesRecord(tc.object, tc.oldObject, api.Kind(tc.kind).WithVersion("version"), namespace, name, api.Resource(tc.resource).WithVersion("version"), tc.subresource, admission.Create, &metav1.UpdateOptions{}, false, nil), nil) if tc.expectError { if err == nil { t.Errorf("%s: unexpected nil error", tc.name) } continue } if tc.expectIgnore { if e, a := api.PullIfNotPresent, pod.Spec.Containers[0].ImagePullPolicy; e != a { t.Errorf("%s: image pull policy was changed to %s", tc.name, a) } continue } if err != nil { t.Errorf("%s: unexpected error: %v", tc.name, err) continue } } </ins> }