Change:
Install a cluster with v1.29.4 with linux node as master and worker node as windows kubeadm upgrade apply v1.30.0 fails with --ignore-preflight-errors` Describe Pod events: It seems that previously the pod was Pending as well, but this was ignored, because the jobs was successfully deleted in defer and return value was overridden with nil. Upgrade health check job expected to be scheduled only on Linux nodes using node selectors, rather than on Windows nodes. See What happened? No response_ <detailskubeadm version (use kubeadm version): v1.30.0 Environment: Kubernetes version (use kubectl version): v1.30.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 22.04.2 LTS, Windows Server 2022 Datacenter Kernel (e.g. uname -a): 5.19.0-1025-aws Container networking plugin (CNI): calico </details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed in 1.30.1 /close
Closing this issue. <detailsIn response to : kubeadm version (use kubeadm version): v1.30.0 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubeadm
we are seeing this issue for 1.30.1 version too with upgrade health check pods getting scheduled on windows worker nodes.
looks like I was mistaken and this issue is different. but the health check uses the pause image, which is multi os, supposedly. the error is strange. can you show the full log output. we might have to apply a node selector on the job spec and backport to 1.31 another bug fix.
I'm not sure if we should add a node selector on job spec. the job uses the pause image to create a container. It doesn't have specific requirements for the node. After I look at the projected volume plugin, I'm confused why the above event occurs. On windows nodes, the function is empty. It doesn't do anything. On linux nodes, the function will set the owner of the volume. FYI: If a log file from the kubelet is provided, it will be helpful to understand the issue.
thanks for checking I also think there should not be an error. maybe we are hitting a core kubelet bug. is the kubelet on the windows node also 1.30?
please ignore the above comment, it's incorrect. I may find the root cause. It's a bug in kubelet when kubelet is running on Windows and the pod configures the SecurityContext with a RunAsUser.
/sig storage /sig windows
/assign I'm trying to fix the kubelet bug.
so this is not a kubeadm bug and we can move it back to the k/k repo?
/transfer kubernetes
/retitle kubelet on Windows fails if a pod has SecurityContext with RunAsUser.
Because of historical reasons, the and in is only designed for the Linux, not Windows. See for more details. adds a new field to the PodSpec, which can be used to identify the OS of the pod. It's an optional field and no default value. If unset, the above fields are not restricted by the api-server. In other words, the following job is allowed to be created. But it will fails to start on Windows nodes. And the root cause is that the is called by the . To fix this issue, the AtomicWriter can check the current OS of the node and skip the if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. I'm not sure if the failing job is expected or not. If it's expected, kubeadm has to change the job manifest to make it work on only Linux nodes. Why I ask this question is that I saw , which is used to set the ownership of the volume, if the os is Windows, it will become a no-op. /cc
/cc
/triage accepted
slack discussion is happening here: but we have agreed that this is a kubelet problem. on the kubeadm side you can use the --ignore-preflight-errors=CreateJob option to skip the check. unclear when or if a kubelet fix will be and to what versions it will be available.
To fix this issue, the AtomicWriter can check the current OS of the node and skip the os.Chown if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. please see: do you want to upstream your carry?
thank you. I raise a PR and add as a co-author. I add an e2e test. do you know which job can test windows?
is the e2e test job for Windows

Discussion:
if fileProjection.FsUser == nil { continue } <del> if err := os.Chown(fullPath, int(*fileProjection.FsUser), -1); err != nil { </del> <ins> if err := w.chown(fullPath, int(*fileProjection.FsUser), -1); err != nil { </ins> klog.Errorf("%s: unable to change file %s with owner %v: %v", w.logContext, fullPath, int(*fileProjection.FsUser), err) return err }

<ins> //go:build linux // +build linux /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import "os" // chown changes the numeric uid and gid of the named file. func (w *AtomicWriter) chown(name string, uid, gid int) error { return os.Chown(name, uid, gid) } </ins>

<ins> //go:build !linux // +build !linux /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "runtime" "k8s.io/klog/v2" ) // chown changes the numeric uid and gid of the named file. // This is a no-op on unsupported platforms. func (w *AtomicWriter) chown(name string, uid, _ /* gid */ int) error { klog.Warningf("%s: skipping change of Linux owner %v for file %s; unsupported on %s", w.logContext, uid, name, runtime.GOOS) return nil } </ins>

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/util/uuid" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> clientset "k8s.io/client-go/kubernetes" <ins> "k8s.io/client-go/kubernetes/scheme" </ins> "k8s.io/kubernetes/pkg/kubelet/events" "k8s.io/kubernetes/test/e2e/feature" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eoutput "k8s.io/kubernetes/test/e2e/framework/pod/output" testutils "k8s.io/kubernetes/test/utils" <ins> "k8s.io/kubernetes/test/utils/format" </ins> imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api" <ins> "k8s.io/utils/ptr" </ins> ) const runAsUserNameContainerName = "run-as-username-container"

}) })) <ins> var _ = sigDescribe(feature.Windows, "SecurityContext", skipUnlessWindows(func() { f := framework.NewDefaultFramework("windows-with-unsupported-fields") f.NamespacePodSecurityLevel = admissionapi.LevelPrivileged ginkgo.It("should be able to create pod and run containers", func(ctx context.Context) { ginkgo.By("Creating 1 pods: run with unsupported fields") pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "run-ignore-unsupported-fields", Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ NodeSelector: map[string]string{"kubernetes.io/os": "windows"}, Containers: []v1.Container{ { Name:  "test-container", Image: imageutils.GetE2EImage(imageutils.Pause), }, }, SecurityContext: &v1.PodSecurityContext{ RunAsUser:    ptr.To[int64](999), // windows does not support RunAsGroup:   ptr.To[int64](999), // windows does not support RunAsNonRoot: ptr.To(true), }, RestartPolicy: v1.RestartPolicyNever, }, } pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(ctx, pod, metav1.CreateOptions{}) framework.ExpectNoError(err, "Error creating pod") podErr := e2epod.WaitForPodRunningInNamespace(ctx, f.ClientSet, pod) // Get the logs and events before calling ExpectNoError, so we can debug any errors. var logs string var events *v1.EventList if err := wait.PollUntilContextTimeout(ctx, 30*time.Second, 2*time.Minute, true, func(ctx context.Context) (done bool, err error) { framework.Logf("polling logs") logs, err = e2epod.GetPodLogs(ctx, f.ClientSet, f.Namespace.Name, pod.Name, pod.Spec.Containers[0].Name) if err != nil { framework.Logf("Error pulling logs: %v", err) return false, nil } events, err = f.ClientSet.CoreV1().Events(pod.Namespace).Search(scheme.Scheme, pod) if err != nil { return false, fmt.Errorf("error in listing events: %w", err) } return true, nil }); err != nil { framework.Failf("Unexpected error getting pod logs/events: %v", err) } else { framework.Logf("Pod logs: n%v", logs) framework.Logf("Pod events: n%v", format.Object(events, 1)) } framework.ExpectNoError(podErr) }) })) </ins> func runAsUserNamePod(username *string) *v1.Pod { podName := "run-as-username-" + string(uuid.NewUUID()) return &v1.Pod{