Change:
I created an ephemeral container for a Static Pod using the command. The command succeed. The API server accepted the ephemeral container: However, the container is not created. Its status is also not available: I expected one of these: (a) the API server rejects the request to create an ephemeral container for a Static Pod (b) kubelet creates the ephemeral container, and update the Static Pod's status sub-resource Using a recent version of , v0.17 or newer, and v1.25.0 or newer, create a cluster and create an ephemeral container running . Since the ephemeral container is not running, the output of is 0. The does not address Static Pods. <details</details<detailsNone </details<details</details<details</details<details docker exec -it kind-control-plane containerd --version containerd v1.6.9 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The umbrella issue for ephemeral containers is
/cc
Control plane machines provisioned using run control plane components (etcd, kube-apiserver, kube-controller-manager, kube-scheduler) as Static Pods. Being able to create ephemeral containers for these Static Pods in order to debug the running processes is a use case worthy of mention. If ephemeral containers are, by design, not intended to be used with Static Pods, let's please justify why that is the case, and try to document an alternative.
Thank you for raising this issue. Indeed we either need to update docs or fix it. Can you please provide the kubelet log? We discussed this on triage meeting and not sure if this is actually supported or not. Kubelet log may give us some information quickly in case it is some bug. /triage needs-information /cc
Thanks for triaging it quickly! Yes, I'll do that. (I didn't do it initially, because even after I bumped kubelet log verbosity to 10, I found no references of the ephemeral container names in the log. I'll just attach the complete kubelet log here.)
Attached is the (compressed) kubelet log running in the kind-control-plane container. I bumped up its log verbosity to 8 prior to creating the ephemeral container. I created the ephemeral container using the command described above at 18:42:31.
hi I was not able to reproduce this. I'm using kind 0.17 and kubectl 1.25.4. Can you try upgrading kubectl? I'd like to know if its some specific version. I looked through the kubelet log and didn't see any signs of trouble. Here's the data from my test run:
Thanks for trying to reproduce this. Your results are consistent with mine. What I am reporting is that, despite having the correct spec, the actual container is not created. However, I failed to explain how to verify that claim. We could verify it, for example, by listing all the processes running in the container. We should find the process, since it would be running in the ephemeral container. In my case, the process is not present: Moreover, since the ephemeral container has a shell executable, I should be able to run . In my case, the command fails: I realize that the is not highly reliable; might not work for other reasons, even if the ephemeral container is created.
I do see a difference in the test results, but I still can't exec into the ephemeral container. I'll run some more tests. Thanks Here is the pod spec showing something for the ephemeral container, but i can't exec into it anyway:
We should not allow ephemeral containers for static pods… the API content is not authoritative for static pods, the manifests on disk are, so I would expect the kubelet to ignore API spec content and the API to disallow adding ephemeral containers to static pods
It sounds like a documentation update would be appropriate here. I think the Static Pods page would be a good place for it. I'll get a PR going. Thanks
/remove-kind bug /kind documentation /sig docs
/remove-triage needs-information Is this good to close? We could additionally document why you can't add an ephemeral container (which is that mirror pods are essentially read-only).
If the /ephemeralcontainers subresource doesn't currently reject attempts to add to a static pod, I think it should, to make it clearer the addition won't be effective
Does the API server actually know that mirror pods are mirror pods? I wonder how you can tell.
it seems it can if it wants
I think that is just updating this function to fail if the annotation is set, right? ... and add tests, of course can you confirm?
yeah, that's right
any taker on this one?
/kind bug
I'll take it, per the discussion here: add a check in to reject the attempt to add an ephemeral container if the request is aimed at a static pod. Also include a test case. Also some additional documantation comments from this ticket and from kubernetes/website
/assign Thanks

Discussion:
allErrs = append(allErrs, validatePodMetadataAndSpec(newPod, opts)...) allErrs = append(allErrs, ValidatePodSpecificAnnotationUpdates(newPod, oldPod, fldPath.Child("annotations"), opts)...) <ins> // static pods don't support ephemeral containers #113935 if _, ok := oldPod.Annotations[core.MirrorPodAnnotationKey]; ok { return field.ErrorList{field.Forbidden(field.NewPath(""), "static pods do not support ephemeral containers")} } </ins> // Part 2: Validate that the changes between oldPod.Spec.EphemeralContainers and // newPod.Spec.EphemeralContainers are allowed. //

makeWindowsHostPod(nil), "spec.ephemeralContainers[0].securityContext.windowsOptions.hostProcess: Invalid value: false: pod hostProcess value must be identical", }, <ins> { "Add ephemeral container to static pod", func() *core.Pod { p := makePod(nil) p.Spec.NodeName = "some-name" p.ObjectMeta.Annotations = map[string]string{ core.MirrorPodAnnotationKey: "foo", } p.Spec.EphemeralContainers = []core.EphemeralContainer{{ EphemeralContainerCommon: core.EphemeralContainerCommon{ Name:                     "debugger1", Image:                    "debian", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }} return p }(), func() *core.Pod { p := makePod(nil) p.Spec.NodeName = "some-name" p.ObjectMeta.Annotations = map[string]string{ core.MirrorPodAnnotationKey: "foo", } return p }(), "Forbidden: static pods do not support ephemeral containers", }, </ins> } for _, tc := range tests {

} } <ins> func TestEphemeralContainerStrategyValidateUpdate(t *testing.T) { test := []struct { name   string newPod *api.Pod oldPod *api.Pod }{ { name: "add ephemeral container to regular pod and expect success", oldPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, newPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, }, }, } // expect no errors for _, tc := range test { t.Run(tc.name, func(t *testing.T) { if errs := EphemeralContainersStrategy.ValidateUpdate(genericapirequest.NewContext(), tc.newPod, tc.oldPod); len(errs) != 0 { t.Errorf("unexpected error:%v", errs) } }) } test = []struct { name   string newPod *api.Pod oldPod *api.Pod }{ { name: "add ephemeral container to static pod and expect failure", oldPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", Annotations:     map[string]string{api.MirrorPodAnnotationKey: "someVal"}, }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, NodeName: "example.com", }, }, newPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", Annotations:     map[string]string{api.MirrorPodAnnotationKey: "someVal"}, }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, NodeName: "example.com", }, }, }, { name: "remove ephemeral container from regular pod and expect failure", newPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, oldPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, }, }, { name: "change ephemeral container from regular pod and expect failure", newPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image2", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, }, oldPod: &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:            "test-pod", Namespace:       "test-ns", ResourceVersion: "1", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSDefault, Containers: []api.Container{ { Name:                     "container", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, EphemeralContainers: []api.EphemeralContainer{ { EphemeralContainerCommon: api.EphemeralContainerCommon{ Name:                     "debugger", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, }, }, }, }, }, } // expect one error for _, tc := range test { t.Run(tc.name, func(t *testing.T) { errs := EphemeralContainersStrategy.ValidateUpdate(genericapirequest.NewContext(), tc.newPod, tc.oldPod) if len(errs) == 0 { t.Errorf("unexpected success:ephemeral containers are not supported for static pods") } else if len(errs) != 1 { t.Errorf("unexpected errors:expected one error about ephemeral containers are not supported for static pods:got:%v:", errs) } }) } } </ins> func TestPodStrategyValidateUpdate(t *testing.T) { test := []struct { name   string