Change:
What happened: Adding a validating webhook with resources section of or breaks ability to add Ephemeral Containers as described in What you expected to happen: Debug command should attach to the running pod and allow user to interact with the ephemeral container image. This works when no validating webhook is configured. How to reproduce it (as minimally and precisely as possible): Using a K8 cluster with Ephemeral Containers feature gate enabled and v1.18.3 a validating webhook (contents don't matter except that resources is "pods/") <details</summary</detailsa test pod <details</summary</detailsto add an Ephemeral Container using alpha debug command Error: Anything else we need to know?: When using , can see that the api server appears to be throwing the error when trying to convert the object for the webhook invocation. API returning the 500 error: Environment: Kubernetes version (use ): v1.18.3 Cloud provider or hardware configuration: Minikube OS (e.g: ): MacOS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
This looks like an issue with the ephemeral container storage. It is inheriting the New function from the pod storage to return a core.Pod{}, but that doesn't match the type the ephemeral storage actually expects. I think the ephemeral storage should probably set up its own New method to return the proper type. Enabling the feature gate for the ephemeral container feature in the webhook admission integration test would exercise this with webhooks as well. /sig node /priority important-longterm /assign
/remove-sig api-machinery
ack, thanks for the report & triage! I'm digging into it now.
I thank what matters for proper typing is that for returns the proper type. This is already implemented for and returns an API object. Here's what I think is happening: gets an update for the subresource. This is translated into an update of the pod spec and handed off to the pod store since ephemeral containers do not have separate storage. update logic triggers admission control for the object with the attributes of the subresource. webhook client attempts to convert the stored object (a ) to the Kind of the ephemeralcontainers attributes ( identified by ), but this conversion doesn't exist. It is actually possible to convert from to , and this fixes the problem except that: The conversion is lossy, which would presumably be a problem for a mutating admission controller that returned an which could not be converted back into a This would result in the being sent in the instead of the , which is not the behavior we want. We could instead implement in , which allows us to override the Kind associated with . This results in no conversion and the is sent to the admission controller, but this also changes discovery info from: to: This fixes the problem, but I'm not sure what other side effects to expect. I imagine if we publish for this subresource then we should accept in addition to . I feel like I might be missing something in the storage configuration, though. do you know of any other possible fixes I could investigate?
For future reference, here's a trace from the validation dispatch that receives the conversion error:
mutating admission is applied here: validating admission methods (createValidation / updateValidation) is called from inside here: You need to wrap the createValidation / updateValidation methods (which expect an EphemeralContainers object) with methods that take a pod (which is what will pass in) and translate it to an EphemeralContainers object. See ScaleREST#Update methods for examples of this for the Scale subresource. Mutating admission is applied earlier, the only problematic call is validating admission Isn't it? The user is submitting an object... so that's what should be sent.
Thanks that set me straight. I have a working fix for this now, but I'm going to make sure it's covered in the e2e tests before sending it. re: for , it was a comment you made long ago that made me think there was a requirement to the contrary, so I must have misunderstood.
I had a chance to re-test this flow with which contains the fix to Although the original issue appears to be fixed, I am hoping to understand what the end goal is for ephemeral containers and their webhook behaviour. In my test, I enabled my to forward all resources . When running to add an ephemeral container to an existing pod, the only webhook that I received was for , which did contain the updated pod spec. However I was hoping that the addition of an ephemeral container would cause a webhook to trigger that wasn't of type . Maybe of type or ? Is there any plans to add a webhook trigger for ephemeral containers specifically? API server logs after doing :
Hi Your admission controller should receive a review request for resource subresource with a kind . When I run a test on using the test webhook and the config: I see this request
I see it have had a misconfiguration on my previous test. Sorry about that. Thanks for your work on this! Having ephemeral container support is very useful!
Great! Thanks for trying out the feature while still in alpha and reporting the bug!

Discussion:
return newPod, nil }) <del> obj, _, err = r.store.Update(ctx, name, updatedPodInfo, createValidation, updateValidation, false, options) </del> <ins> // Validation should be passed the API kind (EphemeralContainers) rather than the storage kind. obj, _, err = r.store.Update(ctx, name, updatedPodInfo, toEphemeralContainersCreateValidation(createValidation), toEphemeralContainersUpdateValidation(updateValidation), false, options) </ins> if err != nil { return nil, false, err } return ephemeralContainersInPod(obj.(*api.Pod)), false, err } <ins> func toEphemeralContainersCreateValidation(f rest.ValidateObjectFunc) rest.ValidateObjectFunc { return func(ctx context.Context, obj runtime.Object) error { return f(ctx, ephemeralContainersInPod(obj.(*api.Pod))) } } func toEphemeralContainersUpdateValidation(f rest.ValidateObjectUpdateFunc) rest.ValidateObjectUpdateFunc { return func(ctx context.Context, obj, old runtime.Object) error { return f(ctx, ephemeralContainersInPod(obj.(*api.Pod)), ephemeralContainersInPod(old.(*api.Pod))) } } </ins> // Extract the list of Ephemeral Containers from a Pod func ephemeralContainersInPod(pod *api.Pod) *api.EphemeralContainers { ephemeralContainers := pod.Spec.EphemeralContainers

"--disable-admission-plugins=ServiceAccount,StorageObjectInUseProtection", // force enable all resources so we can check storage. "--runtime-config=api/all=true", <ins> // enable feature-gates that protect resources to check their storage, too. "--feature-gates=EphemeralContainers=true", </ins> }, etcdConfig) defer server.TearDownFn()