Change:
if serr, ok := err.(*errors.StatusError); ok && serr.Status().Reason == metav1.StatusReasonNotFound && serr.ErrStatus.Details.Name == "" { return nil, "", fmt.Errorf("ephemeral containers are disabled for this cluster (error from server: %q).", err) } <ins> // The Kind used for the /ephemeralcontainers subresource changed in 1.22. When presented with an unexpected // Kind the api server will respond with a not-registered error. When this happens we can optimistically try // using the old API. if runtime.IsNotRegisteredError(err) { klog.V(1).Infof("Falling back to legacy API because server returned error: %v", err) return o.debugByEphemeralContainerLegacy(ctx, pod, debugContainer) } </ins> return nil, "", err } return result, debugContainer.Name, nil } <ins> // debugByEphemeralContainerLegacy adds debugContainer as an ephemeral container using the pre-1.22 /ephemeralcontainers API // This may be removed when we no longer wish to support releases prior to 1.22. func (o *DebugOptions) debugByEphemeralContainerLegacy(ctx context.Context, pod *corev1.Pod, debugContainer *corev1.EphemeralContainer) (*corev1.Pod, string, error) { // We no longer have the v1.EphemeralContainers Kind since it was removed in 1.22, but // we can present a JSON 6902 patch that the api server will apply. patch, err := json.Marshal([]map[string]interface{}{{ "op":    "add", "path":  "/ephemeralContainers/-", "value": debugContainer, }}) if err != nil { return nil, "", fmt.Errorf("error creating JSON 6902 patch for old /ephemeralcontainers API: %s", err) } result := o.podClient.RESTClient().Patch(types.JSONPatchType). Namespace(pod.Namespace). Resource("pods"). Name(pod.Name). SubResource("ephemeralcontainers"). Body(patch). Do(ctx) if err := result.Error(); err != nil { return nil, "", err } newPod, err := o.podClient.Pods(pod.Namespace).Get(ctx, pod.Name, metav1.GetOptions{}) if err != nil { return nil, "", err } return newPod, debugContainer.Name, nil } </ins> // debugByCopy runs a copy of the target Pod with a debug container added or an original container modified func (o *DebugOptions) debugByCopy(ctx context.Context, pod *corev1.Pod) (*corev1.Pod, string, error) { copied, dc, err := o.generatePodCopyWithDebugContainer(pod)

Discussion:
changed the Kind used with the subresource from to . This change will break old clients. (The ephemeral containers API is still in alpha, so this is acceptable.) Currently, compatibility is broken in both directions: versions prior to 1.22 are incompatible with server version 1.22 version 1.22 is incompatible with server versions prior to 1.22 Remediating the first requires complex server-side backwards compatibility logic which is not worth implementing for an alpha API. Remediating the second for should be much easier. The struct hasn't changed, only the path in the object. We could detect the kind returned by the server and change the path used in the JSON patch (or even just naively try the previous path if the server returns an error). This behavior would be removed after the old server version falls out of the support window. This isn't strictly needed, but it would improve UX for what I expect is a common scenario: using the latest kubectl with older servers. /sig cli /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about adding this backwards compatibility behavior to kubectl?
Are we able to detect this and fail gracefully client side with a warning to users to upgrade their client? Agree that this fix is the important one. I wonder if a commit for backwards compatibility is something we can cherry pick to other release branches? Seeing as earlier branches will have both structs it might even be cleaner to do version conditional in kubectl there? is that something we've done before?
Just to make in my head a matrix of effects/noneffects, please confirm for me if these are right: 1.22 will have the compatibility logic baked in. So it will be able to talk to older servers. -picking that logic to older branches would not alter in any way how pre-1.22 kubectls interact with old (pre 1.22) servers.

cherry-pick would only add the needed logic to allow old kubectls to talk to 1.22 and later servers Are these assumptions correct?
Correct. I haven't prototyped it, but this should be possible. We could trigger the logic on a server error that indicates it received an unexpected Kind. That being said, I don't think it's worth cherry-picking forward compatibility into older versions of kubectl. This is for an alpha feature that must be explicitly enabled with the feature flag. It's not unreasonable to require people experimenting with an alpha feature to use a kubectl version = server version. There's far more risk than benefit. It would be more of a burden to keep around old versions of kubectl for older clusters, so I thought adding backwards compatibility in 1.22 would be nice, even if it's not strictly required. I agree it would be nice if we could give users of pre-1.22 kubectl a better error message, but I at least have kubernetes/website out to indicate a minimum kubectl version requirement of 1.22.
I'd like to take up this issue. /assign
How's it going? I'd like to make sure this makes it into 1.22.
Hi , I am able to reproduce this issue. I am looking for some guidance on how to go about fixing this issue. You have mentioned that "detect the kind returned by the server and change the path used in the JSON patch ", where is the kind being returned by the server and what JSON patch are you referring to?
Hi given that time zones are difficult and the code freeze is near, I think it'll be most efficient to communicate with examples. I'll prepare a PR with what I have in mind.