Change:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : I would like to make kubelet wait for CNI resources to be completely cleaned before deleting the pod from the apiserver. Currently it just waits for containers to be deleted and volumes to be unmounted. Why is this needed: There are couple of reason why this would make sense: 1) This will make sure that apiserver would provide the true view of the system. Because when pod is not cleaned from CNI because of any issue, then it's better for apiserver to show that the pod is still not terminated properly instead of cleaning up from etcd and relying on the GC to take care of it. 2) Currently the CNI's rely on the apiserver for additional metadata during creation and cleaning up since the kubelet doesn't pass additional metadata to the CNI which was discussed and with apiserver deleting the pod before CNI is successful means CNI has to store kubernetes metadata separately.
/sig network
/sig node
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/area kubelet
/assign
What platform did you observe the problem? It depends on how container runtime handles the interaction with CNI plugins. For , I am pretty sure it only proceeds to completely remove the pod when CNI DEL completes.
This is on ubuntu with docker are runtime and the issue seems to be on the kubelet itself not the CRI. Kubelet based on the function, it only checks for the containers are deleted and not the CNI resources.
do you know if its expected or not? Or can you assign it to someone who might have more context on it.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
yeah, unfortunately this isn't the way it currently works and it would be a hard lift to make that happen. If you're interested in that, it's best to file a KEP describing the issue and the proposed solution, since this would touch not just networking but apiserver and a bunch of other areas. One of the reasons that Kuberenetes doesn't pass down additional data is that plugins that need this data often need more than just the Pod object, and Kubelet isn't going to pass all of that (Services, Endpoints, Namespaces, NetworkPolicies, etc). So the plugin will already be talking to the apiserver and have a cache (which you get for free with the Go kube client's SharedInformer) of the pod information it can draw from when it gets a DEL event from the runtime. If a plugin is trying to talk to the apiserver when it gets a CNI DEL, it's likely already broken for pod deletion and should be changed to cache the information it needs on DEL, whether that's through a long-running process or checkpoint files or something like that.
Thanks I would definitely be interested in writing a KEP and working on it. I just want to understand what you meant by this would require changes in apiserver and other areas as i was thinking on waiting for deletion of sandboxes which would require changes only in kubelet.
We discussed this in sig-node. The conclusion was: is a bug should introduce a node e2e test which inspects the container runtime to ensure all resources are gone after the pod is deleted need for a KEP. Feel free to open a PR to fix it, and an e2e test that demonstrates the fix. Feel free to assign the PR to me for review.
/milestone v1.19
/reopen Re-opening as the PR was reverted
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is this revert-revert something that sig-node is handling, or is there something sig-net can do?
Yeah, at this point I don't think this is sig-net related anymore...
you mentioned at you are looking into it. Do you need help?
i didn't get time last week to look into the issue... am currently debugging the issue and mostly will try to fix in couple of weeks
/cc
/remove-kind feature /kind bug see /triage accepted
The issue is fixed by .
I also encountered this problem when using multus with kubernetes (version 1.18). I saw two solutions in the issue: which one was used in the latest version of Kubelet.
its

Discussion:
"//staging/src/k8s.io/client-go/util/testing:go_default_library", "//staging/src/k8s.io/component-base/featuregate/testing:go_default_library", "//staging/src/k8s.io/component-base/version:go_default_library", <ins> "//staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2:go_default_library", </ins> "//vendor/github.com/google/cadvisor/info/v1:go_default_library", "//vendor/github.com/google/cadvisor/info/v2:go_default_library", "//vendor/github.com/stretchr/testify/assert:go_default_library", "//vendor/github.com/stretchr/testify/require:go_default_library", "//vendor/k8s.io/utils/mount:go_default_library", <del> ] + select({ "@io_bazel_rules_go//go/platform:android": [ "//staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2:go_default_library", ], "@io_bazel_rules_go//go/platform:linux": [ "//staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2:go_default_library", ], "//conditions:default": [], }), </del> <ins> ], </ins> ) filegroup(

// Status of containers in the pod. ContainerStatuses []*ContainerStatus // Status of the pod sandbox. <del> // Only for kuberuntime now, other runtime may keep it nil. </del> SandboxStatuses []*runtimeapi.PodSandboxStatus }

klog.V(3).Infof("Pod %q is terminated, but some containers have not been cleaned up: %s", format.Pod(pod), statusStr) return false } <ins> // pod's sandboxes should be deleted if len(runtimeStatus.SandboxStatuses) > 0 { var sandboxStr string for _, sandbox := range runtimeStatus.SandboxStatuses { sandboxStr += fmt.Sprintf("%+v ", *sandbox) } klog.V(3).Infof("Pod %q is terminated, but some pod sandboxes have not been cleaned up: %s", format.Pod(pod), sandboxStr) return false } </ins> if kl.podVolumesExist(pod.UID) && !kl.keepTerminatedPodVolumes { // We shouldn't delete pods whose volumes have not been cleaned up if we are not keeping terminated pod volumes klog.V(3).Infof("Pod %q is terminated, but some volumes have not been cleaned up", format.Pod(pod))

// api.Registry.GroupOrDie(v1.GroupName).GroupVersions[0].String() is changed // to "v1"? <ins> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" </ins> _ "k8s.io/kubernetes/pkg/apis/core/install" "k8s.io/kubernetes/pkg/features" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

assert.Equal(t, test.output, output) } } <ins> func TestPodResourcesAreReclaimed(t *testing.T) { type args struct { pod           *v1.Pod status        v1.PodStatus runtimeStatus kubecontainer.PodStatus } tests := []struct { name string args args want bool }{ { "pod with running containers", args{ pod: &v1.Pod{}, status: v1.PodStatus{ ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, }, }, false, }, { "pod with containers in runtime cache", args{ pod:    &v1.Pod{}, status: v1.PodStatus{}, runtimeStatus: kubecontainer.PodStatus{ ContainerStatuses: []*kubecontainer.ContainerStatus{ {}, }, }, }, false, }, { "pod with sandbox present", args{ pod:    &v1.Pod{}, status: v1.PodStatus{}, runtimeStatus: kubecontainer.PodStatus{ SandboxStatuses: []*runtimeapi.PodSandboxStatus{ {}, }, }, }, false, }, } testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kl := testKubelet.kubelet for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { testKubelet.fakeRuntime.PodStatus = tt.args.runtimeStatus if got := kl.PodResourcesAreReclaimed(tt.args.pod, tt.args.status); got != tt.want { t.Errorf("PodResourcesAreReclaimed() = %v, want %v", got, tt.want) } }) } } </ins>

// This call is idempotent, and must not return an error if the sandbox has // already been removed. func (f *RemoteRuntime) RemovePodSandbox(ctx context.Context, req *kubeapi.RemovePodSandboxRequest) (*kubeapi.RemovePodSandboxResponse, error) { <del> err := f.RuntimeService.StopPodSandbox(req.PodSandboxId) </del> <ins> err := f.RuntimeService.RemovePodSandbox(req.PodSandboxId) </ins> if err != nil { return nil, err }