Change:
What happened: It's most visible in the test <img width="323" alt="Screen Shot 2020-05-20 at 6 09 10 PM" src="https://user-"It shows up in kind, kube- and kops jobs The regression was introduced somewhere within , possible culprits are: (FYI (FYI (FYI There are more tests affected, just not as prominently (visually inspecting ) [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data I wasn't sure whether to tag this as kind/regression or not, I'll leave that up to or someone /sig node /sig api-machimery /sig scheduling
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: It's most visible in the test (FYI (FYI (FYI StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications with PVCs [sig-apps] StatefulSet [] Basic StatefulSet functionality [StatefulSetBasic] should perform rolling updates and roll backs of template modifications [Conformance] [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data /sig api-machimery /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
will increase the time for deletion of pod and it should be expected because currently pod is deleted from apiserver before removing the sandbox container but after this fix pod will be deleted from the apiserver only after deleting the sandbox container.
/remove-sig scheduling doesn't affect scheduling running time at all.
Yeah, is almost definitely the culprit. Last time we changed that code path, it also caused a regression. The degradation in performance is expected, but a 5x slowdown is definitely larger than it should be.
do we know if that's because the status used by the path is stale, or if removing sandboxes is actually that slow?
if there's not a quick fix, should we revert and bring it back in once the root issue is understood?
I think a revert is appropriate in this case.
ok, I'll open it
open at
I agree with that 5x is something which shouldn't happen and i can take a look on why thats happening and that might require changing the way how the sandbox is currently deleted which is only during GC.

Discussion:
"//staging/src/k8s.io/client-go/util/testing:go_default_library", "//staging/src/k8s.io/component-base/featuregate/testing:go_default_library", "//staging/src/k8s.io/component-base/version:go_default_library", <del> "//staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2:go_default_library", </del> "//vendor/github.com/google/cadvisor/info/v1:go_default_library", "//vendor/github.com/google/cadvisor/info/v2:go_default_library", "//vendor/github.com/stretchr/testify/assert:go_default_library", "//vendor/github.com/stretchr/testify/require:go_default_library", "//vendor/k8s.io/utils/mount:go_default_library", <del> ], </del> <ins> ] + select({ "@io_bazel_rules_go//go/platform:android": [ "//staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2:go_default_library", ], "@io_bazel_rules_go//go/platform:linux": [ "//staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2:go_default_library", ], "//conditions:default": [], }), </ins> ) filegroup(

// Status of containers in the pod. ContainerStatuses []*ContainerStatus // Status of the pod sandbox. <ins> // Only for kuberuntime now, other runtime may keep it nil. </ins> SandboxStatuses []*runtimeapi.PodSandboxStatus }

// This call is idempotent, and must not return an error if the sandbox has // already been removed. func (f *RemoteRuntime) RemovePodSandbox(ctx context.Context, req *kubeapi.RemovePodSandboxRequest) (*kubeapi.RemovePodSandboxResponse, error) { <del> err := f.RuntimeService.RemovePodSandbox(req.PodSandboxId) </del> <ins> err := f.RuntimeService.StopPodSandbox(req.PodSandboxId) </ins> if err != nil { return nil, err }

klog.V(3).Infof("Pod %q is terminated, but some containers have not been cleaned up: %s", format.Pod(pod), statusStr) return false } <del> // pod's sandboxes should be deleted if len(runtimeStatus.SandboxStatuses) > 0 { var sandboxStr string for _, sandbox := range runtimeStatus.SandboxStatuses { sandboxStr += fmt.Sprintf("%+v ", *sandbox) } klog.V(3).Infof("Pod %q is terminated, but some pod sandboxes have not been cleaned up: %s", format.Pod(pod), sandboxStr) return false } </del> if kl.podVolumesExist(pod.UID) && !kl.keepTerminatedPodVolumes { // We shouldn't delete pods whose volumes have not been cleaned up if we are not keeping terminated pod volumes klog.V(3).Infof("Pod %q is terminated, but some volumes have not been cleaned up", format.Pod(pod))

// api.Registry.GroupOrDie(v1.GroupName).GroupVersions[0].String() is changed // to "v1"? <del> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" </del> _ "k8s.io/kubernetes/pkg/apis/core/install" "k8s.io/kubernetes/pkg/features" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

assert.Equal(t, test.output, output) } } <del> func TestPodResourcesAreReclaimed(t *testing.T) { type args struct { pod           *v1.Pod status        v1.PodStatus runtimeStatus kubecontainer.PodStatus } tests := []struct { name string args args want bool }{ { "pod with running containers", args{ pod: &v1.Pod{}, status: v1.PodStatus{ ContainerStatuses: []v1.ContainerStatus{ runningState("containerA"), runningState("containerB"), }, }, }, false, }, { "pod with containers in runtime cache", args{ pod:    &v1.Pod{}, status: v1.PodStatus{}, runtimeStatus: kubecontainer.PodStatus{ ContainerStatuses: []*kubecontainer.ContainerStatus{ {}, }, }, }, false, }, { "pod with sandbox present", args{ pod:    &v1.Pod{}, status: v1.PodStatus{}, runtimeStatus: kubecontainer.PodStatus{ SandboxStatuses: []*runtimeapi.PodSandboxStatus{ {}, }, }, }, false, }, } testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kl := testKubelet.kubelet for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { testKubelet.fakeRuntime.PodStatus = tt.args.runtimeStatus if got := kl.PodResourcesAreReclaimed(tt.args.pod, tt.args.status); got != tt.want { t.Errorf("PodResourcesAreReclaimed() = %v, want %v", got, tt.want) } }) } } </del>