jiebi/CodeConvo huggingface dataset:
Code change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: Triage board shows flakes here spiked dramatically on 3/4 /sig node /priority critical-urgent /milestone v1.18
are you able to assist if this is related to recent topology work?
sure, I'll take a look
is this the test you just in ?
As mentioned above, this is a new test that was just , and the test sometimes fails. I don't see that any of the recent topology manager changes would cause these failures. I'll be happy to help out and dig more if or others think otherwise.
This is me, the bugs are untriaged containerd / Kubelet failures with low occurrence rates that this test now exposes. Investigating.
still flaking in non-dockershim environments: pod pod-submit-status-0-14 on node kind-worker2 container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2020-03-06 16:16:54 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-819e-4840-8369-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-819e-4840-8369- no such file or directory""": unknown `
Different reason this time. Will continue to enumerate failure modes.

Related discussion:
text: switch { case t.ExitCode == 1: // expected <del> case t.ExitCode == 128 && t.Reason == "ContainerCannotRun" && reBug88766.MatchString(t.Message): </del> <ins> case t.ExitCode == 128 && (t.Reason == "StartError" || t.Reason == "ContainerCannotRun") && reBug88766.MatchString(t.Message): </ins> // pod volume teardown races with container start in CRI, which reports a failure framework.Logf("pod %s on node %s failed with the symptoms of https://github.com/kubernetes/kubernetes/issues/88766") default:
Code change:
What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon

Related discussion:
text: defer runtime.HandleCrash() for event := range outStream { <del> if event.ContainerName == recordEventContainerName { </del> <ins> if event.VictimContainerName == recordEventContainerName { </ins> klog.V(1).Infof("Got sys oom event: %v", event) eventMsg := "System OOM encountered" if event.ProcessName != "" && event.Pid != 0 {
Code change:
What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon

Related discussion:
text: Pid:                 1000, ProcessName:         "fakeProcess", TimeOfDeath:         time.Now(), <del> ContainerName:       recordEventContainerName, VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "some-container", VictimContainerName: recordEventContainerName, </ins> }, } numExpectedOomEvents := len(oomInstancesToStream)
Code change:
What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon

Related discussion:
text: Pid:                 1000, ProcessName:         "fakeProcess", TimeOfDeath:         time.Now(), <del> ContainerName:       recordEventContainerName, VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "some-container", VictimContainerName: recordEventContainerName, </ins> }, { Pid:                 1000, ProcessName:         "fakeProcess", TimeOfDeath:         time.Now(), <del> ContainerName:       "/dont-record-oom-event", VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "kubepods/some-container", VictimContainerName: recordEventContainerName + "kubepods", </ins> }, } numExpectedOomEvents := len(oomInstancesToStream) - numOomEventsWithIncorrectContainerName
Code change:
What happened: A SystemOOM event was not reported even when the memory limit hit was the root cgroup memory limit. What you expected to happen: A SystemOOM event should have been reported. How to reproduce it (as minimally and precisely as possible): Do not set or , and set to disable memory eviction on the kubelet. Run a pod without memory limits which slowly consumes memory, such as with from Use arguments -mem-alloc-size", "100Mi", "-mem-alloc-sleep", "10s". This should trigger an OOM on the root cgroup, but no event will be generated. Anything else we need to know?: cAdvisor populates ContainerName and VictimContainerName from matching the regexp: instead in the kubelet. cc
/sig node /priority important-soon

Related discussion:
text: Pid:                 eventPid, ProcessName:         processName, TimeOfDeath:         time.Now(), <del> ContainerName:       recordEventContainerName, VictimContainerName: "some-container", </del> <ins> ContainerName:       recordEventContainerName + "some-container", VictimContainerName: recordEventContainerName, </ins> }, } numExpectedOomEvents := len(oomInstancesToStream)
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node

Related discussion:
text: } var cgroupRoots []string <del> cgroupRoots = append(cgroupRoots, cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupDriver)) </del> <ins> nodeAllocatableRoot := cm.NodeAllocatableRoot(s.CgroupRoot, s.CgroupsPerQOS, s.CgroupDriver) cgroupRoots = append(cgroupRoots, nodeAllocatableRoot) </ins> kubeletCgroup, err := cm.GetKubeletContainer(s.KubeletCgroups) if err != nil { klog.Warningf("failed to get the kubelet's cgroup: %v.  Kubelet system container metrics may be missing.", err)
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node

Related discussion:
text: } // NodeAllocatableRoot returns the literal cgroup path for the node allocatable cgroup <del> func NodeAllocatableRoot(cgroupRoot, cgroupDriver string) string { root := ParseCgroupfsToCgroupName(cgroupRoot) nodeAllocatableRoot := NewCgroupName(root, defaultNodeAllocatableCgroupName) </del> <ins> func NodeAllocatableRoot(cgroupRoot string, cgroupsPerQOS bool, cgroupDriver string) string { nodeAllocatableRoot := ParseCgroupfsToCgroupName(cgroupRoot) if cgroupsPerQOS { nodeAllocatableRoot = NewCgroupName(nodeAllocatableRoot, defaultNodeAllocatableCgroupName) } </ins> if libcontainerCgroupManagerType(cgroupDriver) == libcontainerSystemd { return nodeAllocatableRoot.ToSystemd() }
Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node

Related discussion:
text: } // NodeAllocatableRoot returns the literal cgroup path for the node allocatable cgroup <del> func NodeAllocatableRoot(cgroupRoot, cgroupDriver string) string { </del> <ins> func NodeAllocatableRoot(cgroupRoot string, cgroupsPerQOS bool, cgroupDriver string) string { </ins> return "" }
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/strategicpatch:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library", "//staging/src/k8s.io/client-go/informers:go_default_library",
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: import ( "context" <ins> "encoding/json" </ins> "fmt" "io/ioutil" "math/rand"
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/strategicpatch" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/informers" coreinformers "k8s.io/client-go/informers/core/v1"
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: func (p *podConditionUpdaterImpl) update(pod *v1.Pod, condition *v1.PodCondition) error { klog.V(3).Infof("Updating pod condition for %s/%s to (%s==%s, Reason=%s)", pod.Namespace, pod.Name, condition.Type, condition.Status, condition.Reason) <del> if podutil.UpdatePodCondition(&pod.Status, condition) { _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), pod, metav1.UpdateOptions{}) </del> <ins> oldData, err := json.Marshal(pod) if err != nil { </ins> return err } <del> return nil </del> <ins> if !podutil.UpdatePodCondition(&pod.Status, condition) { return nil } newData, err := json.Marshal(pod) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") return err </ins> } type podPreemptorImpl struct {
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: } func (p *podPreemptorImpl) setNominatedNodeName(pod *v1.Pod, nominatedNodeName string) error { <ins> klog.V(3).Infof("Setting nominated node name for %s/%s to "%s"", pod.Namespace, pod.Name, nominatedNodeName) if pod.Status.NominatedNodeName == nominatedNodeName { return nil } </ins> podCopy := pod.DeepCopy() <ins> oldData, err := json.Marshal(podCopy) if err != nil { return err } </ins> podCopy.Status.NominatedNodeName = nominatedNodeName <del> _, err := p.Client.CoreV1().Pods(pod.Namespace).UpdateStatus(context.TODO(), podCopy, metav1.UpdateOptions{}) </del> <ins> newData, err := json.Marshal(podCopy) if err != nil { return err } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, &v1.Pod{}) if err != nil { return fmt.Errorf("failed to create merge patch for pod %q/%q: %v", pod.Namespace, pod.Name, err) } _, err = p.Client.CoreV1().Pods(pod.Namespace).Patch(context.TODO(), pod.Name, types.StrategicMergePatchType, patchBytes, metav1.PatchOptions{}, "status") </ins> return err }
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: "os" "path" "reflect" <ins> "regexp" </ins> "sort" "strings" "sync"
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" "k8s.io/kubernetes/plugin/pkg/admission/priority" testutils "k8s.io/kubernetes/test/integration/util" <del> utils "k8s.io/kubernetes/test/utils" </del> <ins> "k8s.io/kubernetes/test/utils" </ins> ) var lowPriority, mediumPriority, highPriority = int32(100), int32(200), int32(300)
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: Labels:    map[string]string{"pod": name}, Resources: defaultPodRes, }) <del> // Setting grace period to zero. Otherwise, we may never see the actual deletion // of the pods in integration tests. </del> pod.Spec.TerminationGracePeriodSeconds = &grace return pod }
Code change:
Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestNominatedNodeCleanUp Testgrid link: Reason for failure: /sig scheduling
/assign
any progress on this one?
I was working on it and tried to find the root cause. I will give my solution within this week
Another one
Still happening cc
/assign
/close dupe
Closing this issue. <detailsIn response to : /close dupe Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: } // Step 5. Check that nominated node name of the high priority pod is set. if err := waitForNominatedNodeName(cs, highPriPod); err != nil { <del> t.Errorf("NominatedNodeName annotation was not set for pod %v/%v: %v", medPriPod.Namespace, medPriPod.Name, err) </del> <ins> t.Errorf("NominatedNodeName annotation was not set for pod %v/%v: %v", highPriPod.Namespace, highPriPod.Name, err) </ins> } // And the nominated node name of the medium priority pod is cleared. if err := wait.Poll(100*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) {
Code change:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

Related discussion:
text: podSpec.EphemeralContainers = nil } <del> if (!utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpath) || !utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpathEnvExpansion)) && !subpathExprInUse(oldPodSpec) { // drop subpath env expansion from the pod if either of the subpath features is disabled and the old spec did not specify subpath env expansion </del> <ins> if !utilfeature.DefaultFeatureGate.Enabled(features.VolumeSubpath) && !subpathExprInUse(oldPodSpec) { // drop subpath env expansion from the pod if subpath feature is disabled and the old spec did not specify subpath env expansion </ins> VisitContainers(podSpec, AllContainers, func(c *api.Container, containerType ContainerType) bool { for i := range c.VolumeMounts { c.VolumeMounts[i].SubPathExpr = ""
Code change:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

Related discussion:
text: } t.Run(fmt.Sprintf("feature enabled=%v, old pod %v, new pod %v", enabled, oldPodInfo.description, newPodInfo.description), func(t *testing.T) { <del> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpathEnvExpansion, enabled)() </del> var oldPodSpec *api.PodSpec if oldPod != nil {
Code change:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

Related discussion:
text: } func TestValidateSubpathMutuallyExclusive(t *testing.T) { <del> // Enable feature VolumeSubpathEnvExpansion and VolumeSubpath defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpathEnvExpansion, true)() </del> <ins> // Enable feature VolumeSubpath </ins> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpath, true)() volumes := []core.Volume{
Code change:
/kind cleanup /sig storage /sig node Description: Feature gate VolumeSubpathEnvExpansion was GA in 1.17 and is marked for removal in 1.19 Related Issues: GA Release:
/assign
/cc

Related discussion:
text: } func TestValidateDisabledSubpathExpr(t *testing.T) { <del> // Enable feature VolumeSubpathEnvExpansion defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumeSubpathEnvExpansion, true)() </del> volumes := []core.Volume{ {Name: "abc", VolumeSource: core.VolumeSource{PersistentVolumeClaim: &core.PersistentVolumeClaimVolumeSource{ClaimName: "testclaim1"}}},