jiebi/CodeConvo huggingface dataset:
Code change:
Namespace: pod.Namespace, } tests := []struct { <del> containers                   []api.Container statuses                     []*kubecontainer.ContainerStatus reasons                      map[string]error oldStatuses                  []api.ContainerStatus expectedState                map[string]api.ContainerState </del> <ins> containers    []api.Container statuses      []*kubecontainer.ContainerStatus reasons       map[string]error oldStatuses   []api.ContainerStatus expectedState map[string]api.ContainerState // Only set expectedInitState when it is different from expectedState expectedInitState            map[string]api.ContainerState </ins> expectedLastTerminationState map[string]api.ContainerState }{ // For container with no historical record, State should be Waiting, LastTerminationState should be retrieved from // old status from apiserver. { <del> []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, []*kubecontainer.ContainerStatus{}, map[string]error{}, []api.ContainerStatus{{ </del> <ins> containers: []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, statuses:   []*kubecontainer.ContainerStatus{}, reasons:    map[string]error{}, oldStatuses: []api.ContainerStatus{{ </ins> Name:                 "with-old-record", LastTerminationState: api.ContainerState{Terminated: &api.ContainerStateTerminated{}}, }}, <del> map[string]api.ContainerState{ </del> <ins> expectedState: map[string]api.ContainerState{ </ins> "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: startWaitingReason, }},

Related discussion:
text: is this a Node bug? cc as next oncall to follow up on this.
This is a node bug due to init containers
I may need help debugging - I see:
So I think the problem is in the error case - we're not properly recording that the start failed, and the next time we come in we somehow think the first container succeeded. I'll add some debugging to see if we can't recreate it.
Let's discuss on instead.
Code change:
Namespace: pod.Namespace, } tests := []struct { <del> containers                   []api.Container statuses                     []*kubecontainer.ContainerStatus reasons                      map[string]error oldStatuses                  []api.ContainerStatus expectedState                map[string]api.ContainerState </del> <ins> containers    []api.Container statuses      []*kubecontainer.ContainerStatus reasons       map[string]error oldStatuses   []api.ContainerStatus expectedState map[string]api.ContainerState // Only set expectedInitState when it is different from expectedState expectedInitState            map[string]api.ContainerState </ins> expectedLastTerminationState map[string]api.ContainerState }{ // For container with no historical record, State should be Waiting, LastTerminationState should be retrieved from // old status from apiserver. { <del> []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, []*kubecontainer.ContainerStatus{}, map[string]error{}, []api.ContainerStatus{{ </del> <ins> containers: []api.Container{{Name: "without-old-record"}, {Name: "with-old-record"}}, statuses:   []*kubecontainer.ContainerStatus{}, reasons:    map[string]error{}, oldStatuses: []api.ContainerStatus{{ </ins> Name:                 "with-old-record", LastTerminationState: api.ContainerState{Terminated: &api.ContainerStateTerminated{}}, }}, <del> map[string]api.ContainerState{ </del> <ins> expectedState: map[string]api.ContainerState{ </ins> "without-old-record": {Waiting: &api.ContainerStateWaiting{ Reason: startWaitingReason, }},

Related discussion:
text: The extra check of last terminated container in your initial code makes me nervous, but we couldn't reproduce it due to that small race windows until we have a faster docker running on the node. :-)
I got "Pods should invoke init containers on a RestartAlways pod" flake (referenced from ): Logs:
All the fixes went in. Please reopen this issue if you see any more flakes.
Code change:
rs := &apps.ReplicaSet{} rc := &api.ReplicationController{} <ins> extGroup := schema.GroupVersion{Group: "apps", Version: "v1"} extCodec := legacyscheme.Codecs.LegacyCodec(extGroup) </ins> <del> extGroup := testapi.Apps defaultGroup := testapi.Default </del> <ins> defaultGroup := schema.GroupVersion{Group: "", Version: "v1"} defaultCodec := legacyscheme.Codecs.LegacyCodec(defaultGroup) </ins> fuzzInternalObject(t, schema.GroupVersion{Group: "apps", Version: runtime.APIVersionInternal}, rs, rand.Int63())

Related discussion:
text: Execute make test WHAT=./pkg/api/testing/ And it failed with information: /kind failing-test
/sig api-machinery
what's the urgency of fixing this? Is it a release blocker?
i don't see it as a release blocker. but the problem is in client-go, so it's important to get fixed.
Code change:
kubeClient := testKubelet.fakeKubeClient kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ { <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, </ins> Spec:       api.NodeSpec{}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{

Related discussion:
text: Looks like some kubelet unit tests perform a lookupIp which ends up depending on the health of the local resolver. Specifically has this failing: Which leads to the test in: And we recently swapped out the node names from testnode to 127.0.0.1, so probably the culprit?
/ref
Interesting. I did swap out the node names in the unit test as part of to be 127.0.0.1, party because I thought that it wouldn't have any problems resolving during unit tests.
/cc
Code change:
kubeClient := testKubelet.fakeKubeClient kubeClient.ReactFn = testclient.NewSimpleFake(&api.NodeList{Items: []api.Node{ { <del> ObjectMeta: api.ObjectMeta{Name: "127.0.0.1"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: testKubeletHostname}, </ins> Spec:       api.NodeSpec{}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{

Related discussion:
text: cc
What is the environment where you are running the tests? They pass for me on my mac, a linux box, and shippable seems to be passing them as well.
I couldn 't reproduce the failure either.
I'm out right now; i'll try them when i get back. For the record, I run hack/test- Paul Morie wrote:
Ahem FTR This acronym is going to go quad-plat Paul Morie wrote:
I ran on both platforms and saw the tests pass.
I'm still seeing it. Won't have bandwidth to dig in personally until tomorrow.
seeing a similar failure after a git pull
I knew I wasn't crazy
environment: fedora 21, golang: 1.4, docker: 1.6
I think I see the problem. When I set the node name to in the test I was assuming it would parse as a valid ip address and not end up at the LookupIP call. But only the node name (and not the kubelet hostname) is being set (except in my new test where I ) so it's trying to do a lookup. Do you think you could test just changing the default hostname from "testnode" to "127.0.0.1" and see if that fixes it for you?
Trying now
I still get failures but no panic / threaddump.
gist coming
this looks like again?
Good that it isn't panicing any longer. I can dig into the test failures tomorrow (I think it should be relatively simple to resolve).
cool, thanks for looking
Passes in master now for me. Closing.
FYI, they will break again once goes in, and then we can merge my PR to see if that fixes it for you.
:+1:
is in -- are these tests breaking again?
I just rebased which I think will fix them.
They are breaking again.
is on it.
Code change:
# # We specify -X to avoid a race condition that can cause minion failure to # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 <del> curl -L http://bootstrap.saltstack.com | sh -s -- -X </del> <ins> curl -L --connect-timeout 20 --retry 6 --retry-delay 10 http://bootstrap.saltstack.com | sh -s -- -X </ins>

Related discussion:
text: Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
this appears to have always been broken, I'm going to disable the job for now. https://k8s-
Failed: https://k8s- Run so broken it didn't make JUnit output!
Code change:
if err := os.MkdirAll(dest, 0o755); err != nil { return err } <del> return utils.WithProcfd(c.root, m.Destination, func(procfd string) error { if err := mount(m.Source, m.Destination, procfd, "cgroup2", uintptr(m.Flags), m.Data); err != nil { // when we are in UserNS but CgroupNS is not unshared, we cannot mount cgroup2 (#2158) if errors.Is(err, unix.EPERM) || errors.Is(err, unix.EBUSY) { src := fs2.UnifiedMountpoint if c.cgroupns && c.cgroup2Path != "" { // Emulate cgroupns by bind-mounting // the container cgroup path rather than // the whole /sys/fs/cgroup. src = c.cgroup2Path } err = mount(src, m.Destination, procfd, "", uintptr(m.Flags)|unix.MS_BIND, "") if c.rootlessCgroups && errors.Is(err, unix.ENOENT) { err = nil } } return err } return nil </del> <ins> err = utils.WithProcfd(c.root, m.Destination, func(procfd string) error { return mount(m.Source, m.Destination, procfd, "cgroup2", uintptr(m.Flags), m.Data) </ins> }) <ins> if err == nil || !(errors.Is(err, unix.EPERM) || errors.Is(err, unix.EBUSY)) { return err } // When we are in UserNS but CgroupNS is not unshared, we cannot mount // cgroup2 (#2158), so fall back to bind mount. bindM := &configs.Mount{ Device:           "bind", Source:           fs2.UnifiedMountpoint, Destination:      m.Destination, Flags:            unix.MS_BIND | m.Flags, PropagationFlags: m.PropagationFlags, } if c.cgroupns && c.cgroup2Path != "" { // Emulate cgroupns by bind-mounting the container cgroup path // rather than the whole /sys/fs/cgroup. bindM.Source = c.cgroup2Path } // mountToRootfs() handles remounting for MS_RDONLY. // No need to set c.fd here, because mountToRootfs() calls utils.WithProcfd() by itself in mountPropagate(). err = mountToRootfs(bindM, c) if c.rootlessCgroups && errors.Is(err, unix.ENOENT) { // ENOENT (for `src = c.cgroup2Path`) happens when rootless runc is being executed // outside the userns+mountns. // // Mask `/sys/fs/cgroup` to ensure it is read-only, even when `/sys` is mounted // with `rbind,ro` (`runc spec --rootless` produces `rbind,ro` for `/sys`). err = utils.WithProcfd(c.root, m.Destination, func(procfd string) error { return maskPath(procfd, c.label) }) } return err </ins> } func doTmpfsCopyUp(m *configs.Mount, rootfs, mountLabel string) (Err error) {

Related discussion:
text: opencontainers/runc has fixed three CVEs: Latest runc is pulled by k/k Check runc version in file No response n/a n/a <details</details<details</details<details</details<details</details
/assign /sig security architecture release
/triage accepted
Code change:
import ( "fmt" <del> "k8s.io/kubernetes/pkg/api" </del> "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/runtime" )

Related discussion:
text: I started the federation-apiserver and went to and got the following output: Expected: I should have got an empty v1.ServiceList Any idea on what we are doing wrong? Have you guys seen this error?
I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote:
Deep copy generation was slightly broken as well - you may want to double check that they are registered. On May 23, 2016, at 5:50 PM, Daniel Smith wrote: I think your deep copiers or conversion functions are not getting registered properly. If you look at my change in , some places are inappropriately registering with api.Scheme instead of the passed in scheme. Maybe you have that problem or a similar one, since I think you guys copied some code. , Nikhil Jindal wrote: — You are receiving this because you are on a team that was mentioned. Reply to this email directly or view it on GitHub
I got the same error with latest branch, as I was swamping on service controller, so I use a previous version and worked around it.
deep copies were indeed missing, but adding them hasnt fixed the issue. Will debug more. is my PR to add the deep copies cc Please feel free to file issues for any such problem you face. We need to fix all these issues. Its good if we know about them early. Thanks!
fixes this
Code change:
framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podNames1, svc1IP, servicePort)) // Restart apiserver <del> initialRestartCount, err := framework.GetApiserverRestartCount(cs) Expect(err).NotTo(HaveOccurred(), "failed to get apiserver's restart count") </del> By("Restarting apiserver") <del> if err := framework.RestartApiserver(cs.Discovery()); err != nil { </del> <ins> if err := framework.RestartApiserver(cs); err != nil { </ins> framework.Failf("error restarting apiserver: %v", err) } <del> By("Waiting for apiserver to be restarted") if err := framework.WaitForApiserverRestarted(cs, initialRestartCount); err != nil { framework.Failf("error while waiting for apiserver to be restarted: %v", err) } </del> By("Waiting for apiserver to come up by polling /healthz") if err := framework.WaitForApiserverUp(cs); err != nil { framework.Failf("error while waiting for apiserver up: %v", err)

Related discussion:
text: This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.
Code change:
framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podNames1, svc1IP, servicePort)) // Restart apiserver <del> initialRestartCount, err := framework.GetApiserverRestartCount(cs) Expect(err).NotTo(HaveOccurred(), "failed to get apiserver's restart count") </del> By("Restarting apiserver") <del> if err := framework.RestartApiserver(cs.Discovery()); err != nil { </del> <ins> if err := framework.RestartApiserver(cs); err != nil { </ins> framework.Failf("error restarting apiserver: %v", err) } <del> By("Waiting for apiserver to be restarted") if err := framework.WaitForApiserverRestarted(cs, initialRestartCount); err != nil { framework.Failf("error while waiting for apiserver to be restarted: %v", err) } </del> By("Waiting for apiserver to come up by polling /healthz") if err := framework.WaitForApiserverUp(cs); err != nil { framework.Failf("error while waiting for apiserver up: %v", err)

Related discussion:
text: Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke.
Code change:
framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podNames1, svc1IP, servicePort)) // Restart apiserver <del> initialRestartCount, err := framework.GetApiserverRestartCount(cs) Expect(err).NotTo(HaveOccurred(), "failed to get apiserver's restart count") </del> By("Restarting apiserver") <del> if err := framework.RestartApiserver(cs.Discovery()); err != nil { </del> <ins> if err := framework.RestartApiserver(cs); err != nil { </ins> framework.Failf("error restarting apiserver: %v", err) } <del> By("Waiting for apiserver to be restarted") if err := framework.WaitForApiserverRestarted(cs, initialRestartCount); err != nil { framework.Failf("error while waiting for apiserver to be restarted: %v", err) } </del> By("Waiting for apiserver to come up by polling /healthz") if err := framework.WaitForApiserverUp(cs); err != nil { framework.Failf("error while waiting for apiserver up: %v", err)

Related discussion:
text: Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/
Code change:
serviceHasSynced cache.InformerSynced // nodeLister knows how to list nodes nodeLister corelisters.NodeLister <ins> // nodeHasSynced indicates whether nodes have been sync'd at least once. // Check this before trusting a response from the node lister. nodeHasSynced cache.InformerSynced </ins> // a list of node labels to register nodeLabels map[string]string

Related discussion:
text: What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.