jiebi/CodeConvo huggingface dataset:
Code change:
execAffinityTestForLBServiceWithTransition(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .
Code change:
execAffinityTestForLBServiceWithTransition(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Related discussion:
text: /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.
Code change:
return fmt.Errorf("timed out waiting for apiserver to be restarted") } <del> func GetApiserverRestartCount(c clientset.Interface) (int32, error) { </del> <ins> func getApiserverRestartCount(c clientset.Interface) (int32, error) { </ins> label := labels.SelectorFromSet(labels.Set(map[string]string{"component": "kube-apiserver"})) listOpts := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(listOpts)

Related discussion:
text: This test is failing in both gce serial suite: http://k8s- /sig network /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone cc /assign PTAL
xref
/milestone v1.10
this also fails in http://k8s-
Failing to connect to Services at all. I can't even ping pod IPs:
So I can ping pods same-node and cross-node right up until it kills the apiserver. Rebooting fixes a given node. The apiserver demonstrably comes back - I can query it for info...
- I could use help - have to shift attention soon.
FWIW, I am testing on a Calico-enabled node
I'll take a look at this as well.
Weird, now even after a reboot, this condition is sticky. I have a cluster in this state, if you want to poke.
ugh, now it is resolved.
I just repro'ed without the e2e - have a cluster, proved ping works, killed apiserver ping fails.
Interesting - docker shows the pods not running, but apiserver says they are alive.
Update: docker sees no running pods. API server thinks kubelet is running them. kubelet is getting "" from apiserver, even after kubelet restart.
Update and recap: Bring up a cluster Run some pods Prove that pods are running and services work pkill -9 apiserver Wait. Watch kubelets "lose" pods that were assigned to them Observe that pods are not running and services do not work Observe that apiserver says the pods are running pkill -9 kubelet Observe that it is still broken kubectl delete one pod Observe that kubelet is fixed for that one pod, but not others Q: Why is kubelet not getting a full listing when it restarts?
What exactly does it mean for it to be fixed for a deleted pod?
Kubelet started running the pod that was deleted - it got the update for that one.
The cluster I was testing was a week or 2 old. I built a cluster from HEAD and I can not repro - said the same. Is this test running against head of master?
I also ran the test manually against my new cluster and it passed
FWIW I was also unable to reproduce this yesterday on a cluster from HEAD.
Code change:
return fmt.Errorf("timed out waiting for apiserver to be restarted") } <del> func GetApiserverRestartCount(c clientset.Interface) (int32, error) { </del> <ins> func getApiserverRestartCount(c clientset.Interface) (int32, error) { </ins> label := labels.SelectorFromSet(labels.Set(map[string]string{"component": "kube-apiserver"})) listOpts := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(listOpts)

Related discussion:
text: Did some investigations and had some findings for this specific failure: Traced down a bit on the history, the first occurrence was https://k8s- From the , seems suspicious because it modified . Also found reported a relevant symptom: The fix () for that was merged a day ago and may explained why we can't repro from HEAD anymore.
nice, is this issue reproducible without the fix ()?
Sorry should have had confirmed this, bringing up a cluster now.
Quick update: I was able to repro the same failure () from HEAD with reverted. Though this test is now flaky on some other errors: https://k8s- https://k8s- They all seem to failed while communicating with apiserver.
Thanks ! So fixed the issue and I am curious about what introduced the issue. Do you happen to know when did this test start to fail? Just trying to find out what went wrong in between. We have some other possibly related issues so finding out the root cause of this one would be super helpful.
/assign
The first occurrence (for that exact failure) I noticed was https://k8s-, which happened on 2018-02-15 09:49 PST.
You all rock! I see a couple greens on it now, but still mostly red?
The test is now flaky mostly on creating an exec pod after restarting apiserver. Some sample failures: To elaborate a bit more, a snippet of the test procedure is: ssh to master and (sending a SIGTERM) Keep polling on /healthz on apiserver until 200 is returned. Create an exec pod for later steps (and failed on various erros --- EOF, connection refused, apiserver terminating). According to above test failures, my suspicion is that the test is assuming the apiserver is restarted too soon. It is possible that apiserver captured SIGTERM and did a graceful termination, so that polling on /healthz may succeed even in the termination phase. I'm not surprised trying to create a pod via a terminating apiserver will hit those errors. Could anyone confirme the SIGTERM theory?
Spoke with folks, sounds like after SIGTERM apiserver does wait a bit for remaining requests to finish. I will take a stab to refine the test. cc
Sent .
Test is now passing on gce but failing on gke.
Code change:
return fmt.Errorf("timed out waiting for apiserver to be restarted") } <del> func GetApiserverRestartCount(c clientset.Interface) (int32, error) { </del> <ins> func getApiserverRestartCount(c clientset.Interface) (int32, error) { </ins> label := labels.SelectorFromSet(labels.Set(map[string]string{"component": "kube-apiserver"})) listOpts := metav1.ListOptions{LabelSelector: label.String()} pods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(listOpts)

Related discussion:
text: Apparently we don't have a way to trigger apiserver restart on gke at this moment. Sending a PR to skip this test on gke.
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Commented too quick. On gke we do a same-version master upgrade. Will send another fix :/
Code change:
"//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/component-base/config:go_default_library", "//staging/src/k8s.io/component-base/metrics/legacyregistry:go_default_library", <ins> "//staging/src/k8s.io/component-base/metrics/prometheus/workqueue:go_default_library", </ins> "//vendor/k8s.io/klog:go_default_library", ], )

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : kube-controller-manager currently does not expose work queue metrics Why is this needed: queuing delay and queue length are important information to debug control plane performance in large scale cluster /assign zhan849 /sig apps
Code change:
const connectionTimeout = 1 * time.Second <del> type connection interface { serverReachable(address string) bool parseServerList(serverList []string) error CheckEtcdServers() (bool, error) } </del> // EtcdConnection holds the Etcd server list type EtcdConnection struct { ServerList []string

Related discussion:
text: /kind bug What happened: Every time a health check is called on the APIServer via the endpoint, an etcd healthcheck is performed. However, as this just starts at the first server in the list passed to the apiserver binary all the healthchecks generate read load on a single etcd member. What you expected to happen: The first server that is checked should either increment (and wrap around) each time or be randomised to spread load around the cluster
/assign
I think a random check is ok.
/sig api-machinery
Code change:
} missingIPs := desiredIPs.Difference(currentIPs) if len(missingIPs) > 0 { <del> glog.V(2).Infof("Current certificate is missing requested IP addresses %v, rotating now", missingIPs.List()) return time.Now() </del> <ins> glog.V(2).Infof("Current certificate is missing requested IP addresses %v", missingIPs.List()) return false </ins> } <ins> currentOrgs := sets.NewString(m.cert.Leaf.Subject.Organization...) desiredOrgs := sets.NewString(template.Subject.Organization...) missingOrgs := desiredOrgs.Difference(currentOrgs) if len(missingOrgs) > 0 { glog.V(2).Infof("Current certificate is missing requested orgs %v", missingOrgs.List()) return false } } return true } func (m *manager) certSatisfiesTemplate() bool { m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() return m.certSatisfiesTemplateLocked() } // nextRotationDeadline returns a value for the threshold at which the // current certificate should be rotated, 80%+/-10% of the expiration of the // certificate. func (m *manager) nextRotationDeadline() time.Time { // forceRotation is not protected by locks if m.forceRotation { m.forceRotation = false return time.Now() } m.certAccessLock.RLock() defer m.certAccessLock.RUnlock() if !m.certSatisfiesTemplateLocked() { return time.Now() </ins> } notAfter := m.cert.Leaf.NotAfter

Related discussion:
text: I'd love to see an integration test that would catch this (but that shouldn't block a fix).
Thanks for the comments! Now that I have the CLA signed, I'll address those and submit a PR.
Just submitted the PR:
Code change:
if longRunningRequestCheck(req) { return nil, "" } <del> return time.After(time.Minute), "" </del> <ins> return time.After(globalTimeout), "" </ins> } if secureLocation != "" {

Related discussion:
text: this job failed with It seems it is imssing some dependency The job history confirms it is totally broken Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Code change:
return volumeToMount.GenerateError("MapVolume failed", fmt.Errorf("Device path of the volume is empty")) } <ins> // When kubelet is containerized, devicePath may be a symlink at a place unavailable to // kubelet, so evaluate it on the host and expect that it links to a device in /dev, // which will be available to containerized kubelet. If still it does not exist, // AttachFileDevice will fail. If kubelet is not containerized, eval it anyway. mounter := og.GetVolumePluginMgr().Host.GetMounter(blockVolumePlugin.GetPluginName()) devicePath, err = mounter.EvalHostSymlinks(devicePath) if err != nil { return volumeToMount.GenerateError("MapVolume.EvalHostSymlinks failed", err) } </ins> // Map device to global and pod device map path volumeMapPath, volName := blockVolumeMapper.GetPodDeviceMapPath() mapErr = blockVolumeMapper.MapDevice(devicePath, globalMapPath, volumeMapPath, volName, volumeToMount.Pod.UID)

Related discussion:
text: /kind bug Using , if a block device is symlinked to the discovery directory e.g. , then MapVolume will fail with since the containerized kubelet cannot see the path on host. is mounted by containerized kubelet, so if before we try to map the device we set devicePath to the evaluation of symlink and end up with a devicePath of , containerized kubelet will see the device and successfully map the device. Are there any issues with this solution? This is on Kubernetes 1.10
/sig storage
ah yes containerized kubelet has many issues when dealing with symlinks. Check out Mounter.GetMode(), it calls a nsenter EvalSymlinks which should be able to do what you need. But actually, one thing to consider, is do you need to do the losetup on the host, or does inside the container work?
Yes it appears to work from inside the container, on the host I run which disappears after I delete the pod using the device. My understanding of running losetup in MapVolume is not that great but I suppose that as long as that /dev/loop0 is created we are fine. Thanks for the quick response, I'll create a PR using that function and tag you
Does it create it on the host or in the containerized kubelet? What happens when containerized kubelet is restarted? The purpose of losetup is to do "refcounting" on the block device to prevent possible naming collisions when devices are attached/detached under unhealthy conditions. Ie, if a device showed up as /dev/sdb and you gave that to a pod, and then for some reason it become unhealthy and disappeared (while the pod is still running), another device (maybe even another volume type) could get attached to the node and reuse /dev/sdb. Now the pod has access to a device it's not supposed to. With filesystem volumes, the refcounting is done with mount points, but for block device, loopback is the equivalent.
It is created on the host (and also visible in the container), and stays around through restart. I understand now, thank you.
Actually after I restart it becomes /vdb instead of /dev/vdb in the losetup output, idk what that is about.
Code change:
return volumeToMount.GenerateError("MapVolume failed", fmt.Errorf("Device path of the volume is empty")) } <ins> // When kubelet is containerized, devicePath may be a symlink at a place unavailable to // kubelet, so evaluate it on the host and expect that it links to a device in /dev, // which will be available to containerized kubelet. If still it does not exist, // AttachFileDevice will fail. If kubelet is not containerized, eval it anyway. mounter := og.GetVolumePluginMgr().Host.GetMounter(blockVolumePlugin.GetPluginName()) devicePath, err = mounter.EvalHostSymlinks(devicePath) if err != nil { return volumeToMount.GenerateError("MapVolume.EvalHostSymlinks failed", err) } </ins> // Map device to global and pod device map path volumeMapPath, volName := blockVolumeMapper.GetPodDeviceMapPath() mapErr = blockVolumeMapper.MapDevice(devicePath, globalMapPath, volumeMapPath, volName, volumeToMount.Pod.UID)

Related discussion:
text: There is no such thing as /vdb and /dev/vdb is still intact of course. Seems inconsequential since it is disappearing after all pods using the device are deleted, i.e. it's still serving as a refcounter
Code change:
// singleCallTimeout is how long to try single API calls (like 'get' or 'list'). Used to prevent // transient failures from failing tests. singleCallTimeout = 5 * time.Minute <ins> // sshBastionEnvKey is the environment variable key for running SSH commands via bastion. sshBastionEnvKey = "KUBE_SSH_BASTION" </ins> ) // GetSigner returns an ssh.Signer for the provider ("gce", etc.) that can be

Related discussion:
text: ci-kubernetes-e2e-gce-scale-correctness Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands kubetest.Test Jul 10 14:01:51 UTC+0200 Kubernetes e2e suite: [sig-node] SSH should SSH to all nodes and run commands: more . Test /sig node
/cc
/triage accepted
Uhhh, this may help here
/milestone v1.22
/sig scalability FYI
I got annoyed that I couldn't click through to see the other logs, so now you can, ref: (spyglass doesn't render build- because it's 500MB, and the limit for rendering is 100MB)
/assign SIG Scalability chairs/TLs, please find someone to address or call this a non-release-blocking failure
Assigning to current scalability oncall (): /assign
This is sig-node issue - should be addres by SIG-node folks. /sig node
is being discussed as a fix for this and I'm not sure it's correct
is it sig scalability's intent that 5k node clusters should support SSH?
The SSH test in question passes in the "default" job: So the fact that this failure is unique to a scalability job means to me that scalability needs to own at least some part of this
I did a little bit of digging and ci-kubernetes-e2e-gce-scale-correctness uses option in kubetest, which according to seems to be setting KUBESSHBASTION env. So I believe this is main difference between our scalability job and "default".
Debugging the test in and providing a fix there. The crictl host PR () is doing something different.