jiebi/CodeConvo huggingface dataset:
Code change:
# Clean up kubectl delete jobs pi "${kube_flags[@]}" # Post-condition: no pods exist. <del> kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </del> <ins> kube::test::wait_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </ins> # Pre-Condition: no Deployment exists kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" ''

Related discussion:
text: <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: test-cmd runkubectlrun_tests Since when has it been failing: According to triage, the first failure was 2019-02-21 12:00 UTC Testgrid link: Reason for failure: Unknown Anything else we need to know: This may be a flake? I can't tell yet. The testgrid link is not a continuous failure, but it could be that not enough PR's have whatever change is causing the failure Triage link: It looks like it may be happening in CI as well
/sig cli /priority critical-urgent /milestone v1.14
/remove-kind failing-test /kind flake I can't see it consistently happening in CI, so it must be a flake?
What I got when running locally...
And I see the same thing in the build- for
Trying to track down the first occurrence. I think there was an XML parsing error that was obscuring this that was fixed recently. Backtracking through pull-kubernetes-integration failures and visually inspecting them. - runkubectlruntests - 2019-02-22 15:45 PST - runkubectlruntests - 2019-02-21 01:13 PST - runkubectlrun_tests - 2019-02-20 16:50 PST ... and then I don't see this particular test failing during 02-19 or 02-18
So going by that, let's cast a net in commits that landed just prior to the earliest PR failure, and N hours prior
Suspects - ~only because it touches nope, rolled back to right after this landed, no repro after 30 attempts - ~enables server side apply <-- this is my guess~ nope, rolled back to right after this landed, no repro after 30 attempts - reduce lock contention in watchcache
the test might be racy in the first place, maybe enabling the feature changes the time profile of the test.
Welp, now that I have what I think is a local repro loop for this, and I'm bisecting, it's looking like the whole commit range I suspected is fine Looking closer, this particular test case is overly optimistic, it deletes a job and immediately gets pods expecting there to be none.
Code change:
# Clean up kubectl delete jobs pi "${kube_flags[@]}" # Post-condition: no pods exist. <del> kube::test::get_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </del> <ins> kube::test::wait_object_assert pods "{{range.items}}{{$id_field}}:{{end}}" '' </ins> # Pre-Condition: no Deployment exists kube::test::get_object_assert deployment "{{range.items}}{{$id_field}}:{{end}}" ''

Related discussion:
text: I'd still like to nail down what timing change surface this before I go change the to a or using a different set of flags for the call
Well, for some reason seems to be when the flakes started showing up
I realize that the flake is hard to reproduce especially since it's hard to run specific tests in . You can comment a lot of the tests by playing with some of these shell scripts so that they mostly just run the test you want to run. if you do so, you might be able to run 10 or 20 times the test in a row at this PR and right before. I'm curious what the difference would be in term of flake rate.
Yeah that's what I've been doing. It's and specifically looks to be . Which has nothing directly to do with pods, so I guess it's just a matter of this slowing the apiserver down just enough to start tripping a poorly written test.
FYI for the above. My plan for now is just to rewrite the test so it's more forgiving. But I wanted to confirm that it's expected / known that apiserver performance changed a bit as a result of this.
/assign /assign
What metric are you seeing suffer with that commit? There should be no impact.
A racy test goes from flaking 0% of the time to flaking 10-20% of the time Jordan Liggitt <:
flakes: It should probably use instead. The thing is, I can't get it to flake before. I hacked up test/cmd/legacy- to run just this, and none of the other test cases
It definitely should. I'll take a look at job-pod deletion propagation time before/after that commit, but there's nothing I can think of in that path a new networking API group would impact.
Code change:
for _, testcase := range testcases { t.Run(testcase.name, func(t *testing.T) { <del> buf := new(bytes.Buffer) errBuf := new(bytes.Buffer) </del> <ins> buf := new(buffer) errBuf := new(buffer) </ins> klog.SetOutputBySeverity("INFO", buf) klog.SetOutputBySeverity("WARNING", errBuf)

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: No data races How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
for the record as we tend to lose the logs in the gcs bucket
/sig instrumentation
Code change:
// Flaky operation failures in an e2e test can be captured through this. flakeReport *FlakeReport <del> // To make sure that this framework cleans up after itself, no matter what, // we install a Cleanup action before each test and clear it after.  If we // should abort, the AfterSuite hook should run all Cleanup actions. cleanupHandle CleanupActionHandle </del> // afterEaches is a map of name to function to be called after each test.  These are not // cleared.  The call order is randomized so that no dependencies can grow between // the various afterEaches

Related discussion:
text: Here is a list of TODOs that could be fixed for Ginkgo migration from v1 to v2. [x] Lost the validation on the SIGDescribe: [x] Ginkgo V2 currently cannot get the number of specs that will run in , this might be fixed in the Ginkgo's minor release of V2 in the future. [x] <delGinkgo dry-run mode has dropped the test spec in V2 (), consider to switch back to use to generate test spec when the feature is ready in . </del--- This has been fixed in Ginkgo 2.1.4 [x] <delMigrate all the custom reporter in , should be lightweight and not sure where those custom reporters are used. </delFixed here: [x] Replace all the camel case flags to kebab case flags, e.g. to . [x] <delRemove the legacy e2e test binary's parameter and in 1.28, those parameters were used to manage report, this is no longer needed for Ginkgo v2. </delno longer needed. [x] upgrade vendor when is migrated to ginkgo v2, this will help to remove the dependency of - [x] <delrevisit to explore the possibility to cover the test in Ginkgo V2. </delfixed by [x] remove the workaround introduced in Ginkgo V1 ( is not called after hitting ^c): partially addressed by: , some cleanup needed for each sig before removing the hook entirely. I think has the pr to fix this entirely, but I cannot get the link for that, mark it as resolved. : [x] New work item (?): capture only a single output text output stream () in JUnit (this issue was resolved along with another one in ) [ ] Due to the junit format change, some tool which use to grab "value" directly cannot get the failure reason as what it does in v1. [x] Test failures are missing information since GinkgoV2 migration [ ] is clustering stack traces, not failure text () Address those leftovers if there is any update from Ginkgo in the future. see the relevant link on the each of the issues.
Code change:
// Flaky operation failures in an e2e test can be captured through this. flakeReport *FlakeReport <del> // To make sure that this framework cleans up after itself, no matter what, // we install a Cleanup action before each test and clear it after.  If we // should abort, the AfterSuite hook should run all Cleanup actions. cleanupHandle CleanupActionHandle </del> // afterEaches is a map of name to function to be called after each test.  These are not // cleared.  The call order is randomized so that no dependencies can grow between // the various afterEaches

Related discussion:
text: No response_ <details</details<detailsall </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/cc
/cc
hello! I would like to work on this issues
I think we cannot do anything now except the last two, and those two should be based on top of the:
I would like to work on task 5 of this issue
no problem, pls wait for an while until that PR got merged, then you can create a pr to fix those deprecated parameters.
Regarding : searching for points out the places where the workaround is used and needs to be replaced once is solved.
I just noticed this comment in
seems like is not used in k/k, but in other sigs.
hmm... I just grepped the test code, will push a pr to fix this.
/open
/re-open
okay, this is not resolved actually, but I have no way to re-open it.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
~ it seems that some more jobs needs to be adapted to not use parallel~ [repo][0]=kubernetes/test-infra~ ~ACTION REQUIRED: Ginkgo V2 doesn't accept go test's flags to parallelize Ginkgo specs, please switch to use or instead.~ Sorry for the noise, the flag is passed to kubetest, and it correctly translate it to , it is not the issue here
np, thanks for the update!
New work item (?): capture only a single output text output stream () in JUnit.
updated the issue, haven't got a chance to take a close look at the item though.
This should be resolved by "open stdout" now should only appear if a test intentionally records SpecReports.
mark it as resolved.
is this complete now?
Code change:
// Flaky operation failures in an e2e test can be captured through this. flakeReport *FlakeReport <del> // To make sure that this framework cleans up after itself, no matter what, // we install a Cleanup action before each test and clear it after.  If we // should abort, the AfterSuite hook should run all Cleanup actions. cleanupHandle CleanupActionHandle </del> // afterEaches is a map of name to function to be called after each test.  These are not // cleared.  The call order is randomized so that no dependencies can grow between // the various afterEaches

Related discussion:
text: , Only two of them left, I need to follow this one, here is the issue for this one opened in the Ginkgo: I think this should be fixed in the third party tools instead of Kubernetes, if we agree with that, we can mark it as completed as well. For the first one, I will continue to follow until there is a update from Ginkgo's side, for the later one, there is no consensus yet IIUC, we can open a separate issue for that, I am gonna to close this one since there is no much thing we can do now. /close
We agreed to fix the tool, but had no issue open for it. I just created one:
Another issue for
Code change:
if err != nil { t.Fatalf("Failed to create scheduler: %v", err) } <ins> // Profiles </ins> profiles := make([]string, 0, len(s.Profiles)) for name := range s.Profiles { profiles = append(profiles, name)

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --here is supposed to be set through , but is not called from anywhere except tests, so always returns . should be set through here. Am I missing something? N/A None Kubernetes version (use ): master () Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: None
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Thanks for pointing that out. The plumbing of extenders to the framework impact calling extenders on preemption. That seems to be a path not covered by tests end to end. Can you submit a PR to fix this? as you mentioned we need to pass the extenders on profile creation. We need to patch the fix back to v1.21 and v1.20 (the release where this was refactored). /cc
I have submitted a PR for a fix.
Code change:
defaultPodResourcesPath    = "/var/lib/kubelet/pod-resources" defaultPodResourcesTimeout = 10 * time.Second defaultPodResourcesMaxSize = 1024 * 1024 * 16 // 16 Mb <ins> kubeletReadOnlyPort        = "10255" kubeletHealthCheckURL      = "http://127.0.0.1:" + kubeletReadOnlyPort + "/healthz" </ins> ) func getNodeSummary() (*stats.Summary, error) {

Related discussion:
text: ci-kubernetes-node-kubelet-serial-hugepages when start the pod with the resources requests that contain multiple hugepages resources should set correct hugetlb mount and limit under the container cgroup Unexpected error: "pod ran to completion", Multiple sizes huge pages is going to be graduated to GA, so having this test job fixed is quite important.
/sig node
/assign
/triage accepted
/cc
I did a small analysis and I can see that the pod failed to start, I can not find any failure messages under the kubelet log, so it really hard to say why it failed.
were you able to reproduce it locally? btw, there is a "right way" to fix this - to fix dynamic update of a hugepage resources. This would eliminate need in a kubelet restart. There is even a PR for that: , we just need to make it working again.
after conversation with on slack I'd say we should stick to the current procedure - restarting kubelet when hugepages allocation changes. So, we should investigate why kubelet doesn't restart properly in this test case and fix it.
Nope, I failed to reproduce it locally, also under the CI, it does not appear often. The best thing will be to describe the pod in case if it failed, so we will see the reason.
under the code we have so from the log , I can see that the kubelet received the pod before the kubelet stopped, so the problem that pod dropped to Failed state because of the kubelet stop.
So some additional details, the one who responsible to restart the kubelet again, is the server service: restart the kubelet service are waiting for hugepages to be available are starting the pod server service restart the kubelet again because it was not healthy for too long - failed to start because of the kubelet restart
great findings! Should we just stop the kubelet and let the server service to start it again? We can also check if the kubelet is restarted successfully before starting the pod.
I tried it under the PR , also I unsure how can we check if the kubelet restarted successfully. All tests that I wrote before I checked if the node has NotReady state and waited until it will be Ready, but in case of the restart probably the node will always have Ready state because the time period is very short. I will continue to investigate and will check locally how can I fix it.
Code change:
defaultPodResourcesPath    = "/var/lib/kubelet/pod-resources" defaultPodResourcesTimeout = 10 * time.Second defaultPodResourcesMaxSize = 1024 * 1024 * 16 // 16 Mb <ins> kubeletReadOnlyPort        = "10255" kubeletHealthCheckURL      = "http://127.0.0.1:" + kubeletReadOnlyPort + "/healthz" </ins> ) func getNodeSummary() (*stats.Summary, error) {

Related discussion:
text: So the only reliable solution that I found, it just trying to start the pod number of times. I tested it 5 times and the test passed. Can you please review
Can we access kubelet service from hugepage test cases? If we can we can use its APIs like or to check if kubelet is ready after the restart. Overall I have a feeling of duplication test/e2enode/services functionality in This is a main reason for this and similar bugs I believe. It would be nice to get rid of this duplication if possible. That would nicely isolate kubelet service controls in one place and eliminate conflicts like this one. wdyt?
not really, the used mostly to start all needed components to emulate the cluster(not only the kubelet) and the test does not have access to it. I think you can separate the readiness check under the importable method, but it worth the separate PR.
I updated the PR to check the healthiness of the kubelet and looks like it works just fine.
thank you. it looks good to me so far. I'll add lgtm label when is merged and tests are all green.
the job is still flaky. can you reopen this issue?
I opened a different one - , the issue can affect any job, not only the hugepages one.
I can open a fix to W/A an issue(start the pod until it will succeed), it should fix all problems, WDYT?
i think it's ok as a workaround, until is fixed
would it be better to wait until api server reports correct resource value before running the pod?
We are waiting for it, the problem that the scheduler and the kubelet see a different picture regarding resources requested by pods.
ok. In this case restarting the pod until it succeeds seems reasonable to me to fix the flakiness until is fixed.
any progress on the workaround fix?
Code change:
} // if we're trying to hit the kube-apiserver and there wasn't an explicit config, use the in-cluster config <del> if server == "kubernetes.default.svc" { </del> <ins> if target == "kubernetes.default.svc" { </ins> // if we can find an in-cluster-config use that.  If we can't, fall through. inClusterConfig, err := rest.InClusterConfig() if err == nil {

Related discussion:
text: We are getting the following error when the apiserver connects the webhook on localhost (configured via URL). We should only use the SSL tunnel for the connections to nodes when the webhooks are running as services. /sig api-machinery /kind bug /assign
/close
Code change:
} } <ins> // printPodsMultiline prints multiple pods with a proper alignment. func printPodsMultiline(w PrefixWriter, title string, pods []api.Pod) { printPodsMultilineWithIndent(w, "", title, "t", pods) } // printPodsMultilineWithIndent prints multiple pods with a user-defined alignment. func printPodsMultilineWithIndent(w PrefixWriter, initialIndent, title, innerIndent string, pods []api.Pod) { w.Write(LEVEL_0, "%s%s:%s", initialIndent, title, innerIndent) if pods == nil || len(pods) == 0 { w.WriteLine("<none>") return } // to print pods in the sorted order sort.Slice(pods, func(i, j int) bool { cmpKey := func(pod api.Pod) string { return pod.Name } return cmpKey(pods[i]) < cmpKey(pods[j]) }) for i, pod := range pods { if i != 0 { w.Write(LEVEL_0, "%s", initialIndent) w.Write(LEVEL_0, "%s", innerIndent) } w.Write(LEVEL_0, "%sn", pod.Name) } } </ins> // printPodTolerationsMultiline prints multiple tolerations with a proper alignment. func printPodTolerationsMultiline(w PrefixWriter, title string, tolerations []api.Toleration) { printTolerationsMultilineWithIndent(w, "", title, "t", tolerations)

Related discussion:
text: However it will meet an error if the pod have no volumes, so I improves it by adding a to check whether the volumes exists or not.
python script to list pods with pvc in namespace(s): Is a part of a cli to set pods with velero/restic annotations. Hope it helps someone :)
For posterity... There is a Used By: field in the output of describe.
client-go didn't give this api... ,PodPVCIndex can solve the problem
Hi thanks worked for me ;) but I want to make it reverse my workarounds with adding after didn't worked do you have any ideas
$ kubectl describe pvc -A Used" Name: ocs-deviceset-lso-volumeset-0-data-1b45dn Namespace: openshift-storage StorageClass: lso-volumeset Used By: rook-ceph-osd-1--8fwpd Also useful ^
the reason it didn't work is because each return does, in point of fact, have a key defined for claimName. That key is just set to null value. So, what you'd actually want to do is test for nulls. As nice as answer is, it's a PITA to parse unstructured human-readable command output, as opposed to command output in a markup language that well known shell parsers like jq and yq can work with. Towards that end, because I had use of this, I came up with the following: That simple not null test at the end plucks out all your nulls. Obviously invert the last conditional if you want all the nulls, but you'll get all sorts of temp things that way that you probably won't care about.
Code change:
NoSnatTestProxy          = ImageConfig{e2eRegistry, "no-snat-test-proxy", "1.0", true} NWayHTTP                 = ImageConfig{e2eRegistry, "n-way-http", "1.0", true} // When these values are updated, also update cmd/kubelet/app/options/options.go <del> Pause               = ImageConfig{gcRegistry, "pause", "3.0", false} </del> <ins> Pause               = ImageConfig{gcRegistry, "pause", "3.0", true} </ins> Porter              = ImageConfig{e2eRegistry, "porter", "1.0", true} PortForwardTester   = ImageConfig{e2eRegistry, "port-forward-tester", "1.0", true} Redis               = ImageConfig{e2eRegistry, "redis", "1.0", true}

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Failures - https://k8s- Part of pause is not used properly for tests for other arches than x86. As the current pause image is not a manifest image, so we need to use a different image for different archs like for , for etc.. With this, I'm using logs-generator from repo What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing
/assign