Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: node-kubelet-master Which test(s) are flaking: [sig-storage] Downward API volume should provide container's memory limit [NodeConformance] [Conformance] [cos-stable1] [sig-storage] Downward API volume should provide container's cpu limit [NodeConformance] [Conformance] [ubuntu] Testgrid link: https://k8s- Reason for failure: Both of these tests are failing with same reason. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Hi this issue needs your attention.
Still flaking
Hi can you call attention of the relevant folks here as well ?
For the first test failure posted here on 09/30. The reason is not enough CPU available. The type of error only happened once, it never happen again for this test. For the test failure mentioned, more than half of the tests (150+) failed including this Downward API tests. It happened quite often for ci-kubernetes-e2e-windows-containerd. It seems something wrong with container runtime. The pods are all stuck on pulling image phase, no other events after that and then timed out. Seems containerd for windows have some issues.. So I think this specific test is not considered as flaky test.
Latest failure: This failure includes, lot of other failing test as well. So could, for another major reason. Time Test -- -- 01/12/2020, 06:31:17 [sig-storage] Downward API volume should provide container's memory limit [NodeConformance] [Conformance] Though most flakes are on windows related jobs. But, there are other instances too. Do you think, above failure is for the same reason.
I am seeing downward api flakes in the Windows tests as well. If I broaden the search a bit it seems to happen for more than just memory limit but also: Test [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's memory request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu limit [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] For Windows they seem to be failing with the following (both docker and containerd): I will can look into this from a Windows perspective. /assign
For Windows, I do not believe this is an issue with the test it's self. I ran this subset of tests continuously over the weekend and got the following results: The is "paging file is too small": I was able to force similar (but not exactly the same) errors by over consuming memory via a test pod. I believe this might be happening due to other tests running on the cluster at the same time. I am looking into this further.
A few tests failed similar to this over the weekend:
FYI , during this saw some of the issues w/ containerd leaks
/sig windows
For the aks-engine Containerd test cases we lowered the number of running to the tests concurrency value we had prior to 1.19 and this has gone away. I am continuing to looking into if this was a regression or if is releated.
We haven't seen this in a long time but it came up today:
Another failure in gce windows test:
another failure here:
/milestone v1.22
another failure:
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten We saw this again in master
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone clear
We have a lead this might be related to working set bytes vs commit bytes and the way that windows handles memory.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/sig node
/remove-lifecycle rotten
/remove-sig It looks like sig storage issue. Please reapply if requires sig-node to triage.
This still flakes. An example: are you still working on this?
I haven't looked it from a Linux perspective. From Windows side, it seems this is caused by a mis-match between the commit memory and working set:
Posting some details here, since the issue above has been mentioned. We've been flaking on these downward API tests for quite a while, with errors like these: Logs taken from: Those error logs are actually from the Pod itself, which is supposed to run the agnhost golang binary. It doesn't always happen, but it does frequent enough to be visible in our Windows dashboards (8 failures in the past week, according to: (%20volume)%3F%20should%20provide%20container%27s). Looking through the downward API E2E tests, we noticed that some memory limits are being set in 8 test cases (). The memory limit set is 64MB, which is a bit low for a Windows Pod, even if it's nanoserver-based (also, the agnhost golang binary is not that small either, it has 44.2 MB on Linux). IMO, we could increase the memory limit to 128MB and we should see fewer flakes.
+1

Discussion:
}, Limits: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("1250m"), <del> v1.ResourceMemory: resource.MustParse("64Mi"), </del> <ins> v1.ResourceMemory: resource.MustParse("128Mi"), </ins> }, }, VolumeMounts: []v1.VolumeMount{

pod := downwardAPIVolumeForContainerResources(podName, "/etc/podinfo/memory_limit") e2epodoutput.TestContainerOutput(ctx, f, "downward API volume plugin", pod, 0, []string{ <del> fmt.Sprintf("67108864n"), </del> <ins> "134217728n", </ins> }) })