Code change:
Which jobs are flaking: pull-kubernetes-e2e-gce pull-kubernetes-e2e-gce-rbe Which test(s) are flaking: Stage Testgrid link: Reason for failure: This isn't technically a test that's failing, it's part of the build-stage-up-test cycle used by our e2e jobs. There are two main clusters of failures that I can tell. One is timeouts. The other looks like: Example -gce jobs: () () () () Example -gce-rbe jobs: (source gist ) Anything else we need to know: Triage link for the job that actually reports to PRs: ends up running Some googling for "bazel error code 14" leads me to believe that bazel (or its jvm) is getting killed due to OOM. which is a value that's been cargo culted around as far as I can tell. I was unable to find the decision that led to this value. I'm wondering if a bump to 8Gi of memory would help reduce occurrences of this flake. I'm referencing the -rbe jobs to point out that this happens even if we use RBE, but they're not blocking
/sig testing Top flake according to
With merged we'll wait and see if the Stage flakes on pull-kubernetes-e2e-gce go down. If they do, there are other presubmit jobs that could benefit from that. If they don't, we should revert.
/priority important-soon
/cc
Still seeing flakes, so I don't think just bumping memory request helped. ref: my theory is that making this non-privileged broke our ability to free memory This happened via around 2019-04-11 - flakes were still occurring prior to this merge - but not prior to 2019-04-01 - and very infrequently prior 2019-03-15 It could be that bazel's memory usage grew when we bumped it to 0.23.2 on 2019-03-29 (or maybe on 2019-04-02 is what actually deployed the change) Either way, I still think it looks like bazel is eating more memory than the container is allowed, and is getting oomkilled. Two ideas: Instead of allowing the bazel server to exist as a long lived process, have kubetest call in between the Build and Stage steps, in addition to trying to flush memory Constrain bazel's host JVM to a hardcoded memory limit via in bazelrc
Idea 3: switch build modes and see how expensive / flaky that is. Including: use RBE use the dockerized build instead
I opened to try and constrain the JVM's memory. Jobs that use RBE behave the same way, so try doing the same to them for comparison's sake.
this is still being seen extremely frequently (seen multiple times in a row on the same PR at )
I opened for the last idea I have for working around bazel's behavior, beyond this I am out of my depth on how to replace bazel for the release/push- script
... but I'm willing to try with a canary job and see what happens
bumping priority, this looks like 99/122 flakes according to /priority critical-urgent
Another occurrence:
I'd like to see if we can root cause what changed around 1/27-1/29 that greatly increased this
kubernetes/kubernetes changes: 1/26, ~6am UTC: 1/28: ~6pm UTC: test-infra changes: 1/25, ~3pm UTC: 1/28: ~8pm UTC: in that delta kubekins changed from v20200123- to v20200127- nothing jumps out at me in those diffs, but wanted to link to them in case it helped anyone else see something
It doesn't have to be one, the CI is run overcommitted with no limits, something else could be upping memory pressure..? On Mon, Feb 10, 2020, 08:38 Jordan Liggitt <:
ok... after sweeping k/k and test-infra for that time period, nothing clear emerged, and the bazel bandaids seem to be holding for now. unsatisfying :-/ /close
Closing this issue. <detailsIn response to : ok... after sweeping k/k and test-infra for that time period, nothing clear emerged, and the bazel bandaids seem to be holding for now. unsatisfying :-/ Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: else bazel-release: bazel build //build/release-tars <ins> bazel shutdown </ins> endif