Change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- Impossible to say, the lane is failing because of the global timeout and does not have any attached artifacts. At least for two weeks
/cc
/sig node
It seems to be the impact after this PR /cc
It has been failing for months, if not years... :/ It did consistently do this before kubernetes/test-infra as well, but we no longer see any OOMs now tho. So it might be possible that the tests run fine, and that we only need to increase the timeout to get it properly working. Would be awesome if someone could take a look at it.
/cc
If selecting , you see that the test run sometimes, even though it looks like it times out every time. This is an example on a test run with artifacts , where it looks like the test kills the system/apiserver/kubelet. There is a strange log message around the time where it start failing:
Hmm, yeah, it looks like the apiserver dies because of too many open files, and the e2e suite doesn't restart it... So all in all, kubernetes/test-infra fixed the OOMs, but there are still other issues On my raspberry pi running normal k8s 1.21 i get the following: Since node e2e tests, and therefore the apiserver, is running under sshd, this might be the problem? Anyone who have more knowledge about this?
/triage accepted /priority important-soon
great catch! :) Now I wonder (and I'll check reviewing the tests) that is intentional for the test to kill the apiserver. I wonder how it is supposed to work in the ad-hoc environment the tests run into.
I know a lot of tests are intentionally killing and restarting kubelet, but I don't think that killing the apiserver is wanted. From the tooling where it is started, it looks like there is no tooling around restarting it;
Yep, this is what I was thinking. Restarting/killing kubelet is fine, and supported, we do it in quite some places (at very least cpu, topology manager e2e tests, also memory manager tests if memory serves). It's killing apiserver and/or etcd, and in general everything-but-kubelet which looks suspicious in the context of the e2enode tests. Maybe these tests are just misplaced? should they be moved in the regular e2e suite, on which they run against a full cluster and not on a mini-, special purpose environment like e2enode tests are running?
Yeah. I think a simple "restart" apiserver if it dies would work. It would however mask potential performance issues, so I think the best option long term is to look at why the apiserver dies; and try to fix that. I think that isn't possible, since the tests in need to be serially executed, and they require full access over kubelet and its lifecycle, together with modifying node stuff. They are also pretty disruptive, especially when testing eviction related things. If the regular e2e tests can allow for this, then I guess that is ok, but I don't think that is the case.
This surely fixes the current issue, and we could do that. But I wonder if it is the right thing to do in the e2enode flow. I mean: should we handle these restarts in the special purpose reduced environment? are these tests already abusing this environment? I'll start doing some archeology in git history as soon as I have more time to spend on this issue. Good point. I agree.
I'll start doing some archeology in git history as soon as I have more time to spend on this issue. Agree! I guess we can do something like this: and see if it mitigates the issue. In case it is actually running with , that isn't a realistic production environment anyways. What do you think about that?
This looks like a fine change and we should most likely fix the open files issue anyway. So thanks for sending the PR!
I also found an instance of , and that doesn't look that good :sweat_smile: Would be nice if anyone could take a look at increasing the timeout in test-infra as well, and see if that makes any difference.
/cc
Still work in progress though.
After some digging, I found out that the reason for the lack of stdout and artifacts from this test suite is the way it is killed upon the timout. When the global timeout is exceeded, SIGINT is sent to all the processes. If we ignore the first SIGINT in node-e2e, we can print logs to stdout and fetch artifacts. In case the test is still running 15 minutes later, it will be killed via SIGKILL. See . Would be nice to get that merged as soon as possible!
this is a good idea, but is seems racy because it seems there is a hidden assumption we have enough time to collect the log and the artifacts. Which is a big step forward anyway! but let me ask: do we have a simple way to, once the process receives SIGINT, start explicitely the process of waiting for childs to end, collect logs + artifacts and then terminate (instead of just ignoring the signal and wait for things to converge implicitely)? If so maybe worth making it a bit more robust; if not, let's go ahead with this fix anyway/
Thanks for looking The executor will wait 15min before sending SIGKILL. That should be ok with time. If that doesn't work, we can increase that later. Since the runner will send SIGINT to all children processes, the SSH connections, they will be terminated by SIGINT. Printing of logs and collection of artifacts will therefore start imminently after getting the SIGINT. This worked well when running locally, and hopefully it will work the same way here. Yes, at a later stage that might be smart. The big issue here is that we use the same timeout for ginkgo and for the global test runner; resulting in the fact that the global test runner will always kill the test before ginkgo hits its timeout. Does that make sense? Tried to describe it here as well:
The thing that worries me it seems we only rely on time and to a lot of implicit assumptions in general. I fully ack we are in deep red zone and we need to fix the lane. I'm totally onboard with the urgency (and the sorry state of the lane in general). I think we should not rush fixes in anyway which could make our life harder in the future. OTOH it seems that better and more robust handling of timeouts and artifacts collection need some more investigation and a bit more work; your change clearly signals what's going on and why we do what we do, so I'm inclined to go ahead with it. Yes, it does make sense to me. Your fix seems to hit the sweet spot between unlocking things without making the overall infra more fragile. Still, this is one more hint the flow as whole need some attention and some improvements. Yep, thanks.
Yeah, completely agree! This definitely needs some more love. There are so many layers of things, making it all fuzzy and hard to reason about, and debug... :/ Maybe someone from sig-testing have more insight into what we should do, and how we should do it.
/cc perhaps?
/assign (since you're actively on it...)
Yes, thanks. I'll report my status on today's sig-node CI group meeting
On receiving the timeout signal the process should begin cleanup / teardown, possibly proceeded by log dumping. It should not ignore it. We set a large grace period on to feasibly allow cleanup. When the kill signal is received that is it. If you're hitting the timeout regularly you need to increase the configured timeout or speed up the tests.
Yes. The current behavior is that the test runner get killed by the it gets from prow (or what you would call it), and the logs will therefore not be printed and artifacts are not fetched. With , the test runner will "ignore" the SIGINT from timeout, while all its children processes will be killed. It will therefore imminently start printing the logs and fetching artifacts. The runner will therefore exit cleanly quite quickly after the initial . In a perfect world we could properly terminate the ginkgo suite, but since we get to all procs, but there is currently no way to do that. Is that an ok solution, or do you have any other ideas. Yeah, that is work in progress. The issue now is that we get no logs or artifacts when tests time out, and that is quite impractical :/
FYI, we now have a job from if we want to test fixes on PRs.
cc this scenario is something to be aware of for kubetest -kubetest2
I'm most likely stating the obvious, but if we exclude the eviction tests, the lane seems to complete. Running locally with initial success, checking if it was a lucky run.
from the PRs above, still timed out in CI after 7 hours, I can try to submit a PR to exclude the eviction tests. Can we test those separately?
:tada:
/reopen we are not done yet
Reopened this issue. <detailsIn response to : /reopen we are not done yet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/close This issue was fixed in the 1.23 release. The job was replaced by and
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
} func getNodeProblemDetectorImage() string { <del> const defaultImage string = "k8s.gcr.io/node-problem-detector:v0.6.2" </del> <ins> const defaultImage string = "k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7" </ins> image := os.Getenv("NODE_PROBLEM_DETECTOR_IMAGE") if image == "" { image = defaultImage