opened
tosi3k
username_0
**Which jobs are failing**:
`ci-npd-e2e-node`

**Which test(s) are failing**:
`SystemLogMonitor should generate node condition and events for corresponding errors`

**Since when has it been failing**:
Last successful run (23 Oct 2020, 10:07 PDT): https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-npd-e2e-node/1319686617375444992
First unsuccessful run (23 Oct 2020, 12:08 PDT): https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-npd-e2e-node/1319717073798041600

**Testgrid link**:
https://testgrid.k8s.io/sig-node-node-problem-detector#ci-npd-e2e-node

**Reason for failure**:
```
_output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_problem_detector_linux.go:245
Timed out after 60.000s.
Expected success, but got an error:
    <*errors.errorString | 0xc001212cd0>: {
        s: "node condition \"TestCondition\" not found",
    }
    node condition "TestCondition" not found
_output/local/go/src/k8s.io/kubernetes/test/e2e_node/node_problem_detector_linux.go:360
```

**Anything else we need to know**:
[pull-npd-e2e-node](https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-npd-e2e-node) presubmit job in k8s.io/node-problem-detector repository also started to fail in the same fashion.

Attempted to debug it myself, found out the following stuff:
* last successful run was accessing the apiserver via HTTP:
```
W1023 17:12:40.514888    2808 test_context.go:420] Unable to find in-cluster config, using default host : http://127.0.0.1:8080
```
* all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail:
```
W1023 19:14:09.903857     995 test_context.go:437] Unable to find in-cluster config, using default host : https://127.0.0.1:6443
(...)
W1023 19:15:40.789213     925 util.go:106] Health check on "https://127.0.0.1:6443/healthz" failed, status=500
```
* perused the history of `k/k/test` to find out that the PR https://github.com/kubernetes/kubernetes/pull/94723 is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests):
  * first, it coincides with the start of the failures of `ci-npd-e2e-node`,
  * second, we changed the default hostname of the apiserver in node tests in that PR.

Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a `--host=http://127.0.0.1:8080` flag to the `--node-test-args` of these jobs.

/assign
/sig node
/cc @username_1 @Random-Liu
[Failing test] SystemLogMonitor should generate node condition and events for corresponding errors
issue
created
tosi3k
username_0
Opened https://github.com/kubernetes/test-infra/pull/19735 for a mitigation that would make those jobs stop failing.
comment
created
tosi3k
username_0
comment
created
knight42
username_1
@username_0 I have filed #96050 to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
comment
created
tosi3k
username_0
Unfortunately I don't :/. My PR was also a wild guess. @Random-Liu could you take a look at the proposed fix?

Thanks once again for your help - really appreciated :).
comment
created
tosi3k
username_0
@username_1 @Random-Liu the submission of #96050 hasn't fixed the problems, unfortunately :/.

Sample failing run: https://prow.k8s.io/view/gcs/kubernetes-jenkins/logs/ci-npd-e2e-node/1324222964127764480

We still have some peculiar problem related to health checking:
```
W1105 05:45:46.863813     983 util.go:106] Health check on "https://127.0.0.1:6443/healthz" failed, status=500
```

and now NPD panics, too:
```
panic: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
goroutine 1 [running]:
k8s.io/node-problem-detector/pkg/exporters/k8sexporter/problemclient.NewClientOrDie(0xc0003eb790, 0xc0000b6d00, 0x0)
	/home/prow/go/src/k8s.io/node-problem-detector/pkg/exporters/k8sexporter/problemclient/problem_client.go:73 +0x3f7
k8s.io/node-problem-detector/pkg/exporters/k8sexporter.NewExporterOrDie(0xc0003eb790, 0xc000492370, 0x1)
	/home/prow/go/src/k8s.io/node-problem-detector/pkg/exporters/k8sexporter/k8s_exporter.go:52 +0x4f
main.main()
	/home/prow/go/src/k8s.io/node-problem-detector/cmd/nodeproblemdetector/node_problem_detector.go:60 +0x13a
```
comment
created
knight42
username_1
@username_0 Thanks for your report, I believe the reason is `ServiceAccount ` admission controller has been disabled in node e2e tests:
https://github.com/kubernetes/kubernetes/blob/4364ef62062d3d18b0a0cec117af9620f8a896ff/test/e2e_node/services/apiserver.go#L64
We need to figure out a new approach now.

/reopen
comment
created
knight42
username_1
filed #96262. I hope we could completely address this issue this time.
comment
created
dims
username_2
/milestone v1.20
comment