Change:
pull-kubernetes-e2e-kind [sig-storage] CSI .* It seems that happens since 7/8 The test timeout the docker daemon is killed, but the process keeps running, but it fails to do anything because all the features are missing, in this case, the port forwarding from the host to the apiserver container, as we can see in the following log entry As consequence, all commands fail, like the ones that gather the logs or clean up the environment The sig storage test take more than 10 minutes we shouldn't have test with that duration on presubmit, those also should be tagged as slow ~The trigger may be related to the ginkgo v2 update, the failures start to happen the same day that merged~ /sig testing /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc I know this people is in vacation, ccing just for visibility /cc
I don't know if correlation implies causality here, but it seems to start the day that the ginkgov2 was merged /cc is there any change in behavior for which the tests keep running, I remember some discussion about this between you and about the failure modes
hey there correlation doesn't necessarily imply causation - but it's definitely the first place to start looking ;) there shouldn't be any explicit changes in V2's behavior here however there were changes around how Ginkgo captures stdout/stderr during parallel test runs (i.e. when running with ) that, in some circumstances involving external processes that hang, can cause to hang. Often when this happens Ginkgo catches it and prints out a message but I assume you haven't seen anything like that? Here's something to try: If: (a) these specs are running in parallel (b) the failure is somewhat reproducible Then: try running with to see if the flaky failure is resolved. If the test consistently exits without hanging with this flag on then that would point to this output-interception issue and potentially help us figure out the next step for debugging things. This sort of failure would also happen in Ginkgo v1 but it would be intermittent. V2 has a much cleaner implementation of output interception which has the side-effect of making this sort of failure more consistent. usually the issue ends up being with a spawned process holding on to an open pipe or failing to clean up somehow.... it can be a gnarly thing to debug :(
it seems that the CSI tests are taking into account the time to delete the namespace, per example takes 10 mins, starts at and takes 5 mins deleting the namespace it feels like this is related on how CSI tests are structured, made some comments about this here it seems that is the problem, the sig-storage test take the time to delete namespace into account , maybe for the way the use the hooks for AfterEach, can this be refactored, the current CI presubmits are very flake because of this problem
we are at a success 47% rate on presubmits, /priority critical-urgent
Hi I'm not sure I understand the issue. Is there a change in cleanup behavior to cause these tests to start timing out? cc who also looked into storage test cleanup in the past.
Yeah, it seems to be related to the ginkgo v2 migration, there were some behaviour changes ... I'm just trying to get some help because I couldn't find the problem and I am lost on those CSI e2e test logic
i put this in the sig-testing slack but in case it's relevant here: one thing to note is that in ginkgo v1 the —timeout flag used to apply to each suite separately. So if you are running 3 suites with a timeout of 50 minutes each suite gets 50 minutes. In v2 the timeout governs the entire test run - that means all 3 suites must finish within 50 minutes. not sure if that is in play here (are there multiple suites running?)
let's try this theory
can you please take on the slow tests from sig-storage that are running on presubmits? I can do it but I'm afraid of breaking something, you can see them here
This seems to have improved
nope, it didn't improve, it may be an artifact of not having many PR or low load on the CI, who knows
hey -- looking at just that last run I'm seeing: 13:18 is the first timestamp I see. Eventually (~around 13:35) the specs start running. Everything is happy and several hundred specs run in parallel and pass. The last three specs then fail - these three are running in parallel. The first spec is: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (block volmode)(allowExpansion)] volume-expand [AfterEach] should resize volume when PVC is edited while pod is using it and has an error trying to delete the test namespace: The seems to wrap it in a try and so this timeout/failure doesn't mark the spec as failed. The storage specific cleanup keeps running... This stuff: appears to succeed - but then this step: seems to hang. Eventually, at 14:18:32 I see: Presumably after three minutes of polling the namespace hasn't been removed. But now that the outer container/infrastructure has timed out the tcp connection attempting to continue polling fails. It looks like that 1 hour timeout is governed by the test setup, nto the Ginkgo test suite. With the container/infrasturcture gone, the spec reports the error "". but this too is wrapped in a try. The spec finally fails in the with: again, because the container/infrasturcture was killed at 14:18:32. This failure now bubbles up as the failure associated with the spec. But in reality the underlying issue seems to be a failure to delete the namespace. The second spec is: [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: CSI Ephemeral-volume (default fs)] ephemeral [AfterEach] should create read-only inline ephemeral volume Here too we see: ^ this appears to be the actual failure. Eventually the reports a failure at 14:18 after the container/infrastructure times out. Finally, the third spec is: [sig-storage] CSI mock volume [AfterEach] Delegate FSGroup to CSI driver [LinuxOnly] should not pass FSGroup to CSI driver if it is set in pod and driver supports VOLUMEMOUNTGROUP here, once again, we see ^ this appears to be the actual failure. As with the other specs, eventually the reports a failure at 14:18 after the container/infrastructure times out. So, it looks like something_ happens around 14:08 that puts the cluster in a bad state and these volume-related namespaces time out trying to delete. This pushes the run time for the entire suite (including the setup bits before Ginkgo runs) past the 1 hour limit imposed by the container/infrastructure layer. I'm not sure this is a V1 vs V2 issue (perhaps some previously implicit behavior aroudn test ordering has changed - but these parallel specs basically run the same way in V1 as in V2). Have there been any changes to the storage stuff introduced around the same time as the V2 upgrade? Perhaps this is an infrastructure resource/exhaustion issue - 700 specs managed to succeed before these last three failed!
I don't know if the custom cleanup and the aftereach are conflicting and hanging
no, i don't believe so. they are running sequentially for a given spec.
we do some crazy things, :smile: We need to minimize the diff to find out the root cause, and we can't have the presubmit jobs with a 45% success rate, I've reverted the AfterEach change and I'm asking sig-storage to tag the slow tests accordenly to current policies (5 mins) The lead now is in:
I am attempting to do some bisection... Here's a job from 7/5/22 where all of the csi tests are finishing in under 3 minutes:
Here's a later run with quick running times: I think I need to be looking at periodics though, not pull jobs
here's the earliest run I found with slow times:
Looking at CI jobs: Here's the first job where tests started taking 10 mins: The run before that was fine:
nicely done ! how does one figure out which commit has those two runs correspond to?
Good run was based on the following commit: Slow run was on the following commit: Not a lot of commits between the two and none of them look particularly suspicious
Towards the beginning of the test log, it shows the kubernetes version used to create the kind cluster:
any changes merge to test-infra around the range between those runs (and maybe slightly before)?
not really:
kind versions also look the same
BTW here's the prow job history page starting on the last good run: There is a very clear step up in test time after that.
I think next step is to try to locally rerun the tests against the good commit and the two commits after the good run to see if we can rule them out.
Cross posting from find the report attached with the logs for successful tests I've checked Test starts It seems the problem is on the namespace deletion The storage tests add a custom hook to delete the namespace that seems that is the one causing they take longer
The storage tests run all these cleanups inside the test context has namespace deletion changed somehow ? any possible idea on how ginkgov2 could affect this workflow?
I run this test repeatedly on my desktop, the test passed every time with around 50 seconds to finish the test. And I did some random and destructive testing - terminate the test execution in the middle of process, and hit the situation that namespace deletion is hanging, I check the log and found that is because there is a pod in that namespace, and both the namespace and pod are hanging at the phase of . And the termination never finish unless I force to remove them. Here is the log I can see from kubelet, show that has already been created. I suspect the hang we saw should hit the situation similarly, something in that namespace, pod? prevent us to delete the namespaces successfully. Current code doesn't dump the resource in that namespace since the namespace deletion is done in the , here is the code for that. So, is that possible we could hit the similar situation when all of the testcases run together? we could be able to dump the everything in that namespace when the termination is hanging. will dump the logs from each components, such as controller manger which might give us some hints on this issue, but I don't know where to get those logs.
The reason why the CSI tests manage namespace deletion is because of deletion ordering issues. Each test installs its own CSI driver, however that CSI driver has to be deleted last otherwise Pods will fail to teardown and PVCs will fail to delete due to an uninstalled CSI driver.
make sense, but the log says the is not found. but I do find that csi driver is presented in that cluster, what am I missing here?
the and stuff in these tests is complex and i do wonder why the code isn't calling and instead. nonetheless i would expect this code as written to work: the will always run before the returns to its caller and Ginkgo v1 and v2 both basically guarantee that for a given process no two s will run in parallel - so all those closure variables shouldn't see any pollution from one test to the other within a given process. but. there's a lot of code here and while i don't see any obvious places where test pollution is creeping in due to these shared variables i can't say i'm 100% confident. also - while i don't yet see how ginkgo v2 vs v1 could be causing this we definitely haven't ruled anything out yet! one question: where are these namespace integers being generated (eg ) are we confident that the other parallel processes will never also pick 2296? or is it possible that multiple processes are picking the same integer and so cross-process pollution is occurring? it could be a good use of time run the suite while focusing just these CSI specs and running them in parallel with verbosity turned on so we can see what's going on. my sense is the current hypothesis is that a passing test is inadvertently deleting a resources that another test running in parallel is using thereby causing it to run slower. since we don't run with in verbose mode we won't see the relevant logs from the passing test. that would be my suggestion for a next step. run just these CSI specs in parallel with and see if we learn more from the logs.
I think I found the root cause. I did some debugging on my own PR, and debugged the GCE run because the kind run does not capture kube-controller-manager logs: test log: kube-controller- The commit that I thought wasn't suspicious is actually suspicious:
cc please help take a look. I think the new tests are disruptive and causing namespace deletion in other tests to fail.
Great to hear that! Can you pls share us how to achieve that if we want to check the log from those components? where is the log file I can get?
For example from my test run above, this is where the control plane logs are: It would be nice to collect this from the kind jobs too.
~ (sorry) :) It's historical reason, my pr here: already commented on that, further actions are needed, this is also the action left in We just need time to get familiar with csi e2e test code and then start the migration. It's generated here: , and used here: In theory, I think this is possible, but the how the probability could be. Good suggestion, I think we can add this parameter along with the to the test code, and make it masked with comment. Anyway, thanks for the root causing!
the info is there, but the failing jobs doesn't have it because the job times out cleaning the kind cluster and the related logs If you take a working job the controller-manager logs are in and this confirms theory and demonstrates once more that correlation doesn't imply causality my fault :(, sorry all for the noise
another observation that, once this test started to run, it caused the job to flake !
great job getting to the bottom of this !
no failures since the offending test was reverted ! confirming msau theory
that is itself a bug, and shouldn't be happening. we should get and handle a TERM from prow well before we get killed and the TERM handler should be capturing logs.
log collection in timed out kind e2e runs should be fixed now , , sorry about that

Discussion:
1.17 will work on the current Aggregator/API-Server. release: v1.17, v1.21 file: test/e2e/apimachinery/aggregator.go <del> - testname: Aggregator, manage lifecycle of an APIService codename: '[sig-api-machinery] Aggregator should manage the lifecycle of an APIService [Conformance]' description: An APIService is created which MUST succeed. The APIService status when replaced MUST succeed. Given the updating of the APIService status, the fields MUST equal the new values. The APIService status when patched MUST succeed. Given the patching of the APIService status, the fields MUST equal the new values. The APIService when replaced MUST succeed. Given the updating of the APIService, the fields MUST equal the new values. It MUST succeed at deleting a collection of APIServices via a label selector. release: v1.25 file: test/e2e/apimachinery/aggregator.go </del> - testname: Custom Resource Definition Conversion Webhook, convert mixed version list codename: '[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin] should be able to convert a non homogeneous list of CRs [Conformance]'

TestSampleAPIServer(f, aggrclient, imageutils.GetE2EImage(imageutils.APIServer)) }) <del> /* Release: v1.25 Testname: Aggregator, manage lifecycle of an APIService Description: An APIService is created which MUST succeed. The APIService status when replaced MUST succeed. Given the updating of the APIService status, the fields MUST equal the new values. The APIService status when patched MUST succeed. Given the patching of the APIService status, the fields MUST equal the new values. The APIService when replaced MUST succeed. Given the updating of the APIService, the fields MUST equal the new values. It MUST succeed at deleting a collection of APIServices via a label selector. */ framework.ConformanceIt("should manage the lifecycle of an APIService", func() { </del> <ins> ginkgo.It("should manage the lifecycle of an APIService [Serial][Disruptive]", func() { </ins> ns := f.Namespace.Name framework.Logf("ns: %v", ns)