Change:
TestRotateLogs from kubelet/logs/containerlogmanagertest Probably since the merge of change No response_ /sig node
/triage accepted /cc
/cc
/priority important-longterm Seems to impact on this arch only?
cc
/sig windows
ack.. Checking
what is the easiest way for me to test this one locally ? using based vm ?
poke and on slack please
unfortunately, I couldn't reproduce this flake locally. Will update here once I am able to.
I am now able to see the flake, when I run the test within a Pod with a resource limit as our CI job does. Steps to reproduce: clone kubernetes test -c -race - TestRotateLogs root stress - TestRotateLogs Please find the attached trace file
This is with or on a native x86_64 as well ?
I could reproduce this even on x8664. Also, the failures do happen even when run on the VM itself and NOT ONLY within the pod as I mentioned above. The flakiness % is higher when run from a pod rather than a VM.
I was able to reproduce this on regular machines as well with while running tests. Let me check and see what needs to be done to fix this.
This seem to take care of the issue. Do you think the poll interval of is too much ? cc
bash stress - TestRotateLogs -test.v 5s: 0 runs so far, 0 failures 10s: 48 runs so far, 0 failures 15s: 96 runs so far, 0 failures 20s: 146 runs so far, 0 failures 25s: 196 runs so far, 0 failures 30s: 244 runs so far, 0 failures 35s: 292 runs so far, 0 failures 40s: 339 runs so far, 0 failures 45s: 387 runs so far, 0 failures
Spoke too early. bash 1m30s: 812 runs so far, 4 failures (0.49%) /var/folders/l3/tf0xj3gj67bfmk13h462668w0000gn/T/go-stress-20240223T144024- === RUN TestRotateLogs Error Trace: Error: "[- test-log-1 - test-log-2.- - test-log-3 - test-log-3.- - test-log-3.- - test-log-4]" should have 5 item(s), but has 6 Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-3.-" actual : "test-log-3" Diff: --- Expected +++ Actual -1 +1 -test-log-3.- +test-log-3 Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-3.-" actual : "test-log-3.-" Diff: --- Expected +++ Actual -1 +1 -test-log-3.- +test-log-3.- Test: TestRotateLogs Error Trace: Error: Not equal: expected: "test-log-4" actual : "test-log-3.-" Diff: --- Expected … 1m35s: 851 runs so far, 5 failures (0.59%)
/assign
/triage accepted
Had to add a few additional logs to identify the issue, but has a fix now. PTAL when possible.

Discussion:
}, } f.SetFakeContainers(testContainers) <del> go c.processQueueItems(ctx, 1) </del> <ins> // Push the items into the queue for before starting the worker to avoid issue with the queue being empty. </ins> require.NoError(t, c.rotateLogs(ctx)) <del> pollTimeoutCtx, cancel := context.WithTimeout(ctx, 10*time.Second) defer cancel() err = wait.PollUntilContextCancel(pollTimeoutCtx, 20*time.Millisecond, false, func(ctx context.Context) (done bool, err error) { return c.queue.Len() == 0, nil }) require.NoError(t, err) </del> <ins> // Start a routine that can monitor the queue and shutdown the queue to trigger the retrun from the processQueueItems // Keeping the monitor duration smaller in order to keep the unwanted delay in the test to a minimal. go func() { pollTimeoutCtx, cancel := context.WithTimeout(ctx, 10*time.Second) defer cancel() err = wait.PollUntilContextCancel(pollTimeoutCtx, 5*time.Millisecond, false, func(ctx context.Context) (done bool, err error) { return c.queue.Len() == 0, nil }) require.NoError(t, err) c.queue.ShutDown() }() // This is a blocking call. But the above routine takes care of ensuring that this is terminated once the queue is shutdown c.processQueueItems(ctx, 1) </ins> timestamp := now.Format(timestampFormat) logs, err := os.ReadDir(dir)

assert.Equal(t, testLogs[4]+compressSuffix, logs[2].Name()) assert.Equal(t, testLogs[2]+"."+timestamp, logs[3].Name()) assert.Equal(t, testLogs[3], logs[4].Name()) <del> c.queue.ShutDown() </del> } func TestClean(t *testing.T) {