Bug: Test Failure: org.apache.cassandra.db.commitlog.GroupCommitLogTest
Status: Resolved
Priority: Normal
Resolution: Duplicate
Created: 22/Mar/22 10:23
Resolved: 25/May/22 08:16

Intermitent failures in {{{}org.apache.cassandra.db.commitlog.GroupCommitLogTest{}}}:
 * [testTruncateWithoutSnapshotNonDurable|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
 * [testRecoveryWithEmptyLog|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
 * [testOutOfOrderLogDiscardWithCompaction|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testTruncateWithoutSnapshotNonDurable_4__cdc_2/]
 * [testExceedRecordLimitWithMultiplePartitions|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testExceedRecordLimitWithMultiplePartitions_5__cdc_2/]
 * [testRecoveryWithShortMutationSize|https://ci-cassandra.apache.org/job/Cassandra-trunk/1024/testReport/org.apache.cassandra.db.commitlog/GroupCommitLogTest/testRecoveryWithShortMutationSize_4__cdc_2/]

{code:java}
Failed 2 times in the last 11 runs. Flakiness: 30%, Stability: 81%
Stacktrace
java.io.UncheckedIOException
	at org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:768)
	at org.apache.cassandra.io.util.PathUtils.propagateUnchecked(PathUtils.java:753)
	at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:255)
	at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:297)
	at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:304)
	at org.apache.cassandra.io.util.File.delete(File.java:158)
	at org.apache.cassandra.io.util.File.delete(File.java:167)
	at org.apache.cassandra.db.commitlog.CommitLogSegmentManagerCDC.discard(CommitLogSegmentManagerCDC.java:75)
	at org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.closeAndDeleteSegmentUnsafe(AbstractCommitLogSegmentManager.java:479)
	at org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.stopUnsafe(AbstractCommitLogSegmentManager.java:452)
	at org.apache.cassandra.db.commitlog.CommitLog.stopUnsafe(CommitLog.java:504)
	at org.apache.cassandra.db.commitlog.CommitLog.resetUnsafe(CommitLog.java:470)
	at org.apache.cassandra.db.commitlog.CommitLogTest.beforeTest(CommitLogTest.java:184)
	at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Caused by: java.nio.file.NoSuchFileException: build/test/cassandra/commitlog/CommitLog-7-1647704321215.log
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249)
	at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105)
	at java.base/java.nio.file.Files.delete(Files.java:1142)
	at org.apache.cassandra.io.util.PathUtils.delete(PathUtils.java:250)
Standard Output
DEBUG [main] 2022-03-19 15:41:04,702 InternalLoggerFactory.java:63 - Using SLF4J as the default logging framework
DEBUG [main] 2022-03-19 15:41:04,712 InternalThreadLocalMap.java:83 - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
DEBUG [main] 2022-03-19 15:41:04,712 InternalThreadLocalMap.java:86 - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
INFO  [main] 2022-03-19 15:41:04,713 CipherFactory.java:68 - initializing CipherFactory
INFO  [main] 2022-03-19 15:41:04,713 JKSKeyPro
...[truncated 1879728 chars]...
tLog-7-1647704466908.log
INFO  [main] 2022-03-19 15:41:26,711 CommitLog.java:175 - No commitlog files found; skipping replay
DEBUG [main] 2022-03-19 15:41:28,150 CommitLogReader.java:266 - Reading build/test/cassandra/commitlog/CommitLog-7-1647704466910.log (CL version 7, messaging version 12, compression org.apache.cassandra.io.compress.ZstdCompressor{})
INFO  [main] 2022-03-19 15:41:28,178 CommitLogReader.java:257 - Finished reading build/test/cassandra/commitlog/CommitLog-7-1647704466910.log
{code}
All these failures belong to the same single run, so it could be an environment issue for that particular run. However, it is possible to hit that failure with the CircleCI test multiplexer in current trunk:
 * [https://app.circleci.com/pipelines/github/adelapena/cassandra/1403/workflows/c02df2b9-732b-478b-825f-f4a0491c0867]
 * [https://app.circleci.com/pipelines/github/adelapena/cassandra/1404/workflows/8acda383-78e6-472b-8a1e-f72684992d6b]

The above runs have 500 and 2000 iterations, and they hit the failure once and twice each, so flakiness seems really low.

These tests look ok in 4.0, as suggested by Butler and [this repeated Circle run|https://app.circleci.com/pipelines/github/adelapena/cassandra/1402/workflows/5c6ee580-d8c8-4f44-a26c-e8a0db5b008b].