Bug: Test Failures: org.apache.cassandra.metrics.KeyspaceMetricsTest.testMetricsCleanupOnDrop
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 24/May/22 18:25
Resolved: 12/Aug/22 12:03

The test {{org.apache.cassandra.metrics.KeyspaceMetricsTest.testMetricsCleanupOnDrop}} seems to be flaky on CircleCI, although I haven't seen it failing on Jenkins.

It was first detected during CASSANDRA-17509, on [this run|https://app.circleci.com/pipelines/github/josh-mckenzie/cassandra/211/workflows/2cbb5465-a970-440b-a502-06e380ce6851/jobs/1977/tests]:
{code}
junit.framework.AssertionFailedError: org.apache.cassandra.metrics.keyspace.AdditionalWrites.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.AllMemtablesLiveDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.AllMemtablesOffHeapDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.AllMemtablesOnHeapDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.AntiCompactionTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.BloomFilterDiskSpaceUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.BloomFilterOffHeapMemoryUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.BytesValidated.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasCommitLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasCommitTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasPrepareLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasPrepareTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasProposeLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CasProposeTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ClientTombstoneAborts.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ClientTombstoneWarnings.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ColUpdateTimeDeltaHistogram.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CompressionMetadataOffHeapMemoryUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CoordinatorReadSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CoordinatorReadSizeAborts.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.CoordinatorReadSizeWarnings.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.IdealCLWriteLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.IdealCLWriteTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.IndexSummaryOffHeapMemoryUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LiveDiskSpaceUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LiveScannedHistogram.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LocalReadSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LocalReadSizeAborts.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.LocalReadSizeWarnings.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableColumnsCount.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableLiveDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableOffHeapDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableOnHeapDataSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.MemtableSwitchCount.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.PartitionsValidated.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.PendingCompactions.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.PendingFlushes.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RangeLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RangeTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ReadLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ReadTotalLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairJobsCompleted.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairJobsStarted.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairPrepareTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairSyncTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairedDataInconsistenciesConfirmed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairedDataInconsistenciesUnconfirmed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairedDataTrackingOverreadRows.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RepairedDataTrackingOverreadTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RowIndexSize.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RowIndexSizeAborts.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.RowIndexSizeWarnings.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.SSTablesPerReadHistogram.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.SpeculativeFailedRetries.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.SpeculativeInsufficientReplicas.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.SpeculativeRetries.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.TombstoneScannedHistogram.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.TotalDiskSpaceUsed.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ValidationTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ViewLockAcquireTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.ViewReadTime.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.WriteFailedIdealCL.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.WriteLatency.keyspacemetricstest_metrics_cleanup,org.apache.cassandra.metrics.keyspace.WriteTotalLatency.keyspacemetricstest_metrics_cleanup expected:<0> but was:<65>
	at org.apache.cassandra.metrics.KeyspaceMetricsTest.testMetricsCleanupOnDrop(KeyspaceMetricsTest.java:74)
{code}

The failure can be reproduced on the multiplexer with 1000 iterations:

{code}
.circleci/generate.sh -m \
  -e REPEATED_UTEST_TARGET=testsome \
  -e REPEATED_UTEST_COUNT=1000 \
  -e REPEATED_UTEST_CLASS=org.apache.cassandra.metrics.KeyspaceMetricsTest
{code}

||Branch||CI||
|4.0  |[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1600/workflows/8fb4ec32-2c95-4632-b49d-ece200a137fc] [j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1600/workflows/6c67767a-faaf-42cc-b707-0ca917e367cd]|
|4.1  |[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1601/workflows/2fd10a07-26d6-4de5-b1e5-25c14eeda1ad] [j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1601/workflows/9f588839-3c06-4270-adf4-9f8a1f177e00]|
|trunk|[j8|https://app.circleci.com/pipelines/github/adelapena/cassandra/1599/workflows/e8f25806-5f84-478a-b238-2cf375da5fbe] [j11|https://app.circleci.com/pipelines/github/adelapena/cassandra/1599/workflows/7ecb9fc6-b737-4b7a-b7c0-41d6cc9eb6fd]|

The failure can be reproduced on 4.1 and trunk, but apparently not in 4.0. Flakiness seems below 10%.