Bug: memtable flush stuck for minutes on slow replaceFlushed view update
Status: Patch Available
Priority: Normal
Created: 20/Dec/24 14:54

We observed slow memtable flushing hence caused write latency spikes when the cluster has large number of SSTables (~15K). I looked into the CPU profiling, seems that most of the CPU was busy doing updates on the View from compaction.

!image-2024-12-20-06-14-54-493.png|width=570,height=476!

And taking a closer look - these checkpoint calls are mostly from maybeReopenEarly. I don't fully understand how this early open mechanism works, but according to some recent investigation, the checkpoint calls can become expensive (observed in CASSANDRA-19596, CASSANDRA-20158)

replaceFlushed update also requires an entire SSTableIntervalTree rebuild, which can take significantly longer time when number of SSTable grows

 
{code:java}
static Function<View, View> replaceFlushed(final Memtable memtable, final Iterable<SSTableReader> flushed)
{
    return new Function<View, View>()
    {
        public View apply(View view)
        {
            List<Memtable> flushingMemtables = copyOf(filter(view.flushingMemtables, not(equalTo(memtable))));
            assert flushingMemtables.size() == view.flushingMemtables.size() - 1;

            if (flushed == null || Iterables.isEmpty(flushed))
                return new View(view.liveMemtables, flushingMemtables, view.sstablesMap,
                                view.compactingMap, view.intervalTree);

            Map<SSTableReader, SSTableReader> sstableMap = replace(view.sstablesMap, emptySet(), flushed);
            return new View(view.liveMemtables, flushingMemtables, sstableMap, view.compactingMap,
                            SSTableIntervalTree.build(sstableMap.keySet()));
        }
    };
} {code}
When a node is busy with compaction, the {{replaceFlushed}} update can easily encounter contention. Assuming 1 thread doing memtable flush and 6 concurrent compactors running full speed, these 7 threads will consume similar time (say T ms) to generate the new View. There is only 1/7 chance that the flush can succeed, which means the expectation finish time will be 7 * T ms...

 

What's worse, replaceFlushed is not only competing the View updates with checkpoint calls from compaction. In some of my testing, I noticed that there is a surge in unmarkCompaction calls every 1 hour (from default index_summary_resize_interval = 60m) and this last for 2 minutes or longer. During this 2 minutes windows, replaceFlushed finishTime is significantly longer and we see pending mutationstage tasks starting to pile up (all of them are waiting for memtable to be flushed) and hence writes started to timeout.

!image-2024-12-20-06-39-04-194.png|width=1259,height=292!

These unmarkCompacting calls was from IndexSummaryRedistribution.redistributeSummaries(). If I understand it correctly, what happens here is that
 # IndexSummaryManager marks all SSTables as compacting, which is done in one-go per cfs by iterating all cfs and add all SSTables to compactingMap
 # in adjustSamplingLevels(), it calculates the sampling level for each sstable (probably also did some updates on SSTableReader?)
 # unmark compacting for the SSTables which don't need downsample

Step 3 is done *one by one* hence causing trouble here
{code:java}
if (remainingSpace > 0)
{
    Pair<List<SSTableReader>, List<ResampleEntry>> result = distributeRemainingSpace(toDownsample, remainingSpace);
    toDownsample = result.right;
    newSSTables.addAll(result.left);
    for (SSTableReader sstable : result.left)
        transactions.get(sstable.metadata().id).cancel(sstable);
} {code}
 

Here I have 2 proposals in improving this:
 # Fix this redistributeSummaries() to group and unmark compacting for these SSTables in one go
 # Make replaceFlushed faster - by supporting addSSTables in IntervalTree. For replaceFlushed calls we don't remove anything from the existing IntervalTree. That being said, insert a new interval into the IntervalTree can also be fast (at least better than rebuild the entire tree)
 ** The only concern here is that addSSTables might create imbalanced tree, but we rebuild the tree very, very frequently - I think this should not be a huge concern

Let me know what you think about this. I'm not very sure why this cancel was done one by one, is it possible to do it in batch grouped by cfs (just like how we mark them as compacting)?