Bug: OOM while running ShortAccordSimulationTest
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 01/Feb/23 20:43
Resolved: 02/Feb/23 21:18

{{ShortAccordSimulationTest}} seems to consistently run out of heap when run locally. This is true even when we scale the number of threads and duration down...

{noformat}
AccordSimulationRunner.main(new String[] { "run", "-n", "3", "-t", "10", "--cluster-action-limit", "-1", "-c", "2", "-s", "10"});
{noformat}

{noformat}
ERROR [CommandStore[0]:1] node1 CS:[0] OP:0x8fb4f8f1 2023-01-31 17:16:27,143 Operation AsyncOperation{RUNNING}-0x8fb4f8f1 failed
java.lang.OutOfMemoryError: Java heap space
   at java.util.Arrays.copyOf(Arrays.java:3181)
   at accord.local.Command$NotifyWaitingOn.push(Command.java:794)
   at accord.local.Command$NotifyWaitingOn.accept(Command.java:757)
   at accord.local.Command.maybeExecute(Command.java:600)
   at accord.local.Command.onChange(Command.java:546)
   at org.apache.cassandra.service.accord.ListenerProxy$CommandListenerProxy.lambda$onChange$0(ListenerProxy.java:148)
   at org.apache.cassandra.service.accord.ListenerProxy$CommandListenerProxy$$Lambda$4606/1457728285.accept(Unknown Source)
   at org.apache.cassandra.service.accord.async.AsyncOperation$ForConsumer.apply(AsyncOperation.java:261)
   at org.apache.cassandra.service.accord.async.AsyncOperation$ForConsumer.apply(AsyncOperation.java:248)
   at org.apache.cassandra.service.accord.async.AsyncOperation.runInternal(AsyncOperation.java:154)
   at org.apache.cassandra.service.accord.async.AsyncOperation.run(AsyncOperation.java:194)
   at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
   at org.apache.cassandra.concurrent.SyncFutureTask.run(SyncFutureTask.java:68)
   at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus.lambda$new$0(InterceptingExecutor.java:584)
   at org.apache.cassandra.simulator.systems.InterceptingExecutor$AbstractSingleThreadedExecutorPlus$$Lambda$768/827906088.run(Unknown Source)
   at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   at java.lang.Thread.run(Thread.java:748)
{noformat}

JVM args make it seem like we’re passing both a 1 GiB and 8 GiB heap size, although that doesn’t seem to have any bearing on the result. Setting only 8 GiB just takes longer to hit the same problem.

{noformat}
INFO  [isolatedExecutor:2] node1 2023-01-31 17:15:56,940 JVM Arguments: [-Dstorage-config=/Users/maedhroz/Forks/cassandra/test/conf, -Djava.awt.headless=true, -javaagent:/Users/maedhroz/Forks/cassandra/lib/jamm-0.3.2.jar, -ea, -Djava.io.tmpdir=/var/folders/4d/zfjs7m7s6x5_l93k33r5k6680000gn/T/, -Dcassandra.debugrefcount=true, -Xss384k, -XX:SoftRefLRUPolicyMSPerMB=0, -XX:HeapDumpPath=build/test, -Dcassandra.test.driver.connection_timeout_ms=10000, -Dcassandra.test.driver.read_timeout_ms=24000, -Dcassandra.memtable_row_overhead_computation_step=100, -Dcassandra.test.use_prepared=true, -Dcassandra.test.sstableformatdevelopment=true, -Djava.security.egd=file:/dev/urandom, -Dcassandra.testtag=, -Dcassandra.keepBriefBrief=${cassandra.keepBriefBrief}, -Dcassandra.strict.runtime.checks=true, -Dcassandra.reads.thresholds.coordinator.defensive_checks_enabled=true, -DQT_SHRINKS=0, -Dlogback.configurationFile=test/conf/logback-simulator.xml, -Dcassandra.ring_delay_ms=10000, -Dcassandra.tolerate_sstable_size=true, -Dcassandra.skip_sync=true, -Dcassandra.debugrefcount=false, -Dcassandra.test.simulator.determinismcheck=strict, -Dcassandra.test.simulator.print_asm=none, -javaagent:/Users/maedhroz/Forks/cassandra/build/test/lib/jars/simulator-asm.jar, -Xbootclasspath/a:/Users/maedhroz/Forks/cassandra/build/test/lib/jars/simulator-bootstrap.jar, -XX:ActiveProcessorCount=4, -XX:-TieredCompilation, -XX:-BackgroundCompilation, -XX:CICompilerCount=1, -XX:Tier4CompileThreshold=1000, -XX:ReservedCodeCacheSize=256M, -Xmx8G, -Xmx1024m]
{noformat}