Bug: 2.2.17 fails to start up with ExceptionInInitializerError
Status: Resolved
Priority: Normal
Resolution: Fixed
Created: 03/Aug/20 22:06
Resolved: 23/Sep/20 15:08

After the upgrade to 2.2.17, Cassandra fails to start with the following error:
{noformat}
INFO  20:28:57 JVM Arguments: [-Dcom.sun.management.jmxremote.port=7199, -Dcom.sun.management.jmxremote.ssl=false, -Dcom.sun.management.jmxremote.authenticate=false, -ea, -javaagent:/opt/cassandra/lib/jamm-0.3.0.jar, -XX:+CMSClassUnloadingEnabled, -XX:+UseThreadPriorities, -XX:ThreadPriorityPolicy=42, -Xms128m, -Xmx128m, -Xmn32m, -XX:+HeapDumpOnOutOfMemoryError, -Xss256k, -XX:StringTableSize=1000003, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -XX:SurvivorRatio=8, -XX:MaxTenuringThreshold=1, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+UseTLAB, -XX:+PerfDisableSharedMem, -XX:CompileCommandFile=/etc/cassandra/hotspot_compiler, -XX:CMSWaitDuration=10000, -XX:+CMSParallelInitialMarkEnabled, -XX:+CMSEdenChunksRecordAlways, -XX:CMSWaitDuration=10000, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintHeapAtGC, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -XX:+PrintPromotionFailure, -Xloggc:/opt/cassandra/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=10, -XX:GCLogFileSize=10M, -Djava.net.preferIPv4Stack=true, -Dcassandra.jmx.local.port=7199, -XX:+DisableExplicitGC, -Djava.library.path=/opt/cassandra/lib/sigar-bin, -Dcassandra.libjemalloc=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1, -XX:OnOutOfMemoryError=kill -9 %p, -Dlogback.configurationFile=logback.xml, -Dcassandra.logdir=/opt/cassandra/logs, -Dcassandra.storagedir=/opt/cassandra/data, -Dcassandra-foreground=yes]
WARN  20:28:57 Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root.
INFO  20:28:57 jemalloc seems to be preloaded from /usr/lib/x86_64-linux-gnu/libjemalloc.so.1
INFO  20:28:57 JMX is enabled to receive remote connections on port: 7199
WARN  20:28:57 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO  20:28:57 Initializing SIGAR library
INFO  20:28:57 Checked OS settings and found them configured for optimal performance.
WARN  20:28:57 Directory /opt/cassandra/data/commitlog doesn't exist
WARN  20:28:57 Directory /opt/cassandra/data/saved_caches doesn't exist
Exception (java.lang.ExceptionInInitializerError) encountered during startup: null
java.lang.ExceptionInInitializerError
	at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:709)
	at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:351)
	at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:109)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:607)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:717)
Caused by: java.lang.IllegalArgumentException: Bad configuration; unable to start server: At least one DataFileDirectory must be specified
	at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:846)
	at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:66)
	... 6 more
ERROR 20:28:58 Exception encountered during startup
java.lang.ExceptionInInitializerError: null
	at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:709) ~[apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.StartupChecks$9.execute(StartupChecks.java:351) ~[apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:109) ~[apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188) [apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:607) [apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:717) [apache-cassandra-2.2.17.jar:2.2.17]
Caused by: java.lang.IllegalArgumentException: Bad configuration; unable to start server: At least one DataFileDirectory must be specified
	at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:846) ~[apache-cassandra-2.2.17.jar:2.2.17]
	at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:66) ~[apache-cassandra-2.2.17.jar:2.2.17]
	... 6 common frames omitted
	at org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:109)
	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:188)
	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:607)
	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:717)
Caused by: java.lang.IllegalArgumentException: Bad configuration; unable to start server: At least one DataFileDirectory must be specified
	at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:846)
	at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:66)
	... 6 more
{noformat}
I've traced this down to what I believe is the issue in [https://github.com/apache/cassandra/commit/257fb0377343cbfdb58327da17f31d4eaed940f5], specifically [https://github.com/apache/cassandra/commit/257fb0377343cbfdb58327da17f31d4eaed940f5#r40944000] – the addition of an empty value for {{data_file_directories}} needs to be accompanied with a change to {{DatabaseDescriptor.java}} to handle an empty array the same way as the previous nil value was (and seed the value of {{cassandra.storagedir}} into that empty array), as was done in [https://github.com/apache/cassandra/commit/b09e60f72bb2f37235d9e9190c25db36371b3c18#diff-b66584c9ce7b64019b5db5a531deeda1] (which I believe is the origin of this change).