hazelcast/hazelcast — pull request #1139 by mkjellman (2013-11-12 07:41:17+00:00)
Title: check that backup is actually enabled on a MapConfig

While investigating a memory leak we have found a fairly serious bug where shouldBackup() returns true in almost all cases where it should not on distributed map operations.

After a heap dump, we found the largest issue was one instance of "com.hazelcast.spi.ipml.OperationServiceImpl" loaded by "sun.misc.Launcer$AppClassLoader @ 0x583ab5710" occupies 6,596,128,928 (97.69%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "".

We found 64 Segments under the ConcurrentHashMap for backupCalls of com.hazelcast.spi.impl.OperationServiceImpl. Each segment contained another ConcurrentHashMap continaing ~743,763 entries. Where each HashEntry was a java.util.Long and the value is a java.util.concurrent.Semaphore$NonfairSync

After debugging a bunch we realized that backupCalls contained so much data because backup events were begin created for all distributed map calls even though we had explicitly set backupCount to 0.

if a map config has backups == 0, it disables backup operations, but various map operations
fail to properly check that backups are disabled in the map config and still queue
the event to the ConcurrentHashMap of pending backup elements leading to an OOM / memory leak
situation

I don't believe this is a complete patch as the scope of the code that appeared to require the check seemed to balloon every time I dug deeper, however this does stop backup events from being created when backups are set to 0 on a distributed MapConfig