Java.lang.OutOfMemoryError: Java heap space on Elasticsearch in AWS
By: Sarasa Gunawardhana
Tags: Elasticsearch, Elk Stack, Docker, Logstash, Aws Ec2

Caused by: java.lang.OutOfMemoryError: Java heap space

[2018-05-30T16:45:39,564][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[CJ_O7Il][fetch_shard_store][T#49]], exiting

java.lang.OutOfMemoryError: Java heap space

When you get this error, AWS instance get stuck, CPU usage goes high or Elasticsearch and other service which depends on Elasticsearch service may keep restarting.

In the situation you are running ELK stack on Docker swarm, You may run bunch of services on you Manager node including Elasticsearch container.

Java applications are only allowed to use a limited amount of memory. This limit is specified during application startup. To make things more complex, Java memory is separated into two different regions. These regions are called Heap space and Permgen (for Permanent Generation):

The size of those regions is set during the Java Virtual Machine (JVM) launch and can be customized by specifying JVM parameters -Xmx and -XX:MaxPermSize. If you do not explicitly set the sizes, platform-specific defaults will be used.

The java.lang.OutOfMemoryError: Java heap space error will be triggered when the application attempts to add more data into the heap space area, but there is not enough room for it.

Note that there might be plenty of physical memory available, but the java.lang.OutOfMemoryError: Java heap space error is thrown whenever the JVM reaches the heap size limit.

What is causing it?

There most common reason for the java.lang.OutOfMemoryError: Java heap space error is simple — you try to fit an XXL application into an S-sized Java heap space. That is — the application just requires more Java heap space than available to it to operate normally. Other causes for this OutOfMemoryError message are more complex and are caused by a programming error:

Spikes in usage/data volume . The application was designed to handle a certain amount of users or a certain amount of data. When the number of users or the volume of data suddenly spikes and crosses that expected threshold, the operation which functioned normally before the spike ceases to operate and triggers the java.lang.OutOfMemoryError: Java heap space error.

. The application was designed to handle a certain amount of users or a certain amount of data. When the number of users or the volume of data suddenly spikes and crosses that expected threshold, the operation which functioned normally before the spike ceases to operate and triggers the java.lang.OutOfMemoryError: Java heap space error. Memory leaks. A particular type of programming error will lead your application to constantly consume more memory. Every time the leaking functionality of the application is used it leaves some objects behind into the Java heap space. Over time the leaked objects consume all of the available Java heap space and trigger the already familiar java.lang.OutOfMemoryError: Java heap space error.

Solutions :

There are two solutions that I’m going to discuss.

1. Increase heap size

In this soultion there are two possible solutions,

1.1. Mount the file by changing Heap size from host to container

You can change Heap size of Elasticsearch from jvm.options file where it located inside the Elasticsearch container in this path :

/usr/share/elasticsearch/config/jvm.options

so you need to mount this file from host to container.

step 1: create a file on your server as jvm.options.

sudo nano ~/jvm.options

step 2: Copy and paste following code into the file.

Note* : You can copy this content it from inside container where this file located in /usr/share/elasticsearch/config/jvm.options. To get inside container, do following way: docker exec -it <container_id> bash.

before Copy this content, make sure to increase size of -Xms & -Xmx more than current size.

## You should always set the min and max JVM heap

## size to the same value. For example, to set

## the heap to 4 GB, set:

##

## -Xms4g

## -Xmx4g

##

## See

## for more information

##

################################################################ #### See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html ## for more information################################################################## # Xms represents the initial size of total heap space

# Xmx represents the maximum size of total heap space -Xms2g

-Xmx2g ################################################################

## Expert settings

################################################################

##

## All settings below this section are considered

## expert settings. Don't tamper with them unless

## you understand what you are doing

##

################################################################ ## GC configuration

-XX:+UseConcMarkSweepGC

-XX:CMSInitiatingOccupancyFraction=75

-XX:+UseCMSInitiatingOccupancyOnly ## optimizations # pre-touch memory pages used by the JVM during initialization

-XX:+AlwaysPreTouch ## basic # force the server VM (remove on 32-bit client JVMs)

-server # explicitly set the stack size (reduce to 320k on 32-bit client JVMs)

-Xss1m # set to headless, just in case

-Djava.awt.headless=true # ensure UTF-8 encoding by default (e.g. filenames)

-Dfile.encoding=UTF-8 # use our provided JNA always versus the system one

-Djna.nosys=true # use old-style file permissions on JDK9

-Djdk.io.permissionsUseCanonicalPath=true # flags to configure Netty

-Dio.netty.noUnsafe=true

-Dio.netty.noKeySetOptimization=true

-Dio.netty.recycler.maxCapacityPerThread=0 # log4j 2

-Dlog4j.shutdownHookEnabled=false

-Dlog4j2.disable.jmx=true

-Dlog4j.skipJansi=true ## heap dumps # generate a heap dump when an allocation from the Java heap fails

# heap dumps are created in the working directory of the JVM

-XX:+HeapDumpOnOutOfMemoryError

After adding this file, change docker-compose.yml file and mount this file into container

elasticsearch:

image: docker.elastic.co/elasticsearch/elasticsearch:5.6.15

environment:

ES_JAVA_OPTS: '-Xms1g -Xmx1g'

xpack.security.enabled: 'false'

xpack.monitoring.enabled: 'false'

xpack.graph.enabled: 'false'

xpack.watcher.enabled: 'false'

volumes:

- loggingvol:/usr/share/elasticsearch/data

- ~/jvm.options:/usr/share/elasticsearch/config/jvm.options

OR

1.2. Override Heap Size inside docker-compose.yml file by overriding ENV variables.

elasticsearch:

image: docker.elastic.co/elasticsearch/elasticsearch:5.6.15

environment:

ES_JAVA_OPTS: '-Xms2g -Xmx2g'

xpack.security.enabled: 'false'

xpack.monitoring.enabled: 'false'

xpack.graph.enabled: 'false'

xpack.watcher.enabled: 'false'

volumes:

- loggingvol:/usr/share/elasticsearch/data

Most of the time current node may does not has enough memeory space to allocate on Elasticsearch. To avoid such a situation you may go to the second solution.

2. Transfer your container into another worker node or manager node:

Before you do it, make sure to choose a server which has plenty of memory space. To do that you have to add some lines into your docker-compose.yml file. By adding those line we force to run Elasticsearch container on worker node.

elasticsearch:

image: docker.elastic.co/elasticsearch/elasticsearch:5.6.15

environment:

xpack.security.enabled: 'false'

xpack.monitoring.enabled: 'false'

xpack.graph.enabled: 'false'

xpack.watcher.enabled: 'false'

volumes:

- loggingvol:/usr/share/elasticsearch/data

deploy:

placement:

constraints:

- "node.role==worker"

You can specify as manger or worker. Also , without adding manger or worker, you can add host name of you server.

This is expected to resolve any Elasticsearch stability issues due to java heap.