What a Recurring OutOfMemory Error Taught Me
By: Antonello Zanini
Tags: Technology, Life Lessons, Kotlin, Spring Boot, Programming

The Recurring Error

The back-end runs flawlessly for about two weeks. Day by day, data becomes bigger, and users making concurrent requests increase. During the third week, I get the first OutOfMemory error. Logs are pretty clear: the VPS failed while trying to create a new thread, running out of memory. I immediately ask for advice from Dario Pellegrini, a colleague and developer with 5+ years of experience. Together we decide to double the RAM of the VPS and the maximum heap size limit consequently.

Problem fixed, right?

Nope!

3 days later, the same OutOfMemory exception strikes again! This time I want to tackle the problem for real and look for a real solution.

These my attempts to fix it…

1. Trying a Practical Solution

Running some tests I find out that a query returning 20k rows can easily become the bottleneck of an application. This is especially true if the @Entity mapping class consists of more than 130 attributes (just like mine). Instead of retrieving 10/20k rows each time, mapping them into Hibernate objects, iterate through them to retrieve a single computed value, I replace it all with a very advanced query. By defining a custom mapping class and writing a highly optimized query, I am able to achieve the same result with a much more efficient single-row query. With this (not so) simple trick, I increase the performance by 70/80% and reduce significantly the heap size.

Not to brag, but this time I’m pretty sure I’ve solved the problem…

Not yet…

2. Looking for Esoteric Reasons

Since the story is still going on, I’m not proud to say that it didn’t work out. 6 days later my back-end fails for the same OutOfMemory error.

I decide to take it seriously this time. I look for someone with a similar issue online and come across some interesting, but somewhat esoteric, topics.

Firstly, a couple of APIs are really complex and require hundreds of queries. Then, the query plan cache provided by Hibernate may be the culprit. By default, the maximum number of entries in the plan cache is 2048. An HQLQueryPlan object occupies approximately 3MB. ~3 * 2048 = ~6GB, and I the heap size is limited to 4GB…

Finally! That must be the cause!

The solution is simple: decreasing the query plan cache size by setting the following properties:

spring.jpa.properties.hibernate.query.plan_cache_max_size : controls the maximum number of entries in the plan cache (defaults to 2048 )

: controls the maximum number of entries in the plan cache (defaults to ) spring.jpa.properties.hibernate.query.plan_parameter_metadata_max_size : manages the number of ParameterMetadata instances in the cache (defaults to 128)

I set them to 1024 and 64 respectively.

Secondly, I find out that IN queries with a variable number of parameters are a bit tricky for the Hibernate query plan cache (further reading here and here).

The solution is easy again. You can improve the statement caching efficiency with IN clause parameter padding by setting the following property to true:

spring.jpa.properties.hibernate.query.in_clause_parameter_padding

Thirdly, my application is multi-thread. Since the error seems to occur while it tries to create a new thread, why not reduce threads stack size from 1024KB to 512KB by adding -Xss512k to my Procfile?

Each of these properties should decrease the amount of heap used and therefore fix the OutOfMemory problem, once and for all. Right? Nope!

3. Surrendering to Reality

Last but not least, I decide to do what I should have done immediately: reviewing my code! The JVM Garbage Collector gathers all the unreachable objects to try to delete them. In the case of too many unreachable objects, the heap will fill up before the GC intervenes. In this case, an OutOfMemory exception is thrown as soon as the application tries to allocate a new object. Reviewing my code I find nothing suspicious.

My application is multi-thread and I used thread pools more than once, especially ForkJoinPool . I have not used it for ages, so the problem could be there. I started looking online for well-known memory leaks and during my researches I come across this interesting GitHub page. I immediately realize what a beginner mistake I made. I forgot to shut down each of the ForkJoinPool objects I used!

Oops!