Garbage Collection & Scala Case Classes
Tags: Programming, JVM, Scala, Software Engineering, Technology

Garbage Collection & Scala Case Classes

Photo by Ocean Cleanup Group on Unsplash

On November 22 2019, at 15:23 UTC, we observed severe service degradation. The release of a core service started the incident. The bad release was rolled back in just a few minutes, however the root cause remained unexplained.

Overview

During the incident, the usage of Scala case classes caused our garbage collection time to more than double, leading to increased CPU utilization.

Modifying these two lines alone replicates the incident

Consequentially, this destabilized the application and caused an outage until rollback. This blog post describes how case classes were utilized, the environment in which they were used, the methods of investigation, followed by the crucial findings and overall learnings.

Use Case & Context

Our application loads data into very large objects/classes. The classes are enriched, copied or modified at runtime, depending on business use case. The cardinality of the data is very high. The classes in question use more than 45 fields each. Large objects are combined to create even larger objects, thousands of which are processed every second.

Under normal operations, these are non-case classes (simple classes) that have many final fields. Aiming to use the automatically generated copy method of case classes, these had been migrated from being simple classes to case classes, as part of the change that lead to the incident.

This migration resulted in elevated garbage collection cycle times, elevated CPU usage but no change in the triggering of garbage collection.

Trials & Findings

As part of the investigation, we aimed to detect the precise root cause of this elevation. However, since the code change was minimal, the investigation lead us deeper into the JVM setup. Although not all of the attempts proved fruitful for this case, they may be a good way of troubleshooting issues for others.

Profiling via YourKit

YourKit is a JVM profiler that helped us deep dive into the internal workloads of our JVMs.

We attempted to profile the application to discover any throttling at the garbage collection level or other memory allocation problems. For this use case, no fruitful discovery was made in regard to the case classes. On the other hand, as part of this effort, it helped us identify other unrelated bottlenecks in our applications, and as a consequence, it lead to separate performance improvements and cost savings.

Simple classes vs. case classes

We attempted to remove the usage of case classes in the hope of seeing improvements in garbage collection times across the entire application.

It appears that most of our case classes had a lot less parameters. The cause of this incident was due to a case class with 45+ fields. Most of the other case classes in our application had less than 10 fields.

This suggested that the GC time is directly dependent on the size of the case classes and not just their general usage.

Comparing different garbage collectors

Prior to the incident, we migrated to ZGC. The ZGC migration lowered our CPU usage and enabled us to scale further in the same stack, resulting in significant cost savings.

One of the trials was to compare the usage of case classes in our classic JVM GC setup versus our new ZGC setup. In both setups, case classes resulted in similarly elevated GC times (~2 times higher than usual) and similar CPU elevations.

The migration to ZGC did not worsen this use case and it overall proved beneficial to us.

Investigating byte code

We compared byte code for the same class structure and fields. As visible below, we noticed code generation of about twice the size for case classes compared to simple classes. This was mainly due to automatically generated methods and references.

javap /path/to/YourScalaClassOrCaseClass.scala

Generated byte code of a case class (left) versus a simple class (right) with the same amount of parameters

Investigating syntax trees

The syntax tree of a case class was 4 times larger than the syntax tree of a simple class. To generate a syntax tree, use the following command:

scalac -print path/to/YourScalaClassOrCaseClass.scala

Again, the comparison was executed against the same class structure and fields.

Code Cache

The code cache metric reflects byte code and syntax tree size increase when compared to the average in production. In the graph below, box ip-10–2–3–104 contains code with case classes, whereas all others use code with simple classes.

CodeCache Utilization increased considerably when using case classes

Clearly our case classes are putting enormous pressure on garbage collection.

Conclusion

All in all, the usage of case classes provides great functionality for a lot of use cases in Scala. However, it’s good to be wary of the size of these case classes, and the impact it may have on your JVM Heap and CodeCache. If you are working with large case classes, consider investigating your CodeCache usage. CodeCache tuning may also be an option. Alternatively, you could use simple classes instead. It may relieve your GC and consequentially your CPU usage, leading to potential significant cost savings.

Learn more about technology at Expedia Group