How to leverage AWS infrastructure to avoid Full GC in production environment.
By: Punit Mnit
Tags: Aws Sns, Aws Cloudwatch, AWS, Aws Ssm, AWS Lambda

How to leverage AWS infrastructure to avoid upstream impact due to Full GC in production environment.

For those supporting complex production environment where multiple layer of middleware technologies coexist, Full GCs in one layer, some times can cause negative impact on production environment. If such issues happen on high volume sales days, it becomes particularly painful.

In this article I want to present how you can solve Full GC issues proactively by leveraging agility and automation provided by public cloud platforms. For this case we will focus on AWS. I would also like to thank Kshitiz Bansal and Rahul Kanna for helping me with test java program and giving their valuable feedback to the article.

In the first part of article i will laydown the approach we want to take and the second part i want to explain how this can be automated with AWS. [This article assume you know what is garbage collection and some familiarity with AWS.]

Understand the solution :

In short we want to do a manual Full GC before JVM gets a chance to do a Full GC on its own, we are making sure when doing manual full GC, JVM is already drained, so there is no impact to upstream systems. [For those wondering why we want to do this, its important to understand Full GC is “stop the world” scenario and we want to stop upstream impact due to this “stop the world”.]

First of all you will need to understand the GC pattern on your JVM. You might have different GC algorithm, traffic patterns on various layers of your infrastructure. All this will govern the Full GC pattern you have. You will need to see typically at what % of old generation utilization does your JVM start doing full GC. There are multiple way of doing this :

a. If you have monitoring tool like Dynatrace/AppDynamics/HPD or any custom tool that monitor and record the heap utilization, you can check Full GC pattern. [Look at old generation and not Eden]

b. You could look at JVM GC log and identify the Full GC pattern. This is usually defined by -Xloggc of your JVM arguments.

c. If you don’t have GC logs or monitoring tool, you could use jmap -heap <PID> command to collect this information periodically and establish the pattern.

You will see that your JVM typically does Full GC at a certain percentage. It could be anywhere from 70% from 99%, but mostly it would be constant behavior. For the rest of article’s sake we would assume that full GC happen at 95%.

2. Once you identify the typical pattern for full GC, we need to drain the JVM [stop sending traffic to it] gracefully before this threshold hits. So in our case we would start draining the JVM at 90% heap utilization. This can be achieved by

a. Removing the JVM port from load balancer. [For this use case we will follow this approach].

b. Having a Layer 7 routing rule and managing the routing at JVM layer without having to go back to LB.

3. Once your JVM is drained, invoke a Full GC from outside, this can be done using any of the following commands

a. jcmd <PID> GC.run b. jmap -histo:live <pid>

4. Once you do manual Full GC, its time to plug back the JVM into LB.

Automate this solution in AWS :

Create an alarm for JVM heap utilization reaching 90%. [Pls note in our case JVM’s typical behavior was to do full GC at 95%, so we are setting the alarm at 90%.]

For this part i create a custom script on EC2 which collect JVM Heap utilization and publishes to AWS cloudwatch :

Pls note you will need to assign appropriate role to EC2 to be able to publish to AWS cloudwatch.

[ec2-user@ip-10–1–4–238 ~]$ cat /home/ec2-user/Java_Heap_Used.ksh

PID=`ps -ef | grep ec2-user | grep java | grep MyServer | grep -v grep | tail -1 | awk -F’ ‘ ‘{print $2}’`

echo $PID

Value=`jmap -heap ${PID} | egrep -A4 “tenured generation” | tail -1 | awk -F’%’ ‘{print $1}’`

echo $Value

aws cloudwatch put-metric-data — metric-name jvm-heap-usage — dimensions Instance=i-022cf8afc40b2e281 — namespace “Custom” — value ${Value}

Setup your alarm to publish to SNS topic, when in alarm state.

2. On the SNS topic that you created in Step 1, set a Lambda function as trigger. I used Python code and invoked SSM document “AWS-RunShellScript” via which i invoked the needed commands.

Need to make sure Lambda has appropriate role to execute SSM.

Also EC2 should have appropriate SSM role.

Below is the working Lambda code. Pls note in my case i have hardcoded Instance ID and JVM PID. For a real use case you can get these details via SNS topic and describe-instance commands.

I copied the code from following GIT repository link and modified as per my needs : https://github.com/srcecde/aws-tutorial-code/commit/478860ecdf9ce702056b41a830a61a10e1f20c30#diff-9c84fedb41974d552df06c6a00213bfabe7bc77d6e2ba05a4e683fab2b75b862

Couple more things to note

a. The ELB/Target Group also have a setting with respect to draining “Deregistration delay”. Make sure this is less than the time you sleep between the deregistration and manual Full GC.

b. There is a sleep in below phtyon code, make sure this is greater than execution time of entire program.

c. Make sure to set the default Lambda timeout to be higher than sleep in above step b.

#-*- coding: utf-8 -*-

import time

import json

import boto3

def lambda_handler(event, context):

# boto3 client

client = boto3.client(‘ec2’)

ssm = boto3.client(‘ssm’)

InstanceId=[‘i-022cf8afc40b2e281’]

# looping through instance ids

for instanceid in InstanceId:

# command to be executed on instance

response = ssm.send_command(

InstanceIds=[instanceid],

DocumentName=”AWS-RunShellScript”,

Parameters={

‘commands’: [

‘aws elbv2 deregister-targets — target-group-arn arn:aws:elasticloadbalancing:ap-southeast-1:923312202780:targetgroup/Test-TG-Jvm-Heap/ba508ff2ebd5e75b — targets Id=i-022cf8afc40b2e281;’

‘sleep 300;’

‘jmap -heap 14780 >> /home/ec2-user/test.txt;’

‘jcmd 14780 GC.run;’

‘jmap -heap 14780 >> /home/ec2-user/test.txt;’

‘aws elbv2 register-targets — target-group-arn arn:aws:elasticloadbalancing:ap-southeast-1:923312202780:targetgroup/Test-TG-Jvm-Heap/ba508ff2ebd5e75b — targets Id=i-022cf8afc40b2e281;’

]

},

) # fetching command id for the output

command_id = response[‘Command’][‘CommandId’] time.sleep(420) # fetching command output

output = ssm.get_command_invocation(

CommandId=command_id,

InstanceId=instanceid

)

print(output)

return {

‘statusCode’: 200,

‘body’: json.dumps(‘Thanks from Srce Cde!’)

}

So to conclude we are able to show how easily by integrating couple of AWS services like CloudWatch/SNS/Lambda/SSM, we are able to automate and resolve a given problem statement.

This is also a good example of how these different service can integrate and work as a whole.

A word of caution, pls be mindful of how many JVMs you take out from rotation for doing manual Full GC. A good rule of thumb could be not take out more than 30–40% of the running JVM under a particular LB. Again all this would depend on your traffic pattern and load etc.

I plan to upload a video of how this integration works in AWS, will put up a link when this is ready.

Happy reading and any feedback is greatly appreciated !!!