Spring Scheduler in multi node environment
By: Eresh Gorantla
Tags: Spring Boot, Java8, Multi Node, Spring Schedulling Task, Scheduled Tasks

This articletalks about handling the scheduled tasks in Spring boot applications when deployed in multi-node environments. Talks about solutions how to handle such situations.

Problem Statement:

When Spring scheduler is scheduled to run every one hour if it is deployed in multi nodes, the scheduler triggers on both the nodes. This is expected as each node is a separate service altogether.

In my earlier assignments, we use to have a separate microservice that has only scheduled tasks with no other functionalities and it is deployed as a single node on a global region. But there might be many scenarios where we have to add a scheduled task in any of the existing microservice which is deployed on multiple nodes.

Solutions to resolve this

You can configure your application to have multiple profiles. For example, use another profile ‘cron’ . And start your application on only one server with this profile. So for example, in a production environment, you have three servers (S1, S2, S3), then you could run on S1 with profile prod and cron( -Dspring.profiles.active=prod,cron ). And on S2 and S3 just use prod profile( -Dspring.profiles.active=prod ). And in code, you can use @Profile("cron") scheduler classes. This way it will be executed only when the cron profile is active. Use a distributed lock. If you have Zookeeper in your environment, you can use this to achieve a distributed locking system. You can use some database(mysql) and create a sample code to get a lock on one of the tables and add an entry. And whichever instance gets the lock, will make an entry in this database and will execute the cron job. You need to put a check in your code if getLock() is successful. only then proceed with execution. Mysql has utilities like LOCK TABLES , which you could use to get away with concurrent read/writes.

For option 3, which works well for a spring-based solution, there is a specific library that takes care of this exclusive locking on schedulers.

There is a dependency that provides this locking mechanism. Please find it here.

Implementation in Spring Boot

If you visit the official page, you will find a lot of different variations of configuration settings for RDBMS and No SQL databases. I have chosen Postgres as DB.

compile('net.javacrumbs.shedlock:shedlock-spring:4.9.3')

compile('net.javacrumbs.shedlock:shedlock-provider-jdbc-template:4.7.1')

Create a table by the name, shedlock. These are default names used by shedLock, we can always override them with the help of JDBC Template. Please look at the official documentation for more details.

Let Spring Boot application configuration know about shedLock

The defaultLock is set for 30 seconds when no value is provided the default time till the lock is enables was 30 seconds for application at the global level.

Scheduler Task -> For demo purposes, I am triggering the task every 2 minutes.

I have started the same application in three different ports 9800, 9801, and 9802. A logger is used to determine which instance has picked up the task.

For every 2 mins the shedlock table gets updated with the times.

If you observe, it is locked at 21:34 until 21:34:30 as we have used max lock time as 30 seconds.

Log statements

2020-12-20 21:34:00.110 INFO 4139 --- [ scheduling-1] c.s.spring.SampleSpringSchedulerTask : Scheduler is Triggered in Spring Scheduler on 9800

2020-12-20 21:36:00.043 INFO 4142 --- [ scheduling-1] c.s.spring.SampleSpringSchedulerTask : Scheduler is Triggered in Spring Scheduler on 9802

2020-12-20 21:38:00.002 INFO 4142 --- [ scheduling-1] c.s.spring.SampleSpringSchedulerTask : Scheduler is Triggered in Spring Scheduler on 9802

The first time 9800 picked up the task, on the other 2 occasions 9802 picked up the task. Now I am stopping the 9802 instance and let's see which node picks up the task now.

Service that is running on 9801 is executed. It is like we can not predict which node instance triggers the task, but it ensures that only one instance executes the task.

Other Optional parameters in ShedLock:

The name in SchedulerLock is enough to make it a unique name. But other options include

First, we’ve added lockAtLeastForString so that we can put some distance between method invocations. Using “PT5M” means that this method will hold the lock for 5 minutes, at a minimum. In other words, that means that this method can be run by ShedLock no more often than every five minutes.

Next, we added lockAtMostForString to specify how long the lock should be kept in case the executing node dies. Using “PT14M” means that it will be locked for no longer than 14 minutes.

In normal situations, ShedLock releases the lock directly after the task finishes. Now, we didn’t have to do that because there is a default provided in @EnableSchedulerLock, but we’ve chosen to override that here.