GeoWebCache/geowebcache — pull request #131 by natalvc (2012-08-23 12:29:02+00:00)
Title: Fix deadlock in statistics home page

This is the fix for following issue:

We have run into an issue where after a couple of weeks of having geowebcache running in a production environment, it will hang, refusing to serve anymore tiles. The problem is caused by a deadlock condition in the home page of geowebcache (containing the runtime statistics -- this is usually the page to which the root is redirected at /geowebcache/home). We have a monitoring service which is consistently polling the page at /geowebcache/home. If a call is made at a certain point in time when a tile request is also made, a race condition is caused locking up the service (we can reproduce it consistently using a JMeter test plan with thousands of calls to both tiles and home page).

The problem is located in \geowebcache\core\src\main\java\org\geowebcache\stats\RuntimeStats.java. The code uses 2 locking variables when using the synchronized keyword, both the class itself (when using synchronized on the 'this' object and on a method name) and on the 'bytes' variable.

The log for the deadlock:

Found one Java-level deadlock:
=============================
"http-8081-75": waiting to lock monitor 0x0000000007083e80 (object 0x00000000c21593f0, a org.geowebcache.stats.RuntimeStats),
which is held by "http-8081-37"
"http-8081-37": waiting to lock monitor 0x0000000006382708 (object 0x00000000c2159580, a [I),
which is held by "Thread-20"
"Thread-20": waiting to lock monitor 0x0000000007083e80 (object 0x00000000c21593f0, a org.geowebcache.stats.RuntimeStats),
which is held by "http-8081-37"

Java stack information for the threads listed above:
===================================================
"http-8081-75":
at org.geowebcache.stats.RuntimeStats.log(RuntimeStats.java:124)
- waiting to lock <0x00000000c21593f0> (a org.geowebcache.stats.RuntimeStats)
at org.geowebcache.GeoWebCacheDispatcher.writeFixedResponse(GeoWebCacheDispatcher.java:536)
at org.geowebcache.GeoWebCacheDispatcher.handleServiceRequest(GeoWebCacheDispatcher.java:359)
"http-8081-37":
at org.geowebcache.stats.RuntimeStats.getHTMLStats(RuntimeStats.java:157)
- waiting to lock <0x00000000c2159580> (a [I)
- locked <0x00000000c21593f0> (a org.geowebcache.stats.RuntimeStats) at org.geowebcache.GeoWebCacheDispatcher.handleFrontPage(GeoWebCacheDisp
atcher.java:447)
"Thread-20":
at org.geowebcache.stats.RuntimeStats.popIntervalData(RuntimeStats.java:139)
- waiting to lock <0x00000000c21593f0> (a org.geowebcache.stats.RuntimeStats)
at org.geowebcache.stats.RuntimeStats$RuntimeStatsThread.updateLists(RuntimeStats.java:350)