elasticsearch/elasticsearch — Issue #1682: GC reporting / usage mismatch
Author: Alex-Ikanow
Created: 2012-02-08 15:07:39+00:00
State: open

In https://groups.google.com/group/elasticsearch/browse_thread/thread/31d87c84dd387367#, I reported unusual CPU activity (continuous 40-60% for at least several days even under idle conditions) after creating 12GB of field cache (across 3 nodes) via a facet on a multi-valued field.

Last night I ran jstack while 2 of the nodes were in this state. All of the user threads were WAITING, THREAD_WAITING, or RUNNABLE (in some poll).

Cross-referencing the process ids from "top -H" with the nids reported in jstack, the following threads were actively using the CPU:

~19 minutes in ""Concurrent Mark-Sweep GC Thread"
~2 minutes in each of 4x "Gang worker#0 (Parallel GC Threads)" (#1,#2,#3)
~3 minutes in "VM Thread" prio=10 tid=0x00002aad1c6d0000 nid=0x7041 runnable

Empirically, ongoing CPU time seemed to be split between either Mark-Sweep and 1 of the GC threads, or Mark-Sweep and the VM thread.

However checking both bigdesk and head for the node in question, the following GC times were reported:
ParNew	12	5 seconds and 61 milliseconds
ConcurrentMarkSweep	4	5 seconds and 884 milliseconds

Presumably there's nothing that can be done about the CPU activity itself, that's just what the VM needs to do when I allocate a large amount of (multi valued) field cache. That's obviously fine if so.

Presuambly you also just read some standard stats to get the GC usage out - but I thought the fact the "actual values" weren't getting reported was issue worthy.