Issue 1:
java version "1.8.0_92"

ES 2.3.4

So, I have 4 nodes, each with 32GB dedicated to the heap space. After about 2 days the heap makes it's way up to over 90% and I start to get rolling outage. Checking the stats I notice that ALL of the heap stays in the old generation pool and never seems to clear out. I've tried switching to G1GC which seemed to help a little bit, I've tried messing with the NewGen Ratio and all sorts of other things, but in the end the old generation never clears and the cluster is unstable until I start restarting nodes.

Current Run Options:

/platform/jdk/bin/java -Xms32g -Xmx32g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/var/log/elasticsearch/gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-UseParNewGC -XX:-UseConcMarkSweepGC -XX:+UseG1GC -XX:+PrintGCDetails -XX:NewRatio=4 -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-2.3.4.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch start -Des.pidfile=/var/run/elasticsearch/elasticsearch.pid -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.conf=/etc/elasticsearch

elasticsearch.yaml

Cluster Settings

cluster.name: elasticsearch

node.rack: "East"

node.name: "es-1"

node.max_local_storage_nodes: 1

Path Settings

path.conf: /etc/elasticsearch

path.data: /storage/elasticsearch

Lock memory for ES

bootstrap.mlockall: true

Networking

network.host: [ local, bond0 ]

discovery.zen.minimum_master_nodes: 3

discovery.zen.ping.unicast.hosts: ["es-1", "es-2", "es-3", "es-4"]

CLUSTER

cluster.routing.allocation.disk.watermark.high: "95%"

cluster.routing.allocation.node_concurrent_recoveries: 30

cluster.routing.allocation.cluster_concurrent_rebalance: 6

INDICES

indices.store.throttle.max_bytes_per_sec: 400mb

Recovery

indices.recovery.max_bytes_per_sec: 1000mb

indices.recovery.concurrent_streams: 10

gateway.recover_after_data_nodes: 2

gateway.recover_after_time: 1m

gateway.expected_data_nodes: 2

Field Data tweaks

indices.fielddata.cache.size: 20% #Evictions a large amount of garbage in memory, which must be garbage collected later on

Circuit Breaker

indices.breaker.fielddata.limit: 25%

indices.breaker.request.limit: 25%

indices.breaker.total.limit: 50%

SLOW LOG

index.search.slowlog.threshold.query.warn: 10s

index.search.slowlog.threshold.fetch.warn: 1s

index.indexing.slowlog.threshold.index.warn: 10s

GC LOGGING

monitor.jvm.gc.young.warn: 1000ms

monitor.jvm.gc.young.info: 700ms

monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s

monitor.jvm.gc.old.info: 5s

monitor.jvm.gc.old.debug: 2s

Issue 2:
We run all queries with timeout=160s, but I understand this only really bounds the collection phase of the search?

We had someone run a query using aggregations today over billions of records, which brought the cluster down.

The cluster never OOM'd, but it did run into constant GC as the heap got full.

Another query was run again over potentially huge amounts of data. The cluster had indexing disabled as it was recovering from the previous event, and since the query was run it's been at 100% CPU for about 40 minutes now. The query, afaict, is still running. But we have no way of knowing what it is, and no way to kill it other than restarting the entire cluster.

I'd like to request a feature that lists all queries that are currently executing on every node, as well a way to kill them while they're in progress.