Issue 1:
Elasticsearch version:

5.0.0~alpha2

JVM version:

java version "1.8.0_91"

Java(TM) SE Runtime Environment (build 1.8.0_91-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

OS version:

Linux ip-10-10-155-12 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Create a large number of documents

Attempt to delete the documents using something like a match_all query and the delete-by-query plugin

Boom!

Provide logs (if relevant):

You may see any combination of these:

[2016-05-15 00:38:29,675][ERROR][action.deletebyquery ] [client] scroll request [...] failed, scrolling document(s) is stopped
Failed to execute phase [query], all shards failed; shardFailures {RemoteTransportException[[worker][10.10.155.231:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.184:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.83:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3897]]; }{RemoteTransportException[[worker][10.10.155.181:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3894]]; }{RemoteTransportException[[worker][10.10.155.192:9300][indices:data/read/search[phase/query/scroll]]]; nested: SearchContextMissingException[No search context found for id [3897]]; }
at org.elasticsearch.action.search.SearchScrollQueryThenFetchAsyncAction.onQueryPhaseFailure(SearchScrollQueryThenFetchAsyncAction.java:155)
at org.elasticsearch.action.search.SearchScrollQueryThenFetchAsyncAction$1.onFailure(SearchScrollQueryThenFetchAsyncAction.java:142)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:795)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:204)
at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:194)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:141)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[ERROR][action.deletebyquery ] [client] scroll request [...] failed, scrolling document(s) is stopped

or:

2016-05-15 23:38:55,644WARNrest.suppressed /_bulk Params: {}
CircuitBreakingException[parent Data too large, data for <http_request> would be larger than limit of 2994274304/2.7gb]
at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:211)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128)
...

Yup, 2.7 gigs!

Since I had quite a few of these queries going in tandem, you can guess what happened next. Several of the worker nodes pegged at 200% CPU and started to use up all swap and ram (44gigs for the elasticsearch process in most cases). Searches would take 15 minutes.

Looking at the task list I see a big backlog of scroll requests and delete-by-query tasks, none of them cancellable. Not sure what I can do here except kill the index and start again.

Looking quickly at https://github.com/elastic/elasticsearch/blob/master/plugins/delete-by-query/src/main/java/org/elasticsearch/action/deletebyquery/TransportDeleteByQueryAction.java I'm guessing that the plugin just builds a bulk delete request without considering how large it might get.

I think some first steps might be:

Make this plugin cancellable, not sure what this involves but there seems plenty of opportunity in between pages of the scroll request to check a cancellation flag.

At least error out before attempting to build queries of such a large size, the search loop could probably check the BulkRequest#estimatedSizeInBytes against the cluster's max request size limits.

Chunk bulk requests into smaller components and deliver them piecemeal

Issue 2:
Elasticsearch version: alpha2

JVM version: build 1.8.0_74-b02

OS version: OS X El Capitan 10.11.3

Description of the problem including expected versus actual behavior:

Steps to reproduce:

Install and run elasticsearch-alpha2, topbeat-alpha2, and kibana-alpha2 (Topbeat is only monitoring the node process on a 20 second interval.)

I am using Kibana to monitor a node process. Here is the query Kibana is using to generate the visualization:

{
"size":0,
"aggs":{
"2":{
"date_histogram":{
"field":"@timestamp",
"interval":"30s",
"time_zone":"America/Los_Angeles",
"min_doc_count":1
},
"aggs":{
"1":{
"max":{
"field":"proc.mem.rss"
}
}
}
}
},
"highlight":{
"pre_tags":[
"@kibana-highlighted-field@"
],
"post_tags":[
"@/kibana-highlighted-field@"
],
"fields":{
"*":{

}
},
"require_field_match":false,
"fragment_size":2147483647
},
"query":{
"bool":{
"must":[
{
"query_string":{
"query":"*",
"analyze_wildcard":true
}
},
{
"match":{
"proc.cmdline":{
"query":"/Users/tyler/.nvm/versions/node/v4.4.3/bin/node ./bin/../src/cli",
"type":"phrase"
}
}
},
{
"range":{
"@timestamp":{
"gte":1462389453519,
"lte":1462390353519,
"format":"epoch_millis"
}
}
}
],
"must_not":[

]
}
}
}

Run for about 30 minutes.

I have 1027 documents, and the total size is 1.6MB.

Provide logs (if relevant):

CircuitBreakingException[[parent] Data too large, data for [<http_request>] would be larger than limit of [726571417/692.9mb]]
at org.elasticsearch.indices.breaker.HierarchyCircuitBreakerService.checkParentLimit(HierarchyCircuitBreakerService.java:211)
at org.elasticsearch.common.breaker.ChildMemoryCircuitBreaker.addEstimateBytesAndMaybeBreak(ChildMemoryCircuitBreaker.java:128)
at org.elasticsearch.http.HttpServer.dispatchRequest(HttpServer.java:109)
at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:489)
at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:65)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.messageReceived(HttpPipeliningHandler.java:85)
at org.jboss.netty.channel.SimpleChannelHandler.handleUpstream(SimpleChannelHandler.java:88)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.handler.codec.http.HttpContentDecoder.messageReceived(HttpContentDecoder.java:108)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:83)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Eventually, all requests to ES will fail with this exception including _stats.