Issue 1:
Elasticsearch version: 2.3.3

JVM version: OpenJDK 1.8.0_91

OS version: Ubuntu 16.04

Description of the problem including expected versus actual behavior:

Documents are not properly indexed when Elasticsearch is restarted just after bulk request is send.

Bulk request is retried because of connection refused error. Finally random documents are duplicated (same content, different id) in the index.

Steps to reproduce:

Start sending 1000 bulk requests with 100 documents each, one after another. First request example:

curl -XPOST 'localhost:9200/_bulk' -d '
{"index":{"_index" : "index1", "_type": "test"}}
{"counter": "1", "package": "1/1000"}
{"index":{"_index" : "index1", "_type": "test"}}
{"counter": "2", "package": "1/1000"}
...
{"index":{"_index" : "index1", "_type": "test"}}
{"counter": "100", "package": "1/1000"}
'

Restart Elasticsearch just after server starts receiving requests.

sudo service elasticsearch restart

Repeat previous step couple of times.

If bulk request fails (Connection refused) because of Elasticsearch restart, repeat sending request until it succeed.

After all bulk requests are sent, index document count is grater then 10000.

curl -XGET http://localhost:9200/index1/_count

{"count":10326,"_shards":{"total":5,"successful":5,"failed":0}}

Provide logs (if relevant):

[2016-07-06 15:49:28,441][INFO ][node ] [ubuntu1-node] stopping ...
[2016-07-06 15:49:28,485][WARN ][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler.
java.util.concurrent.RejectedExecutionException: Worker has already been shutdown
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72)
at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56)
at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36)
at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34)
at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636)
at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496)
at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46)
at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781)
at org.jboss.netty.channel.Channels.write(Channels.java:725)
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784)
at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:146)
at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43)
at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:89)
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)
at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:356)
at org.elasticsearch.action.bulk.TransportBulkAction$2.onFailure(TransportBulkAction.java:351)
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:95)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:567)
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:527)
at org.elasticsearch.transport.TransportService$2.run(TransportService.java:206)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-07-06 15:49:28,512][WARN ][transport ] [ubuntu1-node] Transport response handler not found of id [43]
[2016-07-06 15:49:28,863][INFO ][node ] [ubuntu1-node] stopped
[2016-07-06 15:49:28,863][INFO ][node ] [ubuntu1-node] closing ...
[2016-07-06 15:49:28,868][INFO ][node ] [ubuntu1-node] closed

Issue 2:
From Binh:

“Thank you for reporting this, the plan is to revert back and allow 0 for external versioning. Not sure yet when it will be but likely in the next 1 or 2 releases.”

---------- Forwarded message ----------

Hello team - I just switched over to 1.1.0 and changed from VersionType.EXTERNAL to VersionType.EXTERNAL_GTE. Ever since then, we are seeing the error below.

Any comments?

I looked at ES code and see this newly added line for EXTERNAL_GTE. Does it mean we dont support zero as a valid version number? Our systems use zero as the start version number.

Also to add: it seems that even VersionType.EXTERNAL is also broken in the new version (1.1.0) since it now does the validation of input version number and it does not allow a save if the input version number is zero.

Looking for inputs from ES team on how to proceed on this.

@Override
public boolean validateVersion(long version) {
return version > 0L;
}

org.elasticsearch.action.ActionRequestValidationException: Validation Failed: 1: illegal version value [0] for version type [EXTERNAL_GTE];
at org.elasticsearch.action.ValidateActions.addValidationError(ValidateActions.java:29)
at org.elasticsearch.action.index.IndexRequest.validate(IndexRequest.java:181)
at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:63)
at org.elasticsearch.client.transport.support.InternalTransportClient$2.doWithNode(InternalTransportClient.java:109)
at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:228)
at org.elasticsearch.client.transport.support.InternalTransportClient.execute(InternalTransportClient.java:106)
at org.elasticsearch.client.support.AbstractClient.index(AbstractClient.java:82)
at org.elasticsearch.client.transport.TransportClient.index(TransportClient.java:330)
at org.elasticsearch.action.index.IndexRequestBuilder.doExecute(IndexRequestBuilder.java:314)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:85)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:59)

-Amit