Issue 1:
We created an index and tried to "curl -PUT" a document, failed after a timeout with:

{
"error": "NoShardAvailableActionException[[test][0] No shard available for [[test][test_AS24Elasticsearch][1]: routing [null]]]",
"status": 500
}

The reason was this line in the elasticsearch.yml:

index.search.slowlog.threshold.query.warn: s

When reproducing this i found that the elasticsearch.yml has to be broken at startup of the elasticsearch service.

It produced about 1'000'000 lines in the logfile.

I post this hoping for a more precise error message, an earlier parsing of the elasticsearch.yml or that people running into the same problem will find this post and identify their problem faster.

The wrong stanza in elasticsearch.yml:

index.search.slowlog.level: TRACE
index.search.slowlog.threshold.query.warn: s
index.search.slowlog.threshold.query.info: s
index.search.slowlog.threshold.query.debug: s
index.search.slowlog.threshold.query.trace: ms

(This kind of mistake is, as you will guess, the result of a beginner meddling with Puppet)

In the logfile - only the first 82 lines:

[2013-03-05 09:35:10,805][WARN ][indices.cluster ] [dexxxv001] [sunytest][0] failed to create shard
org.elasticsearch.index.shard.IndexShardCreationException: [sunytest][0] failed to create shard
at org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:323)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:561)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:526)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.ElasticSearchParseException: Failed to parse [s]
at org.elasticsearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:253)
at org.elasticsearch.common.settings.ImmutableSettings.getAsTime(ImmutableSettings.java:191)
at org.elasticsearch.index.search.slowlog.ShardSlowLogSearchService.<init>(ShardSlowLogSearchService.java:132)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:56)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:56)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.SingleParameterInjector.inject(SingleParameterInjector.java:42)
at org.elasticsearch.common.inject.SingleParameterInjector.getAll(SingleParameterInjector.java:66)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:85)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.FactoryProxy.get(FactoryProxy.java:52)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:56)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:812)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:129)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:66)
at org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:321)
... 7 more
Caused by: java.lang.NumberFormatException: empty String
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:1011)
at java.lang.Double.parseDouble(Double.java:540)

Issue 2:
Description

First, I am using version 0.20.6 on a quite large cluster.

Pushed a non-dynamic settings with a invalid value for example;

"index.translog.flush_threshold_period": "30mb"

instead of 30m (ok this is kind of obvious)

The cluster API become unresponsive and unrecoverable due to the log overload (specially on the master). I had to stop the whole cluster, delete the index directory by hand, and restart the cluster to recover.

The cluster should be able to;

1 - first; catch the bad parameter and do not set the setting value;

2 - second; respond to the API, search, and every request, or at least close the index since it is not able to recover.

Steps to reproduce

1 - Create and index with valid parameters

2 - While being open; set invalid parameter as described

3 - Close the index

4 - Try to reopen the index with the invalid parameters set.

5 - Try to close/delete the index or update anythingt (shoudn't be able)

Log samples

Master

[2013-05-06 08:41:53,423][WARN ][cluster.action.shard ] [es84b] received shard failed for [public_20120601][28], node[FvESugqWT3OJphhj6nFIZA], [P], s[INITIALIZING], reason [Failed to create shard, message [IndexShardCreationException[[public_20120601][28] failed to create shard]; nested: ElasticSearchParseException[Failed to parse [60mb]]; nested: NumberFormatException[For input string: "60mb"]; ]]
[2013-05-06 08:41:53,423][WARN ][cluster.action.shard ] [es84b] received shard failed for [public_20120601][22], node[SlES3cBSS8q7Fjqc4hOQeQ], [P], s[INITIALIZING], reason [Failed to create shard, message [IndexShardCreationException[[public_20120601][22] failed to create shard]; nested: ElasticSearchParseException[Failed to parse [60mb]]; nested: NumberFormatException[For input string: "60mb"]; ]]

Node

[2013-05-06 08:41:53,451][WARN ][indices.cluster ] [es84b] [public_20120601][35] failed to create shard [0/2]
org.elasticsearch.index.shard.IndexShardCreationException: [public_20120601][35] failed to create shard
at org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:323)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyInitializingShard(IndicesClusterStateService.java:561)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewOrUpdatedShards(IndicesClusterStateService.java:526)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:171)
at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:315)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.elasticsearch.ElasticSearchParseException: Failed to parse [60mb]
at org.elasticsearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:253)
at org.elasticsearch.common.settings.ImmutableSettings.getAsTime(ImmutableSettings.java:191)
at org.elasticsearch.index.translog.TranslogService.<init>(TranslogService.java:79)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
at org.elasticsearch.common.inject.DefaultConstructionProxyFactory$1.newInstance(DefaultConstructionProxyFactory.java:54)
at org.elasticsearch.common.inject.ConstructorInjector.construct(ConstructorInjector.java:86)
at org.elasticsearch.common.inject.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:98)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:45)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:819)
at org.elasticsearch.common.inject.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:42)
at org.elasticsearch.common.inject.Scopes$1$1.get(Scopes.java:56)
at org.elasticsearch.common.inject.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:200)
at org.elasticsearch.common.inject.InjectorBuilder$1.call(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorImpl.callInContext(InjectorImpl.java:812)
at org.elasticsearch.common.inject.InjectorBuilder.loadEagerSingletons(InjectorBuilder.java:193)
at org.elasticsearch.common.inject.InjectorBuilder.injectDynamically(InjectorBuilder.java:175)
at org.elasticsearch.common.inject.InjectorBuilder.build(InjectorBuilder.java:110)
at org.elasticsearch.common.inject.InjectorImpl.createChildInjector(InjectorImpl.java:129)
at org.elasticsearch.common.inject.ModulesBuilder.createChildInjector(ModulesBuilder.java:66)
at org.elasticsearch.index.service.InternalIndexService.createShard(InternalIndexService.java:321)
... 7 more
Caused by: java.lang.NumberFormatException: For input string: "60mb"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at org.elasticsearch.common.unit.TimeValue.parseTimeValue(TimeValue.java:249)
... 30 more