influxdb/influxdb — pull request #1024 by dgnorton (2014-11-06 01:23:35+00:00)
Title: Hitting open files limit causes influxdb to create shards in loop

Default install on Debian, 0.8.3, no tuning done except changing data dir path.

After InfluxDB hit the open files limit it started to create new shards in the loop, and of course could not write data, resulting in creation of few thousand dirs.
It should either:

* exit, so if it is monitored user can notice before wondering why there is no stats from last shard
* close old/least used shards

Logs:

[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).GetShardToWriteToBySeriesAndTime:796) No matching shards for write at time 1413064200000000u, creating...
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:827) createShards for space long_term: start: Tue Oct 7 02:00:00 +0200 CEST 2014. end: Thu Nov 6 01:00:00 +0100 CET 2014
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /var/lib/influxdb/db/shard_db_v2/04960
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).AddShards:1029) AddShards: error setting local store: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04960/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).doOrProxyCommandOnce:169) Cannot run command &coordinator.CreateShardsCommand{Shards:[]*cluster.NewShardData{(*cluster.NewShardData)(0xc20983ba40)}, SpaceName:""}. open /var/lib/influxdb/db/shard_db_v2/04960/type: too many open files
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).CreateShards:749) RAFT: CreateShards: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04960/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).InterpolateValuesAndCommit:626) Couldn't write data for continuous query: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04960/type: too many open files)
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).GetShardToWriteToBySeriesAndTime:796) No matching shards for write at time 1413064200000000u, creating...
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:827) createShards for space long_term: start: Tue Oct 7 02:00:00 +0200 CEST 2014. end: Thu Nov 6 01:00:00 +0100 CET 2014
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /var/lib/influxdb/db/shard_db_v2/04961
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).AddShards:1029) AddShards: error setting local store: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04961/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).doOrProxyCommandOnce:169) Cannot run command &coordinator.CreateShardsCommand{Shards:[]*cluster.NewShardData{(*cluster.NewShardData)(0xc208edaee0)}, SpaceName:""}. open /var/lib/influxdb/db/shard_db_v2/04961/type: too many open files
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).CreateShards:749) RAFT: CreateShards: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04961/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).InterpolateValuesAndCommit:626) Couldn't write data for continuous query: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04961/type: too many open files)
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).GetShardToWriteToBySeriesAndTime:796) No matching shards for write at time 1413064200000000u, creating...
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:827) createShards for space long_term: start: Tue Oct 7 02:00:00 +0200 CEST 2014. end: Thu Nov 6 01:00:00 +0100 CET 2014
[2014/10/12 00:00:01 CEST] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:162) DATASTORE: opening or creating shard /var/lib/influxdb/db/shard_db_v2/04962
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).AddShards:1029) AddShards: error setting local store: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04962/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).doOrProxyCommandOnce:169) Cannot run command &coordinator.CreateShardsCommand{Shards:[]*cluster.NewShardData{(*cluster.NewShardData)(0xc20a3684d0)}, SpaceName:""}. open /var/lib/influxdb/db/shard_db_v2/04962/type: too many open files
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*RaftServer).CreateShards:749) RAFT: CreateShards: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04962/type: too many open files)
[2014/10/12 00:00:01 CEST] [EROR] (github.com/influxdb/influxdb/coordinator.(*CoordinatorImpl).InterpolateValuesAndCommit:626) Couldn't write data for continuous query: %!(EXTRA *os.PathError=open /var/lib/influxdb/db/shard_db_v2/04962/type: too many open files)