jut-io/kairosdb — pull request #2 by tescherm (2014-12-05 23:14:04+00:00)
Title: Wait for Cassandra startup

If KairosDB is started with a Cassandra data store, where no Cassandra hosts are running, it fails with the following error:

```
Caused by: me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:390) ~[hector-core-1.1-4.jar:na]
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:244) ~[hector-core-1.1-4.jar:na]
at me.prettyprint.cassandra.service.AbstractCluster.describeKeyspace(AbstractCluster.java:199) ~[hector-core-1.1-4.jar:na]
at org.kairosdb.datastore.cassandra.CassandraDatastore$1.execute(CassandraDatastore.java:182) ~[classes/:na]
at org.kairosdb.datastore.cassandra.CassandraDatastore$1.execute(CassandraDatastore.java:179) ~[classes/:na]
at org.kairosdb.util.ExecuteWithRetry.retry(ExecuteWithRetry.java:29) ~[classes/:na]
at org.kairosdb.datastore.cassandra.CassandraDatastore.(CassandraDatastore.java:179) ~[classes/:na]
... 32 common frames omitted
```

To fix this, I added retry logic around the initial cassandra keyspace describe in `CassandraDataStore`. Kairos didn't have a retry utility, so I added one. The usage is quite simple:

```java
final String value = retry(new Retriable() {
@Override
public String execute() {
return "foo";
}
}, 5);
```

The second argument is the maximum number of retry attempts. The duration between attempts defaults to a linear backoff, starting at one second, but we might want to make this configurable as well.