OpenTSDB/opentsdb — pull request #290 by jmangs (2014-02-27 21:36:16+00:00)
Title: Added support for timing out long-running queries (disabled by default)

Addresses feature request in #289

Problem Description:

Basically we had a Ruby script running against our DB looking for metrics to clean-up - I was running 8 concurrent requests to OpenTSDB. We managed to hit a few metrics that have millions of data points - basically it keeps retrieving data from HBase even though it's been running for 500-1000 seconds already.

The VM that this node was on ran out of memory so random parts starting throwing OOM errors without actually crashing the Deferred queries; we had 8 queries each occupying about 400-500MB of JVM memory. In our case, we'd rather have these queries fail so we don't lock up all of our workers running queries that will never finish in a reasonable time.

Feature:
What I did was to add a configurable timeout property that errors out ScannerCB() after a certain amount of time. I'll be sending a PR with my feature; I think it's a nice feature to have. It's also defaulted to infinite wait by setting this timeout property to -1 (more info in the actual PR once it's sent)

By default, this property `tsd.query.timeout` defaults to `-1` since that is the normal functionality in OpenTSDB. This results in `ScannerCB()` never timing out.

If the time spent in HBase exceeds this timeout then an exception is thrown

```java
if (timeout >= 0 && hbase_time > timeout) {
throw new InterruptedException("Query timeout exceeded!");
}
```

Hopefully InterruptedException is acceptable. Right now you get "java.lang.RuntimeException: Shouldn't be here" from the timeout because there's no specific catch for this exception... which makes sense to me since it shouldn't be a common intended occurrence unless you put timeouts to 10ms or something.

I'd consider this a useful feature for production databases. We have many different teams using these metrics and there's a possibility someone could accidentally query something with millions of data points and eat up a lot of resources on a node. In my case, I ran into this problem while trying to search a wide date range for unused metrics (I was checking every single metric one by one)