Project: usergrid
Issue USERGRID-507: Many Histrix timeouts though Cassndra is not under load
Cassandra is not under heavy load but I am seeing a high number of Hystrix timeouts. Should we increase connection pool size? Is there an operational setting that can help this? 2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR org.apache.usergrid.persistence.core.task.NamedTaskExecutorImpl- Unable to execute task. Exception is com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1 timed-out and no fallback available. at com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646) at com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103) at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047) at com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627) at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522) at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431) at org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69) at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187) at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208) at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) at com.sun.proxy.$Proxy92.auditShard(Unknown Source) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.util.concurrent.TimeoutException ... 25 more 2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl- Unable to execute audit for shard of {} com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1 timed-out and no fallback available. at com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646) at com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103) at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047) at com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627) at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522) at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431) at org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69) at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187) at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208) at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) at com.sun.proxy.$Proxy92.auditShard(Unknown Source) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.util.concurrent.TimeoutException ... 25 more 2015-03-24 18:34:17,420 [graphTaskExecutor-3] ERROR org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl- Unable to perform audit. Exception is com.netflix.hystrix.exception.HystrixRuntimeException: HystrixCassandra$1 timed-out and no fallback available. at com.netflix.hystrix.HystrixCommand.getFallbackOrThrowException(HystrixCommand.java:1646) at com.netflix.hystrix.HystrixCommand.access$1900(HystrixCommand.java:103) at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$1.run(HystrixCommand.java:1023) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) at com.netflix.hystrix.HystrixCommand$TimeoutObservable$1$2.tick(HystrixCommand.java:1047) at com.netflix.hystrix.HystrixCommand$1.performBlockingGetWithTimeout(HystrixCommand.java:627) at com.netflix.hystrix.HystrixCommand$1.get(HystrixCommand.java:522) at com.netflix.hystrix.HystrixCommand.execute(HystrixCommand.java:431) at org.apache.usergrid.persistence.core.hystrix.HystrixCassandra.user(HystrixCassandra.java:69) at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.advance(MultiRowColumnIterator.java:187) at org.apache.usergrid.persistence.core.astyanax.MultiRowColumnIterator.hasNext(MultiRowColumnIterator.java:124) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardsColumnIterator.hasNext(ShardsColumnIterator.java:65) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.NodeShardAllocationImpl.auditShard(NodeShardAllocationImpl.java:208) at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) at com.sun.proxy.$Proxy92.auditShard(Unknown Source) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:368) at org.apache.usergrid.persistence.graph.serialization.impl.shard.impl.ShardGroupCompactionImpl$ShardAuditTask.call(ShardGroupCompactionImpl.java:319) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.util.concurrent.TimeoutException ... 25 more
Story points: 3