Project: springxd
Issue XD-716: TapCommandTests hangs when using a lazily instantiated Lettuce connection
A change was made in spring-data-redis to instantiate the shared Lettuce connection lazily instead of when the context is initialized. This caused TapCommandTests to hang due to a Netty worker thread trying to initialize the Lettuce connection (Lettuce uses Netty). The change was temporarily backed out of SDR, but we need to consider using a NettyExecutionHandler in NettyHttpInboundChannelAdapter or making the HTTP module's "input" channel an ExecutorChannel to avoid potentially long operations like from happening in an I/O thread. Also, we need to address why this failure simply hangs the shell. Shell was hung waiting on IO here: org.springframework.http.client.SimpleClientHttpResponse.getRawStatusCode(SimpleClientHttpResponse.java:47) org.springframework.http.client.AbstractClientHttpResponse.getStatusCode(AbstractClientHttpResponse.java:32) org.springframework.xd.shell.command.HttpCommands$1.hasError(HttpCommands.java:93) org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:484) org.springframework.web.client.RestTemplate.execute(RestTemplate.java:460) org.springframework.web.client.RestTemplate.postForEntity(RestTemplate.java:335) org.springframework.xd.shell.command.HttpCommands.postHttp(HttpCommands.java:103) sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:191) org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:64) ringframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:57)	- locked <7fd3c7d40> (a java.lang.Class for org.springframework.shell.core.SimpleExecutionStrategy) org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:127) org.springframework.xd.shell.AbstractShellIntegrationTest.executeCommand(AbstractShellIntegrationTest.java:99) org.springframework.xd.shell.AbstractShellIntegrationTest.httpPostData(AbstractShellIntegrationTest.java:112) org.springframework.xd.shell.command.TapCommandTests.testCreateAndDeployTap(TapCommandTests.java:56) Full stack trace of server exception: Aug 19, 2013 9:59:00 AM org.jboss.netty.channel.SimpleChannelUpstreamHandler WARNING: EXCEPTION, please implement org.springframework.integration.x.http.NettyHttpInboundChannelAdapter$Handler.exceptionCaught() for proper handling. org.springframework.integration.MessageHandlingException: org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis on localhost:6379; nested exception is com.lambdaworks.redis.RedisException: Unable to connect org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:76) org.springframework.integration.handler.ServiceActivatingHandler.handleRequestMessage(ServiceActivatingHandler.java:67) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:137) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.BroadcastingDispatcher.invokeHandler(BroadcastingDispatcher.java:121) org.springframework.integration.dispatcher.BroadcastingDispatcher.dispatch(BroadcastingDispatcher.java:112) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.interceptor.WireTap.preSend(WireTap.java:121) org.springframework.integration.channel.AbstractMessageChannel$ChannelInterceptorList.preSend(AbstractMessageChannel.java:248) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:173) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendMessage(AbstractReplyProducingMessageHandler.java:223) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.sendReplyMessage(AbstractReplyProducingMessageHandler.java:207) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.produceReply(AbstractReplyProducingMessageHandler.java:172) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleResult(AbstractReplyProducingMessageHandler.java:166) org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:144) org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73) org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:115) org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:102) org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:178) org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:149) org.springframework.integration.core.MessagingTemplate.doSend(MessagingTemplate.java:304) org.springframework.integration.core.MessagingTemplate.send(MessagingTemplate.java:165) org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:92) org.springframework.integration.x.http.NettyHttpInboundChannelAdapter.access$200(NettyHttpInboundChannelAdapter.java:59) org.springframework.integration.x.http.NettyHttpInboundChannelAdapter$Handler.messageReceived(NettyHttpInboundChannelAdapter.java:122) org.jboss.netty.handler.codec.http.HttpContentEncoder.messageReceived(HttpContentEncoder.java:81) org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148) org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:485) org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107) org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312) org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88) org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:680) Caused by: org.springframework.data.redis.RedisConnectionFailureException: Unable to connect to Redis on localhost:6379; nested exception is com.lambdaworks.redis.RedisException: Unable to connect org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.createLettuceConnector(LettuceConnectionFactory.java:345) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.initConnection(LettuceConnectionFactory.java:116) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getSharedConnection(LettuceConnectionFactory.java:325) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.getConnection(LettuceConnectionFactory.java:106) org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:81) org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:53) org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:173) org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153) org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86) org.springframework.data.redis.core.DefaultZSetOperations.add(DefaultZSetOperations.java:41) org.springframework.data.redis.core.DefaultBoundZSetOperations.add(DefaultBoundZSetOperations.java:47) org.springframework.xd.store.AbstractRedisRepository.trackMembership(AbstractRedisRepository.java:202) org.springframework.xd.analytics.metrics.redis.RedisCounterRepository.increment(RedisCounterRepository.java:88) org.springframework.xd.analytics.metrics.redis.RedisCounterRepository.increment(RedisCounterRepository.java:82) org.springframework.xd.analytics.metrics.integration.MessageCounterHandler.process(MessageCounterHandler.java:28) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:69) org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:97) org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:82) org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:102) org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:103) org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:126) org.springframework.integration.util.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:230) org.springframework.integration.util.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:129) org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:73)	... 84 more Caused by: com.lambdaworks.redis.RedisException: Unable to connect com.lambdaworks.redis.RedisClient.connect(RedisClient.java:176) com.lambdaworks.redis.RedisClient.connectAsync(RedisClient.java:139) org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory.createLettuceConnector(LettuceConnectionFactory.java:339)	... 111 more Caused by: java.lang.IllegalStateException: await*() in I/O thread causes a dead lock or sudden performance drop. Use addListener() instead or call await*() from a different thread. org.jboss.netty.channel.DefaultChannelFuture.checkDeadLock(DefaultChannelFuture.java:342) org.jboss.netty.channel.DefaultChannelFuture.await(DefaultChannelFuture.java:231) com.lambdaworks.redis.RedisClient.connect(RedisClient.java:166)	... 113 more
Story points: 5