opened
dconnelly
username_0
We recently ran into an issue with one of our services where sending large responses results in transport errors being logged as well as eventual transport failure. I managed to reproduce this behavior in integration test largeUnary as well. When running a single iteration of largeUnary the test passes although connection errors are logged:
```
Apr 20, 2015 10:43:45 AM io.grpc.transport.netty.NettyServerHandler onConnectionError
WARNING: Connection Error
io.netty.handler.codec.http2.Http2Exception: Stream does not exist 3
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:58)
	at io.netty.handler.codec.http2.DefaultHttp2Connection.requireStream(DefaultHttp2Connection.java:110)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onWindowUpdateRead(DefaultHttp2ConnectionDecoder.java:486)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onWindowUpdateRead(Http2InboundFrameLogger.java:124)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readWindowUpdateFrame(DefaultHttp2FrameReader.java:560)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:247)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:130)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:39)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:99)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:291)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:332)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:327)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:230)
	at io.netty.channel.ChannelHandlerInvokerUtil.invokeChannelReadNow(ChannelHandlerInvokerUtil.java:83)
	at io.netty.channel.DefaultChannelHandlerInvoker.invokeChannelRead(DefaultChannelHandlerInvoker.java:153)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:157)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:946)
	at io.netty.channel.local.LocalChannel.finishPeerRead(LocalChannel.java:329)
	at io.netty.channel.local.LocalChannel.access$400(LocalChannel.java:45)
	at io.netty.channel.local.LocalChannel$5.run(LocalChannel.java:315)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:322)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:356)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:703)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
```

However, running largeUnary in a simple loop causes transport failure after a few iterations. This happens with both netty and okhttp client variants, as well as with netty local channel. 

Other details:
```
$ uname -a
Darwin eno.local 14.3.0 Darwin Kernel Version 14.3.0: Mon Mar 23 11:59:05 PDT 2015; root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64
$ java -version
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b25)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
```
- David
Integration test largeUnary failing
issue
created
ejona86
username_1
This looks similar to https://github.com/netty/netty/issues/3643, https://github.com/netty/netty/issues/2861. HTTP/2 in Netty recently removed tracking of recently-closed streams because it was consuming a lot of memory. This and many similar bugs show issues with the current handling of missing streams or the need to re-introduce that history (but an alternative approach that uses less memory).

For the time being, using gRPC (and the associated Netty) from before f920bad is probably the best approach. We hope to have the issue resolved soon™.
comment
created
nmittler
username_2
I believe this should be fixed as of 3ecbd04
comment
closed
nmittler
username_2
issue