Repository: netty/netty
Issue: http2: Compatibility issue: SETTINGS ACK may be reordered
Labels: defect

### Expected behavior

When receiving a SETTINGS frame, the ACK would be sent before any frames that make use of the new settings.

For example, say MAX_CONCURRENT_STREAMS=0 initially. When receiving a SETTINGS with MAX_CONCURRENT_STREAMS=1, the SETTINGS(ACK=true) would be sent before a HEADERS for a new stream. Similarly for INITIAL_WINDOW_SIZE increasing the window size; the SETTINGS(ACK=true) should be sent before any DATA makes use of the increased window.

### Actual behavior

When INITIAL_WINDOW_SIZE is increased, we have seen Netty send DATA using the newly available window before the SETTINGS ACK. (Although we haven't actually confirmed the SETTINGS ACK is sent at all, because the receiver aborts as soon as the broken window is detected, while enabling logging on the java side or swapping to plaintext changes the timing so the bug doesn't manifest due to details in the server's behavior.)

`encoder.remoteSettings()` [is called before](https://github.com/netty/netty/blob/netty-4.1.8.Final/codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java#L452,L455) `writeSettingsAck()`. `encoder.remoteSettings()` [calls](https://github.com/netty/netty/blob/netty-4.1.8.Final/codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionEncoder.java#L113) `flowController().initialWindowSize()` which results in [the flow controller writing pending bytes](https://github.com/netty/netty/blob/netty-4.1.8.Final/codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2RemoteFlowController.java#L659).

Two changes should probably be made:
1. Send the settings ack immediately before processing the settings frame in the decoder. Any failure in processing would probably result in the connection being closed, so this doesn't seem dangerous.
2. Stop triggering `writePendingBytes()` in the flow controller and instead [wait for the `flush()` like normal](https://github.com/netty/netty/blob/netty-4.1.8.Final/codec-http2/src/main/java/io/netty/handler/codec/http2/Http2ConnectionHandler.java#L166).

### Steps to reproduce

Nothing easy yet. We have seen the problem with INITIAL_WINDOW_SIZE. It is pretty racy in our environment and requires a C# gRPC server talking to our Netty-based gRPC client. You can see grpc/grpc-java#2801 and grpc/grpc#9956. With a specially-crafted server, it should be reliable to reproduce, but I didn't want to delay filing the issue.

### Versions

Netty version: Netty 4.1.6, Netty 4.1.8
JVM version: 1.8.0_112
OS version: Ubuntu 14.04

### Appendix

This is the current log we can see on the C# server. It's a bit hard to read/comprehend, so I'm not sure you want to look at it much.

`incoming_window_delta` does not include `INITIAL_WINDOW_SIZE`, so that's why it goes negative. You can see how one SETTINGS ACK is received for the connection handshake. 5 DATA frames are received successfully, but then it fails with "frame of size 11642 overflows incoming window of 0" while decoding the next frame.

```
D0309 00:04:02.466914 139734348199680 src/core/lib/transport/connectivity_state.c:134: CONWATCH: 0x7f1664012118 server_transport: from IDLE [cur=READY] notify=0x7f16640074b0
D0309 00:04:02.467097 139734348199680 src/core/lib/transport/connectivity_state.c:134: CONWATCH: 0x7f1664012118 server_transport: from READY [cur=READY] notify=0x7f16640074b0
D0309 00:04:02.467170 139734348199680 src/core/ext/transport/chttp2/transport/frame_settings.c:130: SEND SETTINGS {INITIAL_WINDOW_SIZE = 65535MAX_HEADER_LIST_SIZE = 16384}
D0309 00:04:02.467237 139734348199680 src/core/ext/transport/chttp2/transport/writing.c:334: FLOW    write: server CREDIT                                        TRANSPORT:incoming_window    65535 by                                                        announced 2147418112 giving 2147483647
D0309 00:04:02.474815 139734348199680 src/core/ext/transport/chttp2/transport/frame_settings.c:152: RECV SETTINGS ACK
D0309 00:04:02.524103 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:397: FLOW    parse: server DEBIT                                   STREAM[3]:incoming_window_delta        0 by                                              incoming_frame_size    16384 giving -16384
D0309 00:04:02.524163 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:413: FLOW    parse: server DEBIT                                         TRANSPORT:incoming_window 2147483647 by                                              incoming_frame_size    16384 giving 2147467263
D0309 00:04:02.524238 139734348199680 src/core/ext/transport/chttp2/transport/frame_settings.c:130: SEND SETTINGS {INITIAL_WINDOW_SIZE = 77177}
D0309 00:04:02.524863 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:397: FLOW    parse: server DEBIT                                   STREAM[3]:incoming_window_delta   -16384 by                                              incoming_frame_size    16384 giving -32768
D0309 00:04:02.524889 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:413: FLOW    parse: server DEBIT                                         TRANSPORT:incoming_window 2147467263 by                                              incoming_frame_size    16384 giving 2147450879
D0309 00:04:02.525761 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:397: FLOW    parse: server DEBIT                                   STREAM[3]:incoming_window_delta   -32768 by                                              incoming_frame_size    16384 giving -49152
D0309 00:04:02.525834 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:413: FLOW    parse: server DEBIT                                         TRANSPORT:incoming_window 2147450879 by                                              incoming_frame_size    16384 giving 2147434495
D0309 00:04:02.529791 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:397: FLOW    parse: server DEBIT                                   STREAM[3]:incoming_window_delta   -49152 by                                              incoming_frame_size    16239 giving -65391
D0309 00:04:02.529863 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:413: FLOW    parse: server DEBIT                                         TRANSPORT:incoming_window 2147434495 by                                              incoming_frame_size    16239 giving 2147418256
D0309 00:04:02.529917 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:397: FLOW    parse: server DEBIT                                   STREAM[3]:incoming_window_delta   -65391 by                                              incoming_frame_size      144 giving -65535
D0309 00:04:02.529970 139734348199680 src/core/ext/transport/chttp2/transport/parsing.c:413: FLOW    parse: server DEBIT                                         TRANSPORT:incoming_window 2147418256 by                                              incoming_frame_size      144 giving 2147418112
D0309 00:04:02.530253 139734348199680 src/core/lib/transport/connectivity_state.c:184: SET: 0x7f1664012118 server_transport: READY --> SHUTDOWN [close_transport] error=0x7f1664004560 {"created":"@1489017842.530171783","description":"Delayed close due to in-progress write","file":"src/core/ext/transport/chttp2/transport/chttp2_transport.c","file_line":517,"grpc_status":14,"referenced_errors":[{"created":"@1489017842.530170049","description":"Failed parsing HTTP/2","file":"src/core/ext/transport/chttp2/transport/chttp2_transport.c","file_line":1971,"referenced_errors":[{"created":"@1489017842.530012238","description":"frame of size 11642 overflows incoming window of 0","file":"src/core/ext/transport/chttp2/transport/parsing.c","file_line":391}]}]}
```

Issue comments:
Reproduction:

I've used it to confirm (1) above resolves the problem (move `encoder.writeSettingsAck` earlier). In this reproduction, the client is sending a large message and misbehaves.

```patch
diff --git a/example/src/main/java/io/netty/example/http2/helloworld/client/Http2Client.java b/example/src/main/java/io/netty/example/http2/helloworld/client/Http2Client.java
index bc477d52e..4845a9677 100644
--- a/example/src/main/java/io/netty/example/http2/helloworld/client/Http2Client.java
+++ b/example/src/main/java/io/netty/example/http2/helloworld/client/Http2Client.java
@@ -58,8 +58,8 @@ public final class Http2Client {
     static final boolean SSL = System.getProperty("ssl") != null;
     static final String HOST = System.getProperty("host", "127.0.0.1");
     static final int PORT = Integer.parseInt(System.getProperty("port", SSL? "8443" : "8080"));
-    static final String URL = System.getProperty("url", "/whatever");
-    static final String URL2 = System.getProperty("url2");
+    static final String URL = System.getProperty("url");
+    static final String URL2 = System.getProperty("url2", "/whatever");
     static final String URL2DATA = System.getProperty("url2data", "test data!");
 
     public static void main(String[] args) throws Exception {
@@ -124,7 +124,7 @@ public final class Http2Client {
             if (URL2 != null) {
                 // Create a simple POST request with a body.
                 FullHttpRequest request = new DefaultFullHttpRequest(HTTP_1_1, POST, URL2,
-                        wrappedBuffer(URL2DATA.getBytes(CharsetUtil.UTF_8)));
+                        wrappedBuffer(new byte[10 * 1000 * 1000]));
                 request.headers().add(HttpHeaderNames.HOST, hostName);
                 request.headers().add(HttpConversionUtil.ExtensionHeaderNames.SCHEME.text(), scheme.name());
                 request.headers().add(HttpHeaderNames.ACCEPT_ENCODING, HttpHeaderValues.GZIP);
diff --git a/example/src/main/java/io/netty/example/http2/helloworld/server/HelloWorldHttp2Handler.java b/example/src/main/java/io/netty/example/http2/helloworld/server/HelloWorldHttp2Handler.java
index a763a9de8..bc8fda032 100644
--- a/example/src/main/java/io/netty/example/http2/helloworld/server/HelloWorldHttp2Handler.java
+++ b/example/src/main/java/io/netty/example/http2/helloworld/server/HelloWorldHttp2Handler.java
@@ -92,12 +92,20 @@ public final class HelloWorldHttp2Handler extends Http2ConnectionHandler impleme
         if (endOfStream) {
             sendResponse(ctx, streamId, data.retain());
         }
-        return processed;
+        return 0;
     }
 
     @Override
     public void onHeadersRead(ChannelHandlerContext ctx, int streamId,
                               Http2Headers headers, int padding, boolean endOfStream) {
+        try {
+            decoder().flowController().incrementWindowSize(connection().connectionStream(), 20 * 1000 * 1000);
+        } catch (Exception ex) {
+            throw new RuntimeException(ex);
+        }
+        io.netty.handler.codec.http2.Http2Settings settings = new io.netty.handler.codec.http2.Http2Settings();
+        settings.initialWindowSize(20 * 1000 * 1000);
+        encoder().writeSettings(ctx, settings, ctx.newPromise());
         if (endOfStream) {
             ByteBuf content = ctx.alloc().buffer();
             content.writeBytes(RESPONSE_BYTES.duplicate());
```

Server-side log (where client is misbehaving). You can see that it sends `RST_STREAM: streamId=3, errorCode=3` before the `SETTINGS: ack=true` is received.

```
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] HEADERS: streamId=3, headers=DefaultHttp2Headers[:path: /whatever, :method: POST, :scheme: http, :authority: 127.0.0.1:8080, accept-encoding: gzip, accept-encoding: deflate], streamDependency=0, weight=16, exclusive=false, padding=0, endStream=false
------------------------------------
16:43:12.255 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------OUTBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] WINDOW_UPDATE: streamId=0, windowSizeIncrement=20000000
------------------------------------
16:43:12.255 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------OUTBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] SETTINGS: ack=false, settings={INITIAL_WINDOW_SIZE=20000000}
------------------------------------
16:43:12.257 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=16384, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.258 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=16384, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.258 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=16384, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.259 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=16143, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.259 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=240, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.260 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=16384, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.260 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------OUTBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] WINDOW_UPDATE: streamId=3, windowSizeIncrement=32768
------------------------------------
16:43:12.261 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------OUTBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] RST_STREAM: streamId=3, errorCode=3
------------------------------------
16:43:12.262 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------OUTBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] WINDOW_UPDATE: streamId=3, windowSizeIncrement=49151
------------------------------------
16:43:12.262 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=16384, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.262 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------OUTBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] RST_STREAM: streamId=3, errorCode=5
------------------------------------
16:43:12.264 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=16384, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.264 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------OUTBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] RST_STREAM: streamId=3, errorCode=5
------------------------------------
16:43:12.264 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] DATA: streamId=3, padding=0, endStream=false, length=16384, bytes=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000...
------------------------------------
16:43:12.265 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------OUTBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] RST_STREAM: streamId=3, errorCode=5
------------------------------------
16:43:12.265 [nioEventLoopGroup-2-2] INFO  i.n.e.h.h.s.HelloWorldHttp2Handler - 
----------------INBOUND--------------------
[id: 0x29fccc8d, L:/127.0.0.1:8080 - R:/127.0.0.1:59317] SETTINGS: ack=true
------------------------------------
```
@ejona86 - Thanks for reporting! I have a unit test for this issue now and will submit a PR soon.
@ejona86 - In terms of your 2nd suggestion I *think* we should be OK with the current setup because `Http2ConnectionHandler` forces a `flush()` in `channelReadComplete()` ... but lets handle that in a followup PR because I think the 1st suggestion is sufficient to fix this issue. any objections to this approach?
@Scottmitch, yeah, just doing (1) now sounds fine and is lower risk.

Pull request review comments:
This doesn't actually check "before settings are used" in the general sense. It only tests `INITIAL_WINDOW_SIZE` handling. Maybe `testSettingsAckIsSentBeforeUsingFlowControl`?

I say that because if the default flow controller avoids calling `writePendingBytes();` from `initialWindowSize(int)`, this test passes. But that wouldn't imply that other settings (e.g., `MAX_CONCURRENT_STREAMS`) would be handled correctly.
sure sgtm
fixed

Patch that resolved the issue:
diff --git a/codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java b/codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java
index 333937113e2..ef643fafadc 100644
--- a/codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java
+++ b/codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2ConnectionDecoder.java
@@ -402,11 +402,13 @@ private void applyLocalSettings(Http2Settings settings) throws Http2Exception {
 
         @Override
         public void onSettingsRead(ChannelHandlerContext ctx, Http2Settings settings) throws Http2Exception {
-            encoder.remoteSettings(settings);
-
-            // Acknowledge receipt of the settings.
+            // Acknowledge receipt of the settings. We should do this before we process the settings to ensure our
+            // remote peer applies these settings before any subsequent frames that we may send which depend upon these
+            // new settings. See https://github.com/netty/netty/issues/6520.
             encoder.writeSettingsAck(ctx, ctx.newPromise());
 
+            encoder.remoteSettings(settings);
+
             listener.onSettingsRead(ctx, settings);
         }
 


Test patch:
diff --git a/codec-http2/src/test/java/io/netty/handler/codec/http2/Http2ConnectionRoundtripTest.java b/codec-http2/src/test/java/io/netty/handler/codec/http2/Http2ConnectionRoundtripTest.java
index cb70ec6225a..f5de427b4a8 100644
--- a/codec-http2/src/test/java/io/netty/handler/codec/http2/Http2ConnectionRoundtripTest.java
+++ b/codec-http2/src/test/java/io/netty/handler/codec/http2/Http2ConnectionRoundtripTest.java
@@ -347,6 +347,95 @@ public void operationComplete(ChannelFuture future) throws Exception {
         verify(clientListener, never()).onRstStreamRead(any(ChannelHandlerContext.class), anyInt(), anyLong());
     }
 
+    @Test
+    public void testSettingsAckIsSentBeforeSettingsAreUsed() throws Exception {
+        bootstrapEnv(1, 1, 1, 1);
+
+        final CountDownLatch serverSettingsAckLatch1 = new CountDownLatch(1);
+        final CountDownLatch serverSettingsAckLatch2 = new CountDownLatch(2);
+        final CountDownLatch serverDataLatch = new CountDownLatch(1);
+        final CountDownLatch clientWriteDataLatch = new CountDownLatch(1);
+        final byte[] data = new byte[] {1, 2, 3, 4, 5};
+        final ByteArrayOutputStream out = new ByteArrayOutputStream(data.length);
+
+        doAnswer(new Answer<Void>() {
+            @Override
+            public Void answer(InvocationOnMock invocationOnMock) throws Throwable {
+                serverSettingsAckLatch1.countDown();
+                serverSettingsAckLatch2.countDown();
+                return null;
+            }
+        }).when(serverListener).onSettingsAckRead(any(ChannelHandlerContext.class));
+        doAnswer(new Answer<Integer>() {
+            @Override
+            public Integer answer(InvocationOnMock in) throws Throwable {
+                ByteBuf buf = (ByteBuf) in.getArguments()[2];
+                int padding = (Integer) in.getArguments()[3];
+                int processedBytes = buf.readableBytes() + padding;
+
+                buf.readBytes(out, buf.readableBytes());
+                serverDataLatch.countDown();
+                return processedBytes;
+            }
+        }).when(serverListener).onDataRead(any(ChannelHandlerContext.class), eq(3),
+                any(ByteBuf.class), eq(0), anyBoolean());
+
+        final Http2Headers headers = dummyHeaders();
+
+        // The server initially reduces the connection flow control window to 0.
+        runInChannel(serverConnectedChannel, new Http2Runnable() {
+            @Override
+            public void run() throws Http2Exception {
+                http2Server.encoder().writeSettings(serverCtx(),
+                        new Http2Settings().copyFrom(http2Server.decoder().localSettings())
+                                .initialWindowSize(0),
+                        serverNewPromise());
+                http2Server.flush(serverCtx());
+            }
+        });
+
+        assertTrue(serverSettingsAckLatch1.await(DEFAULT_AWAIT_TIMEOUT_SECONDS, SECONDS));
+
+        // The client should now attempt to send data, but the window size is 0 so it will be queued in the flow
+        // controller.
+        runInChannel(clientChannel, new Http2Runnable() {
+            @Override
+            public void run() throws Http2Exception {
+                http2Client.encoder().writeHeaders(ctx(), 3, headers, 0, (short) 16, false, 0, false,
+                        newPromise());
+                http2Client.encoder().writeData(ctx(), 3, Unpooled.wrappedBuffer(data), 0, true, newPromise());
+                http2Client.flush(ctx());
+                clientWriteDataLatch.countDown();
+            }
+        });
+
+        assertTrue(clientWriteDataLatch.await(DEFAULT_AWAIT_TIMEOUT_SECONDS, SECONDS));
+
+        // Now the server opens up the connection window to allow the client to send the pending data.
+        runInChannel(serverConnectedChannel, new Http2Runnable() {
+            @Override
+            public void run() throws Http2Exception {
+                http2Server.encoder().writeSettings(serverCtx(),
+                        new Http2Settings().copyFrom(http2Server.decoder().localSettings())
+                                .initialWindowSize(data.length),
+                        serverNewPromise());
+                http2Server.flush(serverCtx());
+            }
+        });
+
+        assertTrue(serverSettingsAckLatch2.await(DEFAULT_AWAIT_TIMEOUT_SECONDS, SECONDS));
+        assertTrue(serverDataLatch.await(DEFAULT_AWAIT_TIMEOUT_SECONDS, SECONDS));
+        assertArrayEquals(data, out.toByteArray());
+
+        // Verify that no errors have been received.
+        verify(serverListener, never()).onGoAwayRead(any(ChannelHandlerContext.class), anyInt(), anyLong(),
+                any(ByteBuf.class));
+        verify(serverListener, never()).onRstStreamRead(any(ChannelHandlerContext.class), anyInt(), anyLong());
+        verify(clientListener, never()).onGoAwayRead(any(ChannelHandlerContext.class), anyInt(), anyLong(),
+                any(ByteBuf.class));
+        verify(clientListener, never()).onRstStreamRead(any(ChannelHandlerContext.class), anyInt(), anyLong());
+    }
+
     @Test
     public void priorityUsingHigherValuedStreamIdDoesNotPreventUsingLowerStreamId() throws Exception {
         bootstrapEnv(1, 1, 2, 0);