Repository: akka/akka-http

Issue #2957: Http2StreamHandling - handleOutgoingEnded received unexpectedly in state Closed
Created: 2020-02-11T09:28:05Z
I had just started an Akka gRPC service and was trying it out with grpcurl. I noticed a log message which said that it might be a bug and to report it so here I go.
The individual message is:
[info] [WARN] [02/11/2020 22:10:46.808] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [Http2ServerDemux(akka://ConsoleAPIServer)] handleOutgoingEnded received unexpectedly in state Closed. This indicates a bug in Akka HTTP, please report it to the issue tracker.

The mostly full log is:
[info] [DEBUG] [02/11/2020 21:55:05.585] [main] [akka.actor.ActorSystemImpl(ConsoleAPIServer)] Binding server using HTTP/2
[info] [DEBUG] [02/11/2020 21:55:05.729] [ConsoleAPIServer-akka.actor.internal-dispatcher-6] [akka://ConsoleAPIServer/system/IO-TCP/selectors/$a/0] Successfully bound to /127.0.0.1:8080
[info] 21:55:05.735 INFO  - Console API Server is bound to: /127.0.0.1:8080 (ConsoleServer.scala:37)
[info] [DEBUG] [02/11/2020 21:57:17.874] [ConsoleAPIServer-akka.actor.internal-dispatcher-2] [akka://ConsoleAPIServer/system/IO-TCP/selectors/$a/0] New connection accepted
[info] [DEBUG] [02/11/2020 21:58:18.095] [ConsoleAPIServer-akka.actor.default-dispatcher-9] [akka://ConsoleAPIServer/system/Materializers/StreamSupervisor-0/flow-1-0-detacher] Aborting tcp connection to /127.0.0.1:55991 because of upstream failure: akka.stream.scaladsl.TcpIdleTimeoutException: TCP idle-timeout encountered on connection to [/127.0.0.1:55991], no bytes passed in the last 1 minute
[info] [DEBUG] [02/11/2020 21:58:18.096] [ConsoleAPIServer-akka.actor.default-dispatcher-9] [TcpConnectionStage$TcpStreamLogic(akka://ConsoleAPIServer)] Aborting connection from 127.0.0.1:55991 because of downstream failure: java.util.concurrent.TimeoutException: No elements passed in the last 1 minute.
[info] [DEBUG] [02/11/2020 22:08:49.586] [ConsoleAPIServer-akka.actor.internal-dispatcher-2] [akka://ConsoleAPIServer/system/IO-TCP/selectors/$a/0] New connection accepted
[info] [WARN] [02/11/2020 22:08:49.593] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [akka.actor.ActorSystemImpl(ConsoleAPIServer)] Illegal request, responding with status '400 Bad Request': Unsupported HTTP method: The HTTP method started with 0x16 rather than any known HTTP method. Perhaps this was an HTTPS request sent to an HTTP endpoint?
[info] [DEBUG] [02/11/2020 22:08:49.595] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [StreamUtils$DelayCancellationStage$$anon$11(akka://ConsoleAPIServer)] Delaying cancellation for 1 minute
[info] [DEBUG] [02/11/2020 22:09:46.631] [ConsoleAPIServer-akka.actor.internal-dispatcher-12] [akka://ConsoleAPIServer/system/IO-TCP/selectors/$a/0] New connection accepted
[info] [DEBUG] [02/11/2020 22:09:46.660] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForNetworkToSendControlFrames
[info] [DEBUG] [02/11/2020 22:09:46.661] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from WaitingForNetworkToSendControlFrames to Idle
[info] [DEBUG] [02/11/2020 22:09:46.665] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForData
[info] [DEBUG] [02/11/2020 22:09:46.670] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from WaitingForData to Idle
[info] [DEBUG] [02/11/2020 22:09:46.670] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForData
[info] [DEBUG] [02/11/2020 22:09:46.671] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Got unhandled event SettingsAckFrame(List())
[info] [DEBUG] [02/11/2020 22:09:46.682] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from WaitingForData to Idle
[info] [DEBUG] [02/11/2020 22:09:46.682] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Received DATA 52 for stream [1], remaining window space now 65483, buffered: 52
[info] [DEBUG] [02/11/2020 22:09:46.684] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForData
[info] [DEBUG] [02/11/2020 22:09:46.686] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from WaitingForData to Idle
[info] [DEBUG] [02/11/2020 22:09:46.690] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForData
[info] [WARN] [02/11/2020 22:09:46.694] [ConsoleAPIServer-akka.actor.default-dispatcher-15] [Http2ServerDemux(akka://ConsoleAPIServer)] handleOutgoingEnded received unexpectedly in state HalfClosedLocal. This indicates a bug in Akka HTTP, please report it to the issue tracker.
[info] [DEBUG] [02/11/2020 22:09:49.617] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [StreamUtils$DelayCancellationStage$$anon$11(akka://ConsoleAPIServer)] Stage was canceled after delay of 1 minute
[info] [DEBUG] [02/11/2020 22:10:46.806] [ConsoleAPIServer-akka.actor.default-dispatcher-14] [akka://ConsoleAPIServer/system/Materializers/StreamSupervisor-0/flow-5-0-detacher] Aborting tcp connection to /127.0.0.1:56141 because of upstream failure: akka.stream.scaladsl.TcpIdleTimeoutException: TCP idle-timeout encountered on connection to [/127.0.0.1:56141], no bytes passed in the last 1 minute
[info] [DEBUG] [02/11/2020 22:10:46.806] [ConsoleAPIServer-akka.actor.default-dispatcher-14] [TcpConnectionStage$TcpStreamLogic(akka://ConsoleAPIServer)] Aborting connection from 127.0.0.1:56141 because of downstream failure: java.util.concurrent.TimeoutException: No elements passed in the last 1 minute.
[info] [WARN] [02/11/2020 22:10:46.808] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [Http2ServerDemux(akka://ConsoleAPIServer)] handleOutgoingEnded received unexpectedly in state Closed. This indicates a bug in Akka HTTP, please report it to the issue tracker.
[info] [DEBUG] [02/11/2020 22:10:46.808] [ConsoleAPIServer-akka.actor.internal-dispatcher-16] [akka://ConsoleAPIServer/system/IO-TCP/selectors/$a/0] New connection accepted
[info] [DEBUG] [02/11/2020 22:10:46.814] [ConsoleAPIServer-akka.actor.default-dispatcher-13] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForNetworkToSendControlFrames
[info] [DEBUG] [02/11/2020 22:10:46.814] [ConsoleAPIServer-akka.actor.default-dispatcher-13] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from WaitingForNetworkToSendControlFrames to Idle
[info] [DEBUG] [02/11/2020 22:10:46.814] [ConsoleAPIServer-akka.actor.default-dispatcher-13] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForData
[info] [DEBUG] [02/11/2020 22:10:46.815] [ConsoleAPIServer-akka.actor.default-dispatcher-13] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from WaitingForData to Idle
[info] [DEBUG] [02/11/2020 22:10:46.815] [ConsoleAPIServer-akka.actor.default-dispatcher-13] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForData
[info] [DEBUG] [02/11/2020 22:10:46.815] [ConsoleAPIServer-akka.actor.default-dispatcher-13] [Http2ServerDemux(akka://ConsoleAPIServer)] Got unhandled event SettingsAckFrame(List())
[info] [DEBUG] [02/11/2020 22:11:46.985] [ConsoleAPIServer-akka.actor.default-dispatcher-13] [akka://ConsoleAPIServer/system/Materializers/StreamSupervisor-0/flow-8-0-detacher] Aborting tcp connection to /127.0.0.1:56142 because of upstream failure: akka.stream.scaladsl.TcpIdleTimeoutException: TCP idle-timeout encountered on connection to [/127.0.0.1:56142], no bytes passed in the last 1 minute
[info] [DEBUG] [02/11/2020 22:11:46.985] [ConsoleAPIServer-akka.actor.default-dispatcher-13] [TcpConnectionStage$TcpStreamLogic(akka://ConsoleAPIServer)] Aborting connection from 127.0.0.1:56142 because of downstream failure: java.util.concurrent.TimeoutException: No elements passed in the last 1 minute.
[info] [DEBUG] [02/11/2020 22:11:46.987] [ConsoleAPIServer-akka.actor.internal-dispatcher-17] [akka://ConsoleAPIServer/system/IO-TCP/selectors/$a/0] New connection accepted
[info] [DEBUG] [02/11/2020 22:11:46.994] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForNetworkToSendControlFrames
[info] [DEBUG] [02/11/2020 22:11:46.994] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from WaitingForNetworkToSendControlFrames to Idle
[info] [DEBUG] [02/11/2020 22:11:46.995] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForData
[info] [DEBUG] [02/11/2020 22:11:46.995] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from WaitingForData to Idle
[info] [DEBUG] [02/11/2020 22:11:46.995] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [Http2ServerDemux(akka://ConsoleAPIServer)] Changing state from Idle to WaitingForData
[info] [DEBUG] [02/11/2020 22:11:46.996] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [Http2ServerDemux(akka://ConsoleAPIServer)] Got unhandled event SettingsAckFrame(List())
[info] [DEBUG] [02/11/2020 22:12:47.193] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [akka://ConsoleAPIServer/system/Materializers/StreamSupervisor-0/flow-10-0-detacher] Aborting tcp connection to /127.0.0.1:56144 because of upstream failure: akka.stream.scaladsl.TcpIdleTimeoutException: TCP idle-timeout encountered on connection to [/127.0.0.1:56144], no bytes passed in the last 1 minute
[info] [DEBUG] [02/11/2020 22:12:47.194] [ConsoleAPIServer-akka.actor.default-dispatcher-4] [TcpConnectionStage$TcpStreamLogic(akka://ConsoleAPIServer)] Aborting connection from 127.0.0.1:56144 because of downstream failure: java.util.concurrent.TimeoutException: No elements passed in the last 1 minute.

This then keeps repeating, presumably the client is reconnecting but not sending anything.
What I did was:

Start the app
Send an empty message with grpcurl:
❯ grpcurl localhost:8080 com.lightbend.console.api.MetricsProtoService/GetMetrics
Failed to dial target host "localhost:8080": tls: first record does not look like a TLS handshake


Did the same again but this time with the -plaintext flag:
❯ grpcurl -plaintext localhost:8080 com.lightbend.console.api.MetricsProtoService/GetMetrics
^C

Duplicate issue #2977: dotnet-core 3.1 grpc client causes "Http2ServerDemux: handleOutgoingEnded received unexpectedly in state Closed"
Created: 2020-02-21T14:23:25Z
Hi,
I created a very simple, hello-world like akka-grpc server (akka 2.5.23, akka-http 10.1.11), supporting a single unary endpoint. If I call it from the akka-grpc client (or even nodejs), it works perfectly fine.
However calling from a very simple, vanilla dotnet-core 3.1 client fails about once every 5-10 tries, and akka-http logs:
Http2ServerDemux: handleOutgoingEnded received unexpectedly in state Closed. This indicates a bug in Akka HTTP, please report it to the issue tracker. "
The dotnet-core client gets:
Error starting gRPC call: An error occurred while sending the request
As far as I can see, my service endpoint implementation completes fine, the issue happens afterwards.
I noticed that the dotnet-core client sends different HTTP2 settings than the akka-grpc or nodejs one (e.g. it sends SETTINGS_HEADER_TABLE_SIZE(0) but it doesn't send SETTINGS_MAX_HEADER_LIST_SIZE(8192), SETTINGS_INITIAL_WINDOW_SIZE(1048576) and SETTINGS_MAX_CONCURRENT_STREAMS(0)).
I attached an example akka-grpc server and dotnet core client project.
To run the server: sbt run
To run the client

install dotnet-core 3.1: https://docs.microsoft.com/en-gb/dotnet/core/install/sdk?pivots=os-linux#download-and-manually-install
then dotnet run. After about 5-10 runs, it fails with this error.

akka-http DEBUG logs:
[info] 2020-02-21 13:48:57,979 [13:48:57.979UTC] [DEBUG] TcpListener: New connection accepted 
[info] 2020-02-21 13:48:57,994 [13:48:57.994UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForNetworkToSendControlFrames 
[info] 2020-02-21 13:48:57,994 [13:48:57.994UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForNetworkToSendControlFrames to Idle 
[info] 2020-02-21 13:48:57,995 [13:48:57.995UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[info] 2020-02-21 13:48:57,995 [13:48:57.995UTC] [DEBUG] Http2ServerDemux: Got 2 settings! 
[info] 2020-02-21 13:48:57,996 [13:48:57.995UTC] [DEBUG] Http2ServerDemux: Ignoring setting SETTINGS_ENABLE_PUSH -> 0 (in Demux) 
[info] 2020-02-21 13:48:58,007 [13:48:58.007UTC] [DEBUG] Http2ServerDemux: Ignoring setting SETTINGS_HEADER_TABLE_SIZE -> 0 (in Demux) 
[info] 2020-02-21 13:48:58,008 [13:48:58.007UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForData to Idle 
[info] 2020-02-21 13:48:58,008 [13:48:58.007UTC] [DEBUG] HeaderCompression$$anon$1: Applied SETTINGS_HEADER_TABLE_SIZE(0) in header compression 
[info] 2020-02-21 13:48:58,008 [13:48:58.007UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[info] 2020-02-21 13:48:58,008 [13:48:58.008UTC] [DEBUG] Http2ServerDemux: Updating outgoing connection window by 67043329 to 67108864 
[info] 2020-02-21 13:48:58,008 [13:48:58.008UTC] [DEBUG] Http2ServerDemux: Got unhandled event SettingsAckFrame(List()) 
[info] 2020-02-21 13:48:58,011 [13:48:58.011UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForData to Idle 
[info] 2020-02-21 13:48:58,012 [13:48:58.011UTC] [DEBUG] Http2ServerDemux: Received DATA 5 for stream [1], remaining window space now 65530, buffered: 5 
[info] 2020-02-21 13:48:58,012 [13:48:58.012UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[info] 2020-02-21 13:48:58,012 [13:48:58.012UTC] [DEBUG] Http2ServerDemux: Received DATA 6 for stream [1], remaining window space now 65524, buffered: 11 
[info] 2020-02-21 13:48:58,012 [13:48:58.012UTC] [DEBUG] Http2ServerDemux: Dispatched chunk of 11 for stream [1], remaining window space now 65524, buffered: 0 
[info] 2020-02-21 13:48:58,012 [13:48:58.012UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForData to Idle 
[info] 2020-02-21 13:48:58,012 [13:48:58.012UTC] [DEBUG] Http2ServerDemux: adjusting con-level window by 0, stream-level window by 446476, remaining window space now 512000, buffered: 0, remaining connection window space now 9999989, total buffered: 0 
[info] 2020-02-21 13:48:58,012 [13:48:58.012UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[info] 2020-02-21 13:48:58,012 [               ] [INFO ] TestGrpcServiceImpl: get(Input(hey!)) -> Output(hey! hey!) 
[info] 2020-02-21 13:48:58,013 [13:48:58.013UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForData to Idle 
[info] 2020-02-21 13:48:58,013 [13:48:58.013UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[info] 2020-02-21 13:48:58,014 [13:48:58.013UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForData to Idle 
[info] 2020-02-21 13:48:58,014 [13:48:58.014UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[info] 2020-02-21 13:48:58,016 [13:48:58.016UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForData to Idle 
[info] 2020-02-21 13:48:58,017 [13:48:58.017UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[info] 2020-02-21 13:48:58,017 [13:48:58.017UTC] [DEBUG] Http2ServerDemux: [1] buffered 33 bytes 
[info] 2020-02-21 13:48:58,017 [13:48:58.017UTC] [DEBUG] Http2ServerDemux: [1] sending 33 bytes, endStream = false, remaining buffer [0], remaining stream-level WINDOW [65502] 
[info] 2020-02-21 13:48:58,017 [13:48:58.017UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForData to Idle 
[info] 2020-02-21 13:48:58,018 [13:48:58.017UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[info] 2020-02-21 13:48:58,018 [13:48:58.018UTC] [DEBUG] Http2ServerDemux: [1] buffered 0 bytes 
[info] 2020-02-21 13:48:58,020 [13:48:58.019UTC] [WARN ] Http2ServerDemux: handleOutgoingEnded received unexpectedly in state Closed. This indicates a bug in Akka HTTP, please report it to the issue tracker. 
[info] 2020-02-21 13:48:58,020 [13:48:58.019UTC] [DEBUG] Http2ServerDemux: Changing state from WaitingForData to Idle 
[info] 2020-02-21 13:48:58,020 [13:48:58.019UTC] [DEBUG] Http2ServerDemux: Changing state from Idle to WaitingForData 
[error] 2020-02-21 13:48:58,020 [13:48:58.019UTC] [WARN ] Http2ServerDemux: handleOutgoingEnded received unexpectedly in state Closed. This indicates a bug in Akka HTTP, please report it to the issue tracker. 

Thanks for any help!

Closing comment (2020-04-06T10:59:38Z): This might have been a duplicate of #2957 - if so then it should now be fixed!