opened
FelixMcFelix
username_0
## Description

This removes a `let _ = ...` which was silently causing some resume failures to become permanent, causing the shard to crash the next time it attempted to use its (closed) WS client. This is replaced with code to trigger a further resume and/or reidentification attempt.

In the worst case, this will attempt one further resume before "promoting" the resume into a full reidentify. The `ShardQueuer` is then already responsible for repeated full reidentify attempts from that point on.

## Type of Change

This fixes a (rare) bug in the shard runner triggered when TLS or socket initialisation spuriously fails.

## How Has This Been Tested?

This has been tested by (locally) adding in extra `ShardRunnerMessage` types to test and access the new code paths individually. The underlying bug has not been observed (using the new `warn!`/`debug!` invocations); I haven't seen this bug reoccur in over two weeks, in spite of it being a nightly occurrence around that time (most likely due to my hosting provider).
Reconnect in event of failed ShardActions
issue
created
FelixMcFelix
username_0
This seems to now catch and correctly act on serial failure states as required. I observed this in my logs this morning:

```
Mar 16 07:09:58.139 DEBUG run: serenity::client::bridge::gateway::shard_runner: [ShardRunner [0, 1]] Reconnecting due to error performing Reconnect(Resume): Tungstenite(Io(Custom { kind: Other, error: Ssl(Error { code: ErrorCode(5), cause: None }, X509VerifyResult { code: 0, error: "ok" }) }))
Mar 16 07:09:58.145 DEBUG run:resume: serenity::gateway::shard: [Shard [0, 1]] Attempting to resume
Mar 16 07:09:58.145 DEBUG run:resume:initialize: serenity::gateway::shard: [Shard [0, 1]] Initializing.
Mar 16 07:10:32.111  WARN run: serenity::client::bridge::gateway::shard_runner: [ShardRunner [0, 1]] Resume failed, reidentifying: Tungstenite(Io(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }))
Mar 16 07:10:32.115 DEBUG run:request_restart: serenity::client::bridge::gateway::shard_runner: [ShardRunner [0, 1]] Requesting restart
Mar 16 07:10:32.119 DEBUG serenity::client::bridge::gateway::shard_queuer: [ShardRunner [0, 1]] Stopping
Mar 16 07:10:32.134  INFO start:start_connection{shard_data=[0, 0, 1]}:run:restart{shard_id=ShardId(0)}: serenity::client::bridge::gateway::shard_manager: Restarting shard 0
Mar 16 07:10:32.138  INFO start:start_connection{shard_data=[0, 0, 1]}:run:restart{shard_id=ShardId(0)}:shutdown{shard_id=ShardId(0) code=4000}: serenity::client::bridge::gateway::shard_manager: Shutting down shard 0
Mar 16 07:10:32.138 DEBUG run: serenity::client::bridge::gateway::shard_queuer: [Shard Queuer] Received to shutdown shard 0 with 4000.
Mar 16 07:10:32.139  INFO run:shutdown{shard_id=ShardId(0) code=4000}: serenity::client::bridge::gateway::shard_queuer: Shutting down shard 0
Mar 16 07:10:32.139  WARN run:shutdown{shard_id=ShardId(0) code=4000}: serenity::client::bridge::gateway::shard_queuer: Failed to cleanly shutdown shard 0 when sending message to shard runner: TrySendError { kind: Disconnected }
Mar 16 07:10:37.139  WARN start:start_connection{shard_data=[0, 0, 1]}:run:restart{shard_id=ShardId(0)}:shutdown{shard_id=ShardId(0) code=4000}: serenity::client::bridge::gateway::shard_manager: Failed to cleanly shutdown shard 0, reached timeout: Elapsed(())
Mar 16 07:10:37.142  INFO start:start_connection{shard_data=[0, 0, 1]}:run:restart{shard_id=ShardId(0)}:boot{shard_info=[ShardId(0), ShardId(1)]}: serenity::client::bridge::gateway::shard_manager: Telling shard queuer to start shard 0
Mar 16 07:10:37.142 DEBUG run: serenity::client::bridge::gateway::shard_queuer: [Shard Queuer] Received to start shard 0 of 1.                                                                                   Mar 16 07:10:37.142 DEBUG run:checked_start{id=0 total=1}: serenity::client::bridge::gateway::shard_queuer: [Shard Queuer] Checked start for shard 0 out of 1
Mar 16 07:10:37.518 DEBUG run:checked_start{id=0 total=1}:start{shard_id=0 shard_total=1}:create_native_tls_client{url=Url { scheme: "wss", username: "", password: None, host: Some(Domain("gateway.discord.gg")), port: None, path: "/", query: Some("v=8"), fragment: None }}: tungstenite::handshake::client: Client handshake done.
Mar 16 07:10:37.519  INFO run: serenity::client::bridge::gateway::shard_runner: [ShardRunner [0, 1]] Running
```

This PR doesn't deal with the transfer of any unhandled `Intermessage`s between `ShardRunner`s after a full restart, which affects songbird somewhat. I think the best method to handle this is to put a (configurable) timeout on those connection attempts for now, as that requires less engineering work.
comment