Repository: sopel-irc/sopel

Issue:
Bot should properly handle ConnectionResetError
### Description

Tin. We've given a lot of attention to things like timeouts, but Sopel still spews "critical exception in core" if its IRC connection is reset.

### Reproduction steps

1. Connect to a network
2. Trigger a connection reset using your favorite network debugging tool (e.g. tcpkill)
3. Sopel spews a "critical exception in core" traceback for the `ConnectionResetError` generated by this event

### Expected behavior

Sopel gracefully handles the connection loss and prepares to reconnect, without hitting the critical-exception handler and logging a traceback, because connection resets just happen sometimes.

### Relevant logs

```shell
[2022-09-24 14:23:08,709] sopel.exceptions     ERROR    - Critical exception in core
Traceback (most recent call last):
  File "/home/leisuresuitlarry/sopel/sopel/cli/run.py", line 78, in run
    p.run(settings.core.host, int(settings.core.port))
  File "/home/leisuresuitlarry/sopel/sopel/irc/__init__.py", line 259, in run
    self.backend.run_forever()
  File "/home/leisuresuitlarry/sopel/sopel/irc/backends.py", line 352, in run_forever
    asyncio.run(self._run_forever())
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/leisuresuitlarry/sopel/sopel/irc/backends.py", line 332, in _run_forever
    await self._read_task
  File "/home/leisuresuitlarry/sopel/sopel/irc/backends.py", line 229, in read_forever
    line: bytes = await self._reader.readuntil(separator=b'\r\n')
  File "/usr/lib/python3.9/asyncio/streams.py", line 632, in readuntil
    await self._wait_for_data('readuntil')
  File "/usr/lib/python3.9/asyncio/streams.py", line 517, in _wait_for_data
    await self._waiter
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
ConnectionResetError: [Errno 104] Connection reset by peer
[2022-09-24 14:23:08,715] sopel.exceptions     ERROR    - ----------------------------------------
```

### Notes

Writing this up after a ping from deadyeti on IRC. May have more logs from an in-the-wild instance where this happens if they share the results of turning up the log level with me/us in channel.

### Sopel version

Recent master, exact commit not known

### Installation method

`pip install`

### Python version

_No response_

### Operating system

_No response_

### IRCd

_No response_

### Relevant plugins

_No response_


Discussion:
Seems that WHO polling also needs to check if the connection is up, or better yet disable those timed jobs if the connection is lost.

<details>
<summary>WHO polling errors while disconnected</summary>

```shell
[2022-09-26 15:31:42,410] sopel.tools.jobs     ERROR    - Error while processing job: Event loop is closed
Exception in thread Thread-47734:
Traceback (most recent call last):
  File "/home/leisuresuitlarry/sopel/sopel/tools/jobs.py", line 190, in _call
    job.execute(self.manager)
  File "/home/leisuresuitlarry/sopel/sopel/tools/jobs.py", line 462, in execute
    return self._handler(manager)
  File "/home/leisuresuitlarry/sopel/sopel/coretasks.py", line 826, in _periodic_send_who
    _send_who(bot, selected_channel)
  File "/home/leisuresuitlarry/sopel/sopel/coretasks.py", line 791, in _send_who
    bot.write(['WHO', channel, 'a%nuachtf,' + CORE_QUERYTYPE])
  File "/home/leisuresuitlarry/sopel/sopel/irc/__init__.py", line 561, in write
    self.backend.send_command(*args, text=text)
  File "/home/leisuresuitlarry/sopel/sopel/irc/abstract_backends.py", line 104, in send_command
    self.irc_send(raw_command.encode('utf-8'))
  File "/home/leisuresuitlarry/sopel/sopel/irc/backends.py", line 175, in irc_send
    asyncio.run_coroutine_threadsafe(self.send(data), self._loop)
  File "/usr/lib/python3.9/asyncio/tasks.py", line 931, in run_coroutine_threadsafe
    loop.call_soon_threadsafe(callback)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 791, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib/python3.9/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/home/leisuresuitlarry/sopel/sopel/tools/jobs.py", line 193, in _call
    self.manager.on_job_error(self, job, error)
  File "/home/leisuresuitlarry/sopel/sopel/bot.py", line 917, in on_job_error
    self.error(exception=exc)
  File "/home/leisuresuitlarry/sopel/sopel/bot.py", line 940, in error
    LOGGER.exception(message)
NameError: name 'message' is not defined
/usr/lib/python3.9/threading.py:956: RuntimeWarning: coroutine 'AsyncioBackend.send' was never awaited
  self._invoke_excepthook(self)
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
[2022-09-26 15:32:12,444] sopel.tools.jobs     ERROR    - Error while processing job: Event loop is closed
Exception in thread Thread-47738:
Traceback (most recent call last):
  File "/home/leisuresuitlarry/sopel/sopel/tools/jobs.py", line 190, in _call
    job.execute(self.manager)
  File "/home/leisuresuitlarry/sopel/sopel/tools/jobs.py", line 462, in execute
    return self._handler(manager)
  File "/home/leisuresuitlarry/sopel/sopel/coretasks.py", line 826, in _periodic_send_who
    _send_who(bot, selected_channel)
  File "/home/leisuresuitlarry/sopel/sopel/coretasks.py", line 791, in _send_who
    bot.write(['WHO', channel, 'a%nuachtf,' + CORE_QUERYTYPE])
  File "/home/leisuresuitlarry/sopel/sopel/irc/__init__.py", line 561, in write
    self.backend.send_command(*args, text=text)
  File "/home/leisuresuitlarry/sopel/sopel/irc/abstract_backends.py", line 104, in send_command
    self.irc_send(raw_command.encode('utf-8'))
  File "/home/leisuresuitlarry/sopel/sopel/irc/backends.py", line 175, in irc_send
    asyncio.run_coroutine_threadsafe(self.send(data), self._loop)
  File "/usr/lib/python3.9/asyncio/tasks.py", line 931, in run_coroutine_threadsafe
    loop.call_soon_threadsafe(callback)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 791, in call_soon_threadsafe
    self._check_closed()
  File "/usr/lib/python3.9/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/home/leisuresuitlarry/sopel/sopel/tools/jobs.py", line 193, in _call
    self.manager.on_job_error(self, job, error)
  File "/home/leisuresuitlarry/sopel/sopel/bot.py", line 917, in on_job_error
    self.error(exception=exc)
  File "/home/leisuresuitlarry/sopel/sopel/bot.py", line 940, in error
    LOGGER.exception(message)
NameError: name 'message' is not defined
```
</details>

(Also peep the secondary exception (`NameError`)… `message` is defined locally inside that method, so wtf?)

Resolving patch:
diff --git a/sopel/irc/backends.py b/sopel/irc/backends.py
--- a/sopel/irc/backends.py
+++ b/sopel/irc/backends.py
@@ -313,9 +313,9 @@ async def read_forever(self) -> None:
         while not self._reader.at_eof():
             try:
                 line: bytes = await self._reader.readuntil(separator=b'\r\n')
-            except asyncio.exceptions.IncompleteReadError as e:
+            except asyncio.exceptions.IncompleteReadError as err:
                 LOGGER.warning('Receiving partial message from IRC.')
-                line = e.partial
+                line = err.partial
             except asyncio.exceptions.LimitOverrunError:
                 LOGGER.exception('Unable to read from IRC server.')
                 break
@@ -500,8 +500,25 @@ async def _run_forever(self) -> None:
         self._read_task = asyncio.create_task(self.read_forever())
         try:
             await self._read_task
+
+        # task was cancelled, i.e. another exception is responsible for that
         except asyncio.CancelledError:
             LOGGER.debug('Read task was cancelled.')
+
+        # connection reset requires a log, but no exception log
+        except ConnectionResetError as err:
+            LOGGER.error('Connection reset on read: %s', err)
+
+        # generic (connection) error requires a specific exception log
+        except ConnectionError as err:
+            LOGGER.error('Connection error on read: %s', err)
+            self.log_exception()
+
+        except Exception as err:
+            LOGGER.error('Unexpected error on read: %s', err)
+            self.log_exception()
+
+        # task done (connection closed without error)
         else:
             LOGGER.debug('Reader received EOF.')
 
@@ -514,7 +531,26 @@ async def _run_forever(self) -> None:
         # nothing to read anymore
         LOGGER.debug('Shutting down writer.')
         self._writer.close()
-        await self._writer.wait_closed()
+        try:
+            await self._writer.wait_closed()
+
+        # task was cancelled, i.e. another exception is responsible for that
+        except asyncio.CancelledError:
+            LOGGER.debug('Writer task was cancelled.')
+
+        # connection reset happened before on the read task
+        except ConnectionResetError as err:
+            LOGGER.debug('Connection reset while closing: %s', err)
+
+        # generic (connection) error requires a specific exception log
+        except ConnectionError as err:
+            LOGGER.error('Connection error while closing: %s', err)
+            self.log_exception()
+
+        except Exception:
+            LOGGER.error('Unexpected error while shutting down socket.')
+            self.log_exception()
+
         LOGGER.debug('All clear, exiting now.')
 
     def run_forever(self) -> None:


Test patch: