Repository: python-telegram-bot/python-telegram-bot
Pull request: Add `post_stop`
Closes #3454 


Problem statement:
[FEATURE] Send message in shutdown handler
### What kind of feature are you missing? Where do you notice a shortcoming of PTB?

I tried using the `post_shutdown` hook in the `Application` class to run some code like this:

```python
async def notify_shutdown(app: Application):
    await app.bot.send_message(chat_id=MY_USER_ID, text="<b>NOTE!</b> Bot shutting down", parse_mode=ParseMode.HTML)
```

Unfortunately an exception was thrown:

```
telegram.error.NetworkError: Unknown error in HTTP implementation: RuntimeError('This HTTPXRequest is not initialized!')
```

### Describe the solution you'd like

I'd like a pre-shutdown hook that runs before `Application.stop()` has completed. This would presumably allow me to send messages as part of the shutdown process.

### Describe alternatives you've considered

I haven't considered any alternatives, as this isn't critical functionality I need.

### Additional context

I'm running version `20.0a6`.

Here's the full stack trace. It's taken from journalctl output, so it also has timestamps and other context.

```
Jan 01 11:42:45 shacklebolt python[1042542]: 2023-01-01 11:42:45,423 - telegram.ext._application - INFO - Application is stopping. This might take a moment.
Jan 01 11:42:45 shacklebolt python[1042542]: 2023-01-01 11:42:45,423 - apscheduler.scheduler - INFO - Scheduler has been shut down
Jan 01 11:42:45 shacklebolt python[1042542]: 2023-01-01 11:42:45,434 - telegram.ext._application - INFO - Application.stop() complete
Jan 01 11:42:45 shacklebolt python[1042542]: Traceback (most recent call last):
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/request/_baserequest.py", line 276, in _request_wrapper
Jan 01 11:42:45 shacklebolt python[1042542]:     code, payload = await self.do_request(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/request/_httpxrequest.py", line 153, in do_request
Jan 01 11:42:45 shacklebolt python[1042542]:     raise RuntimeError("This HTTPXRequest is not initialized!")
Jan 01 11:42:45 shacklebolt python[1042542]: RuntimeError: This HTTPXRequest is not initialized!
Jan 01 11:42:45 shacklebolt python[1042542]: The above exception was the direct cause of the following exception:
Jan 01 11:42:45 shacklebolt python[1042542]: Traceback (most recent call last):
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/local/lib/telegram-bots/shacklebot-admin/app.py", line 183, in <module>
Jan 01 11:42:45 shacklebolt python[1042542]:     tg_app.run_polling()
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/ext/_application.py", line 670, in run_polling
Jan 01 11:42:45 shacklebolt python[1042542]:     return self.__run(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/ext/_application.py", line 869, in __run
Jan 01 11:42:45 shacklebolt python[1042542]:     loop.run_until_complete(self.post_shutdown(self))
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
Jan 01 11:42:45 shacklebolt python[1042542]:     return future.result()
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/local/lib/telegram-bots/shacklebot-admin/app.py", line 53, in notify_shutdown
Jan 01 11:42:45 shacklebolt python[1042542]:     await notify_known_users(app.bot, "Shacklebot Admin shutdown")
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/local/lib/telegram-bots/shacklebot-admin/app.py", line 45, in notify_known_users
Jan 01 11:42:45 shacklebolt python[1042542]:     await bot.send_message(chat_id=user_id, text=msg.strip(), parse_mode=ParseMode.HTML)
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/ext/_extbot.py", line 2474, in send_message
Jan 01 11:42:45 shacklebolt python[1042542]:     return await super().send_message(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/_bot.py", line 313, in decorator
Jan 01 11:42:45 shacklebolt python[1042542]:     result = await func(*args, **kwargs)  # skipcq: PYL-E1102
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/_bot.py", line 723, in send_message
Jan 01 11:42:45 shacklebolt python[1042542]:     return await self._send_message(  # type: ignore[return-value]
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/ext/_extbot.py", line 481, in _send_message
Jan 01 11:42:45 shacklebolt python[1042542]:     result = await super()._send_message(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/_bot.py", line 467, in _send_message
Jan 01 11:42:45 shacklebolt python[1042542]:     result = await self._post(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/_bot.py", line 395, in _post
Jan 01 11:42:45 shacklebolt python[1042542]:     return await self._do_post(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/ext/_extbot.py", line 306, in _do_post
Jan 01 11:42:45 shacklebolt python[1042542]:     return await super()._do_post(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/_bot.py", line 426, in _do_post
Jan 01 11:42:45 shacklebolt python[1042542]:     return await request.post(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/request/_baserequest.py", line 167, in post
Jan 01 11:42:45 shacklebolt python[1042542]:     result = await self._request_wrapper(
Jan 01 11:42:45 shacklebolt python[1042542]:   File "/usr/lib/python3.10/site-packages/telegram/request/_baserequest.py", line 292, in _request_wrapper
Jan 01 11:42:45 shacklebolt python[1042542]:     raise NetworkError(f"Unknown error in HTTP implementation: {repr(exc)}") from exc
Jan 01 11:42:45 shacklebolt python[1042542]: telegram.error.NetworkError: Unknown error in HTTP implementation: RuntimeError('This HTTPXRequest is not initialized!')
```

Hints from issue discussion:
Thanks for your feature request. I see the use case and would be happy to accept a contribution in this direction unless other team members have concerns. Regarding the naming, one could also call this hook `post_stop`, is it would run after `Application.stop()` and `post_*` aligns a bit better with `post_{init, shutdown}`.

Commit message: add post_stop

Patch to review:
diff --git a/telegram/ext/_application.py b/telegram/ext/_application.py
index 73703204ecd..04537d10472 100644
--- a/telegram/ext/_application.py
+++ b/telegram/ext/_application.py
@@ -204,6 +204,11 @@ class Application(Generic[BT, CCT, UD, CD, BD, JQ], AbstractAsyncContextManager)
         post_shutdown (:term:`coroutine function`): Optional. A callback that will be executed by
             :meth:`Application.run_polling` and :meth:`Application.run_webhook` after shutting down
             the application via :meth:`shutdown`.
+        post_stop (:term:`coroutine function`): Optional. A callback that will be executed by
+            :meth:`Application.run_polling` and :meth:`Application.run_webhook` after stopping
+            the application via :meth:`stop`.
+
+            ..versionadded:: 21.0
 
     """
 
@@ -236,6 +241,7 @@ class Application(Generic[BT, CCT, UD, CD, BD, JQ], AbstractAsyncContextManager)
         "persistence",
         "post_init",
         "post_shutdown",
+        "post_stop",
         "update_queue",
         "updater",
         "user_data",
@@ -257,6 +263,9 @@ def __init__(
         post_shutdown: Optional[
             Callable[["Application[BT, CCT, UD, CD, BD, JQ]"], Coroutine[Any, Any, None]]
         ],
+        post_stop: Optional[
+            Callable[["Application[BT, CCT, UD, CD, BD, JQ]"], Coroutine[Any, Any, None]]
+        ],
     ):
         if not was_called_by(
             inspect.currentframe(), Path(__file__).parent.resolve() / "_applicationbuilder.py"
@@ -274,6 +283,7 @@ def __init__(
         self.error_handlers: Dict[Callable, Union[bool, DefaultValue]] = {}
         self.post_init = post_init
         self.post_shutdown = post_shutdown
+        self.post_stop = post_stop
 
         if isinstance(concurrent_updates, int) and concurrent_updates < 0:
             raise ValueError("`concurrent_updates` must be a non-negative integer!")
@@ -568,6 +578,9 @@ async def stop(self) -> None:
             :meth:`telegram.ext.Updater.stop` or use one of :meth:`run_polling` or
             :meth:`run_webhook`.
 
+        Does *not* call :attr:`post_stop` - that is only done by :meth:`run_polling` and
+        :meth:`run_webhook`.
+
         Raises:
             :exc:`RuntimeError`: If the application is not running.
         """
@@ -630,6 +643,9 @@ def run_polling(
         If :attr:`post_shutdown` is set, it will be called after both :meth:`shutdown`
         and :meth:`telegram.ext.Updater.shutdown`.
 
+        If :attr:`post_stop` is set, it will be called between :meth:`stop` and
+        :meth:`telegram.ext.Updater.stop`.
+
         .. include:: inclusions/application_run_tip.rst
 
         .. seealso::
@@ -746,6 +762,9 @@ def run_webhook(
         If :attr:`post_shutdown` is set, it will be called after both :meth:`shutdown`
         and :meth:`telegram.ext.Updater.shutdown`.
 
+        If :attr:`post_stop` is set, it will be called between :meth:`stop` and
+        :meth:`telegram.ext.Updater.stop`.
+
         Important:
             If you want to use this method, you must install PTB with the optional requirement
             ``webhooks``, i.e.
@@ -887,6 +906,8 @@ def __run(
                     loop.run_until_complete(self.updater.stop())  # type: ignore[union-attr]
                 if self.running:
                     loop.run_until_complete(self.stop())
+                if self.post_stop:
+                    loop.run_until_complete(self.post_stop(self))
                 loop.run_until_complete(self.shutdown())
                 if self.post_shutdown:
                     loop.run_until_complete(self.post_shutdown(self))
diff --git a/telegram/ext/_applicationbuilder.py b/telegram/ext/_applicationbuilder.py
index 999050d9a47..46cd20e0607 100644
--- a/telegram/ext/_applicationbuilder.py
+++ b/telegram/ext/_applicationbuilder.py
@@ -142,6 +142,7 @@ class ApplicationBuilder(Generic[BT, CCT, UD, CD, BD, JQ]):
         "_pool_timeout",
         "_post_init",
         "_post_shutdown",
+        "_post_stop",
         "_private_key",
         "_private_key_password",
         "_proxy_url",
@@ -196,6 +197,7 @@ def __init__(self: "InitApplicationBuilder"):
         self._updater: ODVInput[Updater] = DEFAULT_NONE
         self._post_init: Optional[Callable[[Application], Coroutine[Any, Any, None]]] = None
         self._post_shutdown: Optional[Callable[[Application], Coroutine[Any, Any, None]]] = None
+        self._post_stop: Optional[Callable[[Application], Coroutine[Any, Any, None]]] = None
         self._rate_limiter: ODVInput["BaseRateLimiter"] = DEFAULT_NONE
 
     def _build_request(self, get_updates: bool) -> BaseRequest:
@@ -301,6 +303,7 @@ def build(
             context_types=DefaultValue.get_value(self._context_types),
             post_init=self._post_init,
             post_shutdown=self._post_shutdown,
+            post_stop=self._post_stop,
             **self._application_kwargs,  # For custom Application subclasses
         )
 
@@ -1016,6 +1019,42 @@ async def post_shutdown(application: Application) -> None:
         self._post_shutdown = post_shutdown
         return self
 
+    def post_stop(
+        self: BuilderType, post_stop: Callable[[Application], Coroutine[Any, Any, None]]
+    ) -> BuilderType:
+        """
+        Sets a callback to be executed by :meth:`Application.run_polling` and
+        :meth:`Application.run_webhook` *after* executing :meth:`Updater.stop`
+        and :meth:`Application.stop`.
+
+        Tip:
+            This can be used for custom stop logic that requires to await coroutines, e.g.
+            sending message to a chat before shutting down the bot
+
+        Example:
+            .. code::
+
+                async def post_stop(application: Application) -> None:
+                    await application.bot.send_message(123456, "Shutting down...")
+
+                application = Application.builder()
+                                        .token("TOKEN")
+                                        .post_stop(post_stop)
+                                        .build()
+
+        Args:
+            post_stop (:term:`coroutine function`): The custom callback. Must be a
+                :term:`coroutine function` and must accept exactly one positional argument, which
+                is the :class:`~telegram.ext.Application`::
+
+                    async def post_stop(application: Application) -> None:
+
+        Returns:
+            :class:`ApplicationBuilder`: The same builder with the updated argument.
+        """
+        self._post_stop = post_stop
+        return self
+
     def rate_limiter(
         self: "ApplicationBuilder[BT, CCT, UD, CD, BD, JQ]",
         rate_limiter: "BaseRateLimiter[RLARGS]",
diff --git a/tests/test_application.py b/tests/test_application.py
index e66e7825cdc..9b10bbb0284 100644
--- a/tests/test_application.py
+++ b/tests/test_application.py
@@ -131,6 +131,7 @@ def test_manual_init_warning(self, recwarn, updater):
             concurrent_updates=False,
             post_init=None,
             post_shutdown=None,
+            post_stop=None,
         )
         assert len(recwarn) == 1
         assert (
@@ -156,6 +157,9 @@ async def post_init(application: Application) -> None:
         async def post_shutdown(application: Application) -> None:
             pass
 
+        async def post_stop(application: Application) -> None:
+            pass
+
         app = Application(
             bot=bot,
             update_queue=update_queue,
@@ -166,6 +170,7 @@ async def post_shutdown(application: Application) -> None:
             concurrent_updates=concurrent_updates,
             post_init=post_init,
             post_shutdown=post_shutdown,
+            post_stop=post_stop,
         )
         assert app.bot is bot
         assert app.update_queue is update_queue
@@ -178,6 +183,7 @@ async def post_shutdown(application: Application) -> None:
         assert app.concurrent_updates == expected
         assert app.post_init is post_init
         assert app.post_shutdown is post_shutdown
+        assert app.post_stop is post_stop
 
         # These should be done by the builder
         assert app.persistence.bot is None
@@ -199,6 +205,7 @@ async def post_shutdown(application: Application) -> None:
                 concurrent_updates=-1,
                 post_init=None,
                 post_shutdown=None,
+                post_stop=None,
             )
 
     def test_job_queue(self, bot, app, recwarn):
@@ -1501,6 +1508,51 @@ async def post_shutdown(app: Application) -> None:
             "post_shutdown",
         ], "Wrong order of events detected!"
 
+    @pytest.mark.skipif(
+        platform.system() == "Windows",
+        reason="Can't send signals without stopping whole process on windows",
+    )
+    def test_run_polling_post_stop(self, bot, monkeypatch):
+        events = []
+
+        async def get_updates(*args, **kwargs):
+            # This makes sure that other coroutines have a chance of running as well
+            await asyncio.sleep(0)
+            return []
+
+        def thread_target():
+            waited = 0
+            while not app.running:
+                time.sleep(0.05)
+                waited += 0.05
+                if waited > 5:
+                    pytest.fail("App apparently won't start")
+
+            os.kill(os.getpid(), signal.SIGINT)
+
+        async def post_stop(app: Application) -> None:
+            events.append("post_stop")
+
+        app = Application.builder().token(bot.token).post_stop(post_stop).build()
+        app.bot._unfreeze()
+        monkeypatch.setattr(app.bot, "get_updates", get_updates)
+        monkeypatch.setattr(app, "stop", call_after(app.stop, lambda _: events.append("stop")))
+        monkeypatch.setattr(
+            app.updater,
+            "stop",
+            call_after(app.updater.stop, lambda _: events.append("updater.stop")),
+        )
+
+        thread = Thread(target=thread_target)
+        thread.start()
+        app.run_polling(drop_pending_updates=True, close_loop=False)
+        thread.join()
+        assert events == [
+            "updater.stop",
+            "stop",
+            "post_stop",
+        ], "Wrong order of events detected!"
+
     @pytest.mark.skipif(
         platform.system() == "Windows",
         reason="Can't send signals without stopping whole process on windows",
@@ -1753,6 +1805,60 @@ async def post_shutdown(app: Application) -> None:
             "post_shutdown",
         ], "Wrong order of events detected!"
 
+    def test_run_webhook_post_stop(self, bot, monkeypatch):
+        events = []
+
+        async def delete_webhook(*args, **kwargs):
+            return True
+
+        async def set_webhook(*args, **kwargs):
+            return True
+
+        async def get_updates(*args, **kwargs):
+            # This makes sure that other coroutines have a chance of running as well
+            await asyncio.sleep(0)
+            return []
+
+        def thread_target():
+            waited = 0
+            while not app.running:
+                time.sleep(0.05)
+                waited += 0.05
+                if waited > 5:
+                    pytest.fail("App apparently won't start")
+
+            os.kill(os.getpid(), signal.SIGINT)
+
+        async def post_stop(app: Application) -> None:
+            events.append("post_stop")
+
+        app = Application.builder().token(bot.token).post_stop(post_stop).build()
+        app.bot._unfreeze()
+        monkeypatch.setattr(app.bot, "set_webhook", set_webhook)
+        monkeypatch.setattr(app.bot, "delete_webhook", delete_webhook)
+        monkeypatch.setattr(app, "stop", call_after(app.stop, lambda _: events.append("stop")))
+        monkeypatch.setattr(
+            app.updater,
+            "stop",
+            call_after(app.updater.stop, lambda _: events.append("updater.stop")),
+        )
+
+        thread = Thread(target=thread_target)
+        thread.start()
+
+        ip = "127.0.0.1"
+        port = randrange(1024, 49152)
+
+        app.run_webhook(
+            ip_address=ip,
+            port=port,
+            url_path="TOKEN",
+            drop_pending_updates=True,
+            close_loop=False,
+        )
+        thread.join()
+        assert events == ["updater.stop", "stop", "post_stop"], "Wrong order of events detected!"
+
     @pytest.mark.skipif(
         platform.system() == "Windows",
         reason="Can't send signals without stopping whole process on windows",
diff --git a/tests/test_applicationbuilder.py b/tests/test_applicationbuilder.py
index 4d0b40dd6c3..3f1996bd72b 100644
--- a/tests/test_applicationbuilder.py
+++ b/tests/test_applicationbuilder.py
@@ -135,6 +135,7 @@ class Client:
         assert app.persistence is None
         assert app.post_init is None
         assert app.post_shutdown is None
+        assert app.post_stop is None
 
     @pytest.mark.parametrize(
         "method, description", _BOT_CHECKS, ids=[entry[0] for entry in _BOT_CHECKS]
@@ -361,6 +362,9 @@ async def post_init(app: Application) -> None:
         async def post_shutdown(app: Application) -> None:
             pass
 
+        async def post_stop(app: Application) -> None:
+            pass
+
         app = (
             builder.token(bot.token)
             .job_queue(job_queue)
@@ -370,6 +374,7 @@ async def post_shutdown(app: Application) -> None:
             .concurrent_updates(concurrent_updates)
             .post_init(post_init)
             .post_shutdown(post_shutdown)
+            .post_stop(post_stop)
             .arbitrary_callback_data(True)
         ).build()
         assert app.job_queue is job_queue
@@ -383,6 +388,7 @@ async def post_shutdown(app: Application) -> None:
         assert app.concurrent_updates == concurrent_updates
         assert app.post_init is post_init
         assert app.post_shutdown is post_shutdown
+        assert app.post_stop is post_stop
         assert isinstance(app.bot.callback_data_cache, CallbackDataCache)
 
         updater = Updater(bot=bot, update_queue=update_queue)


Code review comments:
diff_hunk: @@ -204,6 +204,11 @@ class Application(Generic[BT, CCT, UD, CD, BD, JQ], AbstractAsyncContextManager)
         post_shutdown (:term:`coroutine function`): Optional. A callback that will be executed by
             :meth:`Application.run_polling` and :meth:`Application.run_webhook` after shutting down
             the application via :meth:`shutdown`.
+        post_stop (:term:`coroutine function`): Optional. A callback that will be executed by
+            :meth:`Application.run_polling` and :meth:`Application.run_webhook` after stopping
+            the application via :meth:`stop`.
+
+            ..versionadded:: 21.0
path: telegram/ext/_application.py
text: @user1:
```suggestion
            .. versionadded:: 21.0
```

@user2:
I don't think that we want to make the next release breaking again, so the next version will be 20.1 :D

@author:
Alright!