Repository: python-telegram-bot/python-telegram-bot

Issue:
[BUG] Updated user data of a different user is not propagated to persistence
### Steps to Reproduce

```py
async def callback(update, context):
  # current_user_id = update.effective_user.id
  # other_user_id = httpx.get("network/call")
  # this is updated in persistence
  context.user_data['key'] = value
  # this is not updated in persistence
  context.application.user_data[other_user_id]['key'] = 'value'
```

### Expected behaviour

`user_data` corresponding to `other_user_id` should be updated in persistence

### Actual behaviour

only `user_data` corresponding to `current_user_id` is updated in persistence

### Operating System

ubuntu 22.04

### Version of Python, python-telegram-bot & dependencies

```shell
python-telegram-bot 20.0
Bot API 6.4
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0]
```


### Relevant log output

_No response_

### Additional Context

I believe this error exists because of the following. Only `update.effective_user.id` is marked for persistence update.

https://github.com/python-telegram-bot/python-telegram-bot/blob/800598ced461189d03868267d628291a9f514a5a/telegram/ext/_application.py#L1363-L1368


Discussion:
One might argue that I could use `bot_data` to store user related information. I guess that would be updated in persistence, but I really think that information specific to a user should be stored in `user_data`.
> One might argue that I could use bot_data to store user related information. I guess that would be updated in persistence

Hi. I would argue exactly that. It will get updated if another update from that user triggers. If we update all user_datas like you suggest, it would mean a potential huge performance hit.

But can't you just call `context.application.persistence.update_chat_data(other_user_id, data)` in the code if you need it done right away/to ensure nothing is lost?

https://docs.python-telegram-bot.org/en/stable/telegram.ext.basepersistence.html#telegram.ext.BasePersistence.update_chat_data

@Poolitzer I agree with Pool on the bot_data topic.

However, manually calling the persistence interface has potential of interfering with his the Application is managing the persistence. I strongly advise against it.
Note that the current favorite solution for #3554 would allow users to manually tell the application to also update the other users data on the next run.
@Poolitzer I cannot rely on another update coming from the user. Consider a use case when I ban a user and store the banned time in user_data. Since I've banned the user, there's no good reason to expect another update from the user and in that case I would never save the banned time in persistence. That being said, I'm following your suggestion right now:
```py
await context.application.persistence.update_user_data(other_user_id, {'key': 'value'})
```

@Bibo-Joshi Good to see that giving users fine-grained control over what data to persist is already being thought of. Until that makes its way into a PR, I'm happy using the current workaround. It's not much of a performance hit in my use case.

I also want to point out that I tried calling the following from within my callback but this didn't seem to work.
```py
context.application._chat_ids_to_be_updated_in_persistence.add(other_user_id)
``` 

Resolving patch:
diff --git a/telegram/ext/_application.py b/telegram/ext/_application.py
--- a/telegram/ext/_application.py
+++ b/telegram/ext/_application.py
@@ -53,7 +53,7 @@
 
 from telegram._update import Update
 from telegram._utils.defaultvalue import DEFAULT_NONE, DEFAULT_TRUE, DefaultValue
-from telegram._utils.types import DVType, ODVInput
+from telegram._utils.types import SCT, DVType, ODVInput
 from telegram._utils.warnings import warn
 from telegram.error import TelegramError
 from telegram.ext._basepersistence import BasePersistence
@@ -1373,6 +1373,36 @@ def _mark_for_persistence_update(self, *, update: object = None, job: "Job" = No
             if job.user_id:
                 self._user_ids_to_be_updated_in_persistence.add(job.user_id)
 
+    def mark_data_for_update_persistence(
+        self, chat_ids: SCT[int] = None, user_ids: SCT[int] = None
+    ) -> None:
+        """Mark entries of :attr:`chat_data` and :attr:`user_data` to be updated on the next
+        run of :meth:`update_persistence`.
+
+        Tip:
+            Use this method sparingly. If you have to use this method, it likely means that you
+            access and modify ``context.application.chat/user_data[some_id]`` within a callback.
+            Note that for data which should be available globally in all handler callbacks
+            independent of the chat/user, it is recommended to use :attr:`bot_data` instead.
+
+        .. versionadded:: NEXT.VERSION
+
+        Args:
+            chat_ids (:obj:`int` | Collection[:obj:`int`], optional): Chat IDs to mark.
+            user_ids (:obj:`int` | Collection[:obj:`int`], optional): User IDs to mark.
+
+        """
+        if chat_ids:
+            if isinstance(chat_ids, int):
+                self._chat_ids_to_be_updated_in_persistence.add(chat_ids)
+            else:
+                self._chat_ids_to_be_updated_in_persistence.update(chat_ids)
+        if user_ids:
+            if isinstance(user_ids, int):
+                self._user_ids_to_be_updated_in_persistence.add(user_ids)
+            else:
+                self._user_ids_to_be_updated_in_persistence.update(user_ids)
+
     async def _persistence_updater(self) -> None:
         # Update the persistence in regular intervals. Exit only when the stop event has been set
         while not self.__update_persistence_event.is_set():
@@ -1399,8 +1429,9 @@ async def update_persistence(self) -> None:
         along with :attr:`~telegram.ext.ExtBot.callback_data_cache` and the conversation states of
         any persistent :class:`~telegram.ext.ConversationHandler` registered for this application.
 
-        For :attr:`user_data`, :attr:`chat_data`, only entries used since the last run of this
-        method are updated.
+        For :attr:`user_data` and :attr:`chat_data`, only those entries are updated which either
+        were used or have been manually marked via :meth:`mark_data_for_update_persistence` since
+        the last run of this method.
 
         Tip:
             This method will be called in regular intervals by the application. There is usually
@@ -1410,7 +1441,8 @@ async def update_persistence(self) -> None:
             Any data is deep copied with :func:`copy.deepcopy` before handing it over to the
             persistence in order to avoid race conditions, so all persisted data must be copyable.
 
-        .. seealso:: :attr:`telegram.ext.BasePersistence.update_interval`.
+        .. seealso:: :attr:`telegram.ext.BasePersistence.update_interval`,
+            :meth:`mark_data_for_update_persistence`
         """
         async with self.__update_persistence_lock:
             await self.__update_persistence()
diff --git a/telegram/ext/_basepersistence.py b/telegram/ext/_basepersistence.py
--- a/telegram/ext/_basepersistence.py
+++ b/telegram/ext/_basepersistence.py
@@ -357,6 +357,11 @@ async def refresh_user_data(self, user_id: int, user_data: UD) -> None:
         :attr:`~telegram.ext.Application.user_data` to a callback. Can be used to update data
         stored in :attr:`~telegram.ext.Application.user_data` from an external source.
 
+        Warning:
+            When using :meth:`~telegram.ext.ApplicationBuilder.concurrent_updates`, this method
+            may be called while a handler callback is still running. This might lead to race
+            conditions.
+
         .. versionadded:: 13.6
 
         .. versionchanged:: 20.0
@@ -375,6 +380,11 @@ async def refresh_chat_data(self, chat_id: int, chat_data: CD) -> None:
         :attr:`~telegram.ext.Application.chat_data` to a callback. Can be used to update data
         stored in :attr:`~telegram.ext.Application.chat_data` from an external source.
 
+        Warning:
+            When using :meth:`~telegram.ext.ApplicationBuilder.concurrent_updates`, this method
+            may be called while a handler callback is still running. This might lead to race
+            conditions.
+
         .. versionadded:: 13.6
 
         .. versionchanged:: 20.0
@@ -393,6 +403,11 @@ async def refresh_bot_data(self, bot_data: BD) -> None:
         :attr:`~telegram.ext.Application.bot_data` to a callback. Can be used to update data stored
         in :attr:`~telegram.ext.Application.bot_data` from an external source.
 
+        Warning:
+            When using :meth:`~telegram.ext.ApplicationBuilder.concurrent_updates`, this method
+            may be called while a handler callback is still running. This might lead to race
+            conditions.
+
         .. versionadded:: 13.6
 
         .. versionchanged:: 20.0


Test patch:
diff --git a/tests/ext/test_basepersistence.py b/tests/ext/test_basepersistence.py
--- a/tests/ext/test_basepersistence.py
+++ b/tests/ext/test_basepersistence.py
@@ -950,6 +950,70 @@ async def test_update_persistence_loop_async_logic(
                 assert not papp.persistence.updated_chat_ids
             assert not papp.persistence.updated_conversations
 
+    @papp_store_all_or_none
+    async def test_update_persistence_loop_manual_mark_for_update_persistence(
+        self, papp: Application, chat_id
+    ):
+        async with papp:
+            papp.chat_data[1].update({"key": "value", "refreshed": True})
+            papp.user_data[1].update({"key": "value", "refreshed": True})
+            await papp.update_persistence()
+
+            # Since no update has been processed, nothing should be marked for update
+            # So we expect the persisted data to differ from the current data
+            assert not papp.persistence.chat_data
+            assert papp.persistence.chat_data is not papp.chat_data
+            assert not papp.persistence.user_data
+            assert papp.persistence.user_data is not papp.user_data
+
+            # Double checking that not marking data doesn't change anything
+            papp.mark_data_for_update_persistence()
+            await papp.update_persistence()
+            assert not papp.persistence.chat_data
+            assert papp.persistence.chat_data is not papp.chat_data
+            assert not papp.persistence.user_data
+            assert papp.persistence.user_data is not papp.user_data
+
+            # marking data should lead to the data being updated
+            papp.mark_data_for_update_persistence(chat_ids=1, user_ids=1)
+            await papp.update_persistence()
+
+            assert papp.persistence.chat_data is not papp.chat_data
+            if papp.persistence.store_data.chat_data:
+                assert papp.persistence.chat_data == {1: {"key": "value", "refreshed": True}}
+                assert papp.persistence.chat_data[1] is not papp.chat_data[1]
+            else:
+                assert not papp.persistence.chat_data
+
+            assert papp.persistence.user_data is not papp.user_data
+            if papp.persistence.store_data.user_data:
+                assert papp.persistence.user_data == {1: {"key": "value", "refreshed": True}}
+                assert papp.persistence.user_data[1] is not papp.chat_data[1]
+            else:
+                assert not papp.persistence.user_data
+
+            # Also testing passing collections
+            papp.chat_data[1].update({"key": "value", "refreshed": False})
+            papp.user_data[1].update({"key": "value", "refreshed": False})
+            papp.mark_data_for_update_persistence(chat_ids={1}, user_ids={1})
+            await papp.update_persistence()
+
+            # marking data should lead to the data being updated
+
+            assert papp.persistence.chat_data is not papp.chat_data
+            if papp.persistence.store_data.chat_data:
+                assert papp.persistence.chat_data == {1: {"key": "value", "refreshed": False}}
+                assert papp.persistence.chat_data[1] is not papp.chat_data[1]
+            else:
+                assert not papp.persistence.chat_data
+
+            assert papp.persistence.user_data is not papp.user_data
+            if papp.persistence.store_data.user_data:
+                assert papp.persistence.user_data == {1: {"key": "value", "refreshed": False}}
+                assert papp.persistence.user_data[1] is not papp.chat_data[1]
+            else:
+                assert not papp.persistence.user_data
+
     @filled_papp
     async def test_drop_chat_data(self, papp: Application):
         async with papp: