Repository: python-telegram-bot/python-telegram-bot

Issue:
PicklePersistence of user_data set within conversations
### Steps to reproduce
1. In [persistentconversationbot.py](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/examples/persistentconversationbot.py#L150), change `persistent=True` to `persistent=False`
2. Start the bot, give the bot some information to store

3. Stop bot
4. Start bot again, send command `/show_data`. It will be empty

### Expected behaviour
`user_data` changed within a conversation should be persisted even if the conversation is not
(i.e., I would expect this)

### Actual behaviour
`user_data` changed within a conversation that is not persisted, will not be persisted

### Configuration
Raspbian GNU/Linux 9 (stretch)


**Version of Python, python-telegram-bot & dependencies:**

```
python-telegram-bot 11.1.0
certifi 2018.01.18
future 0.17.1
Python 3.5.3 (default, Sep 27 2018, 17:25:39)  [GCC 6.3.0 20170516]
```

### Research so far:

1.  When the conversation is persisted, the `user_data` will only be persisted, if `single_file = True`. This is because this will trigger `dump_singlefile()` in [picklepersistence.py](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/telegram/ext/picklepersistence.py#L97) on update in Conversation. When setting `single_file = False`, `user_data` will not be persisted
2. Things go south in [`update_user_data()`](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/telegram/ext/picklepersistence.py#L195). Printing the result of

    ```
    self.user_data.get(user_id) == data:
    ```
    always gives `True` even though `self.user_data.[user_id]` _has_ changed after user input (as one can test by printing it).

I therefore conclude, that `self.user_data[user_id]` must be changed somewhere along the way _before_ `update_user_data()` is called. Still, I can't pinpoint, where that happens. Will try and find it.

If you have a clue, please let me know :)
PicklePersistence of user_data set within conversations
### Steps to reproduce
1. In [persistentconversationbot.py](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/examples/persistentconversationbot.py#L150), change `persistent=True` to `persistent=False`
2. Start the bot, give the bot some information to store

3. Stop bot
4. Start bot again, send command `/show_data`. It will be empty

### Expected behaviour
`user_data` changed within a conversation should be persisted even if the conversation is not
(i.e., I would expect this)

### Actual behaviour
`user_data` changed within a conversation that is not persisted, will not be persisted

### Configuration
Raspbian GNU/Linux 9 (stretch)


**Version of Python, python-telegram-bot & dependencies:**

```
python-telegram-bot 11.1.0
certifi 2018.01.18
future 0.17.1
Python 3.5.3 (default, Sep 27 2018, 17:25:39)  [GCC 6.3.0 20170516]
```

### Research so far:

1.  When the conversation is persisted, the `user_data` will only be persisted, if `single_file = True`. This is because this will trigger `dump_singlefile()` in [picklepersistence.py](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/telegram/ext/picklepersistence.py#L97) on update in Conversation. When setting `single_file = False`, `user_data` will not be persisted
2. Things go south in [`update_user_data()`](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/telegram/ext/picklepersistence.py#L195). Printing the result of

    ```
    self.user_data.get(user_id) == data:
    ```
    always gives `True` even though `self.user_data.[user_id]` _has_ changed after user input (as one can test by printing it).

I therefore conclude, that `self.user_data[user_id]` must be changed somewhere along the way _before_ `update_user_data()` is called. Still, I can't pinpoint, where that happens. Will try and find it.

If you have a clue, please let me know :)


Discussion:
Found the issue:
[`get_user_data()`](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/telegram/ext/picklepersistence.py#L125) returns a *shallow* copy and in [`update_user_data()`](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/telegram/ext/picklepersistence.py#L197) ,`self.user_data[user_id]` is set to the actual entry in `user_data`. Therefore checking for
```
self.user_data.get(user_id) == data
```
will always return `True`, since one is comparing an object with itself.
Similar holds for `chat_data` und `conversations`. In the case of latter it doesn't seem to matter at the moment. But persistence of `conversations` may be enhanced and then it might matter.

A solution to this would be, to make use of `copy.deepcopy`. However, if `user_data` growth large, this may be time consuming. I therefore propose an additional argument `on_update` for `PicklePersistence` that let's you choose wether to check for updates before dumping or not.
If `on_update = False`, checking is obmitted and one can skip time consuming deepcopies. If `on_update = True`, deepcopies are used.
Hi,

Sorry it took a while for me to respond. I have to dive into the code to see the problem. But in any case thanks for reporting the issue. I'm marking it as a bug and will look into it as soon as I can.

Pieter
No problem at all.
I have implemented the proposed solution. I will run some tests and open a pull request if that's okay with you.
I just had a look a the tests routines. It seems to me that they don't indicate in issue because only simple dictionaries are used and hence, simply calling `data.copy()` is a 'deep enough' copy.
Unfortunately I'm not familiar with pytest. As far as I can see, changing [test_persistence.py](https://github.com/python-telegram-bot/python-telegram-bot/blob/7eeb670a59ccfe8242a1bbf6754da42131028979/tests/test_persistence.py#L415) in a way that one could rewrite e.g. line 415 to 

```
        user_data[54321]['test9']['test10'] = 'test 11'
```
should do the trick. That is, if the data, `get_user_data()` needs to return is complex enough (a nested dictionary should be enough), calling `.copy()` will return a shallow copy and the assert statement in line 416 fails.
There are multiple parts of the test that should be updated in this manner.
I tried this 'the dirty way' but am not (yet) able to implement this in a tidy fashion in `test_persistence.py`.
Found the issue:
[`get_user_data()`](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/telegram/ext/picklepersistence.py#L125) returns a *shallow* copy and in [`update_user_data()`](https://github.com/python-telegram-bot/python-telegram-bot/blob/378784f55e0c00c5b85659cdc7edb278557bdc0b/telegram/ext/picklepersistence.py#L197) ,`self.user_data[user_id]` is set to the actual entry in `user_data`. Therefore checking for
```
self.user_data.get(user_id) == data
```
will always return `True`, since one is comparing an object with itself.
Similar holds for `chat_data` und `conversations`. In the case of latter it doesn't seem to matter at the moment. But persistence of `conversations` may be enhanced and then it might matter.

A solution to this would be, to make use of `copy.deepcopy`. However, if `user_data` growth large, this may be time consuming. I therefore propose an additional argument `on_update` for `PicklePersistence` that let's you choose wether to check for updates before dumping or not.
If `on_update = False`, checking is obmitted and one can skip time consuming deepcopies. If `on_update = True`, deepcopies are used.
Hi,

Sorry it took a while for me to respond. I have to dive into the code to see the problem. But in any case thanks for reporting the issue. I'm marking it as a bug and will look into it as soon as I can.

Pieter
No problem at all.
I have implemented the proposed solution. I will run some tests and open a pull request if that's okay with you.
I just had a look a the tests routines. It seems to me that they don't indicate in issue because only simple dictionaries are used and hence, simply calling `data.copy()` is a 'deep enough' copy.
Unfortunately I'm not familiar with pytest. As far as I can see, changing [test_persistence.py](https://github.com/python-telegram-bot/python-telegram-bot/blob/7eeb670a59ccfe8242a1bbf6754da42131028979/tests/test_persistence.py#L415) in a way that one could rewrite e.g. line 415 to 

```
        user_data[54321]['test9']['test10'] = 'test 11'
```
should do the trick. That is, if the data, `get_user_data()` needs to return is complex enough (a nested dictionary should be enough), calling `.copy()` will return a shallow copy and the assert statement in line 416 fails.
There are multiple parts of the test that should be updated in this manner.
I tried this 'the dirty way' but am not (yet) able to implement this in a tidy fashion in `test_persistence.py`.

Resolving patch:
diff --git a/telegram/ext/dictpersistence.py b/telegram/ext/dictpersistence.py
--- a/telegram/ext/dictpersistence.py
+++ b/telegram/ext/dictpersistence.py
@@ -17,6 +17,8 @@
 # You should have received a copy of the GNU Lesser Public License
 # along with this program.  If not, see [http://www.gnu.org/licenses/].
 """This module contains the DictPersistence class."""
+from copy import deepcopy
+
 from telegram.utils.helpers import decode_user_chat_data_from_json,\
     decode_conversations_from_json, enocde_conversations_to_json
 
@@ -129,7 +131,7 @@ def get_user_data(self):
             pass
         else:
             self._user_data = defaultdict(dict)
-        return self.user_data.copy()
+        return deepcopy(self.user_data)
 
     def get_chat_data(self):
         """Returns the chat_data created from the ``chat_data_json`` or an empty defaultdict.
@@ -141,7 +143,7 @@ def get_chat_data(self):
             pass
         else:
             self._chat_data = defaultdict(dict)
-        return self.chat_data.copy()
+        return deepcopy(self.chat_data)
 
     def get_conversations(self, name):
         """Returns the conversations created from the ``conversations_json`` or an empty
diff --git a/telegram/ext/picklepersistence.py b/telegram/ext/picklepersistence.py
--- a/telegram/ext/picklepersistence.py
+++ b/telegram/ext/picklepersistence.py
@@ -19,6 +19,7 @@
 """This module contains the PicklePersistence class."""
 import pickle
 from collections import defaultdict
+from copy import deepcopy
 
 from telegram.ext import BasePersistence
 
@@ -122,7 +123,7 @@ def get_user_data(self):
             self.user_data = data
         else:
             self.load_singlefile()
-        return self.user_data.copy()
+        return deepcopy(self.user_data)
 
     def get_chat_data(self):
         """Returns the chat_data from the pickle file if it exsists or an empty defaultdict.
@@ -142,7 +143,7 @@ def get_chat_data(self):
             self.chat_data = data
         else:
             self.load_singlefile()
-        return self.chat_data.copy()
+        return deepcopy(self.chat_data)
 
     def get_conversations(self, name):
         """Returns the conversations from the pickle file if it exsists or an empty defaultdict.


Test patch: