Repository: opsdroid/opsdroid

Issue:
Telegram Connector : Parsing edited message cause process to freeze
# Description
Hi,

I am experiencing a strange behaviour, when opsdroid is not active and someone send many messages and call them back for modification and send them back. this cause parsing error when opsdroid start receiving those messages and parsing them.

The problem is that the function **async def _parse_message(self, response):** try to parse a json with looking for **message** in it's tests but in our case the entry name became **edited_message** 

An example of an edited message :

```
{'update_id': 987978987, 'edited_message': {'message_id': 142, 'from': {'id': 65464848, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 9849489499, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551697059, 'edit_date': 1551697076, 'text': 'test message'}}
```


## Steps to Reproduce
- Stop opsdroid
- Send messages
- edit the message by clicking on up arrow
- send edited message back
- doing this 4 times :-)
- activate opsdroid


## Expected Functionality
detect if the message is an edited one or an "original" one
in case of an edited one, use the edited_message entry instead of message entry


## Experienced Functionality

here is some logs to help understanding the freeze when the function try to test 
> if "channel" in result["message"]["chat"]["type"]

or any other message instead of edited_message

```
{'ok': True, 'result': [{'update_id': 684648646, 'edited_message': {'message_id': 142, 'from': {'id': 987456897, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 987456897, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551697059, 'edit_date': 1551697076, 'text': 'please launch a vmware instance with those paramaters --os=windows --template=windows2016'}}, {'update_id': 246644493, 'message': {'message_id': 145, 'from': {'id': 987456897, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 987456897, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551697087, 'text': 'please launch a vmware server with those paramaters --os=windows --template=windows2016 --ip=92.0.34.0.7 --hostname=helios --note="it\'s my test machine ".'}}, {'update_id': 246644494, 'message': {'message_id': 146, 'from': {'id': 987456897, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 987456897, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551697540, 'text': 'hi'}}, {'update_id': 246644495, 'message': {'message_id': 147, 'from': {'id': 987456897, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 987456897, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551731191, 'text': 'hi'}}]}
```
Where the freeze occurs :

```
DEBUG opsdroid.connector.websocket: Starting Websocket connector
DEBUG opsdroid.connector.telegram: Loaded telegram connector
DEBUG opsdroid.connector.telegram: Connecting to telegram
DEBUG opsdroid.connector.telegram: {'ok': True, 'result': {'id': 731324986, 'is_bot': True, 'first_name': 'ChamixLocalBot', 'username': 'ChamixLocalBot'}}
DEBUG opsdroid.connector.telegram: Connected to telegram as ChamixLocalBot
INFO opsdroid.core: Opsdroid is now running, press ctrl+c to exit.
INFO opsdroid.web: Started web server on http://127.0.0.1:8080
DEBUG opsdroid.connector.telegram: =======> Testing if "channel" in result["message"]["chat"]["type"]
DEBUG opsdroid.parsers.crontab: Running crontab skills
DEBUG opsdroid.parsers.crontab: Running crontab skills
DEBUG opsdroid.parsers.crontab: Running crontab skills
DEBUG opsdroid.parsers.crontab: Running crontab skills
```

> this is only my debug line :-) : DEBUG opsdroid.connector.telegram: =======> Testing if "channel" in result["message"]["chat"]["type"]


## Versions
- **Opsdroid version: v0.14.1+19.g6360912.dirty**
- **Python version: 3.7.0**

Regards


Discussion:
Thanks for raising this @iobreaker just to clarify, on your log you have an original message and then an edited message? It seems you tried to run the vm command twice. I'm trying figure out if Telegram sends every edited message with the Json response or if it replaces it.

If the Telegram API returns all the edited messages (let's say you edit the same message 3 times) we will have to add some logic to just use the last version of the edited message
Hi @FabioRosado 

I confirm, when you edit and send the message back the error occurs.

Bellow an example I just did that reproduce the error :


> IOBReaker - Initial message- : hello
>
```
DEBUG opsdroid.connector.telegram: =======> <2> : {'update_id': 246644498, 'message': {'message_id': 150, 'from': {'id': 245245245, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 245245245, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551797346, 'text': 'hello'}}
DEBUG opsdroid.connector.telegram: =======> <4> : {'update_id': 246644498, 'message': {'message_id': 150, 'from': {'id': 245245245, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 245245245, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551797346, 'text': 'hello'}}
```

> IOBReaker -recall and edit the same message- : hello2

```
DEBUG opsdroid.connector.telegram: =======> <2> : {'update_id': 246644499, 'edited_message': {'message_id': 150, 'from': {'id': 245245245, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 245245245, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551797346, 'edit_date': 1551797365, 'text': 'hello2'}}
```

> IOBReaker - new message -  : Hi all
```
DEBUG opsdroid.parsers.crontab: Running crontab skills
DEBUG opsdroid.parsers.crontab: Running crontab skills
DEBUG opsdroid.parsers.crontab: Running crontab skills
DEBUG opsdroid.parsers.crontab: Running crontab skills
```

No message will be processed since opsdroid received the edited message

I Do not think that the solution is to send only the edited message because each time Telegram does not retain the original message but send each one after user confirmation.

More than that, the user can send an original message like :

- buy a packet of **chips**

and recall the message to only modify 1 item like

- buy a packet of **candies**

The solution i think can be detecting if the message is an edited one or a standard one, for each message type execute the correct workflow/function

Hope that help :-)

I foresee some issues with implementing the edited message logic. I won't be able to keep track of previous messages - the bot will "consume" messages as they come. 

So if you type an original message like:
`Hello`

And then edit it to `Bye`, opsdroid will reply both to the `hello` and `bye` message. Do you think this would be an alright solution for the connector?
Hi @FabioRosado 

Yes it's what i explained in my previous message using the example of chips and candies :-)

Two solutions to correct this issue

## The quick one 

Changing the key of the returned dict from **edited_message** to **message**:
I checked the dict returned by telegram for edited_message, it has the same structure as the message one.

This solution will work until Telegram decide to change something

**The code : it cost 2 lines of modifications **
```
 async def _parse_message(self, response):
        for result in response["result"]:
            if result.get('edited_message',None):
                result['message'] = result.pop('edited_message')
            if "channel" in result["message"]["chat"]["type"]:
                _LOGGER.debug("Channel message parsing not supported "
                              "- Ignoring message")
            elif "message" in result and "text" in result["message"]:
                user = self.get_user(result)
...
...
```

## The "good" but long one

Let  **_parse_message** function manage the split of response contain and and create an other function like **_extract_message** in charge of turning the right message object to **_parse_message** .

Please note that **handle_user_permission** and **get_user** has to change too (because they receive response as a param)

**The code :** 

```
    @staticmethod
    def get_user(response):
        user = None
        if response.get('message',None):
            if "username" in response["message"]["from"]:
                user = response["message"]["from"]["username"]
            elif "first_name" in response["message"]["from"]:
                user = response["message"]["from"]["first_name"]
        elif response.get('edited_message',None):
            if "username" in response["edited_message"]["from"]:
                user = response["edited_message"]["from"]["username"]
            elif "first_name" in response["edited_message"]["from"]:
                user = response["edited_message"]["from"]["first_name"]

        return user
```

```
    def handle_user_permission(self, response, user):
        if response.get('message',None):
            user_id = response["message"]["from"]["id"]
        elif response.get('edited_message',None):
            ser_id = response["edited_message"]["from"]["id"]

        if not self.whitelisted_users or \
            user in self.whitelisted_users or \
                user_id in self.whitelisted_users:
            return True

        return False
```

```
    async def _extract_message(self, result):
        """Handle logic to extract a received message.

        Since message can be a standard one (initial) or
        a modified one (recalled, modified and sent back), this function
        will detect which kind a message is received and extract information from it

        Args:
            response (dict): result returned by aiohttp.ClientSession.
        Return:
            message (object) : Message object.
        """

        message = None
        if result.get('message',None):
            if "channel" in result["message"]["chat"]["type"]:
                _LOGGER.debug("Channel message parsing not supported - Ignoring message")
            else:
                if "message" in result and "text" in result["message"]:
                    user = self.get_user(result)
                    message = Message( user, result["message"]["chat"], self, result["message"]["text"])
                else:
                    _LOGGER.debug("Unable to extract text message.")
        elif result.get('edited_message',None):
            if "channel" in result["edited_message"]["chat"]["type"]:
                _LOGGER.debug("Channel message parsing not supported - Ignoring message")
            else:
                if "edited_message" in result and "text" in result["edited_message"]:
                    user = self.get_user(result)
                    message = Message( user, result["edited_message"]["chat"], self, result["edited_message"]["text"])
                else:
                    _LOGGER.debug("Unable to extract text message.")
        else:
            _LOGGER.error("Unable to parse the message.")
        
        return message
```

```
    async def _parse_message(self, response):
        for result in response["result"]:
            message = await self._extract_message(result)
            if message:
                user = self.get_user(result)
                if self.handle_user_permission(result, user): 
                    await self.opsdroid.parse(message)
                else:
                    message.text = "Sorry, you're not allowed to speak with this bot."
                    await self.respond(message)
            self.latest_update = result["update_id"] + 1
```


## Tests 

I tested the two methods, both works :

> IObreaker Initial Message : buy a packet of **chips**
```
DEBUG opsdroid.connector.telegram: ========> Calling parse_message using : {'ok': True, 'result': [{'update_id': 246644523, 'message': {'message_id': 168, 'from': {'id': 112233445, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 112233445, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551864488, 'text': 'buy a packet of chips'}}]}
DEBUG opsdroid.connector.telegram: ========> Calling _extract_message using : {'update_id': 246644523, 'message': {'message_id': 168, 'from': {'id': 112233445, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 112233445, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551864488, 'text': 'buy a packet of chips'}}
DEBUG opsdroid.connector.telegram: ========> Parssing standard message
DEBUG opsdroid.connector.telegram: ========> Handlling message Object
DEBUG opsdroid.connector.telegram: ========> Calling self.opsdroid.parse
DEBUG opsdroid.core: Parsing input: buy a packet of chips
DEBUG opsdroid.core: Processing parsers...
```

> IObreaker modified  Message : buy a packet of **candies**
```
DEBUG opsdroid.connector.telegram: ========> Calling parse_message using : {'ok': True, 'result': [{'update_id': 246644524, 'edited_message': {'message_id': 168, 'from': {'id': 112233445, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 112233445, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551864488, 'edit_date': 1551864493, 'text': 'buy a packet of candies'}}]}
DEBUG opsdroid.connector.telegram: ========> Calling _extract_message using : {'update_id': 246644524, 'edited_message': {'message_id': 168, 'from': {'id': 112233445, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 112233445, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551864488, 'edit_date': 1551864493, 'text': 'buy a packet of candies'}}
DEBUG opsdroid.connector.telegram: ========> Parssing edited message
DEBUG opsdroid.connector.telegram: ========> Handlling message Object
DEBUG opsdroid.connector.telegram: ========> Calling self.opsdroid.parse
DEBUG opsdroid.core: Parsing input: buy a packet of candies
DEBUG opsdroid.core: Processing parsers...
```

> IObreaker  new message : forget about it
```
DEBUG opsdroid.connector.telegram: ========> Calling parse_message using : {'ok': True, 'result': [{'update_id': 246644525, 'message': {'message_id': 169, 'from': {'id': 112233445, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 112233445, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551864518, 'text': 'forget about it'}}]}
DEBUG opsdroid.connector.telegram: ========> Calling _extract_message using : {'update_id': 246644525, 'message': {'message_id': 169, 'from': {'id': 112233445, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 112233445, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551864518, 'text': 'forget about it'}}
DEBUG opsdroid.connector.telegram: ========> Parssing standard message
DEBUG opsdroid.connector.telegram: ========> Handlling message Object
DEBUG opsdroid.connector.telegram: ========> Calling self.opsdroid.parse
```

> IObreaker  new message : ok ?
```
DEBUG opsdroid.connector.telegram: ========> Calling parse_message using : {'ok': True, 'result': [{'update_id': 246644526, 'message': {'message_id': 170, 'from': {'id': 112233445, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 112233445, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551864527, 'text': 'ok ?'}}]}
DEBUG opsdroid.connector.telegram: ========> Calling _extract_message using : {'update_id': 246644526, 'message': {'message_id': 170, 'from': {'id': 112233445, 'is_bot': False, 'first_name': 'IOBreaker', 'language_code': 'en'}, 'chat': {'id': 112233445, 'first_name': 'IOBreaker', 'type': 'private'}, 'date': 1551864527, 'text': 'ok ?'}}
DEBUG opsdroid.connector.telegram: ========> Parssing standard message
DEBUG opsdroid.connector.telegram: ========> Handlling message Object
DEBUG opsdroid.connector.telegram: ========> Calling self.opsdroid.parse
DEBUG opsdroid.core: Parsing input: ok ?
DEBUG opsdroid.core: Processing parsers...
```

No freezing detected.

Regards



I would be keen to discuss whether we think bots should respond to edited messages at all.

The main reason I raise this is because it is possible to edit messages way back in the past in many chat clients. This could result in unexpected actions from the bot.
Good question.

If i take the example of sending a big CLI to the bot to execute an action and in case of this CLI is wrong, i would be really happy to just have to modify the error rather than rewriting the whole thing.

if you allow me, I will ask the question in a different way:  when a chat service allows an action (perhaps seen by the user as an excellent action), should opsdroid answer it or not?

P.S : In both cases, the message should be handled (even with no response like for channel message)
I can see the reasoning behind jacobs input. Personally, I think opsdroid should handle the edited message. Taking in consideration what @iobreaker  was trying to do, if you mess up a long code by mistake, it's easier to just press the `up` arrow and get the last message, correct the mistake and sent the command back to the bot.

I am going to start a PR to fix this issue - I have also started using telegram more often and quite like it, even though the documentation kind of sucks haha
Sounds good to me. I'm happy to go down this route, I just wanted us to decide to do it explicitly. I wonder if we should start recording these kind of design decisions somewhere?
Not sure how should we record them to be honest, each connector have their own ways to handle edit messages. I wonder how did @Cadair solved this issue with the matrix connector -I think he mentioned something about edited messages handling as well?
That's kind of my point. I would like the handling to be consistent. We've captured the discussion and decision in this issue, but is that enough? Maybe we should add something to the "creating connectors" section of the docs.
I see your point and yeah I agree that we should maybe add that bit to documentation. In regards of edited messages, I think the best way would be run only the edited message, but in order to get that done we would have to somehow gather a list of messages and handle them - this would cause opsdroid to delay the response so it might not be the best way to do it 🤔 
I think what should happen is that an `Edit` event should be raised, and then skills and connectors can implement them in whatever way they want :grin:  (see #826 )

(matrix dosen't have event editing yet, so not currently a problem)
That's a fair point and I agree an edit event is a good idea. However it would be great to have consistency in implementation.
Implementation in what way? I see an `Edit` event having a `new_message` attribute, and maybe some representation of the diff or something. What did you have in mind?

Resolving patch:
diff --git a/opsdroid/connector/telegram/__init__.py b/opsdroid/connector/telegram/__init__.py
--- a/opsdroid/connector/telegram/__init__.py
+++ b/opsdroid/connector/telegram/__init__.py
@@ -152,6 +152,8 @@ async def _parse_message(self, response):
         """
         for result in response["result"]:
             _LOGGER.debug(result)
+            if result.get('edited_message', None):
+                result['message'] = result.pop('edited_message')
             if "channel" in result["message"]["chat"]["type"]:
                 _LOGGER.debug("Channel message parsing not supported "
                               "- Ignoring message")


Test patch:
diff --git a/tests/test_connector_telegram.py b/tests/test_connector_telegram.py
--- a/tests/test_connector_telegram.py
+++ b/tests/test_connector_telegram.py
@@ -112,6 +112,35 @@ async def test_parse_message_username(self):
             await self.connector._parse_message(response)
             self.assertTrue(mocked_parse.called)
 
+    async def test_parse_edited_message(self):
+        response = {'result': [{
+            'update_id': 246644499,
+            'edited_message': {
+                'message_id': 150,
+                'from': {
+                    'id': 245245245,
+                    'is_bot': False,
+                    'first_name': 'IOBreaker',
+                    'language_code': 'en'},
+                'chat': {
+                    'id': 245245245,
+                    'first_name': 'IOBreaker',
+                    'type': 'private'},
+                'date': 1551797346,
+                'edit_date': 1551797365,
+                'text': 'hello2'}}]}
+        response_copy = list(response)
+        mocked_status = amock.CoroutineMock()
+        mocked_status.status = 200
+        with amock.patch('opsdroid.core.OpsDroid.parse') as mocked_parse, \
+                amock.patch.object(self.connector.session, 'post') \
+                as patched_request:
+            patched_request.return_value = asyncio.Future()
+            patched_request.return_value.set_result(mocked_status)
+            self.assertTrue(response['result'][0].get('edited_message'))
+            await self.connector._parse_message(response)
+
+
     async def test_parse_message_channel(self):
         response = {'result': [{
             "update_id": 427647860,
@@ -137,12 +166,12 @@ async def test_parse_message_channel(self):
             }
         }]}
 
-        with amock.patch('opsdroid.core.OpsDroid.parse') as mocked_parse:
+        with amock.patch('opsdroid.core.OpsDroid.parse'):
             await self.connector._parse_message(response)
             self.assertLogs('_LOGGER', 'debug')
 
     async def test_parse_message_first_name(self):
-        response = { 'result': [{
+        response = {'result': [{
             "update_id": 427647860,
             "message": {
                 "message_id": 12,
@@ -162,7 +191,6 @@ async def test_parse_message_first_name(self):
                 "text": "Hello"
             }
         }]}
-
         with amock.patch('opsdroid.core.OpsDroid.parse') as mocked_parse:
             await self.connector._parse_message(response)
             self.assertTrue(mocked_parse.called)