Repository: python-telegram-bot/python-telegram-bot

Issue:
[FEATURE] tool for translating unicode offsets to utf-16 offsets of MessageEntity
### What kind of feature are you missing? Where do you notice a shortcoming of PTB?

I want to send a message with customized `MessageEntity`. At first I assumed ptb already handled the unicode -> utf-16 translation. It generally works for most texts, but turns out it will fail if there are some emojis in the message.

### Describe the solution you'd like

A new class `UnicodeMessageEntity` is needed. Currently I have a simple solution:
```python
    def fix_entities_offset(self):
        for text, entities in zip(self.texts, self.entities):
            cur_index = 0
            accumulated_len = 0
            for i, entity in enumerate(entities):
                cur_text = text[cur_index:entity.offset]
                accumulated_len += len(cur_text.encode('utf-16-le'))
                cur_off = accumulated_len // 2
                cur_text = text[entity.offset:entity.offset+entity.length]
                accumulated_len += len(cur_text.encode('utf-16-le'))
                cur_len = accumulated_len // 2 - cur_off
                entities[i] = MessageEntity(offset=cur_off, length=cur_len, type=entity.type, language=entity.language)
                cur_index = entity.offset + entity.length
```
It would be nice if this can be automatically applied when send messages, when entity object is an instance of `UnicodeMessageEntity`.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_


Discussion:
Hi, thanks for reaching out. I can't quite follow. I understood that you have troubles specifying the `offset` and `length` argument of `MessageEntity` if you have emoji in your text, but I don't quite get how your snippet is expected to solve that. Can you give an example of how you'd like a usre interface to look like?
I wrote a custom Markdown parser, which cuts super long text (exceeding the limit length of telegram message, usually long code) into many parts, and mark all code blocks manually
```python
def longtext_markdown_split(txt: str) -> tuple[list[str], list[list[UnicodeMessageEntity]]]:
    ...
```
I mean if we can support this
```python
texts, entities_list = longtext_markdown_split(original_text)
for text, entities in zip(texts, entities_list):
    # the type of `entities` is list of `UnicodeMessageEntity | MessageEntity`, and the `UnicodeMessageEntity` objects will be translated into `MessageEntity` internally in ptb
    await bot.send_message(text, entities=entities)
```

Sorry for misleading:
* the `fix_entities_offset` above is my current workaround, for a list of sorted `UnicodeMessageEntity`.
* Edit: ~~utf-8~~ -> unicode
Mh, I still don't get it. `longtext_markdown_split(original_text)` does not accept any entities, only text. Does it receive MD-formatted text and then parse that into plain text + entities? If so, you could probably utf-16-encode the text before you start parsing?

Note that `MessageEntity` is a pure implementation of the TG definition. As it does not have a direct connection to the text covered by the entity, TG does not adjust the offset or length in any way.
> If so, you could probably utf-16-encode the text before you start parsing?

Yes, but if the code of parsing entities is hard to refactor (for example, using a third-party library, which assumes `str` type everywhere), this would not be a good solution.

> Note that `MessageEntity` is a pure implementation of the TG definition. As it does not have a direct connection to the text covered by the entity, TG does not adjust the offset or length in any way.

In my opinion, since `str` is the most common type in processing strings, a tool class for unicode string may ease the burden of developers, and it is actually the thing that a _wrapper_ should do.
Fair points. I agree that PTB can provide more utility functionality for working with MessageEntity, I just took a bit to understand what exactly you're asking for : ) If I understand correctly now, you'd like to have a conversion functionality like

```python
def adjust_message_entities_for_utf_16(text: str, entities: Sequence[MessageEntity]) -> Sequence[MessageEntity]:
    """Utility functionality for converting the offset and length of entities from unicode to UTF-16.

    Tip:
       Useful if you want to express formatting of text containing arbitrary unicode characters with `MessageEntity` objects
       but do not want to take care of caculating in UTF-16 space. Instead you can just count characters with the cursor
       in your editor.

    Args:
        text: The text that the entities belong to
        entities: Sequence of entities with offset and length calculated in UTF-8 encoding

    Returns:
        Sequence[MessageEntity]: 
    """
```

Based on your snippet I've build an mwe. Suing `utf_16_entities` gives the same result as using `adjust_message_entities_for_utf_16(text, unicode_entities)`

<details>

```python
import asyncio
from collections.abc import Sequence
from pprint import pprint

from telegram import Bot, MessageEntity


def adjust_message_entities_for_utf_16(
    text: str, entities: Sequence[MessageEntity]
) -> Sequence[MessageEntity]:
    cur_index = 0
    accumulated_len = 0
    out = []
    for entity in entities:
        filler_text = text[cur_index : entity.offset]
        accumulated_len += len(filler_text.encode("utf-16-le"))
        cur_off = accumulated_len // 2

        entity_text = text[entity.offset: entity.offset + entity.length]
        accumulated_len += len(entity_text.encode("utf-16-le"))
        cur_len = accumulated_len // 2 - cur_off
        out.append(
            MessageEntity(
                offset=cur_off, length=cur_len, type=entity.type, language=entity.language
            )
        )
        cur_index = entity.offset + entity.length

    return out


async def main():
    text = "𠌕 bold 𝄢 italic\nunderlined: 𝛙𝌢𑁍"
    unicode_entities = [
        MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
        MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
        MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
    ]
    utf_16_entities = [
        MessageEntity(offset=3, length=4, type=MessageEntity.BOLD),
        MessageEntity(offset=11, length=6, type=MessageEntity.ITALIC),
        MessageEntity(offset=30, length=6, type=MessageEntity.UNDERLINE),
    ]
    async with Bot(token="token") as bot:
        await bot.send_message(
            chat_id=123,
            text=text,
            entities=adjust_message_entities_for_utf_16(text, unicode_entities),
            # entities=utf_16_entities,
        )


if __name__ == "__main__":
    asyncio.run(main())
```

</details>

However, I see no need for a new class `UnicodeMessageEntity`. This would be a rather tight integration of this conversion into the bot methods, plus a deviation from the Bot API with we generally avoid. For starters, I think adding a helper function should be enough.
I'm now wondering where such a function would be suitable - in `telegram.helpers` as simple function or as static method of `MessageEntity`? The latter may be a tad easier to find, but binding this helper function to the class doesn't give any inherit benefit 🤔 I guess I'd tend for `helpers`. Maybe you can give an opinion on where you'd be looking for this?
I'd prefer static method of `MessageEntity`. It's easier to find, and can also let developers aware there is a problem of translating the offsets at their first use.


Okay, fair point, let's go ahead with those then :)
Would you like to send a PR? We can probably come up with a better function name and also docstring (an example would be nice). In the code snippets, I would like to improve the variable names somewhat to make them more self-speaking :)
I've made a PR #4323



Patch:
diff --git a/telegram/_messageentity.py b/telegram/_messageentity.py
index 2f7fb7d6179..187a21a063d 100644
--- a/telegram/_messageentity.py
+++ b/telegram/_messageentity.py
@@ -18,7 +18,9 @@
 # along with this program.  If not, see [http://www.gnu.org/licenses/].
 """This module contains an object that represents a Telegram MessageEntity."""
 
-from typing import TYPE_CHECKING, Final, List, Optional
+import copy
+import itertools
+from typing import TYPE_CHECKING, Dict, Final, List, Optional, Sequence
 
 from telegram import constants
 from telegram._telegramobject import TelegramObject
@@ -140,6 +142,81 @@ def de_json(cls, data: Optional[JSONDict], bot: "Bot") -> Optional["MessageEntit
 
         return super().de_json(data=data, bot=bot)
 
+    @staticmethod
+    def adjust_message_entities_to_utf_16(
+        text: str, entities: Sequence["MessageEntity"]
+    ) -> Sequence["MessageEntity"]:
+        """Utility functionality for converting the offset and length of entities from
+        Unicode (:obj:`str`) to UTF-16 (``utf-16-le`` encoded :obj:`bytes`).
+
+        Tip:
+            Only the offsets and lengths calulated in UTF-16 is acceptable by the Telegram Bot API.
+            If they are calculated using the Unicode string (:obj:`str` object), errors may occur
+            when the text contains characters that are not in the Basic Multilingual Plane (BMP).
+            For more information, see `Unicode <https://en.wikipedia.org/wiki/Unicode>`_ and
+            `Plane (Unicode) <https://en.wikipedia.org/wiki/Plane_(Unicode)>`_.
+
+        .. versionadded:: NEXT.VERSION
+
+        Examples:
+            Below is a snippet of code that demonstrates how to use this function to convert
+            entities from Unicode to UTF-16 space. The ``unicode_entities`` are calculated in
+            Unicode and the `utf_16_entities` are calculated in UTF-16.
+
+            .. code-block:: python
+
+                text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+                unicode_entities = [
+                    MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
+                    MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
+                    MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
+                ]
+                utf_16_entities = MessageEntity.adjust_message_entities_to_utf_16(
+                    text, unicode_entities
+                )
+                await bot.send_message(
+                    chat_id=123,
+                    text=text,
+                    entities=utf_16_entities,
+                )
+                # utf_16_entities[0]: offset=3, length=4
+                # utf_16_entities[1]: offset=11, length=6
+                # utf_16_entities[2]: offset=30, length=6
+
+        Args:
+            text (:obj:`str`): The text that the entities belong to
+            entities (Sequence[:class:`telegram.MessageEntity`]): Sequence of entities
+                with offset and length calculated in Unicode
+
+        Returns:
+            Sequence[:class:`telegram.MessageEntity`]: Sequence of entities
+            with offset and length calculated in UTF-16 encoding
+        """
+        # get sorted positions
+        positions = sorted(itertools.chain(*((x.offset, x.offset + x.length) for x in entities)))
+        accumulated_length = 0
+        # calculate the length of each slice text[:position] in utf-16 accordingly,
+        # store the position translations
+        position_translation: Dict[int, int] = {}
+        for i, position in enumerate(positions):
+            last_position = positions[i - 1] if i > 0 else 0
+            text_slice = text[last_position:position]
+            accumulated_length += len(text_slice.encode("utf-16-le")) // 2
+            position_translation[position] = accumulated_length
+        # get the final output entites
+        out = []
+        for entity in entities:
+            translated_positions = position_translation[entity.offset]
+            translated_length = (
+                position_translation[entity.offset + entity.length] - translated_positions
+            )
+            new_entity = copy.copy(entity)
+            with new_entity._unfrozen():
+                new_entity.offset = translated_positions
+                new_entity.length = translated_length
+            out.append(new_entity)
+        return out
+
     ALL_TYPES: Final[List[str]] = list(constants.MessageEntityType)
     """List[:obj:`str`]: A list of all available message entity types."""
     BLOCKQUOTE: Final[str] = constants.MessageEntityType.BLOCKQUOTE


Test patch:
diff --git a/tests/test_messageentity.py b/tests/test_messageentity.py
index b14ec79c321..8bab9fec7b9 100644
--- a/tests/test_messageentity.py
+++ b/tests/test_messageentity.py
@@ -16,6 +16,9 @@
 #
 # You should have received a copy of the GNU Lesser Public License
 # along with this program.  If not, see [http://www.gnu.org/licenses/].
+import random
+from typing import List, Tuple
+
 import pytest
 
 from telegram import MessageEntity, User
@@ -81,6 +84,25 @@ def test_enum_init(self):
         entity = MessageEntity(type="url", offset=0, length=1)
         assert entity.type is MessageEntityType.URL
 
+    def test_fix_utf16(self):
+        text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+        inputs_outputs: List[Tuple[Tuple[int, int, str], Tuple[int, int]]] = [
+            ((2, 4, MessageEntity.BOLD), (3, 4)),
+            ((9, 6, MessageEntity.ITALIC), (11, 6)),
+            ((28, 3, MessageEntity.UNDERLINE), (30, 6)),
+        ]
+        random.shuffle(inputs_outputs)
+        unicode_entities = [
+            MessageEntity(offset=_input[0], length=_input[1], type=_input[2])
+            for _input, _ in inputs_outputs
+        ]
+        utf_16_entities = MessageEntity.adjust_message_entities_to_utf_16(text, unicode_entities)
+        for out_entity, input_output in zip(utf_16_entities, inputs_outputs):
+            _, output = input_output
+            offset, length = output
+            assert out_entity.offset == offset
+            assert out_entity.length == length
+
     def test_equality(self):
         a = MessageEntity(MessageEntity.BOLD, 2, 3)
         b = MessageEntity(MessageEntity.BOLD, 2, 3)