Repository: python-telegram-bot/python-telegram-bot
Pull request: Add `adjust_message_entities_to_utf_16` utility
<!--
Hey! You're PRing? Cool!
Please be sure to check out our contribution guide (https://github.com/python-telegram-bot/python-telegram-bot/blob/master/.github/CONTRIBUTING.rst).
Especially, please have a look at the check list for PRs (https://github.com/python-telegram-bot/python-telegram-bot/blob/master/.github/CONTRIBUTING.rst#check-list-for-prs). Feel free to copy (parts of) the checklist to the PR description to remind you or the maintainers of open points or if you have questions on anything.
-->

add  `adjust_message_entities_to_utf_16` for #4319


Problem statement:
[FEATURE] tool for translating unicode offsets to utf-16 offsets of MessageEntity
### What kind of feature are you missing? Where do you notice a shortcoming of PTB?

I want to send a message with customized `MessageEntity`. At first I assumed ptb already handled the unicode -> utf-16 translation. It generally works for most texts, but turns out it will fail if there are some emojis in the message.

### Describe the solution you'd like

A new class `UnicodeMessageEntity` is needed. Currently I have a simple solution:
```python
    def fix_entities_offset(self):
        for text, entities in zip(self.texts, self.entities):
            cur_index = 0
            accumulated_len = 0
            for i, entity in enumerate(entities):
                cur_text = text[cur_index:entity.offset]
                accumulated_len += len(cur_text.encode('utf-16-le'))
                cur_off = accumulated_len // 2
                cur_text = text[entity.offset:entity.offset+entity.length]
                accumulated_len += len(cur_text.encode('utf-16-le'))
                cur_len = accumulated_len // 2 - cur_off
                entities[i] = MessageEntity(offset=cur_off, length=cur_len, type=entity.type, language=entity.language)
                cur_index = entity.offset + entity.length
```
It would be nice if this can be automatically applied when send messages, when entity object is an instance of `UnicodeMessageEntity`.

### Describe alternatives you've considered

_No response_

### Additional context

_No response_

Hints from issue discussion:
Hi, thanks for reaching out. I can't quite follow. I understood that you have troubles specifying the `offset` and `length` argument of `MessageEntity` if you have emoji in your text, but I don't quite get how your snippet is expected to solve that. Can you give an example of how you'd like a usre interface to look like?
I wrote a custom Markdown parser, which cuts super long text (exceeding the limit length of telegram message, usually long code) into many parts, and mark all code blocks manually
```python
def longtext_markdown_split(txt: str) -> tuple[list[str], list[list[UnicodeMessageEntity]]]:
    ...
```
I mean if we can support this
```python
texts, entities_list = longtext_markdown_split(original_text)
for text, entities in zip(texts, entities_list):
    # the type of `entities` is list of `UnicodeMessageEntity | MessageEntity`, and the `UnicodeMessageEntity` objects will be translated into `MessageEntity` internally in ptb
    await bot.send_message(text, entities=entities)
```

Sorry for misleading:
* the `fix_entities_offset` above is my current workaround, for a list of sorted `UnicodeMessageEntity`.
* Edit: ~~utf-8~~ -> unicode
Mh, I still don't get it. `longtext_markdown_split(original_text)` does not accept any entities, only text. Does it receive MD-formatted text and then parse that into plain text + entities? If so, you could probably utf-16-encode the text before you start parsing?

Note that `MessageEntity` is a pure implementation of the TG definition. As it does not have a direct connection to the text covered by the entity, TG does not adjust the offset or length in any way.
> If so, you could probably utf-16-encode the text before you start parsing?

Yes, but if the code of parsing entities is hard to refactor (for example, using a third-party library, which assumes `str` type everywhere), this would not be a good solution.

> Note that `MessageEntity` is a pure implementation of the TG definition. As it does not have a direct connection to the text covered by the entity, TG does not adjust the offset or length in any way.

In my opinion, since `str` is the most common type in processing strings, a tool class for unicode string may ease the burden of developers, and it is actually the thing that a _wrapper_ should do.
Fair points. I agree that PTB can provide more utility functionality for working with MessageEntity, I just took a bit to understand what exactly you're asking for : ) If I understand correctly now, you'd like to have a conversion functionality like

```python
def adjust_message_entities_for_utf_16(text: str, entities: Sequence[MessageEntity]) -> Sequence[MessageEntity]:
    """Utility functionality for converting the offset and length of entities from unicode to UTF-16.

    Tip:
       Useful if you want to express formatting of text containing arbitrary unicode characters with `MessageEntity` objects
       but do not want to take care of caculating in UTF-16 space. Instead you can just count characters with the cursor
       in your editor.

    Args:
        text: The text that the entities belong to
        entities: Sequence of entities with offset and length calculated in UTF-8 encoding

    Returns:
        Sequence[MessageEntity]: 
    """
```

Based on your snippet I've build an mwe. Suing `utf_16_entities` gives the same result as using `adjust_message_entities_for_utf_16(text, unicode_entities)`

<details>

```python
import asyncio
from collections.abc import Sequence
from pprint import pprint

from telegram import Bot, MessageEntity


def adjust_message_entities_for_utf_16(
    text: str, entities: Sequence[MessageEntity]
) -> Sequence[MessageEntity]:
    cur_index = 0
    accumulated_len = 0
    out = []
    for entity in entities:
        filler_text = text[cur_index : entity.offset]
        accumulated_len += len(filler_text.encode("utf-16-le"))
        cur_off = accumulated_len // 2

        entity_text = text[entity.offset: entity.offset + entity.length]
        accumulated_len += len(entity_text.encode("utf-16-le"))
        cur_len = accumulated_len // 2 - cur_off
        out.append(
            MessageEntity(
                offset=cur_off, length=cur_len, type=entity.type, language=entity.language
            )
        )
        cur_index = entity.offset + entity.length

    return out


async def main():
    text = "𠌕 bold 𝄢 italic\nunderlined: 𝛙𝌢𑁍"
    unicode_entities = [
        MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
        MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
        MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
    ]
    utf_16_entities = [
        MessageEntity(offset=3, length=4, type=MessageEntity.BOLD),
        MessageEntity(offset=11, length=6, type=MessageEntity.ITALIC),
        MessageEntity(offset=30, length=6, type=MessageEntity.UNDERLINE),
    ]
    async with Bot(token="token") as bot:
        await bot.send_message(
            chat_id=123,
            text=text,
            entities=adjust_message_entities_for_utf_16(text, unicode_entities),
            # entities=utf_16_entities,
        )


if __name__ == "__main__":
    asyncio.run(main())
```

</details>

However, I see no need for a new class `UnicodeMessageEntity`. This would be a rather tight integration of this conversion into the bot methods, plus a deviation from the Bot API with we generally avoid. For starters, I think adding a helper function should be enough.
I'm now wondering where such a function would be suitable - in `telegram.helpers` as simple function or as static method of `MessageEntity`? The latter may be a tad easier to find, but binding this helper function to the class doesn't give any inherit benefit 🤔 I guess I'd tend for `helpers`. Maybe you can give an opinion on where you'd be looking for this?
I'd prefer static method of `MessageEntity`. It's easier to find, and can also let developers aware there is a problem of translating the offsets at their first use.


Okay, fair point, let's go ahead with those then :)
Would you like to send a PR? We can probably come up with a better function name and also docstring (an example would be nice). In the code snippets, I would like to improve the variable names somewhat to make them more self-speaking :)
I've made a PR #4323

Commit message: fix doc and test

Patch to review:
diff --git a/telegram/_messageentity.py b/telegram/_messageentity.py
index 2f7fb7d6179..0c35f3c0010 100644
--- a/telegram/_messageentity.py
+++ b/telegram/_messageentity.py
@@ -18,7 +18,7 @@
 # along with this program.  If not, see [http://www.gnu.org/licenses/].
 """This module contains an object that represents a Telegram MessageEntity."""
 
-from typing import TYPE_CHECKING, Final, List, Optional
+from typing import TYPE_CHECKING, Final, List, Optional, Sequence
 
 from telegram import constants
 from telegram._telegramobject import TelegramObject
@@ -140,6 +140,92 @@ def de_json(cls, data: Optional[JSONDict], bot: "Bot") -> Optional["MessageEntit
 
         return super().de_json(data=data, bot=bot)
 
+    @staticmethod
+    def adjust_message_entities_to_utf_16(
+        text: str, entities: Sequence["MessageEntity"]
+    ) -> Sequence["MessageEntity"]:
+        """Utility functionality for converting the offset and length of entities from
+        Unicode (`str`) to UTF-16 (`utf-16-le` encoded `bytes`).
+
+        Tip:
+        Only the offsets and lengths calulated in UTF-16 is acceptable by the Telegram Bot API.
+        If they are calculated using the Unicode string (`str` object), errors may occur
+        when the text contains characters that are not in the Basic Multilingual Plane (BMP).
+
+        Examples:
+        Below is a snippet of code that demonstrates how to use this function to convert
+        entities from Unicode to UTF-16 space. The `unicode_entities` are calculated in
+        Unicode and the `utf_16_entities` are calculated in UTF-16.
+
+        ```python
+        text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+        #         ^^^^   ^^^^^^             ^^^
+        unicode_entities = [
+            MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
+            MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
+            MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
+        ]
+        utf_16_entities = MessageEntity.adjust_message_entities_to_utf_16(
+            text, unicode_entities
+        )
+        await bot.send_message(
+            chat_id=123,
+            text=text,
+            entities=utf_16_entities,
+        )
+        # utf_16_entities[0]: offset=3, length=4
+        # utf_16_entities[1]: offset=11, length=6
+        # utf_16_entities[2]: offset=30, length=6
+        ```
+
+        Args:
+            text (:obj:`str`): The text that the entities belong to
+            entities (List[:class:`telegram.MessageEntity`]): Sequence of entities
+                with offset and length calculated in Unicode
+
+        Returns:
+            List[:class:`telegram.MessageEntity`]: Sequence of entities with offset and length
+                calculated in UTF-16 encoding
+        """
+        offsets_set = set()
+        # collect all offsets occured
+        for entity in entities:
+            offsets_set.add(entity.offset)
+            offsets_set.add(entity.offset + entity.length)
+        # get sorted offsets
+        offsets_list = sorted(offsets_set)
+        # indexes_map stores the index of each `cut_offset` in `offsets_list`
+        indexes_map = {cut_offset: i for i, cut_offset in enumerate(offsets_list)}
+        # calculate the length of each slice in utf-16
+        accumulated_length = 0
+        utf16_lengths = []
+        # what utf16_lengths stores:
+        # utf16_lengths[indexes_map[offset]] = len(text[:offset].encode("utf-16-le")) // 2
+        for i, cut_offset in enumerate(offsets_list):
+            last_cut_offset = offsets_list[i - 1] if i > 0 else 0
+            text_slice = text[last_cut_offset:cut_offset]
+            accumulated_length += len(text_slice.encode("utf-16-le")) // 2
+            utf16_lengths.append(accumulated_length)
+        # get the final output entites
+        out = []
+        for entity in entities:
+            translated_offset = utf16_lengths[indexes_map[entity.offset]]
+            translated_length = (
+                utf16_lengths[indexes_map[entity.offset + entity.length]] - translated_offset
+            )
+            new_entity = MessageEntity(
+                type=entity.type,
+                offset=translated_offset,
+                length=translated_length,
+                url=entity.url,
+                user=entity.user,
+                language=entity.language,
+                custom_emoji_id=entity.custom_emoji_id,
+                api_kwargs={**entity.api_kwargs},
+            )
+            out.append(new_entity)
+        return out
+
     ALL_TYPES: Final[List[str]] = list(constants.MessageEntityType)
     """List[:obj:`str`]: A list of all available message entity types."""
     BLOCKQUOTE: Final[str] = constants.MessageEntityType.BLOCKQUOTE
diff --git a/tests/test_messageentity.py b/tests/test_messageentity.py
index b14ec79c321..ec8cb89d249 100644
--- a/tests/test_messageentity.py
+++ b/tests/test_messageentity.py
@@ -96,3 +96,17 @@ def test_equality(self):
 
         assert a != d
         assert hash(a) != hash(d)
+
+    def test_fix_utf16(self):
+        text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+        unicode_entities = [
+            MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
+            MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
+            MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
+        ]
+        utf_16_entities = MessageEntity.adjust_message_entities_to_utf_16(text, unicode_entities)
+        offsets_lengths = [(3, 4), (11, 6), (30, 6)]
+        for entity, offset_length in zip(utf_16_entities, offsets_lengths):
+            offset, length = offset_length
+            assert entity.offset == offset
+            assert entity.length == length


Code review comments:
diff_hunk: @@ -140,6 +140,92 @@ def de_json(cls, data: Optional[JSONDict], bot: "Bot") -> Optional["MessageEntit
 
         return super().de_json(data=data, bot=bot)
 
+    @staticmethod
+    def adjust_message_entities_to_utf_16(
+        text: str, entities: Sequence["MessageEntity"]
+    ) -> Sequence["MessageEntity"]:
+        """Utility functionality for converting the offset and length of entities from
+        Unicode (`str`) to UTF-16 (`utf-16-le` encoded `bytes`).
+
+        Tip:
+        Only the offsets and lengths calulated in UTF-16 is acceptable by the Telegram Bot API.
+        If they are calculated using the Unicode string (`str` object), errors may occur
+        when the text contains characters that are not in the Basic Multilingual Plane (BMP).
+
+        Examples:
+        Below is a snippet of code that demonstrates how to use this function to convert
+        entities from Unicode to UTF-16 space. The `unicode_entities` are calculated in
+        Unicode and the `utf_16_entities` are calculated in UTF-16.
+
+        ```python
+        text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+        #         ^^^^   ^^^^^^             ^^^
path: telegram/_messageentity.py
text: @user1:
it looks like the marking does not align. please make sure that this renders correctly in the html docs

diff_hunk: @@ -140,6 +140,92 @@ def de_json(cls, data: Optional[JSONDict], bot: "Bot") -> Optional["MessageEntit
 
         return super().de_json(data=data, bot=bot)
 
+    @staticmethod
+    def adjust_message_entities_to_utf_16(
+        text: str, entities: Sequence["MessageEntity"]
+    ) -> Sequence["MessageEntity"]:
+        """Utility functionality for converting the offset and length of entities from
+        Unicode (`str`) to UTF-16 (`utf-16-le` encoded `bytes`).
+
+        Tip:
+        Only the offsets and lengths calulated in UTF-16 is acceptable by the Telegram Bot API.
+        If they are calculated using the Unicode string (`str` object), errors may occur
+        when the text contains characters that are not in the Basic Multilingual Plane (BMP).
+
+        Examples:
+        Below is a snippet of code that demonstrates how to use this function to convert
+        entities from Unicode to UTF-16 space. The `unicode_entities` are calculated in
+        Unicode and the `utf_16_entities` are calculated in UTF-16.
+
+        ```python
+        text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+        #         ^^^^   ^^^^^^             ^^^
+        unicode_entities = [
+            MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
+            MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
+            MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
+        ]
+        utf_16_entities = MessageEntity.adjust_message_entities_to_utf_16(
+            text, unicode_entities
+        )
+        await bot.send_message(
+            chat_id=123,
+            text=text,
+            entities=utf_16_entities,
+        )
+        # utf_16_entities[0]: offset=3, length=4
+        # utf_16_entities[1]: offset=11, length=6
+        # utf_16_entities[2]: offset=30, length=6
+        ```
+
+        Args:
+            text (:obj:`str`): The text that the entities belong to
+            entities (List[:class:`telegram.MessageEntity`]): Sequence of entities
+                with offset and length calculated in Unicode
+
+        Returns:
+            List[:class:`telegram.MessageEntity`]: Sequence of entities with offset and length
+                calculated in UTF-16 encoding
+        """
+        offsets_set = set()
+        # collect all offsets occured
+        for entity in entities:
+            offsets_set.add(entity.offset)
+            offsets_set.add(entity.offset + entity.length)
+        # get sorted offsets
+        offsets_list = sorted(offsets_set)
+        # indexes_map stores the index of each `cut_offset` in `offsets_list`
+        indexes_map = {cut_offset: i for i, cut_offset in enumerate(offsets_list)}
+        # calculate the length of each slice in utf-16
+        accumulated_length = 0
+        utf16_lengths = []
+        # what utf16_lengths stores:
+        # utf16_lengths[indexes_map[offset]] = len(text[:offset].encode("utf-16-le")) // 2
path: telegram/_messageentity.py
text: @user1:
this comment is not particularly helpful. A code comment should explain what the meaning of the code is or why it is done. In this case, the meaning of the code would be something like `for each split_mark in unicode-space it stores the corresponding split_mark in utf_16-space`

diff_hunk: @@ -96,3 +96,17 @@ def test_equality(self):
 
         assert a != d
         assert hash(a) != hash(d)
+
+    def test_fix_utf16(self):
path: tests/test_messageentity.py
text: @user1:
please move to before `test_equality`

diff_hunk: @@ -140,6 +140,92 @@ def de_json(cls, data: Optional[JSONDict], bot: "Bot") -> Optional["MessageEntit
 
         return super().de_json(data=data, bot=bot)
 
+    @staticmethod
+    def adjust_message_entities_to_utf_16(
+        text: str, entities: Sequence["MessageEntity"]
+    ) -> Sequence["MessageEntity"]:
+        """Utility functionality for converting the offset and length of entities from
+        Unicode (`str`) to UTF-16 (`utf-16-le` encoded `bytes`).
+
+        Tip:
+        Only the offsets and lengths calulated in UTF-16 is acceptable by the Telegram Bot API.
+        If they are calculated using the Unicode string (`str` object), errors may occur
+        when the text contains characters that are not in the Basic Multilingual Plane (BMP).
+
+        Examples:
+        Below is a snippet of code that demonstrates how to use this function to convert
+        entities from Unicode to UTF-16 space. The `unicode_entities` are calculated in
+        Unicode and the `utf_16_entities` are calculated in UTF-16.
+
+        ```python
+        text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+        #         ^^^^   ^^^^^^             ^^^
+        unicode_entities = [
+            MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
+            MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
+            MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
+        ]
+        utf_16_entities = MessageEntity.adjust_message_entities_to_utf_16(
+            text, unicode_entities
+        )
+        await bot.send_message(
+            chat_id=123,
+            text=text,
+            entities=utf_16_entities,
+        )
+        # utf_16_entities[0]: offset=3, length=4
+        # utf_16_entities[1]: offset=11, length=6
+        # utf_16_entities[2]: offset=30, length=6
+        ```
+
+        Args:
+            text (:obj:`str`): The text that the entities belong to
+            entities (List[:class:`telegram.MessageEntity`]): Sequence of entities
+                with offset and length calculated in Unicode
+
+        Returns:
+            List[:class:`telegram.MessageEntity`]: Sequence of entities with offset and length
path: telegram/_messageentity.py
text: @user1:
```suggestion
            Sequence[:class:`telegram.MessageEntity`]: Sequence of entities with offset and length
```

diff_hunk: @@ -140,6 +140,92 @@ def de_json(cls, data: Optional[JSONDict], bot: "Bot") -> Optional["MessageEntit
 
         return super().de_json(data=data, bot=bot)
 
+    @staticmethod
+    def adjust_message_entities_to_utf_16(
+        text: str, entities: Sequence["MessageEntity"]
+    ) -> Sequence["MessageEntity"]:
+        """Utility functionality for converting the offset and length of entities from
+        Unicode (`str`) to UTF-16 (`utf-16-le` encoded `bytes`).
+
+        Tip:
+        Only the offsets and lengths calulated in UTF-16 is acceptable by the Telegram Bot API.
+        If they are calculated using the Unicode string (`str` object), errors may occur
+        when the text contains characters that are not in the Basic Multilingual Plane (BMP).
+
+        Examples:
+        Below is a snippet of code that demonstrates how to use this function to convert
+        entities from Unicode to UTF-16 space. The `unicode_entities` are calculated in
+        Unicode and the `utf_16_entities` are calculated in UTF-16.
+
+        ```python
path: telegram/_messageentity.py
text: @user1:
This markdown syntax does not work in rst it should be 

.. code-block:: python

Iirc

diff_hunk: @@ -140,6 +140,92 @@ def de_json(cls, data: Optional[JSONDict], bot: "Bot") -> Optional["MessageEntit
 
         return super().de_json(data=data, bot=bot)
 
+    @staticmethod
+    def adjust_message_entities_to_utf_16(
+        text: str, entities: Sequence["MessageEntity"]
+    ) -> Sequence["MessageEntity"]:
+        """Utility functionality for converting the offset and length of entities from
+        Unicode (`str`) to UTF-16 (`utf-16-le` encoded `bytes`).
+
+        Tip:
+        Only the offsets and lengths calulated in UTF-16 is acceptable by the Telegram Bot API.
+        If they are calculated using the Unicode string (`str` object), errors may occur
+        when the text contains characters that are not in the Basic Multilingual Plane (BMP).
+
+        Examples:
+        Below is a snippet of code that demonstrates how to use this function to convert
+        entities from Unicode to UTF-16 space. The `unicode_entities` are calculated in
+        Unicode and the `utf_16_entities` are calculated in UTF-16.
+
+        ```python
+        text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+        #         ^^^^   ^^^^^^             ^^^
+        unicode_entities = [
+            MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
+            MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
+            MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
+        ]
+        utf_16_entities = MessageEntity.adjust_message_entities_to_utf_16(
+            text, unicode_entities
+        )
+        await bot.send_message(
+            chat_id=123,
+            text=text,
+            entities=utf_16_entities,
+        )
+        # utf_16_entities[0]: offset=3, length=4
+        # utf_16_entities[1]: offset=11, length=6
+        # utf_16_entities[2]: offset=30, length=6
+        ```
+
+        Args:
+            text (:obj:`str`): The text that the entities belong to
+            entities (List[:class:`telegram.MessageEntity`]): Sequence of entities
+                with offset and length calculated in Unicode
+
+        Returns:
+            List[:class:`telegram.MessageEntity`]: Sequence of entities with offset and length
+                calculated in UTF-16 encoding
+        """
+        offsets_set = set()
+        # collect all offsets occured
+        for entity in entities:
+            offsets_set.add(entity.offset)
+            offsets_set.add(entity.offset + entity.length)
+        # get sorted offsets
+        offsets_list = sorted(offsets_set)
path: telegram/_messageentity.py
text: @user1:
It should be possible to shorten this with list comprehension or itertools and avaid the temporary set. Additionally the word "offset" is somewhat misleading as the entity length is considered as well. Maybe "split_marks" or "split_positions"?
Something like
```python
split_mark = sorted(itertools.chain(((e.offset, e.offset+e.lenght) for e in entities)))
```
Could work I guess

diff_hunk: @@ -140,6 +140,92 @@ def de_json(cls, data: Optional[JSONDict], bot: "Bot") -> Optional["MessageEntit
 
         return super().de_json(data=data, bot=bot)
 
+    @staticmethod
+    def adjust_message_entities_to_utf_16(
+        text: str, entities: Sequence["MessageEntity"]
+    ) -> Sequence["MessageEntity"]:
+        """Utility functionality for converting the offset and length of entities from
+        Unicode (`str`) to UTF-16 (`utf-16-le` encoded `bytes`).
+
+        Tip:
+        Only the offsets and lengths calulated in UTF-16 is acceptable by the Telegram Bot API.
+        If they are calculated using the Unicode string (`str` object), errors may occur
+        when the text contains characters that are not in the Basic Multilingual Plane (BMP).
+
+        Examples:
+        Below is a snippet of code that demonstrates how to use this function to convert
+        entities from Unicode to UTF-16 space. The `unicode_entities` are calculated in
+        Unicode and the `utf_16_entities` are calculated in UTF-16.
+
+        ```python
+        text = "𠌕 bold 𝄢 italic underlined: 𝛙𝌢𑁍"
+        #         ^^^^   ^^^^^^             ^^^
+        unicode_entities = [
+            MessageEntity(offset=2, length=4, type=MessageEntity.BOLD),
+            MessageEntity(offset=9, length=6, type=MessageEntity.ITALIC),
+            MessageEntity(offset=28, length=3, type=MessageEntity.UNDERLINE),
+        ]
+        utf_16_entities = MessageEntity.adjust_message_entities_to_utf_16(
+            text, unicode_entities
+        )
+        await bot.send_message(
+            chat_id=123,
+            text=text,
+            entities=utf_16_entities,
+        )
+        # utf_16_entities[0]: offset=3, length=4
+        # utf_16_entities[1]: offset=11, length=6
+        # utf_16_entities[2]: offset=30, length=6
+        ```
+
+        Args:
+            text (:obj:`str`): The text that the entities belong to
+            entities (List[:class:`telegram.MessageEntity`]): Sequence of entities
+                with offset and length calculated in Unicode
+
+        Returns:
+            List[:class:`telegram.MessageEntity`]: Sequence of entities with offset and length
+                calculated in UTF-16 encoding
+        """
+        offsets_set = set()
+        # collect all offsets occured
+        for entity in entities:
+            offsets_set.add(entity.offset)
+            offsets_set.add(entity.offset + entity.length)
+        # get sorted offsets
+        offsets_list = sorted(offsets_set)
+        # indexes_map stores the index of each `cut_offset` in `offsets_list`
+        indexes_map = {cut_offset: i for i, cut_offset in enumerate(offsets_list)}
path: telegram/_messageentity.py
text: @user1:
I think this map is superflous - all you need is a mapping from unidoce-positions to utf_16-positions. So something like 

```python
position_mapping: Dict[int, int] = {}
for split_mark in split_marks:
    # do computation
    position_mapping[split_mark] = accumulated_length
```

should already do the trick