Repository: LonamiWebs/Telethon

Issue:
0-length `MessageEntity` is no longer valid but MD/HTML parser still produce them
Formerly, Telegram simply ignored 0-length `MessageEntity`, but several days before, Telegram has silently changed its server-side behavior to throw an `RPCError 400: ENTITY_BOUNDS_INVALID`. However, the Markdown and HTML parsers of Telethon still produce 0-length `MessageEntity` if the user input does have empty tags.

A typical use case (https://github.com/Rongronggg9/RSS-to-Telegram-Bot/blob/15aef44b6e9b870ec9411398cb83ccb8f38acf68/src/command/administration.py#L17-L34) that can reproduce this issue (`value` can be an empty string, which is intended):

```python
parseKeyValuePair = re.compile(r'^/\S+\s+([^\s=]+)(?:\s*=\s*|\s+)?(.+)?$')


@command_gatekeeper(only_manager=True)
async def cmd_set_option(event: Union[events.NewMessage.Event, Message], *_, lang: Optional[str] = None, **__):
    kv = parseKeyValuePair.match(event.raw_text)
    if not kv:  # return options info
        options = db.EffectiveOptions.options
        msg = (
                f'<b>{i18n[lang]["current_options"]}</b>\n\n'
                + '\n'.join(f'<code>{key}</code> = <code>{value}</code> '
                            f'({i18n[lang]["option_value_type"]}: <code>{type(value).__name__}</code>)'
                            for key, value in options.items())
                + '\n\n' + i18n[lang]['cmd_set_option_usage_prompt_html']
        )
        await event.respond(msg, parse_mode='html')
        return
	key, value = kv.groups()
	...
```

****

I consider there are 3 solutions:
1. simply drop 0-length `MessageEntity` (or even all invalid ones) before returning the parse result.
2. like 1, but make a warning.
3. keep the current behavior but update the document and post an announcement on https://t.me/TelethonUpdates.

I'd rather prefer 1 because:
1. The HTTP bot API allows empty tags. ![](https://user-images.githubusercontent.com/15956627/180619477-c9c396e7-3caf-4f9c-b2f0-09ee62c9f1f7.png)
2. The HTTP bot API even allows 0-length entities. ![](https://user-images.githubusercontent.com/15956627/180619634-7760ebf6-3b11-4194-9769-8cad6ef53aac.png)
3. Empty tags in HTML are totally valid (not sure for Markdown).
4. Not to introduce breaking changes (since Telegram formerly allowed them).
5. `f-string`, `str.format()`, `%` operator, or other templating methods are popular. They usually produce empty tags which are intended (at least in my use case, there are several templates that produce intended empty tags, one of which has been shown before).
6. The current documentation does not mention that it can't accept empty tags.
7. The Markdown/HTML parsers provided by Telethon seem to simulate the behavior of the ones provided by HTTP bot API.

Obviously, this is not a bug of Telethon, but I wish Telethon could "fix" it urgently since the change has a wide influence but the use case is common.

****

I consider there are 2 possible "fixes":
1. prevent the Markdown and HTML parsers from creating 0-length entities
2. let `helpers.strip_text()` to delete them

I'd rather prefer 2 because:
1. downstream projects can use it even if they create entities themselves
2. preventing the Markdown parser from creating 0-length entities requires many modifications

I will submit a PR to the `v1.24` branch to fix it according to https://github.com/LonamiWebs/Telethon/issues/3880#issuecomment-1190811979.

I am not sure if I should add `ENTITY_BOUNDS_INVALID` as a new exception since `v1` should not receive new features that is not necessary according to the same context.

Discussion:
Lonami: Thanks, feel free to implement your suggested simplification. Let me know if you want to do that in this or a separate PR.
Rongronggg9: > Thanks, feel free to implement your suggested simplification. Let me know if you want to do that in this or a separate PR.

Done. I've implemented it in this PR and added unit tests and detailed comments.
Lonami: > and added unit tests and detailed comments

The "tests" in this library are pretty much a joke (they're not really ran often, or at all, and their coverage is really low), but if it helped you verify the code works correctly then it's fine.

Patch:
diff --git a/telethon/helpers.py b/telethon/helpers.py
index 6c782b0b..aac058b0 100644
--- a/telethon/helpers.py
+++ b/telethon/helpers.py
@@ -64,40 +64,74 @@ def within_surrogate(text, index, *, length=None):
 
 def strip_text(text, entities):
     """
-    Strips whitespace from the given text modifying the provided entities.
+    Strips whitespace from the given surrogated text modifying the provided
+    entities, also removing any empty (0-length) entities.
 
-    This assumes that there are no overlapping entities, that their length
-    is greater or equal to one, and that their length is not out of bounds.
+    This assumes that the length of entities is greater or equal to 0, and
+    that no entity is out of bounds.
     """
     if not entities:
         return text.strip()
 
-    while text and text[-1].isspace():
-        e = entities[-1]
-        if e.offset + e.length == len(text):
-            if e.length == 1:
-                del entities[-1]
-                if not entities:
-                    return text.strip()
-            else:
-                e.length -= 1
-        text = text[:-1]
-
-    while text and text[0].isspace():
-        for i in reversed(range(len(entities))):
-            e = entities[i]
-            if e.offset != 0:
-                e.offset -= 1
-                continue
-
-            if e.length == 1:
-                del entities[0]
-                if not entities:
-                    return text.lstrip()
+    len_ori = len(text)
+    text = text.lstrip()
+    left_offset = len_ori - len(text)
+    text = text.rstrip()
+    len_final = len(text)
+
+    for i in reversed(range(len(entities))):
+        e = entities[i]
+        if e.length == 0:
+            del entities[i]
+            continue
+
+        if e.offset + e.length > left_offset:
+            if e.offset >= left_offset:
+                #  0 1|2 3 4 5       |       0 1|2 3 4 5
+                #     ^     ^        |          ^
+                #   lo(2)  o(5)      |      o(2)/lo(2)
+                e.offset -= left_offset
+                #     |0 1 2 3       |          |0 1 2 3
+                #           ^        |          ^
+                #     o=o-lo(3=5-2)  |    o=o-lo(0=2-2)
             else:
-                e.length -= 1
+                # e.offset < left_offset and e.offset + e.length > left_offset
+                #  0 1 2 3|4 5 6 7 8 9 10
+                #   ^     ^           ^
+                #  o(1) lo(4)      o+l(1+9)
+                e.length = e.offset + e.length - left_offset
+                e.offset = 0
+                #         |0 1 2 3 4 5 6
+                #         ^           ^
+                #        o(0)  o+l=0+o+l-lo(6=0+6=0+1+9-4)
+        else:
+            # e.offset + e.length <= left_offset
+            #   0 1 2 3|4 5
+            #  ^       ^
+            # o(0)   o+l(4)
+            #        lo(4)
+            del entities[i]
+            continue
 
-        text = text[1:]
+        if e.offset + e.length <= len_final:
+            # |0 1 2 3 4 5 6 7 8 9
+            #   ^                 ^
+            #  o(1)       o+l(1+9)/lf(10)
+            continue
+        if e.offset >= len_final:
+            # |0 1 2 3 4
+            #           ^
+            #       o(5)/lf(5)
+            del entities[i]
+        else:
+            # e.offset < len_final and e.offset + e.length > len_final
+            # |0 1 2 3 4 5 (6) (7) (8) (9)
+            #   ^         ^           ^
+            #  o(1)     lf(6)      o+l(1+8)
+            e.length = len_final - e.offset
+            # |0 1 2 3 4 5
+            #   ^         ^
+            #  o(1) o+l=o+lf-o=lf(6=1+5=1+6-1)
 
     return text
 


Test patch:
diff --git a/tests/telethon/test_helpers.py b/tests/telethon/test_helpers.py
index 689db8af..47705ca2 100644
--- a/tests/telethon/test_helpers.py
+++ b/tests/telethon/test_helpers.py
@@ -7,11 +7,34 @@ from base64 import b64decode
 import pytest
 
 from telethon import helpers
+from telethon.utils import get_inner_text
+from telethon.tl.types import MessageEntityUnknown as Meu
 
 
 def test_strip_text():
-    assert helpers.strip_text(" text ", []) == "text"
-    # I can't interpret the rest of the code well enough yet
+    text = ' text '
+    text_stripped = 'text'
+    entities_before_and_after = (
+        ([], []),
+        ([Meu(i, 0) for i in range(10)], []),  # del ''
+        ([Meu(0, 0), Meu(0, 1), Meu(5, 1)], []),  # del '', ' ', ' '
+        ([Meu(0, 3)], [Meu(0, 2)]),  # ' te' -> 'te'
+        ([Meu(3, 1)], [Meu(2, 1)]),  # 'x'
+        ([Meu(3, 2)], [Meu(2, 2)]),  # 'xt'
+        ([Meu(3, 3)], [Meu(2, 2)]),  # 'xt ' -> 'xt'
+        ([Meu(0, 6)], [Meu(0, 4)]),  # ' text ' -> 'text'
+    )
+    for entities_before, entities_expected in entities_before_and_after:
+        entities_for_test = [Meu(meu.offset, meu.length) for meu in entities_before]  # deep copy
+        text_after = helpers.strip_text(text, entities_for_test)
+        assert text_after == text_stripped
+        assert sorted((e.offset, e.length) for e in entities_for_test) \
+               == sorted((e.offset, e.length) for e in entities_expected)
+        inner_text_before = get_inner_text(text, entities_before)
+        inner_text_before_stripped = [t.strip() for t in inner_text_before]
+        inner_text_after = get_inner_text(text_after, entities_for_test)
+        for t in inner_text_after:
+            assert t in inner_text_before_stripped
 
 
 class TestSyncifyAsyncContext: