Repository: django/django
Issue category: Feature Request

Issue:
Improve utils.text.Truncator &co to use a full HTML parser.
Description
		
(last modified by Carlton Gibson)		
Original description:
I'm using Truncator.chars to truncate wikis, and it sometimes truncates in the middle of &quot; entities, resulting in '<p>some text &qu</p>'
This is a limitation of the regex based implementation (which has had security issues, and presents an intractable problem). 
Better to move to use a HTML parser, for Truncate, and strip_tags(), via html5lib and bleach.
Improve utils.text.Truncator &co to use a full HTML parser.
Description
		
(last modified by Carlton Gibson)		
Original description:
I'm using Truncator.chars to truncate wikis, and it sometimes truncates in the middle of &quot; entities, resulting in '<p>some text &qu</p>'
This is a limitation of the regex based implementation (which has had security issues, and presents an intractable problem). 
Better to move to use a HTML parser, for Truncate, and strip_tags(), via html5lib and bleach.


Hints:
['Hi Thomas. Any chance of an example string (hopefully minimal) that creates the behaviour so we can have a look?', 1565074082.0]
["I think now that the security release are out let's just add bleach as dependency on master and be done with it?", 1565076701.0]
["Here's an example \u200bhttps://repl.it/@tdhooper/Django-truncate-entities-bug", 1565079129.0]
['btw I confused truncator with strip_tags. So in this case the answer would be to rewrite the parser using html5lib, while split_tags would use bleach which in turn then uses html5lib as well.', 1565079313.0]
['Looks like it can be fixed with this regex change \u200bhttps://github.com/django/django/pull/11633/files', 1565085272.0]
['Example implemetation of _truncate_html() using html5lib, by Florian Apolloner', 1565151161.0]
["Right, good news is this isn't a regression from 7f65974f8219729c047fbbf8cd5cc9d80faefe77. The new example case fails on v2.2.3 &co. The suggestion for the regex change is in the part not changed as part of 7f65974f8219729c047fbbf8cd5cc9d80faefe77. (Which is why the new case fails, I suppose :) I don't want to accept a tweaking of the regex here. Rather, we should move to using html5lib as Florian suggests. Possibly this would entail small changes in behaviour around edge cases, to be called out in release notes, but would be a big win overall. This has previously been discussed by the Security Team as the required way forward. I've updated the title/description and will Accept accordingly. I've attached an initial WIP patch by Florian of an html5lib implementation of the core _truncate_html() method. An implementation of strip_tags() using bleach would go something like: bleach.clean(text, tags=[], strip=True, strip_comments=True) Thomas, would taking on making changes like these be something you'd be willing/keen to do? If so, I'm very happy to input to assist in any way. :)", 1565151467.0]
['Hi Carlton, that would be fun, but this is bigger than I have time for now. It looks like you all have it in hand.', 1565172776.0]
["Do we want to make both html5lib and bleach required dependencies of Django? html5lib latest release is now 20 months ago, and when I read issues like \u200bhttps://github.com/html5lib/html5lib-python/issues/419 without any maintainer feedback, I'm a bit worried. What about the security report workflow for those libs? What if a security issue is discovered in html5 lib and the maintainers are unresponsive? Sorry to sound a bit negative, but I think those questions must be asked.", 1565172804.0]
["Yep Claude, absolutely. I think there's two difficulties we could face: trying to successfully sanitize HTML with regexes. (Help) Make sure html5lib-python is maintained. The first of these is intractable. The second not. 🙂 I've put out some feelers to try and find out more. This is pressing for Python and pip now, not for us for a while yet. If we look at \u200bhttps://github.com/html5lib/html5lib-python/issues/361 it seems there's some money on the table from tidelift potentially. We COULD allocate some time in a pinch I think. AND it's just (even with the emphasis, cough) a wrapper around the underlying C library, so whilst 20 months seems a long time, I'm not sure the release cadence is really an issue. BUT, yes, absolutely. Let's hammer this out properly before we commit. 👍 I will open a mailing list thread when I know more.", 1565181330.0]
["AND it's just (even with the emphasis, cough) a wrapper around the underlying C library, so whilst 20 months seems a long time, I'm not sure the release cadence is really an issue. OK, that last one isn't at all true. (Looking at the source it's the entire implementation.) Update: I had \u200b`lxml` in mind.", 1565182199.0]
["To be clear, I'm also convinced parsing is more reliable than regexes. I just think we have to double-think before adding a dependency, because as the name implies, we depend on it and therefore we must be able to trust its maintainers. Some guarantees about the security process and serious bugs fixing should be obtained. Without that, we are just outsourcing problems.", 1565184992.0]
['@Claude: 💯👍 Totally agree.', 1565227033.0]
["Duplicate in #30700, with \u200bfailing test case provided. I've tried contacting maintainers of HTML5lib with no success. I've re-opened \u200bhttps://github.com/django/django/pull/11633 (original regex based suggestion) so we can at least assess it as a possible stop-gap.", 1565577102.0]
["Paging Jon, to ask his opinion on this. Hey Jon, I see you've made a number of PRs to both html5lib, and bleach. To me, at this point, html5lib essentially looks unmaintained. I don't have personal capacity to give to it, as cool as it is as a project. Arguably we (Fellows) could allocate it _some_ time, since we spend a fair bit already messing around with regexes but that would be small, and we couldn't take it on whole, so can I ask your thoughts? Is html5lib in trouble? If so, as a user, what are your plans, if any? And from that, what do you think about Django adopting it? What's the alternative? Thanks for the thought and insight.", 1566304019.0]
["To me, at this point, html5lib essentially looks unmaintained. I agree with this observation. The previous main maintainer looks to have stopped working on the project. Responses to issues and PRs have stopped. Is html5lib in trouble? If so, as a user, what are your plans, if any? And from that, what do you think about Django adopting it? What's the alternative? For my own projects, I'll probably continue using html5lib until its staleness creates an observable bug for me. I haven't hit that point yet. Bleach, on the other hand, looks like maintenance has slowed, but not stopped. I believe they have vendored html5lib to allow them to make changes internally. FWIW, I also still use Bleach. --- I'm not familiar with all the details of this ticket, but would the stdlib HTML parser be sufficient? \u200bhttps://docs.python.org/3/library/html.parser.html", 1566650557.0]
["Hi Jon, Thank you for the comments. I will email Will, the maintainer of Bleach, and ask his thoughts too. Bleach has slowed down, but that's because it's Stable/Mature now I would have thought. ...would the stdlib HTML parser be sufficient? Yes. Maybe. Ideally we just thought to bring in Bleach, and with it html5lib since, in theory, that's already working code. (Florian already had a Truncate prototype...) Anyhow... will follow-up.", 1566794092.0]
["Adding some detail after the last post, since you're looking at it David. There was a discussion (with various folks from html5lib, and Mozilla, and ...) about whether html5lib could be put on a better footing. I'm not sure how that panned out in the medium term. (I didn't check what the rhythm looks like now.) There was also talk about whether bleach (or an alternate) could build off html5ever which is the HTML parser from the Mozilla servo project. \u200bhttps://github.com/servo/html5ever \u200bhttps://github.com/SimonSapin/html5ever-python (CFFI bindings.) \u200bhttps://github.com/PyO3/setuptools-rust/tree/main/examples/html-py-ever (Py03 Bindings) That would be pretty cool, but it was clearly a lot of work, and then 2020 happened, so... The other candidate in this space is Matthias' html-sanitizer: \u200bhttps://github.com/matthiask/html-sanitizer — which is built on lxml. That's just to lay down the notes I had gathered. I'm not sure the way forward, but hopefully it's helpful. Very open to ideas though! Thanks for picking it up.", 1672741702.0]
['Hi all lxml is quite a heavy dependency. It works very well but you\'ll wait for the compilation a long time if you do not have wheels. (see \u200bhttps://pypi.org/project/lxml/#files) I think Python packaging is almost a non-issue these days except when it comes to transitive dependencies, and I wouldn\'t want to be in charge of specifying and updating the supported range of lxml versions. That being said, I encountered almost no breaking changes in lxml since \u200b~2009, I use lxml in almost all projects and I can heartily recommend it to anyone. I\'m sure that the regex-based solution has some problems; I\'m sorry to admit I haven\'t read the full thread but I just cannot imagine a situation where using |strip_tags without |safe would lead to a security issue, and why would you want to combine these? There\'s no point to mark a string as safe after stripping all tags. So it\'s only about the fact that the output sometimes isn\'t nice, something which may be fixed by converting as many entities to their unicode equivalents as possible and only truncating afterwards? Last but not least: I haven\'t benchmarked it ever, but I have the suspicion that running bleach or html-sanitizer during rendering may be wasteful in terms of CPU cycles. I only ever use the sanitizer when saving, never when rendering. |strip_tags is obviously applied when rendering and performs well enough in many situations. So, to me strip_tags is a clear case of \u200ba simple implementation with "worse is better" characteristics. I truly hope this is helpful and not just a cold shower (sorry for using "just" here) Thanks, Matthias', 1672742858.0]
['Hey Matthias — that\'s a very useful input. Thank you for your insight. So, to me strip_tags is a clear case of \u200ba simple implementation with "worse is better" characteristics. Let, me review what happened here tomorrow (it was a long while ago) but assuming it makes sense, wontfix + We\'re not accepting any complications to the algorithm — use ... if you need more sophistication may be the neatest way all round.', 1672748487.0]
["\u200bPR I was thinking about Jon's suggestion of using the HTMLParser from the standard library. Since the last comments on this ticket Adam Johnson has also written a blog post on Truncating HTML with Python's HTMLParser which helped inspire my PR, see \u200bblog post. (I'd cc Adam as I've mentioned his name, but not sure how to do that?!) While my PR still needs more work I thought it worth sharing as it may be helpful to Carlton when reviewing tomorrow.", 1672758237.0]
['Hi Thomas. Any chance of an example string (hopefully minimal) that creates the behaviour so we can have a look?', 1565074082.0]
["I think now that the security release are out let's just add bleach as dependency on master and be done with it?", 1565076701.0]
["Here's an example \u200bhttps://repl.it/@tdhooper/Django-truncate-entities-bug", 1565079129.0]
['btw I confused truncator with strip_tags. So in this case the answer would be to rewrite the parser using html5lib, while split_tags would use bleach which in turn then uses html5lib as well.', 1565079313.0]
['Looks like it can be fixed with this regex change \u200bhttps://github.com/django/django/pull/11633/files', 1565085272.0]
['Example implemetation of _truncate_html() using html5lib, by Florian Apolloner', 1565151161.0]
["Right, good news is this isn't a regression from 7f65974f8219729c047fbbf8cd5cc9d80faefe77. The new example case fails on v2.2.3 &co. The suggestion for the regex change is in the part not changed as part of 7f65974f8219729c047fbbf8cd5cc9d80faefe77. (Which is why the new case fails, I suppose :) I don't want to accept a tweaking of the regex here. Rather, we should move to using html5lib as Florian suggests. Possibly this would entail small changes in behaviour around edge cases, to be called out in release notes, but would be a big win overall. This has previously been discussed by the Security Team as the required way forward. I've updated the title/description and will Accept accordingly. I've attached an initial WIP patch by Florian of an html5lib implementation of the core _truncate_html() method. An implementation of strip_tags() using bleach would go something like: bleach.clean(text, tags=[], strip=True, strip_comments=True) Thomas, would taking on making changes like these be something you'd be willing/keen to do? If so, I'm very happy to input to assist in any way. :)", 1565151467.0]
['Hi Carlton, that would be fun, but this is bigger than I have time for now. It looks like you all have it in hand.', 1565172776.0]
["Do we want to make both html5lib and bleach required dependencies of Django? html5lib latest release is now 20 months ago, and when I read issues like \u200bhttps://github.com/html5lib/html5lib-python/issues/419 without any maintainer feedback, I'm a bit worried. What about the security report workflow for those libs? What if a security issue is discovered in html5 lib and the maintainers are unresponsive? Sorry to sound a bit negative, but I think those questions must be asked.", 1565172804.0]
["Yep Claude, absolutely. I think there's two difficulties we could face: trying to successfully sanitize HTML with regexes. (Help) Make sure html5lib-python is maintained. The first of these is intractable. The second not. 🙂 I've put out some feelers to try and find out more. This is pressing for Python and pip now, not for us for a while yet. If we look at \u200bhttps://github.com/html5lib/html5lib-python/issues/361 it seems there's some money on the table from tidelift potentially. We COULD allocate some time in a pinch I think. AND it's just (even with the emphasis, cough) a wrapper around the underlying C library, so whilst 20 months seems a long time, I'm not sure the release cadence is really an issue. BUT, yes, absolutely. Let's hammer this out properly before we commit. 👍 I will open a mailing list thread when I know more.", 1565181330.0]
["AND it's just (even with the emphasis, cough) a wrapper around the underlying C library, so whilst 20 months seems a long time, I'm not sure the release cadence is really an issue. OK, that last one isn't at all true. (Looking at the source it's the entire implementation.) Update: I had \u200b`lxml` in mind.", 1565182199.0]
["To be clear, I'm also convinced parsing is more reliable than regexes. I just think we have to double-think before adding a dependency, because as the name implies, we depend on it and therefore we must be able to trust its maintainers. Some guarantees about the security process and serious bugs fixing should be obtained. Without that, we are just outsourcing problems.", 1565184992.0]
['@Claude: 💯👍 Totally agree.', 1565227033.0]
["Duplicate in #30700, with \u200bfailing test case provided. I've tried contacting maintainers of HTML5lib with no success. I've re-opened \u200bhttps://github.com/django/django/pull/11633 (original regex based suggestion) so we can at least assess it as a possible stop-gap.", 1565577102.0]
["Paging Jon, to ask his opinion on this. Hey Jon, I see you've made a number of PRs to both html5lib, and bleach. To me, at this point, html5lib essentially looks unmaintained. I don't have personal capacity to give to it, as cool as it is as a project. Arguably we (Fellows) could allocate it _some_ time, since we spend a fair bit already messing around with regexes but that would be small, and we couldn't take it on whole, so can I ask your thoughts? Is html5lib in trouble? If so, as a user, what are your plans, if any? And from that, what do you think about Django adopting it? What's the alternative? Thanks for the thought and insight.", 1566304019.0]
["To me, at this point, html5lib essentially looks unmaintained. I agree with this observation. The previous main maintainer looks to have stopped working on the project. Responses to issues and PRs have stopped. Is html5lib in trouble? If so, as a user, what are your plans, if any? And from that, what do you think about Django adopting it? What's the alternative? For my own projects, I'll probably continue using html5lib until its staleness creates an observable bug for me. I haven't hit that point yet. Bleach, on the other hand, looks like maintenance has slowed, but not stopped. I believe they have vendored html5lib to allow them to make changes internally. FWIW, I also still use Bleach. --- I'm not familiar with all the details of this ticket, but would the stdlib HTML parser be sufficient? \u200bhttps://docs.python.org/3/library/html.parser.html", 1566650557.0]
["Hi Jon, Thank you for the comments. I will email Will, the maintainer of Bleach, and ask his thoughts too. Bleach has slowed down, but that's because it's Stable/Mature now I would have thought. ...would the stdlib HTML parser be sufficient? Yes. Maybe. Ideally we just thought to bring in Bleach, and with it html5lib since, in theory, that's already working code. (Florian already had a Truncate prototype...) Anyhow... will follow-up.", 1566794092.0]
["Adding some detail after the last post, since you're looking at it David. There was a discussion (with various folks from html5lib, and Mozilla, and ...) about whether html5lib could be put on a better footing. I'm not sure how that panned out in the medium term. (I didn't check what the rhythm looks like now.) There was also talk about whether bleach (or an alternate) could build off html5ever which is the HTML parser from the Mozilla servo project. \u200bhttps://github.com/servo/html5ever \u200bhttps://github.com/SimonSapin/html5ever-python (CFFI bindings.) \u200bhttps://github.com/PyO3/setuptools-rust/tree/main/examples/html-py-ever (Py03 Bindings) That would be pretty cool, but it was clearly a lot of work, and then 2020 happened, so... The other candidate in this space is Matthias' html-sanitizer: \u200bhttps://github.com/matthiask/html-sanitizer — which is built on lxml. That's just to lay down the notes I had gathered. I'm not sure the way forward, but hopefully it's helpful. Very open to ideas though! Thanks for picking it up.", 1672741702.0]
['Hi all lxml is quite a heavy dependency. It works very well but you\'ll wait for the compilation a long time if you do not have wheels. (see \u200bhttps://pypi.org/project/lxml/#files) I think Python packaging is almost a non-issue these days except when it comes to transitive dependencies, and I wouldn\'t want to be in charge of specifying and updating the supported range of lxml versions. That being said, I encountered almost no breaking changes in lxml since \u200b~2009, I use lxml in almost all projects and I can heartily recommend it to anyone. I\'m sure that the regex-based solution has some problems; I\'m sorry to admit I haven\'t read the full thread but I just cannot imagine a situation where using |strip_tags without |safe would lead to a security issue, and why would you want to combine these? There\'s no point to mark a string as safe after stripping all tags. So it\'s only about the fact that the output sometimes isn\'t nice, something which may be fixed by converting as many entities to their unicode equivalents as possible and only truncating afterwards? Last but not least: I haven\'t benchmarked it ever, but I have the suspicion that running bleach or html-sanitizer during rendering may be wasteful in terms of CPU cycles. I only ever use the sanitizer when saving, never when rendering. |strip_tags is obviously applied when rendering and performs well enough in many situations. So, to me strip_tags is a clear case of \u200ba simple implementation with "worse is better" characteristics. I truly hope this is helpful and not just a cold shower (sorry for using "just" here) Thanks, Matthias', 1672742858.0]
['Hey Matthias — that\'s a very useful input. Thank you for your insight. So, to me strip_tags is a clear case of \u200ba simple implementation with "worse is better" characteristics. Let, me review what happened here tomorrow (it was a long while ago) but assuming it makes sense, wontfix + We\'re not accepting any complications to the algorithm — use ... if you need more sophistication may be the neatest way all round.', 1672748487.0]
["\u200bPR I was thinking about Jon's suggestion of using the HTMLParser from the standard library. Since the last comments on this ticket Adam Johnson has also written a blog post on Truncating HTML with Python's HTMLParser which helped inspire my PR, see \u200bblog post. (I'd cc Adam as I've mentioned his name, but not sure how to do that?!) While my PR still needs more work I thought it worth sharing as it may be helpful to Carlton when reviewing tomorrow.", 1672758237.0]

Patch:
diff --git a/django/utils/text.py b/django/utils/text.py
index 374fd78f927d..9560ebc67840 100644
--- a/django/utils/text.py
+++ b/django/utils/text.py
@@ -2,12 +2,20 @@
 import re
 import secrets
 import unicodedata
+from collections import deque
 from gzip import GzipFile
 from gzip import compress as gzip_compress
+from html import escape
+from html.parser import HTMLParser
 from io import BytesIO
 
 from django.core.exceptions import SuspiciousFileOperation
-from django.utils.functional import SimpleLazyObject, keep_lazy_text, lazy
+from django.utils.functional import (
+    SimpleLazyObject,
+    cached_property,
+    keep_lazy_text,
+    lazy,
+)
 from django.utils.regex_helper import _lazy_re_compile
 from django.utils.translation import gettext as _
 from django.utils.translation import gettext_lazy, pgettext
@@ -80,6 +88,101 @@ def add_truncation_text(text, truncate=None):
     return f"{text}{truncate}"
 
 
+def calculate_truncate_chars_length(length, replacement):
+    truncate_len = length
+    for char in add_truncation_text("", replacement):
+        if not unicodedata.combining(char):
+            truncate_len -= 1
+            if truncate_len == 0:
+                break
+    return truncate_len
+
+
+class TruncateHTMLParser(HTMLParser):
+    class TruncationCompleted(Exception):
+        pass
+
+    def __init__(self, *, length, replacement, convert_charrefs=True):
+        super().__init__(convert_charrefs=convert_charrefs)
+        self.tags = deque()
+        self.output = ""
+        self.remaining = length
+        self.replacement = replacement
+
+    @cached_property
+    def void_elements(self):
+        from django.utils.html import VOID_ELEMENTS
+
+        return VOID_ELEMENTS
+
+    def handle_startendtag(self, tag, attrs):
+        self.handle_starttag(tag, attrs)
+        if tag not in self.void_elements:
+            self.handle_endtag(tag)
+
+    def handle_starttag(self, tag, attrs):
+        self.output += self.get_starttag_text()
+        if tag not in self.void_elements:
+            self.tags.appendleft(tag)
+
+    def handle_endtag(self, tag):
+        if tag not in self.void_elements:
+            self.output += f"</{tag}>"
+            try:
+                self.tags.remove(tag)
+            except ValueError:
+                pass
+
+    def handle_data(self, data):
+        data, output = self.process(data)
+        data_len = len(data)
+        if self.remaining < data_len:
+            self.remaining = 0
+            self.output += add_truncation_text(output, self.replacement)
+            raise self.TruncationCompleted
+        self.remaining -= data_len
+        self.output += output
+
+    def feed(self, data):
+        try:
+            super().feed(data)
+        except self.TruncationCompleted:
+            self.output += "".join([f"</{tag}>" for tag in self.tags])
+            self.tags.clear()
+            self.reset()
+        else:
+            # No data was handled.
+            self.reset()
+
+
+class TruncateCharsHTMLParser(TruncateHTMLParser):
+    def __init__(self, *, length, replacement, convert_charrefs=True):
+        self.length = length
+        self.processed_chars = 0
+        super().__init__(
+            length=calculate_truncate_chars_length(length, replacement),
+            replacement=replacement,
+            convert_charrefs=convert_charrefs,
+        )
+
+    def process(self, data):
+        self.processed_chars += len(data)
+        if (self.processed_chars == self.length) and (
+            len(self.output) + len(data) == len(self.rawdata)
+        ):
+            self.output += data
+            raise self.TruncationCompleted
+        output = escape("".join(data[: self.remaining]))
+        return data, output
+
+
+class TruncateWordsHTMLParser(TruncateHTMLParser):
+    def process(self, data):
+        data = re.split(r"(?<=\S)\s+(?=\S)", data)
+        output = escape(" ".join(data[: self.remaining]))
+        return data, output
+
+
 class Truncator(SimpleLazyObject):
     """
     An object used to truncate text, either by characters or words.
@@ -108,19 +211,16 @@ def chars(self, num, truncate=None, html=False):
             return ""
         text = unicodedata.normalize("NFC", self._wrapped)
 
-        # Calculate the length to truncate to (max length - end_text length)
-        truncate_len = length
-        for char in add_truncation_text("", truncate):
-            if not unicodedata.combining(char):
-                truncate_len -= 1
-                if truncate_len == 0:
-                    break
         if html:
-            return self._truncate_html(length, truncate, text, truncate_len, False)
-        return self._text_chars(length, truncate, text, truncate_len)
+            parser = TruncateCharsHTMLParser(length=length, replacement=truncate)
+            parser.feed(text)
+            parser.close()
+            return parser.output
+        return self._text_chars(length, truncate, text)
 
-    def _text_chars(self, length, truncate, text, truncate_len):
+    def _text_chars(self, length, truncate, text):
         """Truncate a string after a certain number of chars."""
+        truncate_len = calculate_truncate_chars_length(length, truncate)
         s_len = 0
         end_index = None
         for i, char in enumerate(text):
@@ -149,7 +249,10 @@ def words(self, num, truncate=None, html=False):
         if length <= 0:
             return ""
         if html:
-            return self._truncate_html(length, truncate, self._wrapped, length, True)
+            parser = TruncateWordsHTMLParser(length=length, replacement=truncate)
+            parser.feed(self._wrapped)
+            parser.close()
+            return parser.output
         return self._text_words(length, truncate)
 
     def _text_words(self, length, truncate):
@@ -164,94 +267,6 @@ def _text_words(self, length, truncate):
             return add_truncation_text(" ".join(words), truncate)
         return " ".join(words)
 
-    def _truncate_html(self, length, truncate, text, truncate_len, words):
-        """
-        Truncate HTML to a certain number of chars (not counting tags and
-        comments), or, if words is True, then to a certain number of words.
-        Close opened tags if they were correctly closed in the given HTML.
-
-        Preserve newlines in the HTML.
-        """
-        if words and length <= 0:
-            return ""
-
-        size_limited = False
-        if len(text) > self.MAX_LENGTH_HTML:
-            text = text[: self.MAX_LENGTH_HTML]
-            size_limited = True
-
-        html4_singlets = (
-            "br",
-            "col",
-            "link",
-            "base",
-            "img",
-            "param",
-            "area",
-            "hr",
-            "input",
-        )
-
-        # Count non-HTML chars/words and keep note of open tags
-        pos = 0
-        end_text_pos = 0
-        current_len = 0
-        open_tags = []
-
-        regex = re_words if words else re_chars
-
-        while current_len <= length:
-            m = regex.search(text, pos)
-            if not m:
-                # Checked through whole string
-                break
-            pos = m.end(0)
-            if m[1]:
-                # It's an actual non-HTML word or char
-                current_len += 1
-                if current_len == truncate_len:
-                    end_text_pos = pos
-                continue
-            # Check for tag
-            tag = re_tag.match(m[0])
-            if not tag or current_len >= truncate_len:
-                # Don't worry about non tags or tags after our truncate point
-                continue
-            closing_tag, tagname, self_closing = tag.groups()
-            # Element names are always case-insensitive
-            tagname = tagname.lower()
-            if self_closing or tagname in html4_singlets:
-                pass
-            elif closing_tag:
-                # Check for match in open tags list
-                try:
-                    i = open_tags.index(tagname)
-                except ValueError:
-                    pass
-                else:
-                    # SGML: An end tag closes, back to the matching start tag,
-                    # all unclosed intervening start tags with omitted end tags
-                    open_tags = open_tags[i + 1 :]
-            else:
-                # Add it to the start of the open tags list
-                open_tags.insert(0, tagname)
-
-        truncate_text = add_truncation_text("", truncate)
-
-        if current_len <= length:
-            if size_limited and truncate_text:
-                text += truncate_text
-            return text
-
-        out = text[:end_text_pos]
-        if truncate_text:
-            out += truncate_text
-        # Close any tags still open
-        for tag in open_tags:
-            out += "</%s>" % tag
-        # Return string
-        return out
-
 
 @keep_lazy_text
 def get_valid_filename(name):
diff --git a/docs/releases/5.1.txt b/docs/releases/5.1.txt
index 701d68653252..aca1281a98a6 100644
--- a/docs/releases/5.1.txt
+++ b/docs/releases/5.1.txt
@@ -368,6 +368,11 @@ Miscellaneous
   :meth:`~django.test.SimpleTestCase.assertInHTML` now add ``": "`` to the
   ``msg_prefix``. This is consistent with the behavior of other assertions.
 
+* ``django.utils.text.Truncator`` used by :tfilter:`truncatechars_html` and
+  :tfilter:`truncatewords_html` template filters now uses
+  :py:class:`html.parser.HTMLParser` subclasses. This results in a more robust
+  and faster operation, but there may be small differences in the output.
+
 .. _deprecated-features-5.1:
 
 Features deprecated in 5.1


Test patch:
diff --git a/tests/template_tests/filter_tests/test_truncatewords_html.py b/tests/template_tests/filter_tests/test_truncatewords_html.py
index 32b7c81a7626..0cf41d83aeef 100644
--- a/tests/template_tests/filter_tests/test_truncatewords_html.py
+++ b/tests/template_tests/filter_tests/test_truncatewords_html.py
@@ -24,7 +24,7 @@ def test_truncate2(self):
             truncatewords_html(
                 '<p>one <a href="#">two - three <br>four</a> five</p>', 4
             ),
-            '<p>one <a href="#">two - three …</a></p>',
+            '<p>one <a href="#">two - three <br> …</a></p>',
         )
 
     def test_truncate3(self):
@@ -32,7 +32,7 @@ def test_truncate3(self):
             truncatewords_html(
                 '<p>one <a href="#">two - three <br>four</a> five</p>', 5
             ),
-            '<p>one <a href="#">two - three <br>four …</a></p>',
+            '<p>one <a href="#">two - three <br>four</a> …</p>',
         )
 
     def test_truncate4(self):
@@ -53,7 +53,7 @@ def test_truncate_complex(self):
             truncatewords_html(
                 "<i>Buenos d&iacute;as! &#x00bf;C&oacute;mo est&aacute;?</i>", 3
             ),
-            "<i>Buenos d&iacute;as! &#x00bf;C&oacute;mo …</i>",
+            "<i>Buenos días! ¿Cómo …</i>",
         )
 
     def test_invalid_arg(self):
diff --git a/tests/utils_tests/test_text.py b/tests/utils_tests/test_text.py
index 6004712bf296..b38d8238c523 100644
--- a/tests/utils_tests/test_text.py
+++ b/tests/utils_tests/test_text.py
@@ -111,7 +111,7 @@ def test_truncate_chars_html(self):
             truncator.chars(46, html=True),
         )
         self.assertEqual(
-            '<p id="par"><strong><em>The quick brown fox jumped over the lazy dog.</em>'
+            '<p id="par"><strong><em>The quick brown fox jumped over the lazy dog…</em>'
             "</strong></p>",
             truncator.chars(45, html=True),
         )
@@ -120,7 +120,7 @@ def test_truncate_chars_html(self):
             truncator.chars(10, html=True),
         )
         self.assertEqual(
-            "…",
+            '<p id="par"><strong><em>…</em></strong></p>',
             truncator.chars(1, html=True),
         )
         self.assertEqual("", truncator.chars(0, html=True))
@@ -142,18 +142,16 @@ def test_truncate_chars_html_size_limit(self):
         bigger_len = text.Truncator.MAX_LENGTH_HTML + 1
         valid_html = "<p>Joel is a slug</p>"  # 14 chars
         perf_test_values = [
-            ("</a" + "\t" * (max_len - 6) + "//>", None),
-            ("</p" + "\t" * bigger_len + "//>", "</p" + "\t" * 6 + "…"),
-            ("&" * bigger_len, "&" * 9 + "…"),
-            ("_X<<<<<<<<<<<>", None),
+            ("</a" + "\t" * (max_len - 6) + "//>", "</a>"),
+            ("</p" + "\t" * bigger_len + "//>", "</p>"),
+            ("&" * bigger_len, ""),
+            ("_X<<<<<<<<<<<>", "_X&lt;&lt;&lt;&lt;&lt;&lt;&lt;…"),
             (valid_html * bigger_len, "<p>Joel is a…</p>"),  # 10 chars
         ]
         for value, expected in perf_test_values:
             with self.subTest(value=value):
                 truncator = text.Truncator(value)
-                self.assertEqual(
-                    expected if expected else value, truncator.chars(10, html=True)
-                )
+                self.assertEqual(expected, truncator.chars(10, html=True))
 
     def test_truncate_chars_html_with_newline_inside_tag(self):
         truncator = text.Truncator(
@@ -181,7 +179,7 @@ def test_truncate_chars_html_with_void_elements(self):
             "<br>The <hr/>quick <em>brown…</em>", truncator.chars(16, html=True)
         )
         self.assertEqual("<br>The <hr/>q…", truncator.chars(6, html=True))
-        self.assertEqual("<br>The …", truncator.chars(5, html=True))
+        self.assertEqual("<br>The <hr/>…", truncator.chars(5, html=True))
         self.assertEqual("<br>The…", truncator.chars(4, html=True))
         self.assertEqual("<br>Th…", truncator.chars(3, html=True))
 
@@ -190,11 +188,19 @@ def test_truncate_chars_html_with_html_entities(self):
             "<i>Buenos d&iacute;as! &#x00bf;C&oacute;mo est&aacute;?</i>"
         )
         self.assertEqual(
-            "<i>Buenos d&iacute;as! &#x00bf;C&oacute;mo…</i>",
+            "<i>Buenos días! ¿Cómo está?</i>",
             truncator.chars(40, html=True),
         )
+        self.assertEqual(
+            "<i>Buenos días…</i>",
+            truncator.chars(12, html=True),
+        )
+        self.assertEqual(
+            "<i>Buenos días! ¿Cómo está…</i>",
+            truncator.chars(24, html=True),
+        )
         truncator = text.Truncator("<p>I &lt;3 python, what about you?</p>")
-        self.assertEqual("<p>I &lt;3 python,…</p>", truncator.chars(16, html=True))
+        self.assertEqual("<p>I &lt;3 python, wh…</p>", truncator.chars(16, html=True))
 
     def test_truncate_words(self):
         truncator = text.Truncator("The quick brown fox jumped over the lazy dog.")
@@ -242,7 +248,7 @@ def test_truncate_html_words(self):
             "<p>The  quick \t brown fox jumped over the lazy dog.</p>"
         )
         self.assertEqual(
-            "<p>The  quick \t brown fox…</p>",
+            "<p>The quick brown fox…</p>",
             truncator.words(4, html=True),
         )
 
@@ -277,7 +283,7 @@ def test_truncate_html_words(self):
             "<i>Buenos d&iacute;as! &#x00bf;C&oacute;mo est&aacute;?</i>"
         )
         self.assertEqual(
-            "<i>Buenos d&iacute;as! &#x00bf;C&oacute;mo…</i>",
+            "<i>Buenos días! ¿Cómo…</i>",
             truncator.words(3, html=True),
         )
         truncator = text.Truncator("<p>I &lt;3 python, what about you?</p>")
@@ -292,19 +298,17 @@ def test_truncate_words_html_size_limit(self):
         bigger_len = text.Truncator.MAX_LENGTH_HTML + 1
         valid_html = "<p>Joel is a slug</p>"  # 4 words
         perf_test_values = [
-            ("</a" + "\t" * (max_len - 6) + "//>", None),
-            ("</p" + "\t" * bigger_len + "//>", "</p" + "\t" * (max_len - 3) + "…"),
-            ("&" * max_len, None),  # no change
-            ("&" * bigger_len, "&" * max_len + "…"),
-            ("_X<<<<<<<<<<<>", None),
+            ("</a" + "\t" * (max_len - 6) + "//>", "</a>"),
+            ("</p" + "\t" * bigger_len + "//>", "</p>"),
+            ("&" * max_len, ""),
+            ("&" * bigger_len, ""),
+            ("_X<<<<<<<<<<<>", "_X&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&lt;&gt;"),
             (valid_html * bigger_len, valid_html * 12 + "<p>Joel is…</p>"),  # 50 words
         ]
         for value, expected in perf_test_values:
             with self.subTest(value=value):
                 truncator = text.Truncator(value)
-                self.assertEqual(
-                    expected if expected else value, truncator.words(50, html=True)
-                )
+                self.assertEqual(expected, truncator.words(50, html=True))
 
     def test_wrap(self):
         digits = "1234 67 9"