Repository: wagtail/wagtail

Issue:
Feature request: rich text character count with validation
The UX for rich text fields with a max character count could be greatly improved. 

Ideally we should include live character counts with an indication of total limits. Currently the check is run at page save meaning if no help text has been added you don't even know there is a limit. 

I wrote several paragraphs of text only to be told there was 250 character limit on save. I was then blindly trying to cut the text down. 



Discussion:
I agree. This is extra complicated for editors, as the character count includes the characters in HTML elements like `<p>`. This means that the visible characters only make up a portion of the counted characters, and some invisible characters make up the rest.

I'm currently doing something along these lines in [this](https://github.com/takeflight/wagtail-charcount) module. For richtext fields the html tags are included as part of the count though, which may be confusing.

There are two issues here:

1. UX-wise, there should be a way to know what the limit is and whether it's reached (ideally with a count)
2. For the counting, it should be calculated based on plain text content, independent from rich text features.

I can foresee this only getting worse with a Draft.js editor, ~where the overhead for individual lines of text is much bigger than that of HTML (about 200 chars for one empty `<p></p>`):~

Edit: actually no – content is stored in the same format, so there won't be any impact.


On a positive note, it's really simple to get the content as a plain text string and then calculate the length of that.

Edit: ~just opened #4150 for the second issue.~
What's the ideal solution for this? As far as I understand, the character count is based on the length of the content in its "DB storage" representation, so there is no way for the editor to know what that is before the content is actually saved.

If the character count is there for technical constraints, it does make sense for it to be based on the formatted string. If it's there for authoring constraints, it should probably be based on the plain-text representation.

How do other systems solve this issue?
I’ve had a stab at building something with the new editor that solves those character count issues. The UI is based on Twitter’s character count, which combines a small donut chart and (after content starts to go over) a red text overlay. Here is what it looks like:

![maxlength-example](https://user-images.githubusercontent.com/877585/52483319-3e43fd80-2bbc-11e9-91bd-1e443d7b0b28.gif)

You can also try it out on http://playground.draftail.org/.

It should be pretty straightforward to port this (or implement a similar UI) within Wagtail if someone wants to.
@thibaudcolas How can you enable the [ReadingTime](https://github.com/springload/draftail/blob/49efd64d9831d27399087645e32ae776ac76defa/examples/components/ReadingTime.js) component in Wagtail?
We've gotten feedback that this would be helpful for not just RIchText fields but other fields as well!
Great example of how GDS/Gov.uk do this here: 
https://design-system.service.gov.uk/components/character-count/
FWIW we implemented this for ourselves using a slightly modified fork of:
https://github.com/RoboAndie/django-countable-field
> FWIW we implemented this for ourselves using a slightly modified fork of:
> https://github.com/RoboAndie/django-countable-field

@easherma is it publicly available?
It’s been awhile but I think I did something similar. I don’t think I did it as a separate library
> It’s been awhile but I think I did something similar. I don’t think I did it as a separate library
> -- Eric Sherman 224-402-0004

Thanks, will make an attempt and see how far it goes!
For what it is worth, I implemented some code in a rich text validator which takes a target character count as a limit, separate to the underlying database length limit in the field. It also does a rough adjustment for formatting that creates line breaks.

It doesn't attempt to provide a real-time indication of the character count.

See https://gist.github.com/dkirkham/a1fc6d3a04754d5bb816cf97401fd357 for the code. 
For people interested in this, we are likely going to do this as part of the [page editor 2022](https://github.com/wagtail/wagtail/discussions/7739) project.
Relevant RFC for people interested in this – feedback very welcome: https://github.com/wagtail/rfcs/pull/60#issuecomment-989011931.
Updating this issue based on what we’re moving forward with. Draftail has character count capabilities as of #8246. They’ve yet to be integrated into Wagtail (#8548). Once we do this, we’ll need to make backend changes so developers can configure character counts in their models. So far we’re expecting:

- A way for developers to specify a maximum plain-text character length for a rich text field
- A way to configure whether going over this maximum length should result in a validation error, or a warning

This will need to be calculated based on the fields’ plain-text content. UX-wise, we’re expecting the "error" and "warning" scenarios to look different, but haven’t looked into the specifics yet.

---

We’ll only implement this for rich text fields for the time being. See [RFC 60](https://github.com/wagtail/rfcs/pull/60) for follow-up work on plain-text fields.
#8822 has implemented the server-side aspect of this. #8548 implements the character count. Now we just need to display the maximum length next to the count.

Resolving patch:
diff --git a/wagtail/rich_text/__init__.py b/wagtail/rich_text/__init__.py
--- a/wagtail/rich_text/__init__.py
+++ b/wagtail/rich_text/__init__.py
@@ -132,7 +132,8 @@ class EmbedHandler(EntityHandler):
 class RichTextMaxLengthValidator(MaxLengthValidator):
     """
     A variant of MaxLengthValidator that only counts text (not HTML tags) towards the limit
+    Un-escapes entities for consistency with client-side character count.
     """
 
     def clean(self, x):
-        return len(strip_tags(x))
+        return len(unescape(strip_tags(x)))


Test patch:
diff --git a/client/src/components/Draftail/__snapshots__/index.test.js.snap b/client/src/components/Draftail/__snapshots__/index.test.js.snap
--- a/client/src/components/Draftail/__snapshots__/index.test.js.snap
+++ b/client/src/components/Draftail/__snapshots__/index.test.js.snap
@@ -16,11 +16,7 @@ Object {
   "commandPalette": [Function],
   "commandToolbar": [Function],
   "commands": true,
-  "controls": Array [
-    Object {
-      "meta": [Function],
-    },
-  ],
+  "controls": Array [],
   "decorators": Array [],
   "editorState": null,
   "enableHorizontalRule": Object {
diff --git a/client/src/components/Draftail/controls/MaxLength.test.tsx b/client/src/components/Draftail/controls/MaxLength.test.tsx
--- a/client/src/components/Draftail/controls/MaxLength.test.tsx
+++ b/client/src/components/Draftail/controls/MaxLength.test.tsx
@@ -1,33 +1,30 @@
 import React from 'react';
 import { shallow } from 'enzyme';
-import { EditorState, convertFromHTML, ContentState } from 'draft-js';
+import { EditorState, convertFromRaw, RawDraftContentState } from 'draft-js';
 
-import MaxLength, { countCharacters } from './MaxLength';
+import MaxLength, { countCharacters, getPlainText } from './MaxLength';
 
+/**
+ * Keep those tests up-to-date with RichTextMaxLengthValidator tests server-side.
+ */
 describe('MaxLength', () => {
   it('works', () => {
-    const { contentBlocks } = convertFromHTML('<p>hello</p>');
-    const contentState = ContentState.createFromBlockArray(contentBlocks);
+    const contentState = convertFromRaw({
+      entityMap: {},
+      blocks: [{ text: 'hello' }],
+    } as RawDraftContentState);
     const editorState = EditorState.createWithContent(contentState);
 
     expect(
       shallow(
-        <MaxLength getEditorState={() => editorState} onChange={() => {}} />,
-      ),
-    ).toMatchInlineSnapshot(`
-      <div
-        className="w-inline-block w-tabular-nums w-label-3"
-      >
-        <span
-          className="w-sr-only"
-        >
-          Character count:
-        </span>
-        <span>
-          5
-        </span>
-      </div>
-    `);
+        <MaxLength
+          getEditorState={() => editorState}
+          onChange={() => {}}
+          id="max-length-test"
+          maxLength={10}
+        />,
+      ).text(),
+    ).toBe('Character count:5/10');
   });
 
   it('supports 0', () => {
@@ -36,25 +33,62 @@ describe('MaxLength', () => {
         <MaxLength
           getEditorState={() => EditorState.createEmpty()}
           onChange={() => {}}
+          id="max-length-test"
+          maxLength={10}
+        />,
+      ).text(),
+    ).toBe('Character count:0/10');
+  });
+
+  it('ignores atomic blocks', () => {
+    const contentState = convertFromRaw({
+      entityMap: {},
+      blocks: [
+        { text: 'hello' },
+        { text: ' ', type: 'atomic' },
+        { text: 'world' },
+      ],
+    } as RawDraftContentState);
+    const editorState = EditorState.createWithContent(contentState);
+
+    expect(
+      shallow(
+        <MaxLength
+          getEditorState={() => editorState}
+          onChange={() => {}}
+          id="max-length-test"
+          maxLength={10}
         />,
-      ),
-    ).toMatchInlineSnapshot(`
-      <div
-        className="w-inline-block w-tabular-nums w-label-3"
-      >
-        <span
-          className="w-sr-only"
-        >
-          Character count:
-        </span>
-        <span>
-          0
-        </span>
-      </div>
-    `);
+      ).text(),
+    ).toBe('Character count:10/10');
   });
 });
 
+// Make sure count here matches result in TestRichTextMaxLengthValidator.test_count_characters
+test.each([
+  // Embed blocks should be ignored.
+  [['Plain text'], 'Plain text', 10],
+  // HTML entities should be un-escaped.
+  [["There's quote"], "There's quote", 13],
+  // BR should be ignored.
+  [['Line\nbreak'], 'Linebreak', 9],
+  // Content over multiple blocks should be treated as a single line of text with no joiner.
+  [['Multi', 'blocks'], 'Multiblocks', 11],
+  // Empty blocks should be ignored.
+  [['Empty', '', 'blocks'], 'Emptyblocks', 11],
+])('getPlainText', (blocks, plainText, count) => {
+  const contentState = convertFromRaw({
+    entityMap: {},
+    blocks: blocks.map((block) => ({ text: block })),
+  } as RawDraftContentState);
+  const editorState = EditorState.createWithContent(contentState);
+  const text = getPlainText(editorState);
+  // Check the plain-text version as well to help with troubleshooting.
+  expect(text).toBe(plainText);
+  expect(countCharacters(text)).toBe(count);
+});
+
+// Make sure count here matches result in TestRichTextMaxLengthValidator.test_count_characters
 describe.each`
   text         | result | segmenterLength
   ${'123456'}  | ${6}   | ${6}
diff --git a/client/src/components/Draftail/index.test.js b/client/src/components/Draftail/index.test.js
--- a/client/src/components/Draftail/index.test.js
+++ b/client/src/components/Draftail/index.test.js
@@ -53,6 +53,18 @@ describe('Draftail', () => {
       });
 
       expect(field.draftailEditor.props).toMatchSnapshot();
+      // Make sure we don’t initialise a character count on fields that have the default unlimited max length.
+      expect(field.draftailEditor.props.ariaDescribedBy).toBe(null);
+    });
+
+    it('maxLength', () => {
+      document.body.innerHTML =
+        '<input id="test" value="null" maxlength="50" />';
+      const field = document.querySelector('#test');
+
+      draftail.initEditor('#test', {});
+
+      expect(field.draftailEditor.props.ariaDescribedBy).toBe('test-length');
     });
 
     describe('selector conflicts', () => {
diff --git a/client/src/entrypoints/admin/telepath/widgets.test.js b/client/src/entrypoints/admin/telepath/widgets.test.js
--- a/client/src/entrypoints/admin/telepath/widgets.test.js
+++ b/client/src/entrypoints/admin/telepath/widgets.test.js
@@ -544,7 +544,7 @@ describe('telepath: wagtail.widgets.DraftailRichTextArea', () => {
     ReactTestUtils.act(() =>
       boundWidget.setCapabilityOptions('split', { enabled: false }),
     );
-    expect(inputElement.draftailEditor.props.controls).toHaveLength(2);
+    expect(inputElement.draftailEditor.props.controls).toHaveLength(1);
     expect(window.draftail.getSplitControl).toHaveBeenLastCalledWith(
       parentCapabilities.get('split').fn,
       false,
@@ -552,7 +552,7 @@ describe('telepath: wagtail.widgets.DraftailRichTextArea', () => {
     ReactTestUtils.act(() =>
       boundWidget.setCapabilityOptions('split', { enabled: true }),
     );
-    expect(inputElement.draftailEditor.props.controls).toHaveLength(2);
+    expect(inputElement.draftailEditor.props.controls).toHaveLength(1);
     expect(window.draftail.getSplitControl).toHaveBeenLastCalledWith(
       parentCapabilities.get('split').fn,
       true,
diff --git a/wagtail/tests/test_rich_text.py b/wagtail/tests/test_rich_text.py
--- a/wagtail/tests/test_rich_text.py
+++ b/wagtail/tests/test_rich_text.py
@@ -5,7 +5,7 @@
 from django.utils import translation
 
 from wagtail.models import Locale, Page
-from wagtail.rich_text import RichText, expand_db_html
+from wagtail.rich_text import RichText, RichTextMaxLengthValidator, expand_db_html
 from wagtail.rich_text.feature_registry import FeatureRegistry
 from wagtail.rich_text.pages import PageLinkHandler
 from wagtail.rich_text.rewriters import LinkRewriter, extract_attrs
@@ -275,3 +275,26 @@ def test_max_length_validation(self):
             }
         )
         self.assertTrue(form.is_valid())
+
+
+class TestRichTextMaxLengthValidator(TestCase):
+    def test_count_characters(self):
+        """Keep those tests up-to-date with MaxLength tests client-side."""
+        validator = RichTextMaxLengthValidator(50)
+        self.assertEqual(validator.clean("<p>Plain text</p>"), 10)
+        # HTML entities should be un-escaped.
+        self.assertEqual(validator.clean("<p>There&#x27;s quote</p>"), 13)
+        # BR should be ignored.
+        self.assertEqual(validator.clean("<p>Line<br/>break</p>"), 9)
+        # Content over multiple blocks should be treated as a single line of text with no joiner.
+        self.assertEqual(validator.clean("<p>Multi</p><p>blocks</p>"), 11)
+        # Empty blocks should be ignored.
+        self.assertEqual(validator.clean("<p>Empty</p><p></p><p>blocks</p>"), 11)
+        # HR should be ignored.
+        self.assertEqual(validator.clean("<p>With</p><hr/><p>HR</p>"), 6)
+        # Embed blocks should be ignored.
+        self.assertEqual(validator.clean("<p>With</p><embed/><p>embed</p>"), 9)
+        # Counts symbols with multiple code units (heart unicode + variation selector).
+        self.assertEqual(validator.clean("<p>U+2764 U+FE0F ❤️</p>"), 16)
+        # Counts symbols with zero-width joiners.
+        self.assertEqual(validator.clean("<p>👨‍👨‍👧</p>"), 5)