Repository: wagtail/wagtail

Issue:
Rich text processing converts non-breaking spaces into normal spaces
### Issue Summary

When inserting non-breaking spaces into Draftail (with `option + space` on macOS), they are persisted in the database and rendered in the front-end HTML, but subsequently get lost in the DB -> editor JSON conversion.

Initially reported by @philgyford in [Slack #support](https://wagtailcms.slack.com/archives/C81FGJR2S/p1547462877447200).

### Steps to Reproduce

1. Open the editing UI of a page with a rich text field.
2. Type `test test` in the field, with a non-breaking space between the two words
3. Publish the page
4. Look at the front-end HTML of the page – the Chrome DevTools will display the content as `test&nbsp;test`. Inspecting the page shows the character normally, not as an entity.
5. Disable JavaScript in the browser
6. Reload the editing UI of the same page
7. Use the DevTools console to inspect the content of the hidden field. For example, using `JSON.parse(document.querySelector('[name="promo_text"]').value).blocks[0].text.charCodeAt(4)` with the correct `input` selector – this should output 160 for a nbsp, but instead outputs 32 (normal space).

Any other relevant information. For example, why do you consider this a bug and what did you expect to happen instead?

Non-breaking spaces should be preserved so they can be used to manage line wrapping.

* I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: yes

### Technical details

* Python version: 3.4
* Django version: 2.0
* Wagtail version: 2.5a0
* Browser version: Chrome 71.0.3578.98

---

I suspect this is caused by https://github.com/wagtail/wagtail/blob/057690815b3147be1280834787e275590f71d274/wagtail/admin/rich_text/converters/html_to_contentstate.py#L314-L317

This collapses sequences matching `\s` to ` `.



Discussion:
I wonder if I'm doing something different... if I do steps 1-4, then I _don't_ see a `&nbsp;` in the source of the rendered page. I'm using View Source in Chrome and Safari, not inspecting the element.

This isn't a clean Wagtail install, but I'm on v2.4, Django 2.1.5, Python 3.6.6.
View Source shouldn't show an `&nbsp;`, it's only the DevTools that do so. There is no automated conversion of the characters to HTML entities (except for things that are reserved as HTML syntax), so the character is displayed on the source of the page as it is stored. I tried to explain this in step 4. but had a really had time 😛 .

Here is an example from https://bakerydemo-thibaudcolas3.herokuapp.com/about/ with various means of inspection. You can see the node's `innerHTML`, `innerText`, and the char code for the first space.

![4996-charcodeat](https://user-images.githubusercontent.com/877585/51389502-351fbd80-1b35-11e9-9b4b-60a4ff8299e6.png)


Ah, I took point 4 to mean the opposite of what you meant :)

Short answer: You're right and I've replicated what you described.

But also: It turns out that Safari doesn't show `&nbsp;`s in either View Source _or_ the Elements inspector! This is what lead me to think they were being stripped entirely. If I'd been using Chrome, and seen them in the inspector, I'd have assumed it was all working fine... and then later would have been very confused when they disappeared again!

Resolving patch:
diff --git a/wagtail/admin/rich_text/converters/html_to_contentstate.py b/wagtail/admin/rich_text/converters/html_to_contentstate.py
--- a/wagtail/admin/rich_text/converters/html_to_contentstate.py
+++ b/wagtail/admin/rich_text/converters/html_to_contentstate.py
@@ -12,7 +12,8 @@
 KEEP_WHITESPACE = 1
 FORCE_WHITESPACE = 2
 
-WHITESPACE_RE = re.compile(r'\s+')
+# match one or more consecutive normal spaces, new-lines, tabs and form-feeds
+WHITESPACE_RE = re.compile(r'[ \t\n\f\r]+')
 
 
 class HandlerState:
@@ -315,6 +316,7 @@ def handle_endtag(self, name):
 
     def handle_data(self, content):
         # normalise whitespace sequences to a single space
+        # This is in line with https://www.w3.org/TR/html4/struct/text.html#h-9.1
         content = re.sub(WHITESPACE_RE, ' ', content)
 
         if self.state.current_block is None:


Test patch:
diff --git a/wagtail/admin/tests/test_contentstate.py b/wagtail/admin/tests/test_contentstate.py
--- a/wagtail/admin/tests/test_contentstate.py
+++ b/wagtail/admin/tests/test_contentstate.py
@@ -754,6 +754,24 @@ def test_html_entities(self):
             ]
         })
 
+    def test_collapse_targeted_whitespace_characters(self):
+        # We expect all targeted whitespace characters (one or more consecutively)
+        # to be replaced by a single space. (\xa0 is a non-breaking whitespace)
+        converter = ContentstateConverter(features=[])
+        result = json.loads(converter.from_database_format(
+            '''
+            <p>Multiple whitespaces:     should  be reduced</p>
+            <p>Multiple non-breaking whitespace characters:  \xa0\xa0\xa0  should be preserved</p>
+            '''
+        ))
+        self.assertContentStateEqual(result, {
+            'entityMap': {},
+            'blocks': [
+                {'inlineStyleRanges': [], 'text': 'Multiple whitespaces: should be reduced', 'depth': 0, 'type': 'unstyled', 'key': '00000', 'entityRanges': []},
+                {'inlineStyleRanges': [], 'text': 'Multiple non-breaking whitespace characters: \xa0\xa0\xa0 should be preserved', 'depth': 0, 'type': 'unstyled', 'key': '00000', 'entityRanges': []},
+            ]
+        })
+
     def test_extra_end_tag_before(self):
         converter = ContentstateConverter(features=[])
         result = json.loads(converter.from_database_format(