Repository: mdn/kuma

Issue:
React breaks on certain IDs within the content
Originally filed on https://github.com/mdn/sprints/issues/2148

> Just wanted to point out some rendering issues that are caused by translations (like French language for example) which makes some selectors not valid.
> Take a look : https://developer.mozilla.org/fr/docs/Web/CSS/line-height




Discussion:
Quoting @schalkneethling 


> Thank you for reporting this @DamChtlv
> 
> The problem is because of:
> 
> ```
> #Utiliser_les_transitions_pour_accentuer_les_éléments_d'un_menu ~ * pre[class*=html]' is not a valid selector
> ```
> 
> My first guess is that it is the characters such as `é` and `d'u` causing the problem

The reasons this happened is one thing. The more pressing thing is that we didn't have any test automation, that uses a real browser, that checks if pages render correctly. 
Able to reproduce locally:
<img width="1559" alt="Screen Shot 2019-09-12 at 10 10 52 AM" src="https://user-images.githubusercontent.com/26739/64791387-a5a4a000-d545-11e9-90a7-6792c74b0343.png">

It looks like it might actually be characters such as `'` as an id such as `Relzione_tra_length_e_proprietà_numeriche` does not break it. Specifically referring to the `à` character in that string.

https://developer.mozilla.org/it/docs/Web/JavaScript/Reference/Global_Objects/Array

So far I have only found the problem with the `fr` locale
I wonder, should the ID have become `Relzione_tra_length_e_proprieta_numeriche` (instead of `Relzione_tra_length_e_proprietà_numeriche`) by the Wiki post-processing?
Yep, @schalkneethling is right. My bad. We just need to make sure the `iframes` don't have a `'` in their ID. 
https://codepen.io/peterbe/pen/vYBrOzj?editors=1111
Sorry, not iframes. These:
```html
<h3 id="Utiliser_les_transitions_pour_accentuer_les_éléments_d'un_menu">Utiliser les transitions pour accentuer les éléments d'un menu</h3>
```
Well sorry for the French :)
Rewording the ID could be a workaround
Adding the `'` in `non_url_safe` https://github.com/mozilla/kuma/blob/master/kuma/wiki/content.py#L556 could help as well

I came to the same conclusion. 
Now I'm just curious what we should do to pages that have "already been allowed" to keep the `'`.
> Well sorry for the French :)

Don't apologize for your beautiful language! It's Kuma who should say sorry. 
> what we should do to pages that have "already been allowed" to keep the `'`

I might have an [old crawler of mine](https://gist.github.com/SphinxKnight/f3b232450fea15cf0118ed2a207b6060#file-scanl10n-js)  that stores the content of `?raw` view on files that I could run to detect the pages having such IDs.
Once Kuma would do something differently, a manual save on those pages should be sufficient (they should not be too many)

A drawback is that I have to "choose" which section I want to crawl and I may miss some pages if I don't know the list of sections beforehand.
Another drawback is that it's quite slow to gather all the pages for all the translations for all the sections.

If possible a fallback display in any case would be nice :s

Could it be possible to reopen this one?
Apparently parenthesis have the same issues: https://developer.mozilla.org/fr/docs/Web/CSS/imagefunction (it now contains `#Afficher_une_partie_(sprite)`) 
I reworded the heading on this one to avoid `'` but used parenthesis and had the same error
You're totally right! Back the drawing board on this one. 
An example in English ;) https://developer.mozilla.org/en-US/docs/Web/CSS/CSS_Grid_Layout/Basic_Concepts_of_Grid_Layout

Resolving patch:
diff --git a/kuma/wiki/content.py b/kuma/wiki/content.py
--- a/kuma/wiki/content.py
+++ b/kuma/wiki/content.py
@@ -572,22 +572,16 @@ def gen_id(self):
                 self.known_ids.add(id)
                 return id
 
-    # MindTouch encodes these characters, so we have to encode them
-    # too.
-    non_url_safe = ['"', '#', '$', '%', '&', '+',
-                    ',', '/', ':', ';', '=', '?',
-                    '@', '[', '\\', ']', '^', '`',
-                    '{', '|', '}', '~', "'"]
+    non_url_safe = '"#$%&+,/:;=?@[\\]^`{|}~\')('
+    translate_table = {ord(char): u'' for char in non_url_safe}
 
     def slugify(self, text):
         """
         Turn the text content of a header into a slug for use in an ID
         """
-        non_safe = [c for c in text if c in self.non_url_safe]
-        if non_safe:
-            for c in non_safe:
-                text = text.replace(c, '')
-        # Strip leading, trailing and multiple whitespace, convert remaining whitespace to _
+        text = text.translate(self.translate_table)
+        # Strip leading, trailing and multiple whitespace,
+        # convert remaining whitespace to _.
         text = u'_'.join(text.split())
         return text
 


Test patch:
diff --git a/kuma/wiki/tests/test_content.py b/kuma/wiki/tests/test_content.py
--- a/kuma/wiki/tests/test_content.py
+++ b/kuma/wiki/tests/test_content.py
@@ -480,7 +480,7 @@ def test_non_ascii_section_headers(self):
             (u'Outils facilitant le développement HTML',
              u'Outils_facilitant_le_développement_HTML'),
             (u'字面值(literals)',
-             u'字面值(literals)'),
+             u'字面值literals'),
             (u'Documentação',
              u'Documentação'),
             (u'Lektury uzupełniające',
@@ -489,7 +489,7 @@ def test_non_ascii_section_headers(self):
              u'Атрибуты'),
             (u'HTML5 엘리먼트',
              u'HTML5_엘리먼트'),
-            (u'Non safe title "#$%&+,/:;=?@[\\]^`{|}~',
+            (u'Non safe title "#$%&+,/:;=?@[\\]^`{|}~\')(',
              u'Non_safe_title'),
             (u"Five o'clock",
              u'Five_oclock'),