Repository: byllyfish/precis_i18n

Issue:
Nickname profile strips more whitespace characters than it should.
The Nickname profile includes an "additional mapping rule" that strips white space characters from both the beginning and end of the string. The current implementation uses the str.strip() method without a parameter. Python's default behavior defines a number of control characters as white space. I believe this is a bug; the code should be str.strip(' ') to just strip the standard ascii space.

The result of the Nickname `enforce` step is not affected by this bug. The code is simply over-permissive; it corrects the input more than it should. A diagnostic should tell the person to clean up their nickname string. A proper fix will include a diagnostic error message that reports "invalid_white_space". This will be a new error message.

Current behavior that is not technically correct:

```
Nickname profile:
  " a\n" -> "a"
  "\x1fb\x1f" -> "b"
  "\tc\r\n" -> "c"
```

Note that non-ascii `Zs` white space are converted to ASCII space 0x20 before any stripping takes place. The Precis spec only talks about stripping the `Zs` white space chars.

The following characters are stripped by Python:

- U+09 (Cc)
- U+0A (Cc)
- U+0B (Cc)
- U+0C (Cc)
- U+0D (Cc)
- U+1C (Cc)
- U+1D (Cc)
- U+1E (Cc)
- U+1F (Cc)
- U+85 (Cc)
- U+2028 (Zl)
- U+2029 (Zp)

Of these, I'm only worried about `\n` (U+0A). I might just continue to strip these, even though the spec doesn't say I have to. This is mostly for compatibility with existing software use.

Discussion:

Patch:
diff --git a/precis_i18n/profile.py b/precis_i18n/profile.py
index 915e234..074f893 100644
--- a/precis_i18n/profile.py
+++ b/precis_i18n/profile.py
@@ -291,7 +291,7 @@ class Nickname(Profile):
     def additional_mapping_rule(self, value):
         # Override
         temp = self.base.ucd.map_nonascii_space_to_ascii(value)
-        return re.sub(r'  +', ' ', temp.strip(' \t\n\r'))
+        return re.sub(r'  +', ' ', temp.strip(' '))
 
     def normalization_rule(self, value):
         # Override


Test patch:
diff --git a/test/golden.json b/test/golden.json
index bfff879..72b8412 100644
--- a/test/golden.json
+++ b/test/golden.json
@@ -5896,19 +5896,19 @@
     "profile": "NicknameCaseMapped",
     "input": "\t",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
     "input": "\n",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
     "input": "\r",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
@@ -5997,14 +5997,14 @@
   {
     "profile": "NicknameCaseMapped",
     "input": "\tA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
     "input": "A\t",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
@@ -6015,14 +6015,14 @@
   {
     "profile": "NicknameCaseMapped",
     "input": "\nA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
     "input": "A\n",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
@@ -6033,14 +6033,14 @@
   {
     "profile": "NicknameCaseMapped",
     "input": "\rA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
     "input": "A\r",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped",
@@ -9810,19 +9810,19 @@
     "profile": "NicknameCasePreserved",
     "input": "\t",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
     "input": "\n",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
     "input": "\r",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
@@ -9911,14 +9911,14 @@
   {
     "profile": "NicknameCasePreserved",
     "input": "\tA",
-    "output": "A",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
     "input": "A\t",
-    "output": "A",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
@@ -9929,14 +9929,14 @@
   {
     "profile": "NicknameCasePreserved",
     "input": "\nA",
-    "output": "A",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
     "input": "A\n",
-    "output": "A",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
@@ -9947,14 +9947,14 @@
   {
     "profile": "NicknameCasePreserved",
     "input": "\rA",
-    "output": "A",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
     "input": "A\r",
-    "output": "A",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCasePreserved",
@@ -11766,19 +11766,19 @@
     "profile": "NicknameCaseMapped:ToLower",
     "input": "\t",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
     "input": "\n",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
     "input": "\r",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
@@ -11867,14 +11867,14 @@
   {
     "profile": "NicknameCaseMapped:ToLower",
     "input": "\tA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
     "input": "A\t",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
@@ -11885,14 +11885,14 @@
   {
     "profile": "NicknameCaseMapped:ToLower",
     "input": "\nA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
     "input": "A\n",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
@@ -11903,14 +11903,14 @@
   {
     "profile": "NicknameCaseMapped:ToLower",
     "input": "\rA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
     "input": "A\r",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:ToLower",
@@ -19591,19 +19591,19 @@
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "\t",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "\n",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "\r",
     "output": null,
-    "error": "DISALLOWED/empty"
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",
@@ -19692,14 +19692,14 @@
   {
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "\tA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "A\t",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",
@@ -19710,14 +19710,14 @@
   {
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "\nA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "A\n",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",
@@ -19728,14 +19728,14 @@
   {
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "\rA",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",
     "input": "A\r",
-    "output": "a",
-    "error": null
+    "output": null,
+    "error": "DISALLOWED/controls"
   },
   {
     "profile": "NicknameCaseMapped:CaseFold",