Repository: RedHatInsights/insights-core

Issue:
The parser IniConfigFile runs into exceptions when the ini file contains Chinese or Japanese strings
For the example ini file:

~~~
[Global]
secret-name = "vsphere-creds"
secret-namespace = "kube-system"
insecure-flag = "1"


[Workspace]
server = "xxxxxx"
datacenter = "1-测试部"
default-datastore = "xxxxxxxx"
folder = "/1-测试部/xxxxxxxxx"


[VirtualCenter "xxxxxx"]
datacenters = "1-测试部" 
~~~

The parser IniConfigFile runs into the following exceptions when parsing the above ini file:

~~~
  File "/Users/shan/work/ccx/lib/python3.7/site-packages/insights/core/__init__.py", line 1441, in parse_content
    super(IniConfigFile, self).parse_content(content)
  File "/Users/shan/work/ccx/lib/python3.7/site-packages/insights/core/__init__.py", line 349, in parse_content
    self.doc = self.parse_doc(content)
  File "/Users/shan/work/ccx/lib/python3.7/site-packages/insights/core/__init__.py", line 1438, in parse_doc
    return iniparser.parse_doc("\n".join(content), self, return_defaults=True, return_booleans=False)
  File "/Users/shan/work/ccx/lib/python3.7/site-packages/insights/parsr/iniparser.py", line 100, in parse_doc
    res = Entry(children=Top(content), src=ctx)
  File "/Users/shan/work/ccx/lib/python3.7/site-packages/insights/parsr/__init__.py", line 356, in __call__
    raise Exception(err.read())
Exception: At line 10 column 17:
KVPair -> EOL
    Expected EOL. Got '测'.
KVPair -> EOF
    Expected end of input. Got '测'.
KVPair -> any whitespace
    Expected any whitespace. Got '测'.
KVPair -> Sep
    Expected Sep. Got '测'.
any whitespace
    Expected any whitespace. Got '测'.
Literal'#'
    Expected '#'. Got '测'.
Literal';'
    Expected ';'. Got '测'.
KVPair -> any whitespace
    Expected any whitespace. Got '测'.
KVPair -> any whitespace
    Expected any whitespace. Got '测'.
KVPair
    Expected 1 of [' ', '!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ';', '<', '>', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', '\\', '^', '_', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', '{', '|', '}', '~']. Got '测'.
any whitespace
    Expected any whitespace. Got '测'.
Literal'#'
    Expected '#'. Got '测'.
Literal';'
    Expected ';'. Got '测'.
Header -> any whitespace
    Expected any whitespace. Got '测'.
Header
    Expected [. Got '测'.
any whitespace
    Expected any whitespace. Got '测'.
EOF
    Expected end of input. Got '测'.
~~~


Discussion:
This is caused by the following basic definition of the `parsr` module
https://github.com/RedHatInsights/insights-core/blob/bbc2186e17d039e0f6a3e4a8d3eabcfaec4670f7/insights/parsr/__init__.py#L1252-L1254

The `string.printable` only includes the following ASCII characters, but no Chinese/Japanese characters:
```python
>>> string.printable
'0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c'
```

Since the IniConfigFile allows to use of other languages, like Chinese, Japanese, and Korean in the String value, I think It's necessary to add the characters from these languages to the `QuotedString`. But these languages have too many characters to list all of them in the `String` set. We may need to find a feasible method to re-define the `QuotedString` instead of enumerating all the characters.

What's your idea, @bfahr, @ryan-blakley ?
I like the idea that @koalakangaroo shared with me: we may replace these `"invalid"` characters/words with some particular or proper words formed from the characters in the pool of `valid` before parsing it.

This a quite good approach for us to quickly fix this issue, I think it's feasible, just like IP obfuscation that we do during the collection. 

@bfahr, @ryan-blakley, Thoughts?
@xiangce after playing around with the symbols I did find that the unidecode python module can convert to the unicode characters to ascii characters. But it seems that module isn't available in RHEL. You mentioned replacing the characters do you know of an easier way to replace unicode characters?
@ryan-blakley - Nope, I have no idea about this either.


And I'm also not sure if the `unidecode` is suitable for this case, just like the following example, after the conversion, a `blank space` is added for the proper noun `"北京"` -> `"Bei Jing"`.  In the very beginning, I just thought about "replacing" but not "translating". 

```
>>> from unidecode import unidecode
>>> city = "北京"
>>> print(unidecode(city))
Bei Jing
```
@xiangce - Yeah I noticed the space that was another reason I figured it wouldn't work. If you're good with replacing, I think the below may work then.

```
>>> import unicodedata
>>> unicodedata.normalize('NFKD', u'/1-测试部/xxxxxxxxx').encode('ascii', 'replace')
'/1-???/xxxxxxxxx'
```

Resolving patch:
diff --git a/insights/parsr/iniparser.py b/insights/parsr/iniparser.py
--- a/insights/parsr/iniparser.py
+++ b/insights/parsr/iniparser.py
@@ -1,10 +1,11 @@
 import string
 
 from insights.parsr import (Comma, EOF, EOL, DoubleQuotedString,
-        HangingString, InSet, LeftBracket, Lift, LineEnd, Literal, Many,
-        OneLineComment, Opt, PosMarker, RightBracket, skip_none, String,
-        WithIndent, WS, WSChar)
+                            HangingString, InSet, LeftBracket, Lift, LineEnd, Literal, Many,
+                            OneLineComment, Opt, PosMarker, RightBracket, skip_none, String,
+                            WithIndent, WS, WSChar)
 from insights.parsr.query import Directive, Entry, eq, Section
+from six import PY2
 
 
 class Error(Exception):
@@ -97,5 +98,15 @@ def apply_defaults(cfg, include_defaults):
     Doc = Many(Comment | Sect).map(skip_none)
     Top = Doc << WS << EOF
 
+    if PY2:
+        # For py2 sub all non ascii chars for question marks,
+        # since it doesn't support unicode encoding/decoding well.
+        from re import sub
+        content = sub(r"[^\x00-\x7F]", "?", content)
+    else:
+        # Encode and replace unicode characters,
+        # then decode again before processing content.
+        content = content.encode('ascii', 'replace').decode()
+
     res = Entry(children=Top(content), src=ctx)
     return apply_defaults(res, return_defaults)


Test patch:
diff --git a/insights/parsr/tests/test_iniparser.py b/insights/parsr/tests/test_iniparser.py
--- a/insights/parsr/tests/test_iniparser.py
+++ b/insights/parsr/tests/test_iniparser.py
@@ -1,4 +1,7 @@
+# coding:UTF-8
 from insights.parsr.iniparser import parse_doc
+from insights.parsr.query import last
+from six import PY2
 
 
 DATA = """
@@ -32,6 +35,17 @@
 
 """.strip()
 
+DATA_UNICODE_TEST = """
+[global]
+secret-name = "vsphere-creds"
+secret-namespace = kube-system
+insecure-flag = 1
+
+[workspace]
+datacenter = 1-测试部
+folder = "/1-测试部/xxxxxxxxx"
+""".strip()
+
 
 def test_iniparser():
     res = parse_doc(DATA, None)
@@ -58,3 +72,17 @@ def test_multiple_values():
 def test_no_value():
     res = parse_doc(DATA, None)
     assert res["novalue"]["the_force"][0].value is None
+
+
+def test_unicode():
+    res = parse_doc(DATA_UNICODE_TEST, None)
+    assert res["global"]["insecure-flag"][last].value == "1"
+    assert res["global"]["secret-name"][last].value == '"vsphere-creds"'
+    assert res["global"]["secret-namespace"][last].value == "kube-system"
+
+    if PY2:
+        assert res["workspace"]["datacenter"][last].value == "1-?????????"
+        assert res["workspace"]["folder"][last].value == '"/1-?????????/xxxxxxxxx"'
+    else:
+        assert res["workspace"]["datacenter"][last].value == "1-???"
+        assert res["workspace"]["folder"][last].value == '"/1-???/xxxxxxxxx"'