Repository: RedHatInsights/insights-core

Issue:
TextFileProvider fails to read files containing byte chars in Python 3
When TextFileProvider loads files that have byte chars it will raise an exception and the parser and anything dependent upon the parser will not run.  For instance this log file contains the byte 0xd2:
```
Aug 20 14:13:00 host3 kernel: found SMP MP-table at [mem 0x000f4f80-0x000f4f8f] mapped at [ffff8800000f4f80]
Aug 20 14:13:00 host3 kernel: Using GB pages for direct mapping
Aug 20 14:13:00 host3 kernel: RAMDISK: [mem 0x35981000-0x36cb8fff]
Aug 20 14:13:00 host3 kernel: ACPI: RSDP 00000000000f4f00 00024 (v02 HP    )
Aug 20 14:13:00 host3 kernel: ACPI: XSDT 00000000bddaed00 000E4 (v01 HP     ProLiant 00000002   <D2>? 0000162E)
Aug 20 14:13:00 host3 kernel: ACPI: FACP 00000000bddaee40 000F4 (v03 HP     ProLiant 00000002   <D2>? 0000162E)
Aug 20 14:13:00 host3 kernel: ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20130517/tbfadt-668)
Aug 20 14:13:00 host3 kernel: ACPI BIOS Warning (bug): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20130517/tbfadt-668)
```
This exception will be raised for bare specs and HostContext specs, and potentially archive specs.
```python
Traceback (most recent call last):
  File "/home/bfahr/work/articles/insights-core/insights/core/dr.py", line 953, in run
    result = DELEGATES[component].process(broker)
  File "/home/bfahr/work/articles/insights-core/insights/core/dr.py", line 672, in process
    return self.invoke(broker)
  File "/home/bfahr/work/articles/insights-core/insights/core/plugins.py", line 124, in invoke
    return self.component(dep_value)
  File "/home/bfahr/work/articles/insights-core/insights/core/__init__.py", line 87, in __init__
    self._handle_content(context)
  File "/home/bfahr/work/articles/insights-core/insights/core/__init__.py", line 90, in _handle_content
    self.parse_content(context.content)
  File "/home/bfahr/work/articles/insights-core/insights/core/spec_factory.py", line 119, in content
    self._content = self.load()
  File "/home/bfahr/work/articles/insights-core/insights/core/spec_factory.py", line 250, in load
    return [l.rstrip("\n") for l in f]
  File "/home/bfahr/work/articles/insights-core/insights/core/spec_factory.py", line 250, in <listcomp>
    return [l.rstrip("\n") for l in f]
  File "/usr/lib64/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd2 in position 3160: invalid continuation byte
```


Discussion:

Resolving patch:
diff --git a/insights/core/spec_factory.py b/insights/core/spec_factory.py
--- a/insights/core/spec_factory.py
+++ b/insights/core/spec_factory.py
@@ -243,8 +243,12 @@ def load(self):
             rc, out = self.ctx.shell_out(args, keep_rc=True, env=SAFE_ENV)
             self.rc = rc
             return out
-        with open(self.path, "rU") as f:  # universal newlines
-            return [l.rstrip("\n") for l in f]
+        if six.PY3:
+            with open(self.path, "r", encoding="ascii", errors="surrogateescape") as f:
+                return [l.rstrip("\n") for l in f]
+        else:
+            with open(self.path, "rU") as f:  # universal newlines
+                return [l.rstrip("\n") for l in f]
 
     def _stream(self):
         """
@@ -261,8 +265,12 @@ def _stream(self):
                     with streams.connect(*args, env=SAFE_ENV) as s:
                         yield s
                 else:
-                    with open(self.path, "rU") as f:  # universal newlines
-                        yield f
+                    if six.PY3:
+                        with open(self.path, "r", encoding="ascii", errors="surrogateescape") as f:
+                            yield f
+                    else:
+                        with open(self.path, "rU") as f:  # universal newlines
+                            yield f
         except StopIteration:
             raise
         except Exception as ex:


Test patch: