Repository: easybuilders/easybuild-framework

Issue:
hard crash with UnicodeDecodeError when reading patch file that contains backtick character when using Python 3
While testing https://github.com/easybuilders/easybuild-easyconfigs/pull/9832 on top of Python 3.6, `eb` crashed hard with a `UnicodeDecodeError`.

The underlying cause is a backtick character in the patch file `Grace-5.1.25-5build1.patch` on line 7238:

```
$  sed -n '7238p' easybuild/easyconfigs/g/Grace/Grace-5.1.25-5build1.patch
+  ximage->byte_order=T1_byte_order; /* Set t1lib�s byteorder */
```

Here's the error reproduced with a small script that basically just calls `det_patched_files` on the patch file:

```
Traceback (most recent call last):
  File "t.py", line 5, in <module>
    det_patched_files('easybuild/easyconfigs/g/Grace/Grace-5.1.25-5build1.patch')
  File "/work/easybuild-framework/easybuild/tools/filetools.py", line 996, in det_patched_files
    txt = read_file(path)
  File "/work/easybuild-framework/easybuild/tools/filetools.py", line 189, in read_file
    txt = handle.read()
  File "/usr/local/Cellar/python/3.7.2/Frameworks/Python.framework/Versions/3.7/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 241240: invalid start byte
```


Discussion:

Resolving patch:
diff --git a/easybuild/tools/filetools.py b/easybuild/tools/filetools.py
--- a/easybuild/tools/filetools.py
+++ b/easybuild/tools/filetools.py
@@ -944,7 +944,9 @@ def det_patched_files(path=None, txt=None, omit_ab_prefix=False, github=False, f
     patched_regex = re.compile(patched_regex, re.M)
 
     if path is not None:
-        txt = read_file(path)
+        # take into account that file may contain non-UTF-8 characters;
+        # so, read a byte string, and decode to UTF-8 string (ignoring any non-UTF-8 characters);
+        txt = read_file(path, mode='rb').decode('utf-8', 'replace')
     elif txt is None:
         raise EasyBuildError("Either a file path or a string representing a patch should be supplied")
 


Test patch:
diff --git a/test/framework/filetools.py b/test/framework/filetools.py
--- a/test/framework/filetools.py
+++ b/test/framework/filetools.py
@@ -663,6 +663,18 @@ def test_det_patched_files(self):
         self.assertEqual(ft.det_patched_files(pf), ['b/toy-0.0/toy.source'])
         self.assertEqual(ft.det_patched_files(pf, omit_ab_prefix=True), ['toy-0.0/toy.source'])
 
+        # create a patch file with a non-UTF8 character in it, should not result in problems
+        # (see https://github.com/easybuilders/easybuild-framework/issues/3190)
+        test_patch = os.path.join(self.test_prefix, 'test.patch')
+        patch_txt = b'\n'.join([
+            b"--- foo",
+            b"+++ foo",
+            b"- test line",
+            b"+ test line with non-UTF8 char: '\xa0'",
+        ])
+        ft.write_file(test_patch, patch_txt)
+        self.assertEqual(ft.det_patched_files(test_patch), ['foo'])
+
     def test_guess_patch_level(self):
         "Test guess_patch_level."""
         # create dummy toy.source file so guess_patch_level can work