Repository: easybuilders/easybuild-framework

Issue:
UnicodeDecodeError when adding easyconfig to archive (if archspec is available)
I ran into this crash when installing `MAFFT-7.453-iccifort-2019.5.281-with-extensions.eb` (when using `eb` on top of Python 2):

```
ERROR: Traceback (most recent call last):
  File "/lib/python2.7/site-packages/easybuild_framework-4.2.0.dev0-py2.7.egg/easybuild/main.py", line 114, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "/lib/python2.7/site-packages/easybuild_framework-4.2.0.dev0-py2.7.egg/easybuild/framework/easyblock.py", line 3239, in build_and_install_one
    repo.add_easyconfig(spec, app.name, det_full_ec_version(app.cfg), buildstats, currentbuildstats)
  File "/lib/python2.7/site-packages/easybuild_framework-4.2.0.dev0-py2.7.egg/easybuild/tools/repository/filerepo.py", line 120, in add_easyconfig
    txt += statstxt
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 668: ordinal not in range(128)
```

The culprit is that `archspec` returns a `unicode`-type value as CPU arch name, which gets embedded in `statstxt`, making it also a `unicode`-type value (rather than a regular `str`).

When `statstxt` is added to `txt` (which is just a `str` value), it triggers a decoding using the `ascii` of the `txt` value, which fails because the `description` value of `MAFFT-7.453-iccifort-2019.5.281-with-extensions.eb` contains non-ASCII characters (the `~` in "`for alignment of <∼200 sequences`"`).

It's easy to fix this, by making sure we cast the `unicode` value produced by `archspec` to a regular `str` value, but it does raise the question why `archspec` uses `unicode` values at all, there should be no need for that...


Discussion:

Resolving patch:
diff --git a/easybuild/tools/systemtools.py b/easybuild/tools/systemtools.py
--- a/easybuild/tools/systemtools.py
+++ b/easybuild/tools/systemtools.py
@@ -360,7 +360,7 @@ def get_cpu_arch_name():
     if HAVE_ARCHSPEC:
         res = archspec_cpu_host()
         if res:
-            cpu_arch_name = res.name
+            cpu_arch_name = str(res.name)
 
     if cpu_arch_name is None:
         cpu_arch_name = UNKNOWN


Test patch:
diff --git a/test/framework/toy_build.py b/test/framework/toy_build.py
--- a/test/framework/toy_build.py
+++ b/test/framework/toy_build.py
@@ -1,4 +1,5 @@
-# #
+# -*- coding: utf-8 -*-
+##
 # Copyright 2013-2020 Ghent University
 #
 # This file is part of EasyBuild,
@@ -2606,6 +2607,26 @@ def __exit__(self, type, value, traceback):
         self.assertErrorRegex(EasyBuildError, error_pattern, self.test_toy_build,
                               extra_args=extra_args, raise_error=True, verbose=False)
 
+    def test_toy_build_unicode_description(self):
+        """Test installation of easyconfig file that has non-ASCII characters in description."""
+        # cfr. https://github.com/easybuilders/easybuild-framework/issues/3284
+
+        test_ecs_dir = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'easyconfigs', 'test_ecs')
+        toy_ec = os.path.join(test_ecs_dir, 't', 'toy', 'toy-0.0.eb')
+        toy_ec_txt = read_file(toy_ec)
+
+        # the tilde character included here is a Unicode tilde character, not a regular ASCII tilde (~)
+        descr = "This description includes a unicode tilde character: ∼, for your entertainment."
+        self.assertFalse('~' in descr)
+
+        regex = re.compile(r'^description\s*=.*', re.M)
+        test_ec_txt = regex.sub(r'description = "%s"' % descr, toy_ec_txt)
+
+        test_ec = os.path.join(self.test_prefix, 'test.eb')
+        write_file(test_ec, test_ec_txt)
+
+        self.test_toy_build(ec_file=test_ec, raise_error=True)
+
 
 def suite():
     """ return all the tests in this file """