Repository: mesonbuild/meson

Issue:
configure_file fails for files encoded with not utf-8 or ascii
I'm working on wrapping a project (freeglut), and one of the files in the project that need to be configured (using configure_file) is encoded in ISO-8859, and contains an character that isn't compatible with utf-8. meson dies trying to configure said file.

Here's the output:
```
Could not read input file /home/user/source/mesa-demos/subprojects/freeglut-3.0.0/freeglut.rc.in: 'utf-8' codec can't decode byte 0xa9 in position 740: invalid start byte
```

I don't know if this is a case really worth solving (how many projects use encodings other than ascii or utf-8?). Since it's just changing the encoding I've just copied the file into my wrap patch.


Discussion:
What character is it? We could probably try harder to decode files from such ISO encodings.
Why should it be decoded, at all?
Any file is just sequence of bytes.
Meson files are in utf-8 always so all the `configure` defitions are in utf-8. Mixing two encodings is just asking for trouble.
`configure_file()` input needs to be decoded because we do substitutions on it before writing it out, and we must be able to guarantee that you can use arbitrary UTF-8 inside those substitutions.

However, there is a lossless mapping from the extended Latin charsets (ISO-8859 and others), so we should be able to decode it to UTF-8.
But how do you detect which ISO-8859-X it is? Just try all until one works? And then you write out the file in the same encoding or UTF-8?
Then I see only one solution - add new kwarg `encoding`.
All encoding detection (without some a header or magic number declaring it, which is required for some files such as Python) is heuristic, and it looks like `str.decode()` is bad at it? I expect (hope) there's a python module for doing that detection. If there isn't, we should require people to be sane and use UTF-8 for all files.
There are two use cases here

1. transition of old projects to Meson
2. needing to output ISO-8859-1 or something

For 1 we can just say to copy the source file to a different name while you have two build systems and then delete the old one. Or, better yet, convert to UTF-8 in the existing system. It should be doable for most cases and in fact should probably be done regardless of the build system. Having non-utf-8 source these days is just asking for trouble. Even VS supports it.

Case 2 is something that could require adding an encoding keyword. However that requires proof of usage in the real world and in several different projects.
Python only does explict decoding from binary data to <encoding> or treats the data as a byte-stream, and the caller is responsible for doing the right thing. Unfortunately python doesn't have a stdlib module for doing autodetection, python-magic is generally used for this at least on *nix and macOS.

Honestly from my point of view it only makes sense to support non-utf8 for projects being wrapped, if you're porting your project to meson, just use utf-8. And that seems like kinda a niche and difficult to implement in such a way that only wraps use it, so maybe just catching the decode error and printing a nicer error message would be sufficient, IMHO.
I've hit the same problem in gtk+ with Windows resource files (.rc) which are either in the ANSI (mbcs) encoding or utf-16. I've worked around it by removing all non-ASCII for now.

Just to add another potential use case for this.
@lazka that should be fixed on Windows with Python 3.7, I think: https://www.python.org/dev/peps/pep-0538/
Actually, it should already be fixed in Python 3.6. Can you try with that?
I don't think this is related. Afaics this issue is about the file content encoding which should be locale/code page independent anyway.
No, it talks about both [console encoding](https://www.python.org/dev/peps/pep-0528/) and [filesystem encoding](https://www.python.org/dev/peps/pep-0529/).
By "this issue" I meant #1542 :) console and filesystem encoding are not relevant here.
Same problem here with this file: https://github.com/marcosps/kernel_experiments/blob/master/userspace/namespaces/completions/bash/ns_exec

Using file, it returns us-ascci:
ns_exec: text/x-shellscript; charset=us-ascii

Would this be a problem to meson?

Resolving patch:
diff --git a/mesonbuild/interpreter.py b/mesonbuild/interpreter.py
--- a/mesonbuild/interpreter.py
+++ b/mesonbuild/interpreter.py
@@ -1724,7 +1724,7 @@ def get_cross_property_method(self, args, kwargs):
                     'add_test_setup': {'exe_wrapper', 'gdb', 'timeout_multiplier', 'env'},
                     'benchmark': {'args', 'env', 'should_fail', 'timeout', 'workdir', 'suite'},
                     'build_target': known_build_target_kwargs,
-                    'configure_file': {'input', 'output', 'configuration', 'command', 'copy', 'install_dir', 'install_mode', 'capture', 'install', 'format', 'output_format'},
+                    'configure_file': {'input', 'output', 'configuration', 'command', 'copy', 'install_dir', 'install_mode', 'capture', 'install', 'format', 'output_format', 'encoding'},
                     'custom_target': {'input', 'output', 'command', 'install', 'install_dir', 'install_mode', 'build_always', 'capture', 'depends', 'depend_files', 'depfile', 'build_by_default'},
                     'dependency': {'default_options', 'fallback', 'language', 'main', 'method', 'modules', 'optional_modules', 'native', 'required', 'static', 'version', 'private_headers'},
                     'declare_dependency': {'include_directories', 'link_with', 'sources', 'dependencies', 'compile_args', 'link_args', 'link_whole', 'version'},
@@ -3322,8 +3322,10 @@ def func_configure_file(self, node, args, kwargs):
             mlog.log('Configuring', mlog.bold(output), 'using configuration')
             if inputfile is not None:
                 os.makedirs(os.path.join(self.environment.build_dir, self.subdir), exist_ok=True)
+                file_encoding = kwargs.setdefault('encoding', 'utf-8')
                 missing_variables = mesonlib.do_conf_file(ifile_abs, ofile_abs,
-                                                          conf.held_object, fmt)
+                                                          conf.held_object, fmt,
+                                                          file_encoding)
                 if missing_variables:
                     var_list = ", ".join(map(repr, sorted(missing_variables)))
                     mlog.warning(
@@ -3350,7 +3352,8 @@ def func_configure_file(self, node, args, kwargs):
                                            (res.stdout, res.stderr))
             if 'capture' in kwargs and kwargs['capture']:
                 dst_tmp = ofile_abs + '~'
-                with open(dst_tmp, 'w', encoding='utf-8') as f:
+                file_encoding = kwargs.setdefault('encoding', 'utf-8')
+                with open(dst_tmp, 'w', encoding=file_encoding) as f:
                     f.writelines(res.stdout)
                 if ifile_abs:
                     shutil.copymode(ifile_abs, dst_tmp)
diff --git a/mesonbuild/mesonlib.py b/mesonbuild/mesonlib.py
--- a/mesonbuild/mesonlib.py
+++ b/mesonbuild/mesonlib.py
@@ -623,9 +623,9 @@ def do_mesondefine(line, confdata):
         raise MesonException('#mesondefine argument "%s" is of unknown type.' % varname)
 
 
-def do_conf_file(src, dst, confdata, format):
+def do_conf_file(src, dst, confdata, format, encoding='utf-8'):
     try:
-        with open(src, encoding='utf-8') as f:
+        with open(src, encoding=encoding) as f:
             data = f.readlines()
     except Exception as e:
         raise MesonException('Could not read input file %s: %s' % (src, str(e)))
@@ -652,8 +652,11 @@ def do_conf_file(src, dst, confdata, format):
             missing_variables.update(missing)
         result.append(line)
     dst_tmp = dst + '~'
-    with open(dst_tmp, 'w', encoding='utf-8') as f:
-        f.writelines(result)
+    try:
+        with open(dst_tmp, 'w', encoding=encoding) as f:
+            f.writelines(result)
+    except Exception as e:
+        raise MesonException('Could not write output file %s: %s' % (dst, str(e)))
     shutil.copymode(src, dst_tmp)
     replace_if_different(dst, dst_tmp)
     return missing_variables


Test patch:
diff --git a/test cases/common/16 configure file/config8.h.in b/test cases/common/16 configure file/config8.h.in
new file mode 100644
--- /dev/null
+++ b/test cases/common/16 configure file/config8.h.in	
@@ -0,0 +1,3 @@
+#define MESSAGE "@var@"
+
+#define "non isolatin1 char � fails decode with utf-8"
diff --git a/test cases/common/16 configure file/meson.build b/test cases/common/16 configure file/meson.build
--- a/test cases/common/16 configure file/meson.build	
+++ b/test cases/common/16 configure file/meson.build	
@@ -174,3 +174,13 @@ ret = run_command(check_file, inf, outf)
 if ret.returncode() != 0
   error('Error running command: @0@\n@1@'.format(ret.stdout(), ret.stderr()))
 endif
+
+# Test non isolatin1 encoded input file which fails to decode with utf-8
+conf8 = configuration_data()
+conf8.set('var', 'foo')
+configure_file(
+  input : 'config8.h.in',
+  output : '@BASENAME@',
+  encoding : 'koi8-r',
+  configuration : conf8
+)
diff --git a/test cases/failing/79 non ascii in ascii encoded configure file/config9.h.in b/test cases/failing/79 non ascii in ascii encoded configure file/config9.h.in
new file mode 100644
--- /dev/null
+++ b/test cases/failing/79 non ascii in ascii encoded configure file/config9.h.in	
@@ -0,0 +1 @@
+#define MESSAGE "@var@"
diff --git a/test cases/failing/79 non ascii in ascii encoded configure file/meson.build b/test cases/failing/79 non ascii in ascii encoded configure file/meson.build
new file mode 100644
--- /dev/null
+++ b/test cases/failing/79 non ascii in ascii encoded configure file/meson.build	
@@ -0,0 +1,10 @@
+project('non acsii to ascii encoding', 'c')
+# Writing a non ASCII character with a ASCII encoding should fail
+conf9 = configuration_data()
+conf9.set('var', 'д')
+configure_file(
+  input : 'config9.h.in',
+  output : '@BASENAME@',
+  encoding : 'ascii',
+  configuration : conf9
+)