Repository: mesonbuild/meson

Issue:
Unable to use find's `print0` with meson
On Linux you can have some *super* weird characters in your paths -- include newlines. This means the only safe character you can use is the null terminator.

find and xarg provide tools for doing this;
* From the find man page;
```
       -print0
              True;  print  the  full  file  name on the standard output, followed by a null character
              (instead of the newline character that -print uses).  This allows file names  that  con‐
              tain newlines or other types of white space to be correctly interpreted by programs that
              process the find output.  This option corresponds to the -0 option of xargs.
```

* From the xargs man page;
```
       -0, --null
              Input  items are terminated by a null character instead of by whitespace, and the quotes
              and backslash are not special (every character is taken literally).  Disables the end of
              file  string,  which  is treated like any other argument.  Useful when input items might
              contain white space, quote marks, or backslashes.  The GNU find -print0 option  produces
              input suitable for this mode.
```

However, when you try and to this with meson using the following;
```meson
project('100 print null')

find_file_list = run_command(find_program('find'), '-type', 'f', '-print0')
assert(find_file_list.returncode() == 0, 'Didn\'t find any files.')

found_files = []
foreach l : find_file_list.stdout().split('\0')
	found_files += [files([l])]
endforeach
```

You get the following error output;
```
tansell@tansell:~/github/mesonbuild/meson/test cases/common/100 print null$ meson build .
The Meson build system
Version: 0.45.0.dev1
Source dir: XXXX/github/mesonbuild/meson/test cases/common/100 print null
Build dir: XXXX/github/mesonbuild/meson/test cases/common/100 print null/build
Build type: native build
Project name: 100 print null
Build machine cpu family: x86_64
Build machine cpu: x86_64
Program find found: YES (/usr/bin/find)
Traceback (most recent call last):
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/mesonmain.py", line
363, in run
    app.generate()
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/mesonmain.py", line
150, in generate
    self._generate(env)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/mesonmain.py", line
197, in _generate
    intr.run()
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreter.py", line 2992, in run
    super().run()
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 173, in run
    self.evaluate_codeblock(self.ast, start=1)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 195, in evaluate_codeblock
    raise e
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 189, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 230, in evaluate_statement
    return self.evaluate_foreach(cur)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 404, in evaluate_foreach
    self.evaluate_codeblock(node.block)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 195, in evaluate_codeblock
    raise e
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 189, in evaluate_codeblock
    self.evaluate_statement(cur)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 232, in evaluate_statement
    return self.evaluate_plusassign(cur)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 409, in evaluate_plusassign
    addition = self.evaluate_statement(node.value)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 216, in evaluate_statement
    return self.evaluate_arraystatement(cur)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 243, in evaluate_arraystatement
    (arguments, kwargs) = self.reduce_arguments(cur.args)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 640, in reduce_arguments
    reduced_pos = [self.evaluate_statement(arg) for arg in args.arguments]
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 640, in <listcomp>
    reduced_pos = [self.evaluate_statement(arg) for arg in args.arguments]
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 200, in evaluate_statement
    return self.function_call(cur)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 456, in function_call
    return self.funcs[func_name](node, self.flatten(posargs), kwargs)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 55, in wrapped
    return f(self, node, args, kwargs)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreterbase.py", line 47, in wrapped
    return f(self, node, args, kwargs)
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreter.py", line 1614, in func_files
    return [mesonlib.File.from_source_file(self.environment.source_dir, self.subdir, fname) for fname in
args]
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/interpreter.py", line 1614, in <listcomp>
    return [mesonlib.File.from_source_file(self.environment.source_dir, self.subdir, fname) for fname in
args]
  File "XXXX/.local/lib/python3.5/site-packages/mesonbuild/mesonlib.py", line 230, in from_source_file
    if not os.path.isfile(os.path.join(source_root, subdir, fname)):
  File "/usr/lib/python3.5/genericpath.py", line 30, in isfile
    st = os.stat(path)
ValueError: embedded null byte
```


Discussion:
In Meson it is considered bad practice to use any shell tools, because they are inherently not portable. If you need to do anything more complex, it is recommended to write a Python script do do that instead.

I'm personally totally fine with having a documented limitation that Meson does not handle file names with newlines in them. 
@jpakkane I can replace the find command with an equivalent Python script pretty easily. However, what I don't understand is how the embedded null byte error is happening? Does `xxx.split('\0')` not strip the value? The docs;
> * split(split_character) splits the string at the specified character (or whitespace if not set) and returns the parts in an array.

don't really mention if the split character is included in the returned value?



Strings support a [limited number of escape characters](http://mesonbuild.com/Syntax.html#strings). `\0` is not one of them.
I don't see that mentioned there at all?
@jpakkane Do you want to support `\0` as an escape sequence or do you want to emit errors on all common escape sequences `\X`? I think failing with a stracktrace is not very nice. Since `\` is used as the escape character I think it is wrong to accept single `\` anywhere in a string. Therefore we could easily give an error message if a single `\` is found. Currently I think both `\\` and `\` will result in a `\`.

To me it makes sense to support at least `\0` and `\t` which are not completely uncommon. This would also be the "least surprising behavior" as it would imitate python. The full list can be found here: https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals
@NickeZ I definitely think some type of error (or atleast a warning) on unknown escape codes would be good.

Resolving patch:
diff --git a/mesonbuild/backend/ninjabackend.py b/mesonbuild/backend/ninjabackend.py
--- a/mesonbuild/backend/ninjabackend.py
+++ b/mesonbuild/backend/ninjabackend.py
@@ -169,7 +169,7 @@ def detect_vs_dep_prefix(self, tempfilename):
                 break
         else:
             # None of our compilers are MSVC, we're done.
-            return open(tempfilename, 'a')
+            return open(tempfilename, 'a', encoding='utf-8')
         filename = os.path.join(self.environment.get_scratch_dir(),
                                 'incdetect.c')
         with open(filename, 'w') as f:
@@ -196,7 +196,7 @@ def detect_vs_dep_prefix(self, tempfilename):
             if match:
                 with open(tempfilename, 'ab') as binfile:
                     binfile.write(b'msvc_deps_prefix = ' + match.group(1) + b'\n')
-                return open(tempfilename, 'a')
+                return open(tempfilename, 'a', encoding='utf-8')
         raise MesonException('Could not determine vs dep dependency prefix string.')
 
     def generate(self, interp):
@@ -206,7 +206,7 @@ def generate(self, interp):
             raise MesonException('Could not detect Ninja v1.5 or newer')
         outfilename = os.path.join(self.environment.get_build_dir(), self.ninja_filename)
         tempfilename = outfilename + '~'
-        with open(tempfilename, 'w') as outfile:
+        with open(tempfilename, 'w', encoding='utf-8') as outfile:
             outfile.write('# This is the build file for project "%s"\n' %
                           self.build.get_project())
             outfile.write('# It is autogenerated by the Meson build system.\n')
diff --git a/mesonbuild/backend/vs2010backend.py b/mesonbuild/backend/vs2010backend.py
--- a/mesonbuild/backend/vs2010backend.py
+++ b/mesonbuild/backend/vs2010backend.py
@@ -227,7 +227,7 @@ def get_target_deps(self, t, recursive=False):
 
     def generate_solution(self, sln_filename, projlist):
         default_projlist = self.get_build_by_default_targets()
-        with open(sln_filename, 'w') as ofile:
+        with open(sln_filename, 'w', encoding='utf-8') as ofile:
             ofile.write('Microsoft Visual Studio Solution File, Format '
                         'Version 11.00\n')
             ofile.write('# Visual Studio ' + self.vs_version + '\n')
@@ -575,7 +575,7 @@ def _prettyprint_vcxproj_xml(self, tree, ofname):
         tree.write(ofname, encoding='utf-8', xml_declaration=True)
         # ElementTree can not do prettyprinting so do it manually
         doc = xml.dom.minidom.parse(ofname)
-        with open(ofname, 'w') as of:
+        with open(ofname, 'w', encoding='utf-8') as of:
             of.write(doc.toprettyxml())
 
     def gen_vcxproj(self, target, ofname, guid):
@@ -1119,7 +1119,7 @@ def gen_regenproj(self, project_name, ofname):
         igroup = ET.SubElement(root, 'ItemGroup')
         rulefile = os.path.join(self.environment.get_scratch_dir(), 'regen.rule')
         if not os.path.exists(rulefile):
-            with open(rulefile, 'w') as f:
+            with open(rulefile, 'w', encoding='utf-8') as f:
                 f.write("# Meson regen file.")
         custombuild = ET.SubElement(igroup, 'CustomBuild', Include=rulefile)
         message = ET.SubElement(custombuild, 'Message')
diff --git a/mesonbuild/mparser.py b/mesonbuild/mparser.py
--- a/mesonbuild/mparser.py
+++ b/mesonbuild/mparser.py
@@ -13,9 +13,44 @@
 # limitations under the License.
 
 import re
+import codecs
 from .mesonlib import MesonException
 from . import mlog
 
+# This is the regex for the supported escape sequences of a regular string
+# literal, like 'abc\x00'
+ESCAPE_SEQUENCE_SINGLE_RE = re.compile(r'''
+    ( \\U........      # 8-digit hex escapes
+    | \\u....          # 4-digit hex escapes
+    | \\x..            # 2-digit hex escapes
+    | \\[0-7]{1,3}     # Octal escapes
+    | \\N\{[^}]+\}     # Unicode characters by name
+    | \\[\\'abfnrtv]   # Single-character escapes
+    )''', re.UNICODE | re.VERBOSE)
+
+# This is the regex for the supported escape sequences of a multiline string
+# literal, like '''abc\x00'''. The only difference is that single quote (')
+# doesn't require escaping.
+ESCAPE_SEQUENCE_MULTI_RE = re.compile(r'''
+    ( \\U........      # 8-digit hex escapes
+    | \\u....          # 4-digit hex escapes
+    | \\x..            # 2-digit hex escapes
+    | \\[0-7]{1,3}     # Octal escapes
+    | \\N\{[^}]+\}     # Unicode characters by name
+    | \\[\\abfnrtv]    # Single-character escapes
+    )''', re.UNICODE | re.VERBOSE)
+
+class MesonUnicodeDecodeError(MesonException):
+    def __init__(self, match):
+        super().__init__("%s" % match)
+        self.match = match
+
+def decode_match(match):
+    try:
+        return codecs.decode(match.group(0), 'unicode_escape')
+    except UnicodeDecodeError as err:
+        raise MesonUnicodeDecodeError(match.group(0))
+
 class ParseException(MesonException):
     def __init__(self, text, line, lineno, colno):
         # Format as error message, followed by the line with the error, followed by a caret to show the error column.
@@ -112,7 +147,6 @@ def lex(self, subdir):
         par_count = 0
         bracket_count = 0
         col = 0
-        newline_rx = re.compile(r'(?<!\\)((?:\\\\)*)\\n')
         while loc < len(self.code):
             matched = False
             value = None
@@ -145,12 +179,18 @@ def lex(self, subdir):
                         if match_text.find("\n") != -1:
                             mlog.warning("""Newline character in a string detected, use ''' (three single quotes) for multiline strings instead.
 This will become a hard error in a future Meson release.""", self.getline(line_start), lineno, col)
-                        value = match_text[1:-1].replace(r"\'", "'")
-                        value = newline_rx.sub(r'\1\n', value)
-                        value = value.replace(r" \\ ".strip(), r" \ ".strip())
+                        value = match_text[1:-1]
+                        try:
+                            value = ESCAPE_SEQUENCE_SINGLE_RE.sub(decode_match, value)
+                        except MesonUnicodeDecodeError as err:
+                            raise MesonException("Failed to parse escape sequence: '{}' in string:\n  {}".format(err.match, match_text))
                     elif tid == 'multiline_string':
                         tid = 'string'
                         value = match_text[3:-3]
+                        try:
+                            value = ESCAPE_SEQUENCE_MULTI_RE.sub(decode_match, value)
+                        except MesonUnicodeDecodeError as err:
+                            raise MesonException("Failed to parse escape sequence: '{}' in string:\n{}".format(err.match, match_text))
                         lines = match_text.split('\n')
                         if len(lines) > 1:
                             lineno += len(lines) - 1


Test patch:
diff --git a/test cases/common/190 escape and unicode/file.c.in b/test cases/common/190 escape and unicode/file.c.in
new file mode 100644
--- /dev/null
+++ b/test cases/common/190 escape and unicode/file.c.in	
@@ -0,0 +1,5 @@
+#include<stdio.h>
+const char* does_it_work() {
+    printf("{NAME}\n");
+    return "yes it does";
+}
diff --git a/test cases/common/190 escape and unicode/file.py b/test cases/common/190 escape and unicode/file.py
new file mode 100644
--- /dev/null
+++ b/test cases/common/190 escape and unicode/file.py	
@@ -0,0 +1,10 @@
+#!/usr/bin/env python3
+
+import sys
+import os
+
+with open(sys.argv[1]) as fh:
+    content = fh.read().replace("{NAME}", sys.argv[2])
+
+with open(os.path.join(sys.argv[3]), 'w') as fh:
+    fh.write(content)
diff --git a/test cases/common/190 escape and unicode/find.py b/test cases/common/190 escape and unicode/find.py
new file mode 100644
--- /dev/null
+++ b/test cases/common/190 escape and unicode/find.py	
@@ -0,0 +1,9 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+for fh in os.listdir('.'):
+    if os.path.isfile(fh):
+        if fh.endswith('.c'):
+            sys.stdout.write(fh + '\0')
diff --git a/test cases/common/190 escape and unicode/fun.c b/test cases/common/190 escape and unicode/fun.c
new file mode 100644
--- /dev/null
+++ b/test cases/common/190 escape and unicode/fun.c	
@@ -0,0 +1,3 @@
+int a_fun() {
+    return 1;
+}
diff --git a/test cases/common/190 escape and unicode/main.c b/test cases/common/190 escape and unicode/main.c
new file mode 100644
--- /dev/null
+++ b/test cases/common/190 escape and unicode/main.c	
@@ -0,0 +1,12 @@
+#include <string.h>
+
+const char* does_it_work();
+
+int a_fun();
+
+int main() {
+    if(strcmp(does_it_work(), "yes it does") != 0) {
+        return -a_fun();
+    }
+    return 0;
+}
diff --git a/test cases/common/190 escape and unicode/meson.build b/test cases/common/190 escape and unicode/meson.build
new file mode 100644
--- /dev/null
+++ b/test cases/common/190 escape and unicode/meson.build	
@@ -0,0 +1,24 @@
+project('180 escape', 'c')
+
+gen = generator(find_program('file.py'), arguments:['@INPUT@', 'erd\u0151', '@OUTPUT@'], output: '@BASENAME@')
+
+gen_file = gen.process('file.c.in')
+
+find_file_list = run_command(find_program('find.py'))
+assert(find_file_list.returncode() == 0, 'Didn\'t find any files.')
+
+# Strings should support both octal \ooo and hex \xhh encodings
+
+found_files_oct = []
+foreach l : find_file_list.stdout().strip('\0').split('\000')
+  found_files_oct += [files(l)]
+endforeach
+
+test('first', executable('first', found_files_oct + [gen_file]))
+
+found_files_hex = []
+foreach l : find_file_list.stdout().strip('\x00').split('\x00')
+  found_files_hex += [files(l)]
+endforeach
+
+test('second', executable('second', found_files_hex + [gen_file]))
diff --git a/test cases/common/33 try compile/meson.build b/test cases/common/33 try compile/meson.build
--- a/test cases/common/33 try compile/meson.build	
+++ b/test cases/common/33 try compile/meson.build	
@@ -1,11 +1,11 @@
 project('try compile', 'c', 'cpp')
 
 code = '''#include<stdio.h>
-void func() { printf("Something.\n"); }
+void func() { printf("Something.\\n"); }
 '''
 
 breakcode = '''#include<nonexisting.h>
-void func() { printf("This won't work.\n"); }
+void func() { printf("This won't work.\\n"); }
 '''
 
 foreach compiler : [meson.get_compiler('c'), meson.get_compiler('cpp')]
diff --git a/test cases/common/39 tryrun/meson.build b/test cases/common/39 tryrun/meson.build
--- a/test cases/common/39 tryrun/meson.build	
+++ b/test cases/common/39 tryrun/meson.build	
@@ -13,8 +13,8 @@ endif
 
 ok_code = '''#include<stdio.h>
 int main(int argc, char **argv) {
-  printf("%s\n", "stdout");
-  fprintf(stderr, "%s\n", "stderr");
+  printf("%s\\n", "stdout");
+  fprintf(stderr, "%s\\n", "stderr");
   return 0;
 }
 '''
diff --git a/test cases/common/42 string operations/meson.build b/test cases/common/42 string operations/meson.build
--- a/test cases/common/42 string operations/meson.build	
+++ b/test cases/common/42 string operations/meson.build	
@@ -77,21 +77,21 @@ assert('"1.1.20"'.strip('"') == '1.1.20', '" badly stripped')
 assert('"1.1.20"'.strip('".') == '1.1.20', '". badly stripped')
 assert('"1.1.20"   '.strip('" ') == '1.1.20', '". badly stripped')
 
-bs_b = '''\b'''
-bs_bs_b = '''\\b'''
+bs_c = '''\c'''
+bs_bs_c = '''\\\c'''
 nl = '''
 '''
-bs_n = '''\n'''
+bs_n = '''\\n'''
 bs_nl = '''\
 '''
-bs_bs_n = '''\\n'''
-bs_bs_nl = '''\\
+bs_bs_n = '''\\\\n'''
+bs_bs_nl = '''\\\\
 '''
 
-assert('\b' == bs_b, 'Single backslash broken')
-assert('\\b' == bs_b, 'Double backslash broken')
-assert('\\\b' == bs_bs_b, 'Three backslash broken')
-assert('\\\\b' == bs_bs_b, 'Four backslash broken')
+assert('\c' == bs_c, 'Single backslash broken')
+assert('\\c' == bs_c, 'Double backslash broken')
+assert('\\\c' == bs_bs_c, 'Three backslash broken')
+assert('\\\\c' == bs_bs_c, 'Four backslash broken')
 assert('\n' == nl, 'Newline escape broken')
 assert('\\n' == bs_n, 'Double backslash broken before n')
 assert('\\\n' == bs_nl, 'Three backslash broken before n')
diff --git a/test cases/failing/72 invalid escape char/meson.build b/test cases/failing/72 invalid escape char/meson.build
new file mode 100644
--- /dev/null
+++ b/test cases/failing/72 invalid escape char/meson.build	
@@ -0,0 +1,4 @@
+# Make sure meson exits on invalid string
+# The string below contains an invalid unicode code point
+
+'my name is what \uxyzo who are you'