Repository: mesonbuild/meson

Issue:
fix: exception during exception-handling when ast/CodeBlockNode is not set
```
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\src\x\.venv\lib\site-packages\mesonbuild\mesonmain.py", line 184, in run
    return options.run_func(options)
  File "c:\src\x\.venv\lib\site-packages\mesonbuild\msetup.py", line 342, in run
    app.generate()
  File "c:\src\x\.venv\lib\site-packages\mesonbuild\msetup.py", line 171, in generate
    return self._generate(env, capture, vslite_ctx)
  File "c:\src\x\.venv\lib\site-packages\mesonbuild\msetup.py", line 193, in _generate
    intr = interpreter.Interpreter(b, user_defined_options=user_defined_options)
  File "c:\src\x\.venv\lib\site-packages\mesonbuild\interpreter\interpreter.py", line 289, in __init__
    self.load_root_meson_file()
  File "c:\src\x\.venv\lib\site-packages\mesonbuild\interpreterbase\interpreterbase.py", line 113, in load_root_meson_file
    self.handle_meson_version_from_ast()
  File "c:\src\x\.venv\lib\site-packages\mesonbuild\interpreter\interpreter.py", line 525, in handle_meson_version_from_ast
    if not self.ast.lines:
AttributeError: 'NoneType' object has no attribute 'lines'

ERROR: Unhandled python exception
```


Discussion:
For context, can you show an example of a meson.build that triggers this issue?

Also: please apply fixups to previous commits using `git commit --amend` and force pushing rather than creating multiple commits. You can also use `git rebase -I origin/master` and change the "pick" line for the fixup commit from "pick" to "f" to amend it after the fact.
> For context, can you show an example of a meson.build that triggers this issue?
> 
> Also: please apply fixups to previous commits using `git commit --amend` and force pushing rather than creating multiple commits. You can also use `git rebase -I origin/master` and change the "pick" line for the fixup commit from "pick" to "f" to amend it after the fact.

I think I had a malformed meson.build file with "invisible characters" that tripped up the lexer - and then this follow-up-exception occured in vscode.
My feeling is that the bad character should be detected sooner, at the Lexer level.
Exactly. I'd still like to see an actual example meson.build file that reproduces the problem with sufficient fidelity that it can be checked into the repository as a test case -- not "a malformed file that tripped up the lexer". :)

(In fact we should have a test case regardless of whether we fix it in the AST interpreter or the lexer.)
> Exactly. I'd still like to see an actual example meson.build file that reproduces the problem with sufficient fidelity that it can be checked into the repository as a test case -- not "a malformed file that tripped up the lexer". :)
> 
> (In fact we should have a test case regardless of whether we fix it in the AST interpreter or the lexer.)

I just recently figured out why python (in general, not just meson) had so much trouble on my machine.
system:
- windows (any 10+ should work, possibly 7+)
- vscode

steps to reproduce:
- add this setting to vscode: `"files.encoding": "utf8bom"` (which python calls encoding="utf-8-sig")
- edit a meson.build-file with vscode and safe it.
- run meson on that file

Python will try to open the file with utf-8 by default and fail to parse the BOM-"header"(?), tripping up the lexer and then the errors cascade from there.

expected behavior:
1. proper error-message from the lexer: finding invisible characters is near impossible if the tool doesn't help you.
meson does not forward the full exception IIRC which makes this even harder. line-number and index would be really helpful.
3. no secondary error when `self.ast is None` (which I was trying to fix)
4. ideally: auto-attempting utf-8-sig import? maybe?

Opening build files using `utf-8-sig` encoding seems reasonable to me. This will still work with conventional utf-8 files, but will allow to silently drop the BOM when it is present.
As an aside, I'm reworking some of the state handling in the Interpreter, and I've changed the ast to use a dummy `CodeBlockNode`, which should fix some of these errors.

Also, Either using `utf-8-sig` or specifying that the format must be utf-8 no-bom and then adding some exception handling code for that seems appropriate. We could always do that with a wrapper function that sets the encoding properly and just use that instead of the raw `open` command
> Python will try to open the file with utf-8 by default and fail to parse the BOM-"header"(?), tripping up the lexer and then the errors cascade from there.

Can you add a new test case:

```
test cases/failing/130 buildfile encoding/meson.build
```

containing this issue? That way the testsuite will detect this case and make sure it does not regress.

You can see how it is done in

```
test cases/failing/96 invalid option file/
```

for inspiration, but we want to check meson.build instead of meson_options.txt this time.

Resolving patch:
diff --git a/mesonbuild/interpreterbase/interpreterbase.py b/mesonbuild/interpreterbase/interpreterbase.py
--- a/mesonbuild/interpreterbase/interpreterbase.py
+++ b/mesonbuild/interpreterbase/interpreterbase.py
@@ -107,10 +107,11 @@ def load_root_meson_file(self) -> None:
             self.handle_meson_version_from_ast()
         except mparser.ParseException as me:
             me.file = mesonfile
-            # try to detect parser errors from new syntax added by future
-            # meson versions, and just tell the user to update meson
-            self.ast = me.ast
-            self.handle_meson_version_from_ast()
+            if me.ast:
+                # try to detect parser errors from new syntax added by future
+                # meson versions, and just tell the user to update meson
+                self.ast = me.ast
+                self.handle_meson_version_from_ast()
             raise me
 
     def parse_project(self) -> None:
diff --git a/mesonbuild/mparser.py b/mesonbuild/mparser.py
--- a/mesonbuild/mparser.py
+++ b/mesonbuild/mparser.py
@@ -96,6 +96,10 @@ def __eq__(self, other: object) -> bool:
 
 class Lexer:
     def __init__(self, code: str):
+        if code.startswith(codecs.BOM_UTF8.decode('utf-8')):
+            line, *_ = code.split('\n', maxsplit=1)
+            raise ParseException('Builder file must be encoded in UTF-8 (with no BOM)', line, lineno=0, colno=0)
+
         self.code = code
         self.keywords = {'true', 'false', 'if', 'else', 'elif',
                          'endif', 'and', 'or', 'not', 'foreach', 'endforeach',


Test patch:
diff --git a/test cases/failing/130 utf8 with bom/meson.build b/test cases/failing/130 utf8 with bom/meson.build
new file mode 100644
--- /dev/null
+++ b/test cases/failing/130 utf8 with bom/meson.build	
@@ -0,0 +1,3 @@
+﻿project('utf8 with bom')
+
+subdir('subdir')
diff --git a/test cases/failing/130 utf8 with bom/test.json b/test cases/failing/130 utf8 with bom/test.json
new file mode 100644
--- /dev/null
+++ b/test cases/failing/130 utf8 with bom/test.json	
@@ -0,0 +1,8 @@
+{
+    "stdout": [
+      {
+        "line": "test cases/failing/130 utf8 with bom/meson.build:0:0: ERROR: Builder file must be encoded in UTF-8 (with no BOM)"
+      }
+    ]
+  }
+  
\ No newline at end of file
diff --git a/test cases/failing/131 utf8 with bom subdir/meson.build b/test cases/failing/131 utf8 with bom subdir/meson.build
new file mode 100644
--- /dev/null
+++ b/test cases/failing/131 utf8 with bom subdir/meson.build	
@@ -0,0 +1,3 @@
+project('utf8 with bom subdir')
+
+subdir('subdir')
diff --git a/test cases/failing/131 utf8 with bom subdir/subdir/meson.build b/test cases/failing/131 utf8 with bom subdir/subdir/meson.build
new file mode 100644
--- /dev/null
+++ b/test cases/failing/131 utf8 with bom subdir/subdir/meson.build	
@@ -0,0 +1 @@
+﻿a = 'Hello, World!'
diff --git a/test cases/failing/131 utf8 with bom subdir/test.json b/test cases/failing/131 utf8 with bom subdir/test.json
new file mode 100644
--- /dev/null
+++ b/test cases/failing/131 utf8 with bom subdir/test.json	
@@ -0,0 +1,8 @@
+{
+    "stdout": [
+      {
+        "line": "test cases/failing/131 utf8 with bom subdir/subdir/meson.build:0:0: ERROR: Builder file must be encoded in UTF-8 (with no BOM)"
+      }
+    ]
+  }
+  
\ No newline at end of file
diff --git a/test cases/failing/132 utf8 with bom options/meson.build b/test cases/failing/132 utf8 with bom options/meson.build
new file mode 100644
--- /dev/null
+++ b/test cases/failing/132 utf8 with bom options/meson.build	
@@ -0,0 +1 @@
+project('utf8 with bom options')
diff --git a/test cases/failing/132 utf8 with bom options/meson.options b/test cases/failing/132 utf8 with bom options/meson.options
new file mode 100644
--- /dev/null
+++ b/test cases/failing/132 utf8 with bom options/meson.options	
@@ -0,0 +1 @@
+﻿option('someoption', type : 'string', value : 'optval', description : 'An option')
diff --git a/test cases/failing/132 utf8 with bom options/test.json b/test cases/failing/132 utf8 with bom options/test.json
new file mode 100644
--- /dev/null
+++ b/test cases/failing/132 utf8 with bom options/test.json	
@@ -0,0 +1,8 @@
+{
+    "stdout": [
+      {
+        "line": "test cases/failing/132 utf8 with bom options/meson.options:0:0: ERROR: Builder file must be encoded in UTF-8 (with no BOM)"
+      }
+    ]
+  }
+  
\ No newline at end of file