Repository: mesonbuild/meson

Issue:
Spurious "The test program '...' does not exist. Cannot run tests before building them."
Hi,

Starting with c85f7a39a80ccbadb9ffb76102da30c05e3db66f I'm seeing postgres' tests (in my WIP branch using meson) fail on windows with $subject. I think the problem is the os.path.isfile() test introduced in the patch. Previously one could use a shebang that worked across operating systems (i.e. no .exe, with the binary being found via PATH), but now that'll fail because of course the file only exists with the file extension.

Of course I can work around that by not relying on the extracted executable. But I bet this will bite not just us.

Shouldn't this be shutil.which() instead of `os.path.isfile(testentry)`? And probably also gated by self.test.cmd_is_built?

```diff
@@ -1347,9 +1347,12 @@ class SingleTestRunner:
             self.console_mode = ConsoleUser.LOGGER
 
     def _get_test_cmd(self) -> T.Optional[T.List[str]]:
-        if self.test.fname[0].endswith('.jar'):
+        testentry = self.test.fname[0]
+        if self.options.no_rebuild and not os.path.isfile(testentry):
+            raise TestException(f'The test program {testentry!r} does not exist. Cannot run tests before building them.')
+        if testentry.endswith('.jar'):
             return ['java', '-jar'] + self.test.fname
-        elif not self.test.is_cross_built and run_with_mono(self.test.fname[0]):
```

CC: @eli-schwartz 

- Andres


Discussion:
> Shouldn't this be shutil.which() instead of `os.path.isfile(testentry)`? And probably also gated by self.test.cmd_is_built?

This seems like a plausible explanation... but if it is not built, it must surely exist either way, right?

Alternatively, do we only ever not know the exact filename, Windows extensions and all, when it is a not-built script? Maybe we only need one or the other of these checks? (In which case cmd_is_built is faster since it doesn't stat the filesystem.)
> This seems like a plausible explanation... but if it is not built, it must surely exist either way, right?

Yea. But even if built, it still could make sense to not include a file ending in some cases. I think. So gating first by cmd_is_built and then shutil seems to make sense?
I'm trying to think in what case a built file that has a ninja output rule might end up with an unknown file ending, but I'm drawing a blank.
Also on an unrelated note if the test program we are checking for is a script interpreter inserted by a shebang, then that defeats the purpose of this check which is supposed to check that the script (not its interpreter) exists. I guess the best we could hope for there is a silent pass, but it seems like the ideal would be to actually know and check the actual file... :( not sure how to handle that. Not even sure if it can ever happen for a built output either, so it may be irrelevant.
> Also on an unrelated note if the test program we are checking for is a script interpreter inserted by a shebang, then that defeats the purpose of this check which is supposed to check that the script (not its interpreter) exists. 

Yea.  It gets extracted during interpetation from what I can tell.

ISTM that the proper fix for #10006 isn't to error out conditional on no-rebuild or such, but instead to handle the failure nicer at test time instead of crashing with the backtrace that #10006 reported. It should just be a failed test imo.
> I'm trying to think in what case a built file that has a ninja output rule might end up with an unknown file ending, but I'm drawing a blank.

Possibly when cross-building? I don't know the details of how wrappers etc end up being set.
But it's not a failed test. It's a failed precondition for running `meson test`.

The contract for `meson test` is that you test a built project. There is no value to successfully running the project tests, but having individual test units marked as failing because you never actually built the project. That was my initial rationale for rejecting the idea of making it a "failed test" and making it instead be "a failed `meson test`".

Anyway, this doesn't seem especially relevant for the issue at hand, which is avoiding false positive errors. :D
> Possibly when cross-building? I don't know the details of how wrappers etc end up being set.

In mtest.py, in the same function I modified to do `isfile()`, it also checks for `.needs_exe_wrapper` and if so, prepends that. *After* the check I added. (i.e. the wrapper is a separate property from the test command+args.) So that at least shouldn't end up with an unknown file ending coming from the wrapper itself.
> Anyway, this doesn't seem especially relevant for the issue at hand, which is avoiding false positive errors. :D

I think it's somewhat relevant, because checking the file exists doesn't mean it's accessible, executable etc.   If I e.g. have an executable file with a shebang pointing to a non-executable but existing file I get a backtrace even with your change:

```
Traceback (most recent call last):
  File "/home/andres/src/meson/mesonbuild/mesonmain.py", line 149, in run
    return options.run_func(options)
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 2016, in run
    return th.doit()
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1693, in doit
    self.run_tests(runners)
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1832, in run_tests
    loop.run_until_complete(self._run_tests(runners))
  File "/usr/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1929, in _run_tests
    await complete_all(futures)
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1195, in complete_all
    check_futures(done)
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1185, in check_futures
    f.result()
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1860, in test_done
    f.result()
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1855, in run_test
    res = await test.run(self)
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1399, in run
    await self._run_cmd(harness, cmd)
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1453, in _run_cmd
    p = await self._run_subprocess(cmd + extra_cmd,
  File "/home/andres/src/meson/mesonbuild/mtest.py", line 1427, in _run_subprocess
    p = await asyncio.create_subprocess_exec(*args,
  File "/usr/lib/python3.9/asyncio/subprocess.py", line 236, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1661, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "/usr/lib/python3.9/asyncio/unix_events.py", line 197, in _make_subprocess_transport
    transp = _UnixSubprocessTransport(self, protocol, args, shell,
  File "/usr/lib/python3.9/asyncio/base_subprocess.py", line 36, in __init__
    self._start(args=args, shell=shell, stdin=stdin, stdout=stdout,
  File "/usr/lib/python3.9/asyncio/unix_events.py", line 789, in _start
    self._proc = subprocess.Popen(
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: '/tmp/fail/f'
```

You can't ensure that popen won't fail, there's just too many types of failures. Which means that the exception needs to be handled better inside mtest, instead of failing with a backtrace.
I feel like we should validate that during setup, not mtest. Although really, having a shebang not point to a valid executable seems like a really crazy thing to ever happen.
> I feel like we should validate that during setup, not mtest. 

The mtest bit is about handling error reasonably. execve can fail for many reasons, many of which can't be prevented by validation. OOM, process count limits for example. And running a bunch of tests concurrently obviously can run into those. 

> though really, having a shebang not point to a valid executable seems like a really crazy thing to ever happen.

I've actually seen it a few times. It's pretty easy to get it wrong across platforms etc. But really the point is just that there's many ways execution can fail, and it was the first thing I could trigger. Partially because I ran into an exception at interpretation time that I ought to report separately :(.
> The mtest bit is about handling error reasonably. execve can fail for many reasons, many of which can't be prevented by validation. OOM, process count limits for example. And running a bunch of tests concurrently obviously can run into those.

I guess that seems like a reasonable case where it would make sense to mark one specific test as an ERROR (not a failure or a skip!) independent of pre-test sanity checks like "does the command to be run even superficially show up on disk where expected".

I haven't actually checked what kinds of errors manifest in various places, but as an execv failure  it does indeed seem likely we don't catch that. I do think it should be a fallback while still catching the quickly verifiable case of "nothing was ever built". :)
> I haven't actually checked what kinds of errors manifest in various places, but as an execv failure it does indeed seem likely we don't catch that. I do think it should be a fallback while still catching the quickly verifiable case of "nothing was ever built". :)

No argument against that, as long as it doesn't break my tests spuriously :)

Resolving patch:
diff --git a/mesonbuild/backend/backends.py b/mesonbuild/backend/backends.py
--- a/mesonbuild/backend/backends.py
+++ b/mesonbuild/backend/backends.py
@@ -224,6 +224,7 @@ class TestSerialisation:
     protocol: TestProtocol
     priority: int
     cmd_is_built: bool
+    cmd_is_exe: bool
     depends: T.List[str]
     version: str
     verbose: bool
@@ -1147,6 +1148,7 @@ def create_test_serialisation(self, tests: T.List['Test']) -> T.List[TestSeriali
                                    t.is_parallel, cmd_args, t.env,
                                    t.should_fail, t.timeout, t.workdir,
                                    extra_paths, t.protocol, t.priority,
+                                   isinstance(exe, build.Target),
                                    isinstance(exe, build.Executable),
                                    [x.get_id() for x in depends],
                                    self.environment.coredata.version,
diff --git a/mesonbuild/mtest.py b/mesonbuild/mtest.py
--- a/mesonbuild/mtest.py
+++ b/mesonbuild/mtest.py
@@ -1349,18 +1349,18 @@ def __init__(self, test: TestSerialisation, env: T.Dict[str, str], name: str,
 
     def _get_test_cmd(self) -> T.Optional[T.List[str]]:
         testentry = self.test.fname[0]
-        if self.options.no_rebuild and not os.path.isfile(testentry):
+        if self.options.no_rebuild and self.test.cmd_is_built and not os.path.isfile(testentry):
             raise TestException(f'The test program {testentry!r} does not exist. Cannot run tests before building them.')
         if testentry.endswith('.jar'):
             return ['java', '-jar'] + self.test.fname
         elif not self.test.is_cross_built and run_with_mono(testentry):
             return ['mono'] + self.test.fname
-        elif self.test.cmd_is_built and self.test.is_cross_built and self.test.needs_exe_wrapper:
+        elif self.test.cmd_is_exe and self.test.is_cross_built and self.test.needs_exe_wrapper:
             if self.test.exe_wrapper is None:
                 # Can not run test on cross compiled executable
                 # because there is no execute wrapper.
                 return None
-            elif self.test.cmd_is_built:
+            elif self.test.cmd_is_exe:
                 # If the command is not built (ie, its a python script),
                 # then we don't check for the exe-wrapper
                 if not self.test.exe_wrapper.found():


Test patch: