Repository: mesonbuild/meson

Issue:
Test priorities/ordering
Right now, the order of tests during `ninja -C build test` is the order they were defined in `meson.build`. In my case, some tests take forever, others are very quick. I know that if one of the quick tests fails, the test suite fill fail so there's no point waiting an hour for the rest of the tests to finish.

Moving the quick tests up in the meson.build file means moving them outside their logical scope. e.g. if I build a helper library that's used by the main code, I want the tests for that helper library defined right there, not in some other part of `meson.build`.

So what I'd like is some way to influence the test order, e.g. by assigning a numerical priority to the test. Tests will then be executed in decreasing priority, with all the tests defaulting to 0.

Is this something worth implementing?


Discussion:
You could use the `--num-processes` argument to `meson test` to run in parallel, I think.
In general there will be many tests and the slow ones can get picked first, despite default testing in parallel. Being able to select test dependency is a needed feature, which is available in some detail in Ctest.
for example if test A fails, don't try to run test B. 
Featurewise this seems like a useful thing. How would this look in practice? Would there be priority levels between 1-10 with 1 being most important and 10 least, with 5 being the default?
I think that test dependency is the most necessary, and possibly the most straightforward to implement.
I would *really* like (and need for certain projects) to have a meson.build snippet like

```meson
big_exe = exectuable(...)
little_validator_exe = exectuable(...)

little_test = test('little', little_validator_exe, ...)
big_test = test('big', big_exe, depends: little_test, ...)
```

where `test(depends: foo)` looks at a boolean property foo.passed.
So in this example, if little_test doesn't pass, big_test doesn't run. But other tests would run unless they have a dependency that failed


---

CMake test properties reference:
* [FIXTURES](https://cmake.org/cmake/help/latest/prop_test/FIXTURES_REQUIRED.html#prop_test:FIXTURES_REQUIRED) this is how Ctest does dependencies. Probably excessive for Meson.
* [COST](https://cmake.org/cmake/help/latest/prop_test/COST.html) is for ordering (lower priority to implement in my opinion)

> Featurewise this seems like a useful thing. How would this look in practice? Would there be priority levels between 1-10 with 1 being most important and 10 least, with 5 being the default?

first gut reaction would be: make it centered on 0. positive numbers increase priority, negative ones decrease priority and there's no need to add a fixed range. It's just numbers anyway and it gives anyone the flexibility to pick a priority of -1000 for the least important tests.

question is then what to do with subprojects: if our tests are 0, 10, 500 and a subproject has 0, 1000 do we run them in the implied order? or do we need to associate an offset with the subproject to rebase it into our project?

but i agree with @scivision, implementing test dependencies would largely obsolete the priority order but provide some more useful features on top. at least for my use-case it'd be enough because the initial test everything relies on is quick.

but there is an issue with pure serial dependencies though, similar to CPU branch prediction - tests may only have a pseudo-dependency on each other: if test A failed I don't want to start test B. But if test A is still running and I have CPU cycles to spare, I still want to start test B for the optimistic case that A will succeed.
> implementing test dependencies would largely obsolete the priority order but provide some more useful features on top

However it means that you have to add that dependency to _every_ other test declaration. Having it with higher priority and bailing out on first error makes it a lot simpler to use.
Yes if having an integer priority instead of dependency is easier, that is fine too--whatever would allow some way to not run expensive tests if cheap preliminary tests fails.
A workaround for this at least for CI is to specify test name(s) or suite(s) at the command line, so that cheap preliminary tests are run first, and then following `meson test` invocations specify the expensive tests. I didn't realize this was possible until https://github.com/mesonbuild/meson/pull/5679

example in CI:

```yaml
- script: meson test cheap1 cheap2
- script: meson test big1 
```


Resolving patch:
diff --git a/mesonbuild/backend/backends.py b/mesonbuild/backend/backends.py
--- a/mesonbuild/backend/backends.py
+++ b/mesonbuild/backend/backends.py
@@ -86,7 +86,7 @@ def __init__(self, name: str, project: str, suite: str, fname: typing.List[str],
                  is_parallel: bool, cmd_args: typing.List[str],
                  env: build.EnvironmentVariables, should_fail: bool,
                  timeout: typing.Optional[int], workdir: typing.Optional[str],
-                 extra_paths: typing.List[str], protocol: str):
+                 extra_paths: typing.List[str], protocol: str, priority: int):
         self.name = name
         self.project_name = project
         self.suite = suite
@@ -103,6 +103,7 @@ def __init__(self, name: str, project: str, suite: str, fname: typing.List[str],
         self.workdir = workdir
         self.extra_paths = extra_paths
         self.protocol = protocol
+        self.priority = priority
 
 class OptionProxy:
     def __init__(self, value):
@@ -727,7 +728,7 @@ def write_test_file(self, datafile):
 
     def create_test_serialisation(self, tests):
         arr = []
-        for t in tests:
+        for t in sorted(tests, key=lambda tst: -1 * tst.priority):
             exe = t.get_exe()
             if isinstance(exe, dependencies.ExternalProgram):
                 cmd = exe.get_command()
@@ -769,7 +770,8 @@ def create_test_serialisation(self, tests):
                     raise MesonException('Bad object in test command.')
             ts = TestSerialisation(t.get_name(), t.project_name, t.suite, cmd, is_cross,
                                    exe_wrapper, t.is_parallel, cmd_args, t.env,
-                                   t.should_fail, t.timeout, t.workdir, extra_paths, t.protocol)
+                                   t.should_fail, t.timeout, t.workdir,
+                                   extra_paths, t.protocol, t.priority)
             arr.append(ts)
         return arr
 
diff --git a/mesonbuild/interpreter.py b/mesonbuild/interpreter.py
--- a/mesonbuild/interpreter.py
+++ b/mesonbuild/interpreter.py
@@ -895,7 +895,8 @@ class Test(InterpreterObject):
     def __init__(self, name: str, project: str, suite: List[str], exe: build.Executable,
                  depends: List[Union[build.CustomTarget, build.BuildTarget]],
                  is_parallel: bool, cmd_args: List[str], env: build.EnvironmentVariables,
-                 should_fail: bool, timeout: int, workdir: Optional[str], protocol: str):
+                 should_fail: bool, timeout: int, workdir: Optional[str], protocol: str,
+                 priority: int):
         InterpreterObject.__init__(self)
         self.name = name
         self.suite = suite
@@ -909,6 +910,7 @@ def __init__(self, name: str, project: str, suite: List[str], exe: build.Executa
         self.timeout = timeout
         self.workdir = workdir
         self.protocol = protocol
+        self.priority = priority
 
     def get_exe(self):
         return self.exe
@@ -2036,7 +2038,7 @@ def get_cross_property_method(self, args, kwargs):
                     'subdir': {'if_found'},
                     'subproject': {'version', 'default_options', 'required'},
                     'test': {'args', 'depends', 'env', 'is_parallel', 'should_fail', 'timeout', 'workdir',
-                             'suite', 'protocol'},
+                             'suite', 'protocol', 'priority'},
                     'vcs_tag': {'input', 'output', 'fallback', 'command', 'replace_string'},
                     }
 
@@ -3366,6 +3368,7 @@ def func_benchmark(self, node, args, kwargs):
         self.add_test(node, args, kwargs, False)
 
     @FeatureNewKwargs('test', '0.46.0', ['depends'])
+    @FeatureNewKwargs('test', '0.52.0', ['priority'])
     @permittedKwargs(permitted_kwargs['test'])
     def func_test(self, node, args, kwargs):
         self.add_test(node, args, kwargs, True)
@@ -3436,8 +3439,11 @@ def add_test(self, node, args, kwargs, is_base_test):
         for dep in depends:
             if not isinstance(dep, (build.CustomTarget, build.BuildTarget)):
                 raise InterpreterException('Depends items must be build targets.')
+        priority = kwargs.get('priority', 0)
+        if not isinstance(priority, int):
+            raise InterpreterException('Keyword argument priority must be an integer.')
         t = Test(args[0], prj, suite, exe.held_object, depends, par, cmd_args,
-                 env, should_fail, timeout, workdir, protocol)
+                 env, should_fail, timeout, workdir, protocol, priority)
         if is_base_test:
             self.build.tests.append(t)
             mlog.debug('Adding test', mlog.bold(args[0], True))
diff --git a/mesonbuild/mintro.py b/mesonbuild/mintro.py
--- a/mesonbuild/mintro.py
+++ b/mesonbuild/mintro.py
@@ -321,6 +321,7 @@ def get_test_list(testdata):
         to['timeout'] = t.timeout
         to['suite'] = t.suite
         to['is_parallel'] = t.is_parallel
+        to['priority'] = t.priority
         result.append(to)
     return result
 


Test patch:
diff --git a/test cases/common/224 test priorities/meson.build b/test cases/common/224 test priorities/meson.build
new file mode 100644
--- /dev/null
+++ b/test cases/common/224 test priorities/meson.build	
@@ -0,0 +1,22 @@
+project('test priorities', 'c')
+
+test_prog = find_program('test.sh')
+
+test('priority 0', test_prog,
+     args : ['0'],
+)
+
+test('priority neg 10', test_prog,
+     args : ['-10'],
+     priority : -10
+)
+
+test('priority 1000', test_prog,
+     args : ['1000'],
+     priority : 1000
+)
+
+test('priority 50', test_prog,
+     args : ['50'],
+     priority : 50
+)
diff --git a/test cases/common/224 test priorities/test.sh b/test cases/common/224 test priorities/test.sh
new file mode 100644
--- /dev/null
+++ b/test cases/common/224 test priorities/test.sh	
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+echo "$1"