Repository: mesonbuild/meson

Issue:
Reported benchmark times too big
I just tried using Meson's provided `benchmark` directive and noticed something odd. The times reported by running the benchmarks through meson are much bigger compared to running them on their own.

This probably also affects the times reported for unit tests, but I honestly don't care about them.

The related project is available on GitHub [here](https://github.com/W4RH4WK/mCc).
To reproduce this, make sure you have `flex` and `bison` installed and run the following:

    $ git clone https://github.com/W4RH4WK/mCc
    $ cd mCc
    $ git checkout 700ce09ea8ebe5b82256843d13a6224ef9da86c6
    $ meson builddir
    $ cd builddir
    $ ninja benchmark

The project is built and the benchmarks are run.
My system yields the following output:

```
...
[22/22] Running benchmark suite.
1/2 parser/binary_op: 100000                OK       0.43 s
2/2 parser/nested_expression: 100000        OK       0.46 s

OK:         2
FAIL:       0
SKIP:       0
TIMEOUT:    0

Full log written to /home/w4rh4wk/git/mCc/builddir/meson-logs/benchmarklog.txt
```

Now if I run the corresponding benchmark binary, with the same number of repetitions (first argument), I get significantly faster reported runtime:

```
$ \time -v ./bench_parser_binary_op 100000
	Command being timed: "./bench_parser_binary_op 100000"
	User time (seconds): 0.22
	System time (seconds): 0.06
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.29
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1432
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 69
	Voluntary context switches: 1
	Involuntary context switches: 1
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
```

- - -

I am on a Lenovo T460s:

```
$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 78
Model name:            Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
Stepping:              3
CPU MHz:               757.226
CPU max MHz:           3000.0000
CPU min MHz:           400.0000
BogoMIPS:              4992.46
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              3072K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb invpcid_single intel_pt spec_ctrl kaiser tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
```


Discussion:
Interesting bug! It would be nice if you could investigate a bit more and try to figure out why it's happening. I suspect it might have something to do with the multiprocess executor we use.
Is the multiprocess executor used for benchmarks? I thought the main point of benchmarks (in Meson) is that they are not executed in parallel.
In benchmark mode, the same infrastructure is used but num_processes is set to 1.
Well, it seems like the overhead is caused by establishing a new environment for the child process.

If I comment out [this line](https://github.com/mesonbuild/meson/blob/86365d9ac2576f1b4799f37be306f25b2ec47e44/mesonbuild/mtest.py#L250), the (reported) runtime is reduced significantly. I also tried moving the `starttime` and `endtime` closer to the `Popen` call, following that up with `p.wait()` -- this did not impact the (reported) runtime notably.

So the times themselves aren't actually wrong. Meson just requires more setup for running a test than simply running it through `time`.

PS: I wasn't able to reproduce times as low as reported by the `time` utility. Yet the lowest reported runtime is about the same as I get by using bash's built-in `time`.
> Well, it seems like the overhead is caused by establishing a new environment for the child process.

That sounds like a Python bug. There's nothing special about setting the environment before calling a process which should cause large slowdowns.

What numbers do you get compared to `/usr/bin/time` and built-in time?
Ah sorry, `/usr/bin/time` and the built-in time command indeed show the same time. I just looked at the wrong line in that ultra verbose output (looked at user-time instead of *Elapsed (wall clock) time*).

`/usr/bin/time`, `time`, and Meson without the environment show about 0.29 s for the `parser/binary_op` benchmark. With environment setup Meson reports about 0.43 s.
Just got out of the shower when it hit me like a drunken hobo. *Of course* the runtime is longer with the environment setup -- `MALLOC_PERTURB_` is set which alters runtime performance. Let me just do...

```
$ MALLOC_PERTURB_=0 /usr/bin/time -v ./bench_parser_binary_op
	Command being timed: "./bench_parser_binary_op"
	User time (seconds): 0.21
	System time (seconds): 0.09
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.31
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1448
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 70
	Voluntary context switches: 1
	Involuntary context switches: 37
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

$ MALLOC_PERTURB_=1 /usr/bin/time -v ./bench_parser_binary_op
	Command being timed: "./bench_parser_binary_op"
	User time (seconds): 0.35
	System time (seconds): 0.10
	Percent of CPU this job got: 99%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.45
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 1460
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 74
	Voluntary context switches: 1
	Involuntary context switches: 1
	Swaps: 0
	File system inputs: 0
	File system outputs: 0
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
```

... see, the numbers line up. So this is actually not a bug in Meson or Python. *Phew!*

Should we set `MALLOC_PERTURB_` only for regular tests and not for benchmarks?
Seems reasonable.

Resolving patch:
diff --git a/mesonbuild/mtest.py b/mesonbuild/mtest.py
--- a/mesonbuild/mtest.py
+++ b/mesonbuild/mtest.py
@@ -228,10 +228,10 @@ def run_single_test(self, wrap, test):
 
             # If MALLOC_PERTURB_ is not set, or if it is set to an empty value,
             # (i.e., the test or the environment don't explicitly set it), set
-            # it ourselves. We do this unconditionally because it is extremely
-            # useful to have in tests.
+            # it ourselves. We do this unconditionally for regular tests
+            # because it is extremely useful to have.
             # Setting MALLOC_PERTURB_="0" will completely disable this feature.
-            if 'MALLOC_PERTURB_' not in child_env or not child_env['MALLOC_PERTURB_']:
+            if ('MALLOC_PERTURB_' not in child_env or not child_env['MALLOC_PERTURB_']) and not self.options.benchmark:
                 child_env['MALLOC_PERTURB_'] = str(random.randint(1, 255))
 
             setsid = None


Test patch: