Repository: nodejs/node
Pull request: benchmark: add per-suite setup option

This allows us to set up fixtures for the benchmark suite only once, which can save quite a bit of time when running benchmarks that require tens of thousands of fixture files or more (e.g. the module benchmarks).

Fixes: https://github.com/nodejs/node/issues/58488

```
$ node git:(main) ✗ time node benchmark/module/module-require.js
module/module-require.js n=10000 type=".js": 23,802.994478117074
module/module-require.js n=10000 type=".json": 39,399.95833454406
module/module-require.js n=10000 type="dir": 17,146.092855523166

User time: 1.08s
System time: 14.75s
CPU time: 83%
Total time: 18.91s

$  node git:(main) ✗ git checkout benchmark-setup
Switched to branch 'benchmark-setup'
$ node git:(benchmark-setup) ✗ time node benchmark/module/module-require.js
module/module-require.js n=10000 type=".js": 23,515.56292405791
module/module-require.js n=10000 type=".json": 35,404.88808736156
module/module-require.js n=10000 type="dir": 16,478.902155500055

User time: 0.86s
System time: 5.79s
CPU time: 87%
Total time: 7.61s
```
<!--
Before submitting a pull request, please read:

- the CONTRIBUTING guide at https://github.com/nodejs/node/blob/HEAD/CONTRIBUTING.md
- the commit message formatting guidelines at
  https://github.com/nodejs/node/blob/HEAD/doc/contributing/pull-requests.md#commit-message-guidelines

For code changes:
1. Include tests for any bug fixes or new features.
2. Update documentation if relevant.
3. Ensure that `make -j4 test` (UNIX), or `vcbuild test` (Windows) passes.

If you believe this PR should be highlighted in the Node.js CHANGELOG
please add the `notable-change` label.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.
-->


Changed files:
filename: benchmark/common.js
patch: @@ -28,6 +28,7 @@ class Benchmark {
     const argv = process.argv.slice(2);
     const parsed_args = this._parseArgs(argv, configs, options);
 
+    this.originalOptions = options;
     this.options = parsed_args.cli;
     this.extra_options = parsed_args.extra;
     this.combinationFilter = typeof options.combinationFilter === 'function' ? options.combinationFilter : allow;
@@ -207,6 +208,12 @@ class Benchmark {
       });
     }
 
+    if (this.originalOptions.setup) {
+      // Only do this from the root process. _run() is only ever called from the root,
+      // in child processes main is run directly.
+      this.originalOptions.setup(this.queue);
+    }
+
     const recursive = (queueIndex) => {
       const config = this.queue[queueIndex];
 

filename: benchmark/module/module-require.js
patch: @@ -8,12 +8,15 @@ const benchmarkDirectory = tmpdir.resolve('nodejs-benchmark-module');
 const bench = common.createBenchmark(main, {
   type: ['.js', '.json', 'dir'],
   n: [1e4],
+}, {
+  setup(configs) {
+    tmpdir.refresh();
+    const maxN = configs.reduce((max, c) => Math.max(max, c.n), 0);
+    createEntryPoint(maxN);
+  },
 });
 
 function main({ type, n }) {
-  tmpdir.refresh();
-  createEntryPoint(n);
-
   switch (type) {
     case '.js':
       measureJSFile(n);
@@ -24,8 +27,6 @@ function main({ type, n }) {
     case 'dir':
       measureDir(n);
   }
-
-  tmpdir.refresh();
 }
 
 function measureJSFile(n) {

filename: doc/contributing/writing-and-running-benchmarks.md
patch: @@ -591,6 +591,32 @@ The arguments of `createBenchmark` are:
     containing a combination of benchmark parameters. It should return `true`
     or `false` to indicate whether the combination should be included or not.
 
+  * `setup` {Function} A function that will be run once in the root process
+    before the benchmark combinations are executed in child processes.
+    It can be used to setup any global state required by the benchmark. Note
+    that the JavaScript heap state will not be shared with the benchmark processes,
+    so don't try to access any variables created in the `setup` function from
+    the `main` function, for example.
+    The argument passed into it is an array of all the combinations of
+    configurations that will be executed.
+    If tear down is necessary, register a listener for the `exit` event on
+    `process` inside the `setup` function. In the example below, that's done
+    by `tmpdir.refresh()`.
+
+    ```js
+    const tmpdir = require('../../test/common/tmpdir');
+    const bench = common.createBenchmark(main, {
+      type: ['fast', 'slow'],
+      n: [1e4],
+    }, {
+      setup(configs) {
+        tmpdir.refresh();
+        const maxN = configs.reduce((max, c) => Math.max(max, c.n), 0);
+        setupFixturesReusedForAllBenchmarks(maxN);
+      },
+    });
+    ```
+
 `createBenchmark` returns a `bench` object, which is used for timing
 the runtime of the benchmark. Run `bench.start()` after the initialization
 and `bench.end(n)` when the benchmark is done. `n` is the number of operations