Repository: aai-institute/nnbench

Issue:
Benchmark results do not contain parameters
Reported by @samuelburbulla (thanks!).

Simply put: In the lightning demo,

```python
import nnbench


@nnbench.benchmark
def product(a: int, b: int) -> int:
    return a * b


@nnbench.benchmark
def power(a: int, b: int) -> int:
    return a ** b


runner = nnbench.BenchmarkRunner()
# run the above benchmarks with the parameters `a=2, b=10`...
record = runner.run("__main__", params={"a": 2, "b": 10})
rep = nnbench.BenchmarkReporter()
rep.display(record)  # ...and print the results to the terminal.

# results in a table like the following:
# name     function    date                   value    time_ns
# -------  ----------  -------------------  -------  ---------
# product  product     2024-03-07T10:14:21       20       1000
# power    power       2024-03-07T10:14:21     1024        750
```

we would expect to find the actual parameter values (`a=2, b=10`) in the table, since they are important to contextualize results, and provide filtering opportunities (for example, "show me each `product` result where `a=2`").

As before, we should think about how to deal with these parameters. Inserting them raw into the result record is the simplest solution, but that again balloons up the schema of multiple concurring experiments taking different sets of parameters.

In contrast, inserting the parameters as a struct/dictionary makes access a little more difficult, but keeps the schema easier (and some modern SQL engines support dotted access for struct columns, see e.g. the duckDB example).

Checklist:
- [ ] Provide parameter struct as a key in the benchmark record, inserted in `runner.run()` (take the union of `params` and the defaults from the benchmark interface).
- [ ] Show an application (maybe as a screenshot of application code or a unit test) of filtering a record by parameter values.

Discussion:

Patch:
diff --git a/README.md b/README.md
index 3b39b81..afcf421 100644
--- a/README.md
+++ b/README.md
@@ -49,11 +49,11 @@ record = runner.run("__main__", params={"a": 2, "b": 10})
 rep = nnbench.BenchmarkReporter()
 rep.display(record)  # ...and print the results to the terminal.
 
-# results in a table like the following:
-# name     function    date                   value    time_ns
-# -------  ----------  -------------------  -------  ---------
-# product  product     2024-03-07T10:14:21       20       1000
-# power    power       2024-03-07T10:14:21     1024        750
+# results in a table look like the following:
+# name     function    date                 parameters         value    time_ns
+# -------  ----------  -------------------  -----------------  -------  ---------
+# product  product     2024-03-08T18:03:48  {'a': 2, 'b': 10}       20       1000
+# power    power       2024-03-08T18:03:48  {'a': 2, 'b': 10}     1024        750
 ```
 
 For a more realistic example of how to evaluate a trained model with a benchmark suite, check the [Quickstart](https://aai-institute.github.io/nnbench/latest/quickstart/).
diff --git a/src/nnbench/runner.py b/src/nnbench/runner.py
index 1dacccb..d3417b5 100644
--- a/src/nnbench/runner.py
+++ b/src/nnbench/runner.py
@@ -255,6 +255,7 @@ class BenchmarkRunner:
         results: list[dict[str, Any]] = []
         for benchmark in self.benchmarks:
             bmparams = {k: v for k, v in dparams.items() if k in benchmark.interface.names}
+            bmdefaults = {k: v for (k, _, v) in benchmark.interface.variables}
             # TODO: Wrap this into an execution context
             res: dict[str, Any] = {
                 "name": benchmark.name,
@@ -263,6 +264,7 @@ class BenchmarkRunner:
                 "date": datetime.now().isoformat(timespec="seconds"),
                 "error_occurred": False,
                 "error_message": "",
+                "parameters": bmdefaults | bmparams,
             }
             try:
                 benchmark.setUp(**bmparams)


Test patch:
diff --git a/tests/test_runner.py b/tests/test_runner.py
index 2fa4a38..c1aa2ca 100644
--- a/tests/test_runner.py
+++ b/tests/test_runner.py
@@ -71,3 +71,18 @@ def test_error_on_duplicate_context_keys_in_runner(testfolder: str) -> None:
             params={"x": 1, "y": 1},
             context=context_providers,
         )
+
+
+def test_filter_benchmarks_on_params(testfolder: str) -> None:
+    @nnbench.benchmark
+    def prod(a: int, b: int = 1) -> int:
+        return a * b
+
+    r = nnbench.BenchmarkRunner()
+    r.benchmarks.append(prod)
+    # TODO (nicholasjng): This is hacky
+    rec1 = r.run("", params={"a": 1, "b": 2})
+    assert rec1.benchmarks[0]["parameters"] == {"a": 1, "b": 2}
+    # Assert that the defaults are also present if not overridden.
+    rec2 = r.run("", params={"a": 1})
+    assert rec2.benchmarks[0]["parameters"] == {"a": 1, "b": 1}