Repository: Lightning-AI/lightning-thunder

Issue:
[Reporting Tool] Modular report classes for saving non-Thunder-specific repro scripts
For the next PR, I think it'd be useful to create an FXReport object. What I'm thinking is:

- Create a new function called `fx_report` (not Thunder-specific)
- Return a FXReport object from `fx_report`
- The FXReport object has a sequence of FXGraphReport objects, one per fx graph in the original program
- Each FXGraphReport has a method to write a reproducer script
  - The method should be able to easily select whether the reproducer uses PyTorch eager, torch.compile, or thunderfx (with some supported options)
  - For total control, the method should allow writing an arbitrary executor string and allow for extending what's imported with arbitrary import strings, in case someone wants to use a very special version of thunderfx or torch.compile or PyTorch eager
- Each FXGraphReport should allow its FXGraph's inputs to be queried (they can just be FakeTensors for now)

So someone could do something like:

```
report: FXReport = fx_report(foo)

fxgr: FXGraphReport
for fxgr in report.graph_reports:
  print(fxgr.inputs)
  fxgr.write_eager_reproducer(filename0)
  fxgr.write_torchcompile_reproducer(filename1)
  fxgr.write_thunder_reproducer(filename2)
  fxgr.write_reproducer(filename3, executor_string="my_crazy_executor", import_strings=("import my_crazy_module", "from my_crazy_module import my_crazy_executor"))
```

And that would generate four reproducers for each of the FX graphs in the program, each one using the different requested executor.

Writing the reproducers directly like this may not be super interesting in isolation, but once that's working we can add more of the data that `thunderfx_report` makes available, and that will become really interesting really fast. It would also be great if the existing reporting functionality, like `thunderfx_report` could be updated to use `FXGraphReport` and `FXReport` objects internally for consistency. Using modular Python objects for the report data and analysis should make it easier for us to develop and for others to extend.

What are your thoughts, @kiya00?

_Originally posted by @mruberry in https://github.com/Lightning-AI/lightning-thunder/issues/1636#issuecomment-2610898585_
            

Discussion:
kiya00: Hi @mruberry , I added some interfaces, mainly the base class `FXGraphReportBase` and the subclasses `FXGraphBenchmarkReport` and `FXGraphReport` that override the method `write_repro(folder: str | PathLike, custom_executor_name: str, custom_import_str: str,  custom_executor_str: str, save_input_tensor: bool, inputs: Sequence[torch.Tensor | ExampleInputMetaData] = None,)` to generate repro/benchmark script. An example of its usage can be found in the test case, what do you think about the interfaces? would you mind review this interface part first and then I'll update the current `ThunderCompiler.save_reproducer_to_folder` to use these new interfaces.

<details>
<summary>the generated repro script `graph0_mythunder.py`:</summary>

```
from math import inf
from math import nan
NoneType = type(None)
import torch
from torch import device
import torch.fx._pytree as fx_pytree
import torch.utils._pytree as pytree
import thunder
from thunder.dev_utils.nvtx_profile_transform import NvtxProfileTransform
from thunder import nvfuser_executor
from thunder.transforms.cudagraph import CUDAGraphTransform


import argparse

parser = argparse.ArgumentParser(description="Script for executing an FX graph with specified configurations.")

parser.add_argument(
    "--check_consistency",
    type=bool,
    default=False,
    help="Whether to check consistency (default: False)"
)
parser.add_argument(
    "--compute_type",
    type=str,
    choices=["forward", "forward+backward"],
    default="forward",
    help="Type of computation to perform (forward, forward+backward)"
)
args = parser.parse_args()
compute_type = args.compute_type
check_acc = args.check_consistency

def test_graph0():
    class DynamoModule(torch.nn.Module):
      def forward(self, L_x_ : torch.Tensor):
          l_x_ = L_x_
          y = l_x_.sin();  l_x_ = None
          return (y,)

    inputs = [
        torch.testing.make_tensor((4, 4), dtype=torch.float32,  device='cuda:0', requires_grad=True, low=None, high=None,).as_strided((4, 4), (4, 1)),

    ]

    model = DynamoModule()
    from thunder.dynamo.report import run_repro1

    compiled_model = thunder.jit(model, transforms=[NvtxProfileTransform(),CUDAGraphTransform()], executors=[nvfuser_executor])
    result = run_repro1(compiled_model, compute_type, *inputs)
    if check_acc:
        eager_result = run_repro1(model, compute_type, *inputs)
        for (compute_t, eager_v), (_, cur_v) in zip(eager_result.items(), result.items()):
            torch.testing.assert_close(eager_v, cur_v, msg=lambda e : f'{compute_t}: {e}')


test_graph0()

```

</details>

<details>
<summary>the generated benchmark script `graph0_mythunder.py`</summary>

```
# NOTE: This script requires `pytest-benchmark==4.0.0` to be installed.
# To execute the script, run `pytest graph0.py --benchmark-timer=torch.utils.benchmark.utils.timer.timer --benchmark-warmup=on --benchmark-group-by=param:compute_type`
# To check the peak allocated CUDA memory, use --benchmark-json=json_file_name and look at the "max_allocated_memory_MB" field in the json file
# To run tests for a specific compute_type, use the pytest `-k` option.
# For example:
#   - `-k "forward"` will run only the forward pass.
#
# Available options:
#   - compute_type: "forward", "backward"

import pytest
from thunder.benchmarks.targets import parametrize_compute_type_only_training, benchmark_for_compute_type
from math import inf
from math import nan
NoneType = type(None)
import torch
from torch import device
import torch.fx._pytree as fx_pytree
import torch.utils._pytree as pytree
import thunder
from thunder.dev_utils.nvtx_profile_transform import NvtxProfileTransform
from thunder import nvfuser_executor
from thunder.transforms.cudagraph import CUDAGraphTransform


@parametrize_compute_type_only_training
def test_graph0(benchmark, compute_type):
    class DynamoModule(torch.nn.Module):
      def forward(self, L_x_ : torch.Tensor):
          l_x_ = L_x_
          y = l_x_.sin();  l_x_ = None
          return (y,)

    inputs = [
        torch.testing.make_tensor((4, 4), dtype=torch.float32,  device='cuda:0', requires_grad=True, low=None, high=None,).as_strided((4, 4), (4, 1)),

    ]

    model = DynamoModule()
    compiled_model = thunder.jit(model, transforms=[NvtxProfileTransform(),CUDAGraphTransform()], executors=[nvfuser_executor])
    benchmark_for_compute_type(compute_type, benchmark, compiled_model, inputs, {})
```

</details>

And one thing we like about using pytest benchmark is its ability to generate a performance comparison table across multiple executors. So I'm also thinking about extending `FXGraphBenchmarkReport.write_repro(...)` to accept a list of executors so that all the executors can be run in a single benchmark script to give the tabular like the previous PR showed
mruberry: <img src="https://media4.giphy.com/media/QmcuB58dRfmZx1KW1Y/giphy.gif"/>

Great start! I made some comments inline for you review. Once those have been resolved let's get this merged, and then we can focus on the next extension.

In the next PR, I think what we'll want to do next is make the timing data programmatically available, so that the timing data can be queried directly to produce interesting reports from Python. That could be a good opportunity to review how we're timing, too.


mruberry: Updates sound great; let me know when this is ready to be reviewed!
kiya00: Hi @mruberry @IvanYashchuk it's ready to review, thank you in advance!

Patch:
diff --git a/thunder/benchmarks/targets.py b/thunder/benchmarks/targets.py
index dd2e22e4..8d69ae6d 100644
--- a/thunder/benchmarks/targets.py
+++ b/thunder/benchmarks/targets.py
@@ -4,6 +4,7 @@ import os
 from collections.abc import Callable
 from enum import auto, Enum
 from collections.abc import Sequence
+from contextlib import nullcontext
 
 import pytest
 import torch
@@ -81,8 +82,9 @@ parametrize_compute_type_only_training = pytest.mark.parametrize(
 )
 
 
-def benchmark_for_compute_type(compute_type: ComputeType, benchmark, fn: Callable, args, kwargs):
-    with record_peak_allocated_memory(benchmark):
+def benchmark_for_compute_type(compute_type: ComputeType, benchmark, fn: Callable, args, kwargs, has_cuda: bool = True):
+    context = record_peak_allocated_memory(benchmark) if has_cuda else nullcontext()
+    with context:
         match compute_type:
             case ComputeType.INFERENCE | ComputeType.TRAINING_FORWARD:
                 benchmark(fn, *args, **kwargs)
diff --git a/thunder/dynamo/compiler.py b/thunder/dynamo/compiler.py
index 24b152b7..aa5bc1c2 100644
--- a/thunder/dynamo/compiler.py
+++ b/thunder/dynamo/compiler.py
@@ -8,9 +8,13 @@ from pathlib import Path
 
 import torch
 
-from thunder.core.baseutils import run_once
-from thunder.core.utils import safe_zip
-from thunder.dynamo.utils import recompile_graph, remove_empty_autocast, reproducer, CompilerType
+from thunder.dynamo.utils import (
+    recompile_graph,
+    remove_empty_autocast,
+    CompilerType,
+    get_split_reasons_string,
+    thunder_options_to_str,
+)
 from thunder.dynamo.splitter import _splitter
 from thunder.core.utils import check
 
@@ -80,7 +84,7 @@ class ThunderCompiler:
         self,
         reproducer_folder: str | PathLike,
         use_pytest_benchmark: bool = False,
-        save_input_tensor=False,
+        serialize_inputs=False,
     ):
         """
         Save the reproducer script for the GraphModule executed by Thunder to the specified ``reproducer_folder``.
@@ -92,38 +96,73 @@ class ThunderCompiler:
         Args:
             reproducer_folder: The folder where the reproducer code will be written. Can be specified as an absolute or relative path.
             use_pytest_benchmark: Determines the type of script to create. When :obj:`False`, create a reproducer script.
-                Otherwise, creats a benchmark script to compare the reproducer's performance with other backends, including Torch eager, torch.compile,
-                and torch.compile with ``backend="eager"``.
+                Otherwise, creats a benchmark script to compare the reproducer's performance with other backends, including Torch eager, torch.compile.
         """
         if not self.subgraph_infos:
             raise TypeError(f"{self} doesn't seem to have been called yet.")
         reproducer_folder = Path(reproducer_folder)
         reproducer_folder.mkdir(exist_ok=True, parents=True)
+        thunder_options_str = thunder_options_to_str(self.thunder_options)
+        thunder_ex_str = f"partial(thunder.jit, {thunder_options_str})" if thunder_options_str else "thunder.jit"
 
         for graph_idx, subgraph_info in enumerate(self.subgraph_infos):
             thunder_module_names = []
             for node in subgraph_info.split_graph_module.graph.nodes:
                 target = node.target
                 if isinstance(target, str) and target.startswith("thunder_"):
-                    thunder_module_names.append(target)
+                    thunder_module_names.append(f"graph{graph_idx}_{target}")
             original_thunder_modules = (
                 m
                 for m, compiled_m in subgraph_info.submodule_to_compiled_functions.items()
                 if compiled_m.compiler == CompilerType.THUNDER
             )
             example_inputs = subgraph_info.thunder_compiled_fns_example_inputs
-            for cur_module, example_input, cur_name in safe_zip(
-                original_thunder_modules, example_inputs, thunder_module_names
-            ):
-                reproducer(
-                    cur_module,
-                    subgraph_info,
-                    self.thunder_options,
-                    example_input,
+            from thunder.dynamo.report import FXReport
+
+            result = FXReport(original_thunder_modules, thunder_module_names)
+            split_reason_str = get_split_reasons_string(subgraph_info)
+            for subgraph_idx, report in enumerate(result.fx_graph_reports):
+                has_cuda_args = any(
+                    hasattr(arg, "device") and arg.device.type == "cuda" for arg in example_inputs[subgraph_idx]
+                )
+                import_str = ["import thunder", "from functools import partial"]
+                if has_cuda_args:
+                    # Since Thunder compile options don't clearly indicate required imports,
+                    # we include commonly used transforms by default.
+                    import_str.extend(
+                        [
+                            "from thunder.transforms.cudagraph import CUDAGraphTransform",
+                            "from thunder.dev_utils.nvtx_profile_transform import NvtxProfileTransform",
+                        ]
+                    )
+                if not use_pytest_benchmark:
+                    report.write_repro(
+                        reproducer_folder,
+                        f"{report.graph_name}_repro.py",
+                        executor_str=thunder_ex_str,
+                        import_str=import_str,
+                        serialize_inputs=serialize_inputs,
+                        inputs=example_inputs[subgraph_idx],
+                        extra_comment_str=split_reason_str,
+                    )
+                    continue
+
+                executor_names_list = ["thunder", "torch_inductor", "eager"]
+                executors = [thunder_ex_str, "torch_inductor", "None"]
+
+                if has_cuda_args:
+                    executor_names_list.append("thunder_cudagraph")
+                    executors.append("partial(thunder.jit, transform=CUDAGraphTransform())")
+
+                report.write_benchmark_repro(
                     reproducer_folder,
-                    f"graph{graph_idx}_{cur_name}",
-                    use_pytest_benchmark,
-                    save_input_tensor=save_input_tensor,
+                    f"{report.graph_name}_benchmark.py",
+                    executor_names_list,
+                    executor_str=executors,
+                    import_str=import_str,
+                    serialize_inputs=serialize_inputs,
+                    inputs=example_inputs[subgraph_idx],
+                    extra_comment_str=split_reason_str,
                 )
 
 
diff --git a/thunder/dynamo/report.py b/thunder/dynamo/report.py
index 16a3474c..34d55edc 100644
--- a/thunder/dynamo/report.py
+++ b/thunder/dynamo/report.py
@@ -1,19 +1,27 @@
 from __future__ import annotations
-from typing import TYPE_CHECKING
+
+import json
 import subprocess
 import sys
+from functools import partial
 from pathlib import Path
-import json
-import torch
+from typing import TYPE_CHECKING
+import textwrap
 
-from thunder.dynamo.compiler import thunderfx
-from thunder.core.utils import sequencify
+import torch
 from thunder.core.pytree import tree_flatten
+from thunder.core.utils import sequencify
+from thunder.dynamo.compiler import thunderfx
+from thunder.dynamo.utils import _get_example_inputs_from_placeholder, _readable, arg_like, get_env
+from thunder.dynamo.repro_script_template import benchmark_multi_exe_code_template, repro_code_template
 
 
 if TYPE_CHECKING:
-    from os import PathLike
     from collections.abc import Callable
+    from os import PathLike
+    from collections.abc import Sequence
+
+    from thunder.dynamo.utils import ExampleInputMetaData
 
 
 def run_backward(fn, *args, **kwargs):
@@ -35,30 +43,17 @@ def run_backward(fn, *args, **kwargs):
     return result, [t.grad for t in forward_inputs]
 
 
-def run_repro(executor_dict, executor_name, model, compute_type, *inputs) -> dict[str, float]:
+def run_repro(compiled_fn, compute_type, *inputs) -> dict[str, float]:
     """Helper function to execute the forward or backward pass based on the `compute_type` using the executor specified by `executor_name` in `executor_dict`.
     If the execution fails, an error is raised. On success, the function returns a dictionary containing the forward results and gradient results.
     """
-    if executor_name == "eager":
-        compiled_fn = model
-    elif executor_name == "torch_inductor":
-        compiled_fn = executor_dict[executor_name](model, inputs)
-    else:
-        compiled_fn = executor_dict[executor_name](model)
-
     results = {}
     match compute_type:
         case "forward":
-            try:
-                result = compiled_fn(*inputs)
-            except Exception as e:
-                raise e
+            result = compiled_fn(*inputs)
             results["forward"] = result
         case "forward+backward":
-            try:
-                forward_result, grads = run_backward(compiled_fn, *inputs)
-            except Exception as e:
-                raise e
+            forward_result, grads = run_backward(compiled_fn, *inputs)
             results["forward"] = forward_result
             results["backward"] = grads
         case _:
@@ -75,8 +70,8 @@ def thunderfx_report(
     folder_path: str | PathLike = "/tmp/thunderfx_report",
     check_consistency: bool = True,
     check_benchmark: bool = True,
-    save_consistency_inputs: bool = False,
-    save_benchmark_inputs: bool = False,
+    serialize_consistency_inputs: bool = False,
+    serialize_benchmark_inputs: bool = False,
     **kwargs,
 ):
     try:
@@ -104,10 +99,10 @@ def thunderfx_report(
         print("\nVerifying consistency between Thunder and Torch eager ...")
         consistency_folder = folder / "consistency"
         consistency_folder.mkdir(exist_ok=True)
-        compiled._backend.save_reproducer_to_folder(consistency_folder, save_input_tensor=save_consistency_inputs)
+        compiled._backend.save_reproducer_to_folder(consistency_folder, serialize_inputs=serialize_consistency_inputs)
         for file in consistency_folder.glob("*.py"):
             g_name = file.name.rstrip(".py")
-            cmd = [sys.executable, folder / file, "--check_consistency=True", "--compute_type=forward+backward"]
+            cmd = [sys.executable, file, "--check_consistency=True", "--compute_type=forward+backward"]
             consistency_result = subprocess.run(cmd, capture_output=True, text=True)
             if consistency_result.returncode:
                 error = consistency_result.stderr
@@ -121,7 +116,7 @@ def thunderfx_report(
         benchmark_folder = folder / "benchmark"
         benchmark_folder.mkdir(exist_ok=True)
         compiled._backend.save_reproducer_to_folder(
-            benchmark_folder, save_input_tensor=save_benchmark_inputs, use_pytest_benchmark=True
+            benchmark_folder, serialize_inputs=serialize_benchmark_inputs, use_pytest_benchmark=True
         )
 
         benchmark_json_files = []
@@ -132,7 +127,7 @@ def thunderfx_report(
                     sys.executable,
                     "-m",
                     "pytest",
-                    benchmark_folder / file,
+                    file,
                     "--benchmark-timer=torch.utils.benchmark.utils.timer.timer",
                     "--benchmark-warmup=on",
                     "--benchmark-group-by=param:compute_type",
@@ -145,13 +140,11 @@ def thunderfx_report(
             )
             g_name = file.name.rstrip(".py")
             if benchmark_result.returncode:
-                print(
-                    f"Failed to run the benchmarking script({benchmark_folder / file}), exception: {benchmark_result.stderr}"
-                )
+                print(f"Failed to run the benchmarking script({file}), exception: {benchmark_result.stderr}")
             else:
                 print(f"{g_name}:\n{benchmark_result.stdout}\n")
 
-        print("Max allocated memory usage:")
+        print("Max allocated CUDA memory usage:")
         for tmp_json in benchmark_json_files:
             with open(tmp_json) as file:
                 data = json.load(file)
@@ -159,10 +152,315 @@ def thunderfx_report(
 
             def print_sorted_memory_info(compute_t: str):
                 filtered_bench = [b for b in bench if compute_t in b["param"]]
-                filtered_bench_sorted = sorted(filtered_bench, key=lambda x: x["extra_info"]["max_allocated_memory_MB"])
+                filtered_bench_sorted = sorted(
+                    filtered_bench, key=lambda x: x["extra_info"].get("max_allocated_memory_MB", 0)
+                )
                 for bk in filtered_bench_sorted:
-                    print(f"{bk['name'].lstrip('test_')}: {bk['extra_info']['max_allocated_memory_MB']/1000} GB")
+                    print(f"{bk['name'].lstrip('test_')}: {bk['extra_info'].get('max_allocated_memory_MB', 0)/1000} GB")
                 print("\n")
 
             print_sorted_memory_info("forward")
             print_sorted_memory_info("backward")
+
+
+class FXGraphReport:
+    """
+    Encapsulates an FX Graph module with metadata to aid in generating
+    reproduction and benchmarking scripts for various executors.
+    """
+
+    def __init__(self, graph: torch.fx.GraphModule, graph_name: str):
+        self.graph = graph
+        self.graph_name = graph_name
+
+    def get_input_metadata(self):
+        placeholders = list(n for n in self.graph.graph.nodes if n.op == "placeholder")
+        example_input_metadata = map(partial(_get_example_inputs_from_placeholder, only_metadata=True), placeholders)
+        return list(example_input_metadata)
+
+    def write_eager_repro(self, folder, use_benchmark: bool = False, serialize_inputs: bool = False):
+        if use_benchmark:
+            self.write_benchmark_repro(
+                folder,
+                f"{self.graph_name}_benchmark_eager.py",
+                ["eager"],
+                executor_str=["None"],
+                serialize_inputs=serialize_inputs,
+            )
+        else:
+            self.write_repro(
+                folder,
+                f"{self.graph_name}_repro_eager.py",
+                executor_str="None",
+                serialize_inputs=serialize_inputs,
+            )
+
+    def write_thunder_repro(self, folder, use_benchmark: bool = False, serialize_inputs: bool = False):
+        thunder_compile_str = "thunder.jit"
+        thunder_import_str = ["import thunder"]
+        if use_benchmark:
+            self.write_benchmark_repro(
+                folder,
+                f"{self.graph_name}_benchmark_thunder.py",
+                ["thunder"],
+                [thunder_compile_str],
+                thunder_import_str,
+            )
+        else:
+            self.write_repro(
+                folder,
+                f"{self.graph_name}_repro_thunder.py",
+                thunder_compile_str,
+                thunder_import_str,
+                serialize_inputs,
+            )
+
+    def write_inductor_repro(self, folder, use_benchmark: bool = False, serialize_inputs: bool = False):
+        inductor_compile_str = "torch.compile"
+        inductor_import_str = ["import torch"]
+        if use_benchmark:
+            self.write_benchmark_repro(
+                folder,
+                f"{self.graph_name}_benchmark_torchcompile.py",
+                ["torchcompile"],
+                [inductor_compile_str],
+                inductor_import_str,
+                serialize_inputs,
+            )
+        else:
+            self.write_repro(
+                folder,
+                f"{self.graph_name}_repro_torchcompile.py",
+                inductor_compile_str,
+                inductor_import_str,
+                serialize_inputs,
+            )
+
+    def _get_input_str(self, folder, inputs, serialize_inputs):
+        input_str = ""
+        if any(arg is None for arg in inputs):
+            input_str += f"# Warning: The inputs that cannot be inferred are set to None, requiring the user to manually give inputs according to the code\n"
+        if serialize_inputs:
+            example_inputs = eval(f"[\n{chr(10).join(arg_like(a) for a in inputs)}]")
+            input_file_name = folder / f"{self.graph_name}_inputs.pt"
+            torch.save(example_inputs, input_file_name)
+            input_str += f"inputs = torch.load('{input_file_name}')\n"
+        else:
+            input_str = "inputs = [\n"
+            input_str += textwrap.indent("\n".join(arg_like(a) for a in inputs), "    ")
+            input_str += "\n]"
+        return input_str
+
+    def write_benchmark_repro(
+        self,
+        folder: str | PathLike,
+        file_name: str,
+        executor_name_str: list[str],
+        executor_str: list[str],
+        import_str: list[str] = None,
+        serialize_inputs: bool = False,
+        inputs: Sequence[torch.Tensor | ExampleInputMetaData] = None,
+        **kwargs,
+    ) -> None:
+        """
+        Generates a benchmark reproduction script for a given FX graph module using various executors and writes it to the specified file.
+
+        Args:
+            folder (str | PathLike): The target directory where the script will be saved.
+            file_name (str): The name of the output script file.
+            executor_name_str (list[str]): A list of executor names to be used in the script.
+            executor_str (list[str]): A list of compilation commands to run the benchmark.
+                Each command should be applicable to the original model, e.g.,
+                ``compile_command(original_model)``.
+                Example values: ``"thunder.jit"`` or ``"partial(thunder.jit, executors=[nvfuser_executor])"``.
+            import_str (list[str], optional): A list of necessary import statements for the script.
+                For example, if using the ``nvfuser_executor``, include:
+                ``import_str=["from thunder import nvfuser_executor"]``.
+            serialize_inputs (bool, optional): Whether to serialize the inputs for reproducibility. Defaults to False.
+                If enabled, all inputs will be serialized into a single file: "{graph_name}_input.pt".
+            inputs (Sequence[torch.Tensor | ExampleInputMetaData], optional):
+                The input tensors or metadata for the FX graph module. Defaults to None.
+                If not provided, inputs will be inferred from the placeholders in the FX graph.
+            **kwargs: Additional arguments for customization.
+
+        Example:
+            See the example in :func:`fx_report`.
+        """
+
+        folder = Path(folder)
+        folder.mkdir(exist_ok=True, parents=True)
+        if inputs == None:
+            inputs = self.get_input_metadata()
+        has_cuda_args = any(hasattr(arg, "device") and arg.device.type == "cuda" for arg in inputs)
+        has_requires_grad_args = any(hasattr(arg, "requires_grad") and arg.requires_grad for arg in inputs)
+        torch_env, thunder_pkgs = get_env()
+        readable = textwrap.indent(_readable(self.graph, "DynamoModule", print_output=False), "    ")
+        # The packages that are likely to be used by the code generated from the Torch GraphModule
+        torch_import_str = "\n".join([v.import_str for v in torch.fx.graph._custom_builtins.values()])
+        import_str = "" if import_str == None else "\n".join(import_str)
+        input_str = textwrap.indent(self._get_input_str(folder, inputs, serialize_inputs), "    ")
+        call_bench_str = f"benchmark_for_compute_type(compute_type, benchmark, compiled_model, inputs, {{}}, has_cuda={True if has_cuda_args else False})"
+        compute_type_decorator = (
+            "@parametrize_compute_type_only_training"
+            if has_requires_grad_args
+            else """@pytest.mark.parametrize("compute_type", (ComputeType.INFERENCE,), ids=("inference",))"""
+        )
+
+        executor_names_str = f"executor_names={executor_name_str}"
+        executors_str = "executors=[\n    " + ",\n    ".join(executor_str) + "\n]"
+        extra_comment_str = kwargs.get("extra_comment_str") if "extra_comment_str" in kwargs else ""
+        code_str = benchmark_multi_exe_code_template.format(
+            torch_env=torch_env,
+            thunder_pkgs=thunder_pkgs,
+            torch_import_str=torch_import_str,
+            import_str=import_str,
+            dynamo_module=readable,
+            inputs=input_str,
+            executors=executors_str,
+            graph_name=self.graph_name,
+            call_benchmark=call_bench_str,
+            executor_names=executor_names_str,
+            compute_type_decorator=compute_type_decorator,
+            extra_comment_str=extra_comment_str,
+        )
+        with open(folder / file_name, "w") as f:
+            print(code_str, file=f)
+
+    def write_repro(
+        self,
+        folder: str | PathLike,
+        file_name: str,
+        executor_str: str,
+        import_str: list[str] = None,
+        serialize_inputs: bool = False,
+        inputs: Sequence[torch.Tensor | ExampleInputMetaData] = None,
+        **kwargs,
+    ) -> None:
+        """
+        Generates a reproduction script for a given FX graph module and writes it to a file.
+
+        Args:
+            folder (str | PathLike): The target directory where the script will be saved.
+            file_name (str): The name of the output script file.
+            executor_str (str): compilation commands to compile the graph. e.g.: ``"partial(thunder.jit, executors=[nvfuser_executor])"``
+            import_str (list[str], optional): A list of necessary import statements for the script.
+                For example, if using the ``nvfuser_executor``, include:
+                ``import_str=["from thunder import nvfuser_executor"]``.
+            serialize_inputs (bool, optional): Whether to serialize the inputs for reproducibility.
+                Defaults to False. If enabled, all inputs will be saved to a file named
+                "{graph_name}_input.pt".
+            inputs (Sequence[torch.Tensor | ExampleInputMetaData], optional):
+                The input tensors or metadata for the FX graph module. Defaults to None, in
+                which case the inputs will be inferred from the placeholders in the FX graph.
+            **kwargs: Additional arguments for customization.
+
+        Example:
+            See the example in :func:`fx_report`.
+        """
+        folder = Path(folder)
+        folder.mkdir(exist_ok=True, parents=True)
+        if inputs == None:
+            inputs = self.get_input_metadata()
+        torch_env, thunder_pkgs = get_env()
+        readable = textwrap.indent(_readable(self.graph, "DynamoModule", print_output=False), "    ")
+        # The packages that are likely to be used by the code generated from the Torch GraphModule
+        torch_import_str = "\n".join([v.import_str for v in torch.fx.graph._custom_builtins.values()])
+        import_str = "" if import_str == None else "\n".join(import_str)
+        input_str = textwrap.indent(self._get_input_str(folder, inputs, serialize_inputs), "    ")
+        extra_comment_str = kwargs.get("extra_comment_str") if "extra_comment_str" in kwargs else ""
+
+        code_str = repro_code_template.format(
+            torch_env=torch_env,
+            thunder_pkgs=thunder_pkgs,
+            torch_import_str=torch_import_str,
+            import_str=import_str,
+            dynamo_module=readable,
+            inputs=input_str,
+            executor_str=executor_str,
+            graph_name=self.graph_name,
+            extra_comment_str=extra_comment_str,
+        )
+
+        with open(folder / file_name, "w") as f:
+            print(code_str, file=f)
+
+
+class FXReport:
+    """
+    This class stores a list of FXGraphReport instances, each of which wraps an FX graph
+    module and provides methods to generate reproduction and benchmark scripts.
+    """
+
+    def __init__(self, graphs: list[torch.fx.GraphModule], graph_names: list[str] = None):
+        if graph_names is None:
+            graph_names = [f"graph{idx}" for idx in range(len(graphs))]
+        self.fx_graph_reports: list[FXGraphReport] = [FXGraphReport(g, name) for name, g in zip(graph_names, graphs)]
+
+    def __str__(self):
+        return f"<FXReport with {len(self.fx_graph_reports)} FXGraphReports accessible via .fx_graph_reports>"
+
+
+def fx_report(fn: Callable, *args, compile_options: dict = None, **kwargs) -> FXReport:
+    """
+    This function compiles a given function using :func:`torch.compile` with specified ``compile_options``,
+    and applies a custom backend that intercepts and collects FX graph modules during execution.
+    The function then returns an :class:`FXReport`, which contains utilities to generate
+    reproduction and benchmark scripts for each collected FX graph module.
+
+    Args:
+        fn (Callable): The function to be compiled and analyzed.
+        *args: Positional arguments of ``fn``.
+        compile_options(dict): the options that are passed to :func:`torch.compile`
+        **kwargs: Keyword arguments of ``fn``.
+
+    Example:
+        .. code-block:: python
+
+        import tempfile
+
+        import torch
+        from thunder.dynamo.report import fx_report
+
+        def model(x):
+            return x * 2
+
+        report = fx_report(model, torch.randn((2, 2), requires_grad=True, device="cuda"), compile_options={"dynamic": False})
+        print(len(report.fx_graph_reports))
+        with tempfile.TemporaryDirectory() as tmpdir:
+            for graph_report in report.fx_graph_reports:
+                graph_report.write_eager_repro(tmpdir)
+                graph_report.write_thunder_repro(tmpdir, use_benchmark=True)
+                graph_report.write_inductor_repro(tmpdir)
+
+                # Executes using the user-defined executor.
+                my_executor = "partial(thunder.jit, transforms=[NvtxProfileTransform()], executors=[nvfuser_executor])"
+                my_imports = [
+                    "import thunder",
+                    "from thunder.dev_utils.nvtx_profile_transform import NvtxProfileTransform",
+                    "from thunder import nvfuser_executor",
+                    "from functools import partial",
+                ]
+                graph_report.write_repro(
+                    tmpdir, f"{graph_report.graph_name}_mythunder_repro.py", executor_str=my_executor, import_str=my_imports
+                )
+                graph_report.write_benchmark_repro(
+                    tmpdir,
+                    f"{graph_report.graph_name}_mythunder_benchmark.py",
+                    executor_name_str=["mythunder"],
+                    executor_str=[my_executor],
+                    import_str=my_imports,
+                )
+    """
+    graphs = []
+
+    def helper_backend(gm, example_inputs):
+        """Helper function to collect FX graphs."""
+        graphs.append(gm)
+        return gm.forward
+
+    if compile_options is None:
+        compile_options = {}
+    compiled = torch.compile(fn, **compile_options, backend=helper_backend)
+    compiled(*args, **kwargs)
+
+    return FXReport(graphs)
diff --git a/thunder/dynamo/repro_script_template.py b/thunder/dynamo/repro_script_template.py
new file mode 100644
index 00000000..e29271c3
--- /dev/null
+++ b/thunder/dynamo/repro_script_template.py
@@ -0,0 +1,115 @@
+repro_code_template = '''
+"""
+Environment information get from `torch.utils.collect_env.get_pretty_env_info()`:
+{torch_env}
+
+Versions of Thunder related libraries:
+{thunder_pkgs}
+
+{extra_comment_str}
+"""
+{torch_import_str}
+{import_str}
+import argparse
+
+parser = argparse.ArgumentParser(description="Script for executing an FX graph with specified configurations.")
+
+parser.add_argument(
+    "--check_consistency",
+    type=bool,
+    default=False,
+    help="Whether to check consistency (default: False)"
+)
+parser.add_argument(
+    "--compute_type",
+    type=str,
+    choices=["forward", "forward+backward"],
+    default="forward",
+    help="Type of computation to perform (forward, forward+backward)"
+)
+args = parser.parse_args()
+compute_type = args.compute_type
+check_acc = args.check_consistency
+
+def test_{graph_name}():
+{dynamo_module}
+
+{inputs}
+
+    model = DynamoModule()
+    from thunder.dynamo.report import run_repro
+
+    executor={executor_str}
+    if executor == None:
+        compiled_model = model
+    else:
+        compiled_model = executor(model)
+    result = run_repro(compiled_model, compute_type, *inputs)
+    if check_acc:
+        eager_result = run_repro(model, compute_type, *inputs)
+        for (compute_t, eager_v), (_, cur_v) in zip(eager_result.items(), result.items()):
+            torch.testing.assert_close(eager_v, cur_v, msg=lambda e : f'{{compute_t}}: {{e}}')
+
+
+test_{graph_name}()
+'''
+
+
+benchmark_multi_exe_code_template = '''
+"""
+Environment information get from `torch.utils.collect_env.get_pretty_env_info()`:
+{torch_env}
+
+Versions of Thunder related libraries:
+{thunder_pkgs}
+
+{extra_comment_str}
+"""
+# NOTE: This script requires `pytest-benchmark==4.0.0` to be installed.
+# To execute the script, run `pytest {graph_name}_benchmark.py --benchmark-timer=torch.utils.benchmark.utils.timer.timer --benchmark-warmup=on --benchmark-group-by=param:compute_type`
+# To check the peak allocated CUDA memory, use --benchmark-json=json_file_name and look at the "max_allocated_memory_MB" field in the json file
+# To run tests for a specific compute_type, use the pytest `-k` option.
+# For example:
+#   - `-k "forward"` will run only the forward pass.
+#
+# Available options:
+#   - compute_type: "forward", "backward"
+
+import pytest
+from thunder.benchmarks.targets import parametrize_compute_type_only_training, benchmark_for_compute_type, ComputeType
+{torch_import_str}
+{import_str}
+
+# NOTE: The reproducer function has already been processed by TorchDynamo.
+# If we let it go through TorchDynamo again, it could be segmented further.
+# To avoid this, we directly use Inductor here.
+# See issue https://github.com/Lightning-AI/lightning-thunder/issues/1521
+def torch_inductor(fn, inputs):
+    from torch._inductor import compile as inductor_compile
+    from torch.fx import symbolic_trace
+
+    fx_graph = symbolic_trace(fn)
+    return inductor_compile(fx_graph, inputs)
+
+{executors}
+{executor_names}
+@pytest.mark.parametrize(
+    "executor",
+    executors,
+    ids=executor_names,
+)
+{compute_type_decorator}
+def test_{graph_name}(benchmark, executor, compute_type):
+{dynamo_module}
+
+{inputs}
+
+    model = DynamoModule()
+    if executor == None:
+        compiled_model = model
+    elif executor == torch_inductor:
+        compiled_model = executor(model, inputs)
+    else:
+        compiled_model = executor(model)
+    {call_benchmark}
+'''
diff --git a/thunder/dynamo/utils.py b/thunder/dynamo/utils.py
index fc827d34..ef2cc08b 100644
--- a/thunder/dynamo/utils.py
+++ b/thunder/dynamo/utils.py
@@ -666,7 +666,7 @@ def _readable(
     module_code = verbose_python_code.src
     module_code = module_code.lstrip("\n")
     module_code = f"class {module_name}(torch.nn.Module):\n" + module_code
-    module_code = _addindent(module_code, 2)
+    module_code = _addindent(module_code, 4)
 
     submodule_code_list = [""]
     for submodule_name, submodule in module.named_children():
@@ -729,178 +729,3 @@ def get_split_reasons_string(subgraph_info: SubgraphInfo) -> str:
     else:
         split_reason_str += "The original graph is not split, and is entirely run by Thunder.\n"
     return split_reason_str
-
-
-def reproducer(
-    gm: torch.fx.GraphModule,
-    subgraph_info: SubgraphInfo,
-    thunder_options: dict,
-    args: tuple[torch.Tensor | ExampleInputMetaData],
-    folder: str | os.PathLike,
-    graph_name: str,
-    use_pytest_benchmark: bool = False,
-    save_input_tensor: bool = False,
-):
-    folder = Path(folder)
-    folder.mkdir(exist_ok=True)
-    torch_env, thunder_pkgs = get_env()
-    # Ideally we'd use print_readable, but we want verbose=False and there's no
-    # way to set that with print_readable.
-    readable = _readable(gm, "DynamoModule", print_output=False)
-    has_cuda_args = any(hasattr(arg, "device") and arg.device.type == "cuda" for arg in args)
-    thunder_options_str = thunder_options_to_str(thunder_options)
-    TORCH_INDUCTOR_FUNCTION_STR = """
-# NOTE: The reproducer function has already been processed by TorchDynamo.
-# If we let it go through TorchDynamo again, it could be segmented further.
-# To avoid this, we directly use Inductor here.
-# See issue https://github.com/Lightning-AI/lightning-thunder/issues/1521
-def torch_inductor(fn, inputs):
-    from torch._inductor import compile as inductor_compile
-    from torch.fx import symbolic_trace
-
-    fx_graph = symbolic_trace(fn)
-    return inductor_compile(fx_graph, inputs)
-"""
-    EXECUTOR_DICT_CODE_STR = f"""{TORCH_INDUCTOR_FUNCTION_STR}
-
-bench_executors_dict = {{}}
-bench_executors_dict["thunder"]=partial(thunder.jit, {thunder_options_str})
-bench_executors_dict["torch_inductor"]=torch_inductor
-bench_executors_dict["eager"]=None
-"""
-    if has_cuda_args:
-        EXECUTOR_DICT_CODE_STR = f"""{EXECUTOR_DICT_CODE_STR}
-bench_executors_dict["thunder_cugraph"]=partial(thunder.jit, transform=CUDAGraphTransform())
-"""
-
-    COMMAND_LINE_ARGS = """
-import argparse
-
-parser = argparse.ArgumentParser(description="Script for executing an FX graph with specified configurations.")
-
-parser.add_argument(
-    "--check_consistency",
-    type=bool,
-    default=False,
-    help="Whether to check consistency (default: False)"
-)
-parser.add_argument(
-    "--compute_type",
-    type=str,
-    choices=["forward", "forward+backward"],
-    default="forward",
-    help="Type of computation to perform (forward, forward+backward)"
-)
-parser.add_argument(
-    "--executor",
-    type=str,
-    choices=["torch_inductor", "thunder", "eager"],
-    default="thunder",
-    help="Type of executors (thunder, torch_inductor, or eager)"
-)
-
-args = parser.parse_args()
-ex_name = args.executor
-compute_type = args.compute_type
-check_acc = args.check_consistency
-"""
-
-    # split reason
-    split_reason_str = get_split_reasons_string(subgraph_info)
-
-    comment_str = f'''"""
-Environment information get from `torch.utils.collect_env.get_pretty_env_info()`:
-{torch_env}
-
-Versions of Thunder related libraries:
-{thunder_pkgs}
-
-{split_reason_str}"""
-'''
-    del split_reason_str, torch_env, thunder_pkgs
-
-    code_str = ""
-    if use_pytest_benchmark:
-        code_str += f"""# NOTE: This script requires `pytest-benchmark==4.0.0` to be installed.
-# To execute the script, run `pytest {graph_name}.py --benchmark-timer=torch.utils.benchmark.utils.timer.timer --benchmark-warmup=on --benchmark-group-by=param:compute_type`
-# To check the peak allocated CUDA memory, use --benchmark-json=json_file_name and look at the "max_allocated_memory_MB" field in the json file
-# To run tests for a specific compute_type or executor, use the pytest `-k` option.
-# For example:
-#   - `-k "thunder and forward"` will run only the forward pass with the Thunder executor.
-#
-# Available options:
-#   - compute_type: "forward", "backward"
-#   - executor: "eager", "thunder", "torch_inductor", "thunder_cugraph" (requires CUDA for thunder_cugraph)
-
-import pytest
-from thunder.benchmarks.targets import parametrize_compute_type_only_training, benchmark_for_compute_type
-"""
-    # The packages that are likely to be used by the code generated from the Torch GraphModule
-    code_str += "\n".join([v.import_str for v in torch.fx.graph._custom_builtins.values()])
-    code_str += f"\nfrom functools import partial\nimport thunder\n"
-    if has_cuda_args:
-        code_str += "from thunder.transforms.cudagraph import CUDAGraphTransform\n"
-        code_str += "from thunder.dev_utils.nvtx_profile_transform import NvtxProfileTransform\n"
-
-    code_str += f"{EXECUTOR_DICT_CODE_STR}"
-    if use_pytest_benchmark:
-        code_str += f"""
-executors = list(bench_executors_dict.values())
-executor_ids = list(bench_executors_dict.keys())
-
-@pytest.mark.parametrize(
-"executor,",
-executors,
-ids=executor_ids,
-)
-@parametrize_compute_type_only_training"""
-        func_str = f"def test_{graph_name}(benchmark, executor, compute_type):\n{readable}\n"
-    else:
-        code_str += f"\n{COMMAND_LINE_ARGS}"
-        func_str = f"def test_{graph_name}():\n{readable}\n"
-
-    if any(arg is None for arg in args):
-        func_str += f"# Warning: The inputs that cannot be inferred are set to None, requiring the user to manually give inputs according to the code\n"
-    if save_input_tensor:
-        example_inputs = eval(f"[\n{chr(10).join(arg_like(a) for a in args)}]")
-        input_file_name = folder / f"{graph_name}_inputs.pt"
-        torch.save(example_inputs, input_file_name)
-        func_str += f"inputs = torch.load('{input_file_name}')\n"
-    else:
-        input_str = f"""inputs = [\n{chr(10).join(arg_like(a) for a in args)}\n"""
-        func_str += f"{_addindent(input_str, 4)}\n]\n"
-        del input_str
-
-    if not use_pytest_benchmark:
-        func_str += f"""
-mod = DynamoModule()
-from thunder.dynamo.report import run_repro
-
-result = run_repro(bench_executors_dict, ex_name, mod, compute_type, *inputs)
-if check_acc:
-    eager_result = run_repro(bench_executors_dict, "eager", mod, compute_type, *inputs)
-    for (compute_t, eager_v), (_, cur_v) in zip(eager_result.items(), result.items()):
-        torch.testing.assert_close(eager_v, cur_v, msg=lambda e : f'{{compute_t}}: {{e}}')
-"""
-    else:
-        func_str = f"""{func_str}
-mod = DynamoModule()
-if executor == None:
-    compiled = mod
-elif executor == torch_inductor:
-    compiled = executor(mod, inputs)
-else:
-    compiled = executor(mod)
-"""
-        if not has_cuda_args:
-            func_str += f"""benchmark(compiled, *inputs)"""
-        else:
-            func_str += f"""benchmark_for_compute_type(compute_type, benchmark, compiled, inputs, {{}})"""
-
-    with open(folder / f"{graph_name}.py", "w") as f:
-        print(comment_str, file=f)
-        print(code_str, file=f)
-        print(_addindent(func_str, 4), file=f)
-
-        if not use_pytest_benchmark:
-            print(f"\ntest_{graph_name}()", file=f)


Test patch:
diff --git a/thunder/tests/test_dynamo.py b/thunder/tests/test_dynamo.py
index 34eb5e92..b0f9c1d8 100644
--- a/thunder/tests/test_dynamo.py
+++ b/thunder/tests/test_dynamo.py
@@ -841,7 +841,6 @@ def test_dynamo_reproducer_2graph(executor, device: str, dtype: dtypes.dtype, us
         cfunc = thunderfx(
             func,
             transforms=[
-                NvtxProfileTransform(),
                 CUDAGraphTransform(),
             ],
             executors=[nvfuser_executor],
@@ -857,8 +856,9 @@ def test_dynamo_reproducer_2graph(executor, device: str, dtype: dtypes.dtype, us
     out = cfunc(x)
     cfunc._backend.save_reproducer_to_folder(tmp_path, use_pytest_benchmark=use_pytest_benchmark)
 
-    s1 = f"{tmp_path}/graph0_thunder_0.py"
-    s2 = f"{tmp_path}/graph1_thunder_0.py"
+    suffix = "_benchmark" if use_pytest_benchmark else "_repro"
+    s1 = f"{tmp_path}/graph0_thunder_0{suffix}.py"
+    s2 = f"{tmp_path}/graph1_thunder_0{suffix}.py"
     assert os.path.exists(s1)
     assert os.path.exists(s2)
     cmd = [sys.executable]
@@ -866,8 +866,8 @@ def test_dynamo_reproducer_2graph(executor, device: str, dtype: dtypes.dtype, us
         cmd = cmd + ["-m", "pytest"]
     cmd1 = cmd + [s1]
     cmd2 = cmd + [s2]
-    result1 = subprocess.run(cmd1, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
-    result2 = subprocess.run(cmd2, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+    result1 = subprocess.run(cmd1, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+    result2 = subprocess.run(cmd2, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
 
     assert result1.returncode == 0, f"Reproducer {s1} failed: {result1}"
     assert result2.returncode == 0, f"Reproducer {s2} failed: {result2}"
@@ -896,7 +896,8 @@ def test_dynamo_reproducer_submodules(use_pytest_benchmark, tmp_path):
     out = jf(x)
     jf._backend.save_reproducer_to_folder(tmp_path, use_pytest_benchmark=use_pytest_benchmark)
 
-    s1 = f"{tmp_path}/graph0_thunder_0.py"
+    suffix = "_benchmark" if use_pytest_benchmark else "_repro"
+    s1 = f"{tmp_path}/graph0_thunder_0{suffix}.py"
     assert os.path.exists(s1)
     cmd = [sys.executable]
     if use_pytest_benchmark:
@@ -962,9 +963,10 @@ def test_dynamo_reproducer_split(executor, device: str, dtype: dtypes.dtype, use
         result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
         assert result.returncode == 0, f"Reproducer {file_name} failed: {result}"
 
-    s1 = f"{tmp_path}/graph0_thunder_0.py"
-    s2 = f"{tmp_path}/graph0_thunder_2.py"
-    s3 = f"{tmp_path}/graph0_thunder_4.py"
+    suffix = "_benchmark" if use_pytest_benchmark else "_repro"
+    s1 = f"{tmp_path}/graph0_thunder_0{suffix}.py"
+    s2 = f"{tmp_path}/graph0_thunder_2{suffix}.py"
+    s3 = f"{tmp_path}/graph0_thunder_4{suffix}.py"
     cmd = [sys.executable]
     if use_pytest_benchmark:
         cmd = cmd + ["-m", "pytest"]
@@ -1059,7 +1061,7 @@ def test_report(tmp_path, capsys):
     assert "Verifying consistency" in msg
     assert "Analyzing performance through benchmarking" in msg
     assert "The input callable can be successfully executed by ThunderFX." in msg
-    assert "Max allocated memory usage:" in msg
+    assert "Max allocated CUDA memory usage:" in msg
 
     with patch("torch.compile", side_effect=Exception("compilation raises exception")):
         thunderfx_report(foo, x, folder_path=tmp_path, check_consistency=False)
@@ -1067,3 +1069,57 @@ def test_report(tmp_path, capsys):
         assert not captured.err
         assert "Failed to run the function using ThunderFX" in captured.out
         assert "Failed to save reproducer" in captured.out
+
+
+@pytest.mark.parametrize("use_benchmark", (True, False), ids=("benchmark", "repro"))
+@instantiate(
+    dtypes=NOTHING,
+    executors=[DynamoThunderExecutor],
+    decorators=(
+        pytest.mark.parametrize("use_benchmark", (True, False), ids=("benchmark", "repro")),
+        pytest.mark.xfail(
+            condition=IS_WINDOWS,
+            strict=True,
+            reason="torch.compile Windows support is still WIP - https://github.com/pytorch/pytorch/issues/122094",
+        ),
+    ),
+)
+def test_fxreport(executor, device: str, dtype: dtypes.dtype, use_benchmark, tmp_path):
+    from thunder.dynamo.report import fx_report
+
+    def foo(x, y):
+        return x + y
+
+    x = torch.randn(4, 4, device=device, requires_grad=True)
+    y = torch.randn(4, 4, device=device, requires_grad=True)
+    results = fx_report(foo, x, y, compile_options={"dynamic": True})
+    for r in results.fx_graph_reports:
+        r.write_eager_repro(tmp_path, use_benchmark=use_benchmark)
+        r.write_thunder_repro(tmp_path, use_benchmark=use_benchmark)
+        r.write_inductor_repro(tmp_path, use_benchmark=use_benchmark)
+        my_exe = "partial(thunder.jit, executors=[pytorch_executor])"
+        my_imports = [
+            "import thunder",
+            "from thunder import pytorch_executor",
+            "from functools import partial",
+        ]
+        if not use_benchmark:
+            r.write_repro(tmp_path, f"{r.graph_name}_mythunder_repro.py", executor_str=my_exe, import_str=my_imports)
+        else:
+            r.write_benchmark_repro(
+                tmp_path, f"{r.graph_name}_mythunder_benchmark.py", ["mythunder"], [my_exe], my_imports
+            )
+
+    def check(file_name, cmd):
+        cmd = cmd + [file_name]
+        result = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+        assert result.returncode == 0, f"Script {file_name} failed: {result}"
+
+    cmd = [sys.executable]
+    if use_benchmark:
+        cmd = cmd + ["-m", "pytest"]
+    py_files = list(tmp_path.glob("*.py"))
+    assert len(py_files) == 4
+
+    for file in py_files:
+        check(tmp_path / file, cmd)