Repository: tenstorrent/tt-torch
Pull request #749: Add Stable Diffusion 3.5 #332
Author: login: ddilbazTT
Created: 2025-05-09T18:41:52Z | Merged: 2025-05-17T23:36:58Z

### Ticket
#332 

### Problem description
StableDiffusion (SD) 3.5 is a big model. The model uses SD3Transformer2DModel. I added two different tests - one that simply uses SD3Transformer2DModel and one that runs SD3.5. To get SD3.5 working, I made some model level optimizations and included memory cleanup in `pass_pipeline`. Even then, the
`tests/models/stable_diffusion/test_stable_diffusion_3_5.py` test can only run the medium variant op-by-op. It cannot run compile/ execute, and it cannot run the large variant.

Also added licensing material.

### What's changed
Added 2 test files for SD3.5 - one using the backbone transformer and one for the complete model. 
Edited workflow files to run SD3.5
Added some memory cleanup to backend



Diff:
diff --git a/.github/workflows/run-e2e-compile-tests.yml b/.github/workflows/run-e2e-compile-tests.yml
index dce39d3e7..b59bce290 100644
--- a/.github/workflows/run-e2e-compile-tests.yml
+++ b/.github/workflows/run-e2e-compile-tests.yml
@@ -53,7 +53,7 @@ jobs:
                   tests/models/mistral/test_mistral.py::test_mistral[full-mistral7b-eval]
                   tests/models/mistral/test_mistral.py::test_mistral[full-ministral8b-eval]
             "
-          }
+          },
         ]
     runs-on:
       - ${{ matrix.build.runs-on }}
diff --git a/.github/workflows/run-full-model-execution-tests-nightly.yml b/.github/workflows/run-full-model-execution-tests-nightly.yml
index e55dc3065..1fca01c8e 100644
--- a/.github/workflows/run-full-model-execution-tests-nightly.yml
+++ b/.github/workflows/run-full-model-execution-tests-nightly.yml
@@ -115,6 +115,7 @@ jobs:
             # Approximately 40 minutes.
             runs-on: wormhole_b0, name: "eval_6", tests: "
                   tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval]
+                  tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[full-SD3.5-medium-transformer-eval]
             "
           },
           {
@@ -161,6 +162,9 @@ jobs:
                 tests/models/falcon/test_falcon3.py::test_falcon[full-tiiuae/Falcon3-7B-Base-eval]
                 tests/models/falcon/test_falcon3.py::test_falcon[full-tiiuae/Falcon3-10B-Base-eval]
                 tests/models/vovnet/test_vovnet_onnx.py::test_vovnet_onnx[full-eval]
+                tests/models/stable_diffusion/test_stable_diffusion_3_5.py::test_stable_diffusion_3_5[full-SD3.5-medium-eval]
+                tests/models/stable_diffusion/test_stable_diffusion_3_5.py::test_stable_diffusion_3_5[full-SD3.5-large-eval]
+                tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[full-SD3.5-large-transformer-eval]
             "
           },
         ]
diff --git a/.github/workflows/run-op-by-op-model-tests-nightly.yml b/.github/workflows/run-op-by-op-model-tests-nightly.yml
index 6e5a76c8e..915b97353 100644
--- a/.github/workflows/run-op-by-op-model-tests-nightly.yml
+++ b/.github/workflows/run-op-by-op-model-tests-nightly.yml
@@ -163,6 +163,13 @@ jobs:
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_schnell-eval]
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_dev-eval]
               "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
+              tests/models/stable_diffusion/test_stable_diffusion_3_5.py::test_stable_diffusion_3_5[op_by_op_torch-SD3.5-medium-eval]
+              tests/models/stable_diffusion/test_stable_diffusion_3_5.py::test_stable_diffusion_3_5[op_by_op_torch-SD3.5-large-eval]
+              tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[op_by_op_torch-SD3.5-large-transformer-eval]
+              "
           }
         ]
     runs-on:
diff --git a/.github/workflows/run-op-by-op-model-tests-weekly.yml b/.github/workflows/run-op-by-op-model-tests-weekly.yml
index d0a493e93..252b5c820 100644
--- a/.github/workflows/run-op-by-op-model-tests-weekly.yml
+++ b/.github/workflows/run-op-by-op-model-tests-weekly.yml
@@ -226,6 +226,11 @@ jobs:
             runs-on: wormhole_b0, name: "falcon3", tests: "
               tests/models/falcon/test_falcon3.py::test_falcon[op_by_op_torch-tiiuae/Falcon3-1B-Base-eval]
               "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-3.5", tests: "
+              tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[op_by_op_torch-SD3.5-medium-transformer-eval]
+              "
           }
         ]
     runs-on:
diff --git a/tests/models/stable_diffusion/test_stable_diffusion_3_5.py b/tests/models/stable_diffusion/test_stable_diffusion_3_5.py
new file mode 100644
index 000000000..29c02cc4d
--- /dev/null
+++ b/tests/models/stable_diffusion/test_stable_diffusion_3_5.py
@@ -0,0 +1,105 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import StableDiffusion3Pipeline, AutoencoderTiny
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_dict = dict(model_info_list)
+        model_path = model_dict[self.model_name]
+        pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_path,
+            text_encoder_3=None,
+            tokenizer_3=None,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        # memory optimization recommended by: https://huggingface.co/docs/diffusers/en/api/pipelines/stable_diffusion/stable_diffusion_3#tiny-autoencoder-for-stable-diffusion-3
+        pipe.vae = AutoencoderTiny.from_pretrained(
+            "madebyollin/taesd3", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True
+        )
+        pipe.enable_attention_slicing()
+        return pipe
+
+    def _load_inputs(self):
+        prompt = [
+            "a photo of an astronaut riding a horse on mars",
+        ]
+
+        negative_prompt = ""
+        height = 512
+        width = 512
+        guidance_scale = 7.0
+        arguments = {
+            "prompt": prompt,
+            "negative_prompt": negative_prompt,
+            "height": height,
+            "width": width,
+            "guidance_scale": guidance_scale,
+        }
+
+        return arguments
+
+
+model_info_list = [
+    ("SD3.5-medium", "stabilityai/stable-diffusion-3.5-medium"),
+    ("SD3.5-large", "stabilityai/stable-diffusion-3.5-large"),
+]
+
+
+@pytest.mark.parametrize(
+    "mode",
+    ["eval"],
+)
+@pytest.mark.parametrize(
+    "model_info",
+    model_info_list,
+    ids=[model_info[0] for model_info in model_info_list],
+)
+@pytest.mark.parametrize(
+    "op_by_op",
+    [OpByOpBackend.STABLEHLO, OpByOpBackend.TORCH, None],
+    ids=["op_by_op_stablehlo", "op_by_op_torch", "full"],
+)
+def test_stable_diffusion_3_5(record_property, model_info, mode, op_by_op):
+    model_group = "red"
+    model_name, model_path = model_info
+
+    cc = CompilerConfig()
+    cc.enable_consteval = True
+    # consteval_parameters is disabled because it results in a memory related crash
+    if op_by_op:
+        cc.compile_depth = CompileDepth.EXECUTE_OP_BY_OP
+        if op_by_op == OpByOpBackend.STABLEHLO:
+            cc.op_by_op_backend = OpByOpBackend.STABLEHLO
+
+    skip_full_eval_test(
+        record_property,
+        cc,
+        model_name,
+        bringup_status="FAILED_RUNTIME",
+        reason="Cannot lower StableHLO --> TTIR : results/mlir_tests/stable_hlo/aten::round.decimals_0.mlir:3:10: error: failed to legalize operation 'stablehlo.round_nearest_even' - https://github.com/tenstorrent/tt-torch/issues/769",
+        model_group=model_group,
+    )
+
+    tester = ThisTester(
+        model_name,
+        mode,
+        compiler_config=cc,
+        record_property_handle=record_property,
+        assert_atol=False,
+        assert_pcc=False,
+        model_group="red",
+    )
+    results = tester.test_model()
+    if mode == "eval":
+        image = results.images[0]
+        image.save("generated_image.png")
+
+    tester.finalize()
diff --git a/tests/models/stable_diffusion/test_stable_diffusion_transformer.py b/tests/models/stable_diffusion/test_stable_diffusion_transformer.py
new file mode 100644
index 000000000..1a269a028
--- /dev/null
+++ b/tests/models/stable_diffusion/test_stable_diffusion_transformer.py
@@ -0,0 +1,128 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import (
+    StableDiffusion3Pipeline,
+    FlowMatchEulerDiscreteScheduler,
+    SD3Transformer2DModel,
+)
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_dict = dict(model_info_list)
+        model_path = model_dict[self.model_name]
+        self.pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_path,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        self.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
+            model_path, subfolder="scheduler"
+        )
+        self.transformer = SD3Transformer2DModel.from_pretrained(
+            model_path, subfolder="transformer", torch_dtype=torch.bfloat16
+        )
+        return self.transformer
+
+    def _load_inputs(self):
+        prompt = "A futuristic cityscape at sunset"
+        num_images_per_prompt = 1
+        height = 512
+        width = 512
+        (
+            prompt_embeds,
+            negative_prompt_embeds,
+            pooled_prompt_embeds,
+            negative_pooled_prompt_embeds,
+        ) = self.pipe.encode_prompt(
+            prompt=prompt,
+            prompt_2=prompt,  # Using the same prompt for all encoders for simplicity
+            prompt_3=prompt,
+            num_images_per_prompt=num_images_per_prompt,
+            do_classifier_free_guidance=False,  # For simplicity in this direct call
+        )
+
+        num_channels_latents = self.transformer.config.in_channels
+        latents = self.pipe.prepare_latents(
+            batch_size=num_images_per_prompt,
+            num_channels_latents=num_channels_latents,
+            height=height,
+            width=width,
+            dtype=prompt_embeds.dtype,
+            device=None,
+            generator=None,
+        )
+
+        self.scheduler.set_timesteps(28)
+        timestep = self.scheduler.timesteps[0].expand(latents.shape[0])
+        arguments = {
+            "hidden_states": latents,
+            "timestep": timestep,
+            "encoder_hidden_states": prompt_embeds,
+            "pooled_projections": pooled_prompt_embeds,
+            "joint_attention_kwargs": {},
+            "return_dict": False,
+        }
+        return arguments
+
+
+model_info_list = [
+    ("SD3.5-medium-transformer", "stabilityai/stable-diffusion-3.5-medium"),
+    ("SD3.5-large-transformer", "stabilityai/stable-diffusion-3.5-large"),
+]
+
+
+@pytest.mark.parametrize(
+    "mode",
+    ["eval"],
+)
+@pytest.mark.parametrize(
+    "model_info",
+    model_info_list,
+    ids=[model_info[0] for model_info in model_info_list],
+)
+@pytest.mark.parametrize(
+    "op_by_op",
+    [OpByOpBackend.STABLEHLO, OpByOpBackend.TORCH, None],
+    ids=["op_by_op_stablehlo", "op_by_op_torch", "full"],
+)
+def test_stable_diffusion_transformer(record_property, model_info, mode, op_by_op):
+    model_group = "red"
+    model_name, model_path = model_info
+
+    cc = CompilerConfig()
+    cc.enable_consteval = True
+    cc.consteval_parameters = True
+    if op_by_op:
+        cc.compile_depth = CompileDepth.EXECUTE_OP_BY_OP
+        if op_by_op == OpByOpBackend.STABLEHLO:
+            cc.op_by_op_backend = OpByOpBackend.STABLEHLO
+
+    skip_full_eval_test(
+        record_property,
+        cc,
+        model_name,
+        bringup_status="FAILED_RUNTIME",
+        reason="Model compilation exceeds CI timeout limit (240 minutes), compiles e2e locally. Model execution fails with `Not enough space to allocate 47316992 B DRAM buffer across 12 banks.`",
+        model_group=model_group,
+        model_name_filter="SD3.5-large-transformer",
+    )
+
+    tester = ThisTester(
+        model_name,
+        mode,
+        compiler_config=cc,
+        record_property_handle=record_property,
+        assert_atol=False,
+        assert_pcc=False,
+        model_group=model_group,
+    )
+    results = tester.test_model()
+
+    tester.finalize()
diff --git a/tt_torch/dynamo/passes.py b/tt_torch/dynamo/passes.py
index 64a78100c..c5e3666d9 100644
--- a/tt_torch/dynamo/passes.py
+++ b/tt_torch/dynamo/passes.py
@@ -2,6 +2,7 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 import torch
+import gc
 from torch.fx.experimental import const_fold
 from typing import List, Optional, Union
 from torch.export.graph_signature import InputKind
@@ -103,6 +104,7 @@ def bypass_dtype_promotion(gm, compiler_config):
 
 def constant_fold(gm):
     gm = const_fold.split_const_subgraphs(gm)
+    gc.collect()
     gm.run_folding()
 
     gm.graph.eliminate_dead_code()


Reviews:
user: login: kmabeeTT
body: Nice changes! Left some comments.

Do you have more details on the failure reasons here other than "cannot run" we can document in this PR to help save time in the future?

> Even then, the tests/models/stable_diffusion/test_stable_diffusion_3_5.py test can only run the medium variant op-by-op. It cannot run compile/ execute, and it cannot run the large variant.


My attempt to document the 4 variants added here and where they are added:

```
tests/models/stable_diffusion/test_stable_diffusion_transformer.py medium => full-model-execute
tests/models/stable_diffusion/test_stable_diffusion_transformer.py large => op-by-op
tests/models/stable_diffusion/test_stable_diffusion_3_5.py medium => op-by-op
tests/models/stable_diffusion/test_stable_diffusion_3_5.py large => not run anywhere
```
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T21:22:02Z
user: login: jameszianxuTT
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T21:34:51Z
user: login: jameszianxuTT
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-09T21:36:26Z
user: login: jameszianxuTT
body: Other than duplicate group naming, I think this looks fine. Thanks!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-09T21:36:57Z
user: login: ddilbazTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T21:47:27Z
user: login: ddilbazTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T21:47:44Z
user: login: ddilbazTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T21:48:18Z
user: login: ddilbazTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T21:51:26Z
user: login: ddilbazTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T21:52:20Z
user: login: kmabeeTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T22:38:18Z
user: login: kmabeeTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T22:48:01Z
user: login: kmabeeTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T22:48:59Z
user: login: AleksKnezevic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T00:07:23Z
user: login: AleksKnezevic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-13T01:35:29Z
user: login: AleksKnezevic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T15:53:36Z
user: login: AleksKnezevic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T15:56:45Z
user: login: AleksKnezevic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T16:00:22Z
user: login: AleksKnezevic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T17:11:51Z
user: login: ddilbazTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T18:36:40Z
user: login: AleksKnezevic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T19:14:49Z
user: login: AleksKnezevic
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T19:36:41Z
user: login: kmabeeTT
body: Hmm some unexpected changes here to comment out tests in run-e2e-compile-tests.yml?
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T20:01:36Z
user: login: kmabeeTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T20:34:35Z
user: login: kmabeeTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T20:35:52Z
user: login: kmabeeTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T20:39:45Z
user: login: ddilbazTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T20:50:11Z
user: login: kmabeeTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T21:09:04Z
user: login: kmabeeTT
body: Thank you! 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T21:18:54Z

Review comments:
diff_hunk: @@ -163,6 +163,16 @@ jobs:
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_schnell-eval]
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_dev-eval]
               "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
+              tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[op_by_op_torch-stable-diffusion-v3.5-large-eval]
+              "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
path: .github/workflows/run-op-by-op-model-tests-nightly.yml
user: login: kmabeeTT
body: I think we've never had 2 groups with same name (`stable-diffusion-v3.5`) before, not sure if this will cause problems, but probably good idea to differentiate...
created_at: 2025-05-09T20:39:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -110,6 +110,7 @@ jobs:
             # Approximately 40 minutes.
             runs-on: wormhole_b0, name: "eval_6", tests: "
                   tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval]
+                  tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[full-stable-diffusion-v3.5-medium-eval]
path: .github/workflows/run-full-model-execution-tests-nightly.yml
user: login: kmabeeTT
body: We can do a follow up PR to address this, but if the plan is to make visible that we have the other variants existing but not yet running, then the idea is to add them here as well , with the skip_full_eval_test() in each test with reasons and bringup_status, I see one of them was added, but not used since not added here.


created_at: 2025-05-09T21:16:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,127 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import (
+    StableDiffusion3Pipeline,
+    FlowMatchEulerDiscreteScheduler,
+    SD3Transformer2DModel,
+)
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_info = self.model_name
+        self.pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_info,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        self.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
+            model_info, subfolder="scheduler"
+        )
+        self.transformer = SD3Transformer2DModel.from_pretrained(
+            model_info, subfolder="transformer", torch_dtype=torch.bfloat16
+        )
+        return self.transformer
+
+    def _load_inputs(self):
+        prompt = "A futuristic cityscape at sunset"
+        num_images_per_prompt = 1
+        height = 512
+        width = 512
+        (
+            prompt_embeds,
+            negative_prompt_embeds,
+            pooled_prompt_embeds,
+            negative_pooled_prompt_embeds,
+        ) = self.pipe.encode_prompt(
+            prompt=prompt,
+            prompt_2=prompt,  # Using the same prompt for all encoders for simplicity
+            prompt_3=prompt,
+            num_images_per_prompt=num_images_per_prompt,
+            do_classifier_free_guidance=False,  # For simplicity in this direct call
+        )
+
+        num_channels_latents = self.transformer.config.in_channels
+        latents = self.pipe.prepare_latents(
+            batch_size=num_images_per_prompt,
+            num_channels_latents=num_channels_latents,
+            height=height,
+            width=width,
+            dtype=prompt_embeds.dtype,
+            device=None,
+            generator=None,
+        )
+
+        self.scheduler.set_timesteps(28)
+        timestep = self.scheduler.timesteps[0].expand(latents.shape[0])
+        arguments = {
+            "hidden_states": latents,
+            "timestep": timestep,
+            "encoder_hidden_states": prompt_embeds,
+            "pooled_projections": pooled_prompt_embeds,
+            "joint_attention_kwargs": {},
+            "return_dict": False,
+        }
+        return arguments
+
+
+model_info_list = [
+    ("stable-diffusion-v3.5-medium", "stabilityai/stable-diffusion-3.5-medium"),
+    ("stable-diffusion-v3.5-large", "stabilityai/stable-diffusion-3.5-large"),
+]
+
+
+@pytest.mark.parametrize(
+    "mode",
+    ["eval"],
+)
+@pytest.mark.parametrize(
+    "model_info",
+    model_info_list,
+    ids=[model_info[0] for model_info in model_info_list],
+)
+@pytest.mark.parametrize(
+    "op_by_op",
+    [OpByOpBackend.STABLEHLO, OpByOpBackend.TORCH, None],
+    ids=["op_by_op_stablehlo", "op_by_op_torch", "full"],
+)
+def test_stable_diffusion_transformer(record_property, model_info, mode, op_by_op):
+    model_group = "red"
+    _, model_name = model_info
+
+    cc = CompilerConfig()
+    cc.enable_consteval = True
+    cc.consteval_parameters = True
+    if op_by_op:
+        cc.compile_depth = CompileDepth.EXECUTE_OP_BY_OP
+        if op_by_op == OpByOpBackend.STABLEHLO:
+            cc.op_by_op_backend = OpByOpBackend.STABLEHLO
+
+    skip_full_eval_test(
+        record_property,
+        cc,
+        model_name,
+        bringup_status="FAILED_RUNTIME",
+        reason="Model is too large to fit on single device during execution.",
path: tests/models/stable_diffusion/test_stable_diffusion_transformer.py
user: login: kmabeeTT
body: This is probably eventually true (since seems like an 8B param model?), but is this really the failure reason, or something else?  When we encounter this, it's of this type of error at runtime (does it make it to runtime)?

```
Out of Memory: Not enough space to allocate 165183488 B DRAM buffer across 12 banks, where each bank needs to store 13766656 B
```
created_at: 2025-05-09T21:18:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,6 +163,16 @@ jobs:
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_schnell-eval]
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_dev-eval]
               "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
+              tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[op_by_op_torch-stable-diffusion-v3.5-large-eval]
+              "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
path: .github/workflows/run-op-by-op-model-tests-nightly.yml
user: login: jameszianxuTT
body: I think that has a chance of blowing up a couple of actions like fetch job-id and upload artifact, which enforce non-duplication policies and pass in as input the `matrix.build.name`; agree to differentiate these names
created_at: 2025-05-09T21:34:51Z
author_association: COLLABORATOR
diff_hunk: @@ -154,4 +157,5 @@ def pass_pipeline(gm: torch.fx.GraphModule, example_inputs, compiler_config):
 
     # Need to run shape_prop again to populate tensor_meta
     run_shape_prop(program.graph_module, constant_inputs + example_inputs)
+    gc.collect()
path: tt_torch/dynamo/passes.py
user: login: jameszianxuTT
body: Was this the fix for memory overrun? Would be interested to see if it helps with GPTNeo and Flux oom issues
created_at: 2025-05-09T21:36:26Z
author_association: COLLABORATOR
diff_hunk: @@ -154,4 +157,5 @@ def pass_pipeline(gm: torch.fx.GraphModule, example_inputs, compiler_config):
 
     # Need to run shape_prop again to populate tensor_meta
     run_shape_prop(program.graph_module, constant_inputs + example_inputs)
+    gc.collect()
path: tt_torch/dynamo/passes.py
user: login: ddilbazTT
body: This and also in constant_fold, I delete the unused graph. 
created_at: 2025-05-09T21:47:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,127 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import (
+    StableDiffusion3Pipeline,
+    FlowMatchEulerDiscreteScheduler,
+    SD3Transformer2DModel,
+)
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_info = self.model_name
+        self.pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_info,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        self.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
+            model_info, subfolder="scheduler"
+        )
+        self.transformer = SD3Transformer2DModel.from_pretrained(
+            model_info, subfolder="transformer", torch_dtype=torch.bfloat16
+        )
+        return self.transformer
+
+    def _load_inputs(self):
+        prompt = "A futuristic cityscape at sunset"
+        num_images_per_prompt = 1
+        height = 512
+        width = 512
+        (
+            prompt_embeds,
+            negative_prompt_embeds,
+            pooled_prompt_embeds,
+            negative_pooled_prompt_embeds,
+        ) = self.pipe.encode_prompt(
+            prompt=prompt,
+            prompt_2=prompt,  # Using the same prompt for all encoders for simplicity
+            prompt_3=prompt,
+            num_images_per_prompt=num_images_per_prompt,
+            do_classifier_free_guidance=False,  # For simplicity in this direct call
+        )
+
+        num_channels_latents = self.transformer.config.in_channels
+        latents = self.pipe.prepare_latents(
+            batch_size=num_images_per_prompt,
+            num_channels_latents=num_channels_latents,
+            height=height,
+            width=width,
+            dtype=prompt_embeds.dtype,
+            device=None,
+            generator=None,
+        )
+
+        self.scheduler.set_timesteps(28)
+        timestep = self.scheduler.timesteps[0].expand(latents.shape[0])
+        arguments = {
+            "hidden_states": latents,
+            "timestep": timestep,
+            "encoder_hidden_states": prompt_embeds,
+            "pooled_projections": pooled_prompt_embeds,
+            "joint_attention_kwargs": {},
+            "return_dict": False,
+        }
+        return arguments
+
+
+model_info_list = [
+    ("stable-diffusion-v3.5-medium", "stabilityai/stable-diffusion-3.5-medium"),
+    ("stable-diffusion-v3.5-large", "stabilityai/stable-diffusion-3.5-large"),
+]
+
+
+@pytest.mark.parametrize(
+    "mode",
+    ["eval"],
+)
+@pytest.mark.parametrize(
+    "model_info",
+    model_info_list,
+    ids=[model_info[0] for model_info in model_info_list],
+)
+@pytest.mark.parametrize(
+    "op_by_op",
+    [OpByOpBackend.STABLEHLO, OpByOpBackend.TORCH, None],
+    ids=["op_by_op_stablehlo", "op_by_op_torch", "full"],
+)
+def test_stable_diffusion_transformer(record_property, model_info, mode, op_by_op):
+    model_group = "red"
+    _, model_name = model_info
+
+    cc = CompilerConfig()
+    cc.enable_consteval = True
+    cc.consteval_parameters = True
+    if op_by_op:
+        cc.compile_depth = CompileDepth.EXECUTE_OP_BY_OP
+        if op_by_op == OpByOpBackend.STABLEHLO:
+            cc.op_by_op_backend = OpByOpBackend.STABLEHLO
+
+    skip_full_eval_test(
+        record_property,
+        cc,
+        model_name,
+        bringup_status="FAILED_RUNTIME",
+        reason="Model is too large to fit on single device during execution.",
path: tests/models/stable_diffusion/test_stable_diffusion_transformer.py
user: login: ddilbazTT
body: This is the exact error it got:
`Out of Memory: Not enough space to allocate 47316992 B DRAM buffer across 12 banks, where each bank needs to store 3949568 B` [link](https://github.com/tenstorrent/tt-torch/actions/runs/14888956747/job/41817359246)
created_at: 2025-05-09T21:47:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,6 +163,16 @@ jobs:
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_schnell-eval]
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_dev-eval]
               "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
+              tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[op_by_op_torch-stable-diffusion-v3.5-large-eval]
+              "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
path: .github/workflows/run-op-by-op-model-tests-nightly.yml
user: login: ddilbazTT
body: Oh sorry, yeah i overlooked this. I'll append one of them with -transformer
created_at: 2025-05-09T21:48:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -110,6 +110,7 @@ jobs:
             # Approximately 40 minutes.
             runs-on: wormhole_b0, name: "eval_6", tests: "
                   tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval]
+                  tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[full-stable-diffusion-v3.5-medium-eval]
path: .github/workflows/run-full-model-execution-tests-nightly.yml
user: login: ddilbazTT
body: Yeah, could you let me know what exactly I should do? I had a feeling I needed to do something like this for the large one, but if you could write it out I'd appreciate it. 
created_at: 2025-05-09T21:51:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -163,6 +163,16 @@ jobs:
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_schnell-eval]
               tests/models/flux/test_flux.py::test_flux[op_by_op_torch-flux_dev-eval]
               "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
+              tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[op_by_op_torch-stable-diffusion-v3.5-large-eval]
+              "
+          },
+          {
+            runs-on: wormhole_b0, name: "stable-diffusion-v3.5", tests: "
path: .github/workflows/run-op-by-op-model-tests-nightly.yml
user: login: ddilbazTT
body: Or actually I'll just merge them in the same group, what am I saying
created_at: 2025-05-09T21:52:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,94 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester
+from diffusers import StableDiffusion3Pipeline, AutoencoderTiny
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_info = self.model_name
+        pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_info,
+            text_encoder_3=None,
+            tokenizer_3=None,
+            torch_dtype=torch.float16,
+            low_cpu_mem_usage=True,
+        )
+        # memory optimization recommended by: https://huggingface.co/docs/diffusers/en/api/pipelines/stable_diffusion/stable_diffusion_3#tiny-autoencoder-for-stable-diffusion-3
+        pipe.vae = AutoencoderTiny.from_pretrained(
+            "madebyollin/taesd3", torch_dtype=torch.float16, low_cpu_mem_usage=True
+        )
+        pipe.enable_attention_slicing()
+        return pipe
+
+    def _load_inputs(self):
+        prompt = [
+            "a photo of an astronaut riding a horse on mars",
+        ]
+
+        negative_prompt = ""
+        height = 512
+        width = 512
+        guidance_scale = 7.0
+        arguments = {
+            "prompt": prompt,
+            "negative_prompt": negative_prompt,
+            "height": height,
+            "width": width,
+            "guidance_scale": guidance_scale,
+        }
+
+        return arguments
+
+
+model_info_list = [
+    ("stable-diffusion-v3.5-medium", "stabilityai/stable-diffusion-3.5-medium"),
path: tests/models/stable_diffusion/test_stable_diffusion_3_5.py
user: login: kmabeeTT
body: Another problem I just noticed, these have same model names ie "stable-diffusion-v3.5-medium" as the _transformer.py test, so they will clash in reporting in various places (XLS, database), I think.  There is also a character limit in excel sheet names so we only show 28 characters that we have to work with.  Can we use short names something like these as the model names (key of model_info_list) in these tests?

```
SD3.5-medium
SD3.5-large
SD3.5-medium-transformer
SD3.5-large-transformer
```
created_at: 2025-05-09T22:38:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -110,6 +110,7 @@ jobs:
             # Approximately 40 minutes.
             runs-on: wormhole_b0, name: "eval_6", tests: "
                   tests/models/stable_diffusion/test_stable_diffusion_unet.py::test_stable_diffusion_unet[full-eval]
+                  tests/models/stable_diffusion/test_stable_diffusion_transformer.py::test_stable_diffusion_transformer[full-stable-diffusion-v3.5-medium-eval]
path: .github/workflows/run-full-model-execution-tests-nightly.yml
user: login: kmabeeTT
body: Sure, I don't have the failing reasons you've encountered so I can't say for sure about the content, but the idea is to follow that commit I shared https://github.com/tenstorrent/tt-torch/commit/48c2ca89c8f30ee85d6b4766b6a77abbdec03671 as in:

1. Add `skip_full_eval_test()` targeting each case that has issues we want to pursue with details on current failing reason as best we know today and correct `bringup_status` (will be used for reporting where this model is failing). Should be placed before model is loaded, right after `CompilerConfig` is created, just like you did in this PR already for 1 of the tests (looks great). Available bringup statuses:

```
tests/utils.py:        bringup_status: The bringup status of the test (FAILED_FE_COMPILATION, FAILED_TTMLIR_COMPILATION, FAILED_RUNTIME, INCORRECT_RESULT, PASSED)
```

2. add the test to `run-full-model-execution-tests-nightly.yml` group called `skip_tests_red_models` where all the other similiar tests are listed that have issues.


For bullet 1, can filter by single string model name, list of model names, or no filter at all, example:

```
    skip_full_eval_test(
        record_property,
        cc,
        model_name,
        bringup_status="FAILED_RUNTIME",
        reason="Model is too large to fit on single device during execution.",
        model_group=model_group,
        model_name_filter=[
            "mistralai/Mistral-7B-v0.1",
            "mistralai/Ministral-8B-Instruct-2410",
        ],
    )
```

If we care about all 4 variants, since 1 is already running, would expect to see `skip_full_eval_test()` covering 3 different failures, and 3 entries added to `skip_tests_red_models` group.
created_at: 2025-05-09T22:48:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,127 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import (
+    StableDiffusion3Pipeline,
+    FlowMatchEulerDiscreteScheduler,
+    SD3Transformer2DModel,
+)
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_info = self.model_name
+        self.pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_info,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        self.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
+            model_info, subfolder="scheduler"
+        )
+        self.transformer = SD3Transformer2DModel.from_pretrained(
+            model_info, subfolder="transformer", torch_dtype=torch.bfloat16
+        )
+        return self.transformer
+
+    def _load_inputs(self):
+        prompt = "A futuristic cityscape at sunset"
+        num_images_per_prompt = 1
+        height = 512
+        width = 512
+        (
+            prompt_embeds,
+            negative_prompt_embeds,
+            pooled_prompt_embeds,
+            negative_pooled_prompt_embeds,
+        ) = self.pipe.encode_prompt(
+            prompt=prompt,
+            prompt_2=prompt,  # Using the same prompt for all encoders for simplicity
+            prompt_3=prompt,
+            num_images_per_prompt=num_images_per_prompt,
+            do_classifier_free_guidance=False,  # For simplicity in this direct call
+        )
+
+        num_channels_latents = self.transformer.config.in_channels
+        latents = self.pipe.prepare_latents(
+            batch_size=num_images_per_prompt,
+            num_channels_latents=num_channels_latents,
+            height=height,
+            width=width,
+            dtype=prompt_embeds.dtype,
+            device=None,
+            generator=None,
+        )
+
+        self.scheduler.set_timesteps(28)
+        timestep = self.scheduler.timesteps[0].expand(latents.shape[0])
+        arguments = {
+            "hidden_states": latents,
+            "timestep": timestep,
+            "encoder_hidden_states": prompt_embeds,
+            "pooled_projections": pooled_prompt_embeds,
+            "joint_attention_kwargs": {},
+            "return_dict": False,
+        }
+        return arguments
+
+
+model_info_list = [
+    ("stable-diffusion-v3.5-medium", "stabilityai/stable-diffusion-3.5-medium"),
+    ("stable-diffusion-v3.5-large", "stabilityai/stable-diffusion-3.5-large"),
+]
+
+
+@pytest.mark.parametrize(
+    "mode",
+    ["eval"],
+)
+@pytest.mark.parametrize(
+    "model_info",
+    model_info_list,
+    ids=[model_info[0] for model_info in model_info_list],
+)
+@pytest.mark.parametrize(
+    "op_by_op",
+    [OpByOpBackend.STABLEHLO, OpByOpBackend.TORCH, None],
+    ids=["op_by_op_stablehlo", "op_by_op_torch", "full"],
+)
+def test_stable_diffusion_transformer(record_property, model_info, mode, op_by_op):
+    model_group = "red"
+    _, model_name = model_info
+
+    cc = CompilerConfig()
+    cc.enable_consteval = True
+    cc.consteval_parameters = True
+    if op_by_op:
+        cc.compile_depth = CompileDepth.EXECUTE_OP_BY_OP
+        if op_by_op == OpByOpBackend.STABLEHLO:
+            cc.op_by_op_backend = OpByOpBackend.STABLEHLO
+
+    skip_full_eval_test(
+        record_property,
+        cc,
+        model_name,
+        bringup_status="FAILED_RUNTIME",
+        reason="Model is too large to fit on single device during execution.",
path: tests/models/stable_diffusion/test_stable_diffusion_transformer.py
user: login: kmabeeTT
body: Okay cool, looks good, thx for link and confirming!
created_at: 2025-05-09T22:48:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,25 @@
+STABILITY AI COMMUNITY LICENSE AGREEMENT Last Updated: July 5, 2024
path: tests/models/stable_diffusion/LICENSE.txt
user: login: AleksKnezevic
body: Could have sworn I already commented here, but I don't see my comments... Anyway, we're not reproducing their code or their model. We don't need a licence. 
created_at: 2025-05-13T00:07:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,3 @@
+This repository includes Stability AI Model.
path: tests/models/stable_diffusion/NOTICE.txt
user: login: AleksKnezevic
body: Ditto on the notice. 
created_at: 2025-05-13T01:35:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,3 @@
+This repository includes Stability AI Model.
path: tests/models/stable_diffusion/NOTICE.txt
user: login: AleksKnezevic
body: Can you remove the notice file please? 
created_at: 2025-05-16T15:53:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -103,11 +104,13 @@ def bypass_dtype_promotion(gm, compiler_config):
 
 
 def constant_fold(gm):
-    gm = const_fold.split_const_subgraphs(gm)
-    gm.run_folding()
+    split_gm = const_fold.split_const_subgraphs(gm)
path: tt_torch/dynamo/passes.py
user: login: AleksKnezevic
body: If the idea is to delete the original gm before you run folding (if this is to solve potential out of memory issues) I would prefer something like:
```python
gm = const_fold.split_const_subgraphs(gm)
gc.collect()
gm.run_folding()
```
created_at: 2025-05-16T15:56:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -154,4 +157,5 @@ def pass_pipeline(gm: torch.fx.GraphModule, example_inputs, compiler_config):
 
     # Need to run shape_prop again to populate tensor_meta
     run_shape_prop(program.graph_module, constant_inputs + example_inputs)
+    gc.collect()
path: tt_torch/dynamo/passes.py
user: login: AleksKnezevic
body: Is this really needed, everything goes out of scope after the return? 
created_at: 2025-05-16T16:00:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,13 +104,12 @@ def bypass_dtype_promotion(gm, compiler_config):
 
 
 def constant_fold(gm):
-    split_gm = const_fold.split_const_subgraphs(gm)
-    del gm
-    split_gm.run_folding()
-
-    split_gm.graph.eliminate_dead_code()
+    gm = const_fold.split_const_subgraphs(gm)
+    gc.collect()
path: tt_torch/dynamo/passes.py
user: login: AleksKnezevic
body: This collect is needed? 
created_at: 2025-05-16T17:11:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,13 +104,12 @@ def bypass_dtype_promotion(gm, compiler_config):
 
 
 def constant_fold(gm):
-    split_gm = const_fold.split_const_subgraphs(gm)
-    del gm
-    split_gm.run_folding()
-
-    split_gm.graph.eliminate_dead_code()
+    gm = const_fold.split_const_subgraphs(gm)
+    gc.collect()
path: tt_torch/dynamo/passes.py
user: login: ddilbazTT
body: We need some memory cleanup here to not crash due to leaked semaphore. My reasoning was that `gm = const_fold.split_const_subgraphs(gm)` assigns a new value to gm without releasing memory occupied by the previous gm. 
created_at: 2025-05-16T18:36:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -104,13 +104,12 @@ def bypass_dtype_promotion(gm, compiler_config):
 
 
 def constant_fold(gm):
-    split_gm = const_fold.split_const_subgraphs(gm)
-    del gm
-    split_gm.run_folding()
-
-    split_gm.graph.eliminate_dead_code()
+    gm = const_fold.split_const_subgraphs(gm)
+    gc.collect()
path: tt_torch/dynamo/passes.py
user: login: AleksKnezevic
body: sounds good. I guess the garbage collector happens to not run before run_folding. 
created_at: 2025-05-16T19:14:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,108 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import StableDiffusion3Pipeline, AutoencoderTiny
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_dict = {
+            "SD3.5-medium": "stabilityai/stable-diffusion-3.5-medium",
+            "SD3.5-large": "stabilityai/stable-diffusion-3.5-large",
+        }
+        model_path = model_dict[self.model_name]
+        pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_path,
+            text_encoder_3=None,
+            tokenizer_3=None,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        # memory optimization recommended by: https://huggingface.co/docs/diffusers/en/api/pipelines/stable_diffusion/stable_diffusion_3#tiny-autoencoder-for-stable-diffusion-3
+        pipe.vae = AutoencoderTiny.from_pretrained(
+            "madebyollin/taesd3", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True
+        )
+        pipe.enable_attention_slicing()
+        return pipe
+
+    def _load_inputs(self):
+        prompt = [
+            "a photo of an astronaut riding a horse on mars",
+        ]
+
+        negative_prompt = ""
+        height = 512
+        width = 512
+        guidance_scale = 7.0
+        arguments = {
+            "prompt": prompt,
+            "negative_prompt": negative_prompt,
+            "height": height,
+            "width": width,
+            "guidance_scale": guidance_scale,
+        }
+
+        return arguments
+
+
+model_info_list = [
+    ("SD3.5-medium", "stabilityai/stable-diffusion-3.5-medium"),
+    ("SD3.5-large", "stabilityai/stable-diffusion-3.5-large"),
+]
+
+
+@pytest.mark.parametrize(
+    "mode",
+    ["eval"],
+)
+@pytest.mark.parametrize(
+    "model_info",
+    model_info_list,
+    ids=[model_info[0] for model_info in model_info_list],
+)
+@pytest.mark.parametrize(
+    "op_by_op",
+    [OpByOpBackend.STABLEHLO, OpByOpBackend.TORCH, None],
+    ids=["op_by_op_stablehlo", "op_by_op_torch", "full"],
+)
+def test_stable_diffusion_3_5(record_property, model_info, mode, op_by_op):
+    model_group = "red"
+    model_name, model_path = model_info
+
+    cc = CompilerConfig()
+    cc.enable_consteval = True
+    # consteval_parameters is disabled because it results in a memory related crash
+    if op_by_op:
+        cc.compile_depth = CompileDepth.EXECUTE_OP_BY_OP
+        if op_by_op == OpByOpBackend.STABLEHLO:
+            cc.op_by_op_backend = OpByOpBackend.STABLEHLO
+
+    skip_full_eval_test(
+        record_property,
+        cc,
+        model_name,
+        bringup_status="FAILED_RUNTIME",
+        reason="Cannot lower StableHLO --> TTIR",
+        model_group=model_group,
+    )
path: tests/models/stable_diffusion/test_stable_diffusion_3_5.py
user: login: kmabeeTT
body: bringup_status is wrong here, plz use this (I merged the text from my previous comment with your reason here):

```
    skip_full_eval_test(
        record_property,
        cc,
        model_name,
        bringup_status="FAILED_TTMLIR_COMPILATION",
        reason="Cannot lower StableHLO --> TTIR : results/mlir_tests/stable_hlo/aten::round.decimals_0.mlir:3:10: error: failed to legalize operation 'stablehlo.round_nearest_even' - https://github.com/tenstorrent/tt-torch/issues/769",
        model_group=model_group,
    )
```
created_at: 2025-05-16T20:34:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,131 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import (
+    StableDiffusion3Pipeline,
+    FlowMatchEulerDiscreteScheduler,
+    SD3Transformer2DModel,
+)
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_dict = {
+            "SD3.5-medium-transformer": "stabilityai/stable-diffusion-3.5-medium",
+            "SD3.5-large-transformer": "stabilityai/stable-diffusion-3.5-large",
+        }
+        model_path = model_dict[self.model_name]
+        self.pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_path,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        self.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
+            model_path, subfolder="scheduler"
+        )
+        self.transformer = SD3Transformer2DModel.from_pretrained(
+            model_path, subfolder="transformer", torch_dtype=torch.bfloat16
+        )
+        return self.transformer
+
+    def _load_inputs(self):
+        prompt = "A futuristic cityscape at sunset"
+        num_images_per_prompt = 1
+        height = 512
+        width = 512
+        (
+            prompt_embeds,
+            negative_prompt_embeds,
+            pooled_prompt_embeds,
+            negative_pooled_prompt_embeds,
+        ) = self.pipe.encode_prompt(
+            prompt=prompt,
+            prompt_2=prompt,  # Using the same prompt for all encoders for simplicity
+            prompt_3=prompt,
+            num_images_per_prompt=num_images_per_prompt,
+            do_classifier_free_guidance=False,  # For simplicity in this direct call
+        )
+
+        num_channels_latents = self.transformer.config.in_channels
+        latents = self.pipe.prepare_latents(
+            batch_size=num_images_per_prompt,
+            num_channels_latents=num_channels_latents,
+            height=height,
+            width=width,
+            dtype=prompt_embeds.dtype,
+            device=None,
+            generator=None,
+        )
+
+        self.scheduler.set_timesteps(28)
+        timestep = self.scheduler.timesteps[0].expand(latents.shape[0])
+        arguments = {
+            "hidden_states": latents,
+            "timestep": timestep,
+            "encoder_hidden_states": prompt_embeds,
+            "pooled_projections": pooled_prompt_embeds,
+            "joint_attention_kwargs": {},
+            "return_dict": False,
+        }
+        return arguments
+
+
+model_info_list = [
path: tests/models/stable_diffusion/test_stable_diffusion_transformer.py
user: login: kmabeeTT
body: Not broken, but I don't love that short model name to weights name is listed here (not used) and then duplicated above in model_dict. The change I gave as cherry pick simplified this part so it was just model_name list.
created_at: 2025-05-16T20:35:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,131 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import (
+    StableDiffusion3Pipeline,
+    FlowMatchEulerDiscreteScheduler,
+    SD3Transformer2DModel,
+)
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_dict = {
+            "SD3.5-medium-transformer": "stabilityai/stable-diffusion-3.5-medium",
+            "SD3.5-large-transformer": "stabilityai/stable-diffusion-3.5-large",
+        }
+        model_path = model_dict[self.model_name]
+        self.pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_path,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        self.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
+            model_path, subfolder="scheduler"
+        )
+        self.transformer = SD3Transformer2DModel.from_pretrained(
+            model_path, subfolder="transformer", torch_dtype=torch.bfloat16
+        )
+        return self.transformer
+
+    def _load_inputs(self):
+        prompt = "A futuristic cityscape at sunset"
+        num_images_per_prompt = 1
+        height = 512
+        width = 512
+        (
+            prompt_embeds,
+            negative_prompt_embeds,
+            pooled_prompt_embeds,
+            negative_pooled_prompt_embeds,
+        ) = self.pipe.encode_prompt(
+            prompt=prompt,
+            prompt_2=prompt,  # Using the same prompt for all encoders for simplicity
+            prompt_3=prompt,
+            num_images_per_prompt=num_images_per_prompt,
+            do_classifier_free_guidance=False,  # For simplicity in this direct call
+        )
+
+        num_channels_latents = self.transformer.config.in_channels
+        latents = self.pipe.prepare_latents(
+            batch_size=num_images_per_prompt,
+            num_channels_latents=num_channels_latents,
+            height=height,
+            width=width,
+            dtype=prompt_embeds.dtype,
+            device=None,
+            generator=None,
+        )
+
+        self.scheduler.set_timesteps(28)
+        timestep = self.scheduler.timesteps[0].expand(latents.shape[0])
+        arguments = {
+            "hidden_states": latents,
+            "timestep": timestep,
+            "encoder_hidden_states": prompt_embeds,
+            "pooled_projections": pooled_prompt_embeds,
+            "joint_attention_kwargs": {},
+            "return_dict": False,
+        }
+        return arguments
+
+
+model_info_list = [
+    ("SD3.5-medium-transformer", "stabilityai/stable-diffusion-3.5-medium"),
+    ("SD3.5-large-transformer", "stabilityai/stable-diffusion-3.5-large"),
+]
+
+
+@pytest.mark.parametrize(
+    "mode",
+    ["eval"],
+)
+@pytest.mark.parametrize(
+    "model_info",
+    model_info_list,
+    ids=[model_info[0] for model_info in model_info_list],
+)
+@pytest.mark.parametrize(
+    "op_by_op",
+    [OpByOpBackend.STABLEHLO, OpByOpBackend.TORCH, None],
+    ids=["op_by_op_stablehlo", "op_by_op_torch", "full"],
+)
+def test_stable_diffusion_transformer(record_property, model_info, mode, op_by_op):
+    model_group = "red"
+    model_name, model_path = model_info
+
+    cc = CompilerConfig()
+    cc.enable_consteval = True
+    cc.consteval_parameters = True
+    if op_by_op:
+        cc.compile_depth = CompileDepth.EXECUTE_OP_BY_OP
+        if op_by_op == OpByOpBackend.STABLEHLO:
+            cc.op_by_op_backend = OpByOpBackend.STABLEHLO
+
+    skip_full_eval_test(
+        record_property,
+        cc,
+        model_name,
+        bringup_status="FAILED_RUNTIME",
+        reason="Model compilation exceeds CI timeout limit (240 minutes), compiles e2e locally. Model execution fails with `Not enough space to allocate 47316992 B DRAM buffer across 12 banks.`",
+        model_group=model_group,
+        model_name_filter="stabilityai/stable-diffusion-3.5-large",
path: tests/models/stable_diffusion/test_stable_diffusion_transformer.py
user: login: kmabeeTT
body: After the most recent force push which started to use the shorter model names and remove duplicate model names across py tests (thank you), this is incorrect:

```
model_name_filter="stabilityai/stable-diffusion-3.5-large",
```

should be:


```
model_name_filter="SD3.5-large-transformer",
```
created_at: 2025-05-16T20:39:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,131 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import (
+    StableDiffusion3Pipeline,
+    FlowMatchEulerDiscreteScheduler,
+    SD3Transformer2DModel,
+)
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_dict = {
+            "SD3.5-medium-transformer": "stabilityai/stable-diffusion-3.5-medium",
+            "SD3.5-large-transformer": "stabilityai/stable-diffusion-3.5-large",
+        }
+        model_path = model_dict[self.model_name]
+        self.pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_path,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        self.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
+            model_path, subfolder="scheduler"
+        )
+        self.transformer = SD3Transformer2DModel.from_pretrained(
+            model_path, subfolder="transformer", torch_dtype=torch.bfloat16
+        )
+        return self.transformer
+
+    def _load_inputs(self):
+        prompt = "A futuristic cityscape at sunset"
+        num_images_per_prompt = 1
+        height = 512
+        width = 512
+        (
+            prompt_embeds,
+            negative_prompt_embeds,
+            pooled_prompt_embeds,
+            negative_pooled_prompt_embeds,
+        ) = self.pipe.encode_prompt(
+            prompt=prompt,
+            prompt_2=prompt,  # Using the same prompt for all encoders for simplicity
+            prompt_3=prompt,
+            num_images_per_prompt=num_images_per_prompt,
+            do_classifier_free_guidance=False,  # For simplicity in this direct call
+        )
+
+        num_channels_latents = self.transformer.config.in_channels
+        latents = self.pipe.prepare_latents(
+            batch_size=num_images_per_prompt,
+            num_channels_latents=num_channels_latents,
+            height=height,
+            width=width,
+            dtype=prompt_embeds.dtype,
+            device=None,
+            generator=None,
+        )
+
+        self.scheduler.set_timesteps(28)
+        timestep = self.scheduler.timesteps[0].expand(latents.shape[0])
+        arguments = {
+            "hidden_states": latents,
+            "timestep": timestep,
+            "encoder_hidden_states": prompt_embeds,
+            "pooled_projections": pooled_prompt_embeds,
+            "joint_attention_kwargs": {},
+            "return_dict": False,
+        }
+        return arguments
+
+
+model_info_list = [
path: tests/models/stable_diffusion/test_stable_diffusion_transformer.py
user: login: ddilbazTT
body: replaced the static dict with `dict(model_info)`
created_at: 2025-05-16T20:50:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,131 @@
+# SPDX-FileCopyrightText: (c) 2025 Tenstorrent AI ULC
+#
+# SPDX-License-Identifier: Apache-2.0
+import torch
+import pytest
+from tests.utils import ModelTester, skip_full_eval_test
+from diffusers import (
+    StableDiffusion3Pipeline,
+    FlowMatchEulerDiscreteScheduler,
+    SD3Transformer2DModel,
+)
+
+from tt_torch.tools.utils import CompilerConfig, CompileDepth, OpByOpBackend
+
+
+class ThisTester(ModelTester):
+    def _load_model(self):
+        model_dict = {
+            "SD3.5-medium-transformer": "stabilityai/stable-diffusion-3.5-medium",
+            "SD3.5-large-transformer": "stabilityai/stable-diffusion-3.5-large",
+        }
+        model_path = model_dict[self.model_name]
+        self.pipe = StableDiffusion3Pipeline.from_pretrained(
+            model_path,
+            torch_dtype=torch.bfloat16,
+            low_cpu_mem_usage=True,
+        )
+        self.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(
+            model_path, subfolder="scheduler"
+        )
+        self.transformer = SD3Transformer2DModel.from_pretrained(
+            model_path, subfolder="transformer", torch_dtype=torch.bfloat16
+        )
+        return self.transformer
+
+    def _load_inputs(self):
+        prompt = "A futuristic cityscape at sunset"
+        num_images_per_prompt = 1
+        height = 512
+        width = 512
+        (
+            prompt_embeds,
+            negative_prompt_embeds,
+            pooled_prompt_embeds,
+            negative_pooled_prompt_embeds,
+        ) = self.pipe.encode_prompt(
+            prompt=prompt,
+            prompt_2=prompt,  # Using the same prompt for all encoders for simplicity
+            prompt_3=prompt,
+            num_images_per_prompt=num_images_per_prompt,
+            do_classifier_free_guidance=False,  # For simplicity in this direct call
+        )
+
+        num_channels_latents = self.transformer.config.in_channels
+        latents = self.pipe.prepare_latents(
+            batch_size=num_images_per_prompt,
+            num_channels_latents=num_channels_latents,
+            height=height,
+            width=width,
+            dtype=prompt_embeds.dtype,
+            device=None,
+            generator=None,
+        )
+
+        self.scheduler.set_timesteps(28)
+        timestep = self.scheduler.timesteps[0].expand(latents.shape[0])
+        arguments = {
+            "hidden_states": latents,
+            "timestep": timestep,
+            "encoder_hidden_states": prompt_embeds,
+            "pooled_projections": pooled_prompt_embeds,
+            "joint_attention_kwargs": {},
+            "return_dict": False,
+        }
+        return arguments
+
+
+model_info_list = [
path: tests/models/stable_diffusion/test_stable_diffusion_transformer.py
user: login: kmabeeTT
body: Nice! Even better - thanks a lot!
created_at: 2025-05-16T21:09:03Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov-commenter
created_at: 2025-05-09T19:22:24Z
body: ## [Codecov](https://app.codecov.io/gh/tenstorrent/tt-torch/pull/749?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 69.28%. Comparing base [(`b062ba9`)](https://app.codecov.io/gh/tenstorrent/tt-torch/commit/b062ba97e7b158a541f759a8a060d7693ec07268?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent) to head [(`1e00685`)](https://app.codecov.io/gh/tenstorrent/tt-torch/commit/1e006853b72c338ef409af11f22bcd6047b5bad2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).

:white_check_mark: All tests successful. No failed tests found.

> :exclamation:  There is a different number of reports uploaded between BASE (b062ba9) and HEAD (1e00685). Click for more details.
> 
> <details><summary>HEAD has 18 uploads less than BASE</summary>
>
>| Flag | BASE (b062ba9) | HEAD (1e00685) |
>|------|------|------|
>||19|1|
></details>

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #749      +/-   ##
==========================================
- Coverage   75.22%   69.28%   -5.94%     
==========================================
  Files          18       18              
  Lines        2886     2888       +2     
==========================================
- Hits         2171     2001     -170     
- Misses        715      887     +172     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/tenstorrent/tt-torch/pull/749?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).

author_association: NONE
user: login: ddilbazTT
created_at: 2025-05-09T21:50:01Z
body: @kmabeeTT 
> Do you have more details on the failure reasons here other than "cannot run" we can document in this PR to help save time in the future?

I'll try to launch a run for all of them together in op-by-op, compile, and execute. They take really long and usually get killed due to OOM. I'll post the link. I know the files I am adding might not be the cleanest way but at least it ticks of SD3.5 somehow. 

author_association: CONTRIBUTOR
user: login: ddilbazTT
created_at: 2025-05-13T15:09:11Z
body: I wanted to post an update. I ran the designs in a nightly flow, and except for one, none of them executed. However, the nightly runs didn't really capture why they weren't executing. When I ran `pytest -s tests/models/stable_diffusion/test_stable_diffusion_3_5.py::test_stable_diffusion_3_5[full-SD3.5-large-eval]` locally, I noticed it throws the following error: 
```
 RuntimeError: TT_THROW @ /localdev/ddilbaz/tt-torch-2/third_party/tt-mlir/src/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:141: tt::exception
E       info:
E       Out of Memory: Not enough space to allocate 47316992 B DRAM buffer across 12 banks, where each bank needs to store 3949568 B
```
[Link](https://github.com/tenstorrent/tt-torch/actions/runs/14983356034)
author_association: CONTRIBUTOR
user: login: ddilbazTT
created_at: 2025-05-13T16:24:27Z
body: I discovered that the reason SD3.5-medium is failing during execute is this:

```
python3.10: /localdev/ddilbaz/tt-torch-2/tt_torch/csrc/bindings.cpp:65: tt::target::DataType torch_scalar_type_to_dt(torch::ScalarType): Assertion `false && "Unsupported scalar type"' failed.
Fatal Python error: Aborted
```
I'll try to use gdb and see why it's hitting this. 
author_association: CONTRIBUTOR
user: login: AleksKnezevic
created_at: 2025-05-13T16:55:27Z
body: > I discovered that the reason SD3.5-medium is failing during execute is this:
> 
> ```
> python3.10: /localdev/ddilbaz/tt-torch-2/tt_torch/csrc/bindings.cpp:65: tt::target::DataType torch_scalar_type_to_dt(torch::ScalarType): Assertion `false && "Unsupported scalar type"' failed.
> Fatal Python error: Aborted
> ```
> 
> I'll try to use gdb and see why it's hitting this.

Fails op by op or e2e? 
author_association: CONTRIBUTOR
user: login: ddilbazTT
created_at: 2025-05-13T16:59:58Z
body: > Fails op by op or e2e?

e2e
author_association: CONTRIBUTOR
user: login: kmabeeTT
created_at: 2025-05-13T17:05:37Z
body: > I'll try to use gdb and see why it's hitting this.

If full model execute no need to debug here, goal is op by op tests, skipped during full model execute for tracking purposes ideally with some minimal reason for reference. 
author_association: CONTRIBUTOR
user: login: ddilbazTT
created_at: 2025-05-13T17:08:34Z
body: @kmabeeTT this could be a bug though, `torch_scalar_type_to_dt` shouldn't really hit an uncovered case when the model is using bfloat16. I would like to debug further why this is happening
author_association: CONTRIBUTOR
user: login: AleksKnezevic
created_at: 2025-05-13T17:29:32Z
body: There's an intermediate value that's not bfloat16, it's fine. I agree with @kmabeeTT that we can just add the op by op test and continue. 
author_association: CONTRIBUTOR
user: login: kmabeeTT
created_at: 2025-05-13T18:31:15Z
body: Also remember that passing op-by-op flow exists to debug, and is what typically shows compiler/runtime failures per op along with repro details for single ops.  To use it, need to generate and open up the op-by-op flow generated XLS to see details, in this case the linked CI [run](https://github.com/tenstorrent/tt-torch/actions/runs/14983356034/job/42092715437) above accidentally had the XLS generation step commented out though. It's useful to keep, this part:

```
  download-report:
    if: success() || failure()
    needs: test_op_by_op
    uses: ./.github/workflows/generate-model-report.yml
    secrets: inherit
```

Upon downloading the artifacts and generating it myself manually (steps below) I see a few small problems... 

```
 4658  13/05/25 17:35:20 python3 results/download_artifacts.py --run-id 14983356034 --filter reports
 4661  13/05/25 17:35:37 cd nightly-tests_artifacts_20250512_run_id_14983356034
 4662  13/05/25 17:35:41 for f in `find *.tar`; do tar -xvf $f; done;
 4667  13/05/25 17:36:17 mv models ../results
 4668  13/05/25 17:36:18 cd ..
 4671  13/05/25 17:36:30 python3 results/parse_op_by_op_results.py

Processing json (1/4) stabilityai_stable_diffusion
Processing json (2/4) stabilityai_stable_diffusion_1
Processing json (3/4) stabilityai_stable_diffusion_2
Processing json (4/4) stabilityai_stable_diffusion_3
Creating summary for 4 models
```

Screenshot and attached XLS ([models_op_per_op_sd3.5.xlsx](https://github.com/user-attachments/files/20193870/models_op_per_op_sd3.5.xlsx)):

![image](https://github.com/user-attachments/assets/574045b5-36cc-4a3a-a475-6cef30f01248)


1. The model names (with cutoff at 28 characters) still appear the same, my previous suggestion wasn't enough I just realized, need to find some way to have `ModelTester` have `self.model_name` set to to the abbreviated `SD3.5-medium` version and the model loading use the longer string `stabilityai/stable-diffusion-3.5-medium`. 
2. Something is off here in this run, and 99% of the error messages on per-model sheets aren't able to be extracted from stderr. Not sure what is up there, never seen this before... not seen in nightly regression,  so I wonder if something wrong with the flow or model, or branch.  Medium SD3.5 test is quick to run (15 mins on CI) in op-by-op flow to reproduce, I can give some tips to make it even shorter by targeting just a specific op type.
3. For my earlier point, `stabilityai_stable_diffusion_2 and stabilityai_stable_diffusion_3` must be the full (non-transformer) models you mentioned have real errors, so they do not look close at all to be ready for full-model-execute running and debug, which is okay.

author_association: CONTRIBUTOR
user: login: ddilbazTT
created_at: 2025-05-13T18:35:48Z
body: @kmabeeTT I have been running again and I still can't find the actual error why SD3.5 (the complete model) fail. It's like every time I run I get a different error. Even if not debugging, I just want to note down the exact reason why they're failing in the tests themselves. For example, when I reran SD3.5 medium, I got the following error this time:

```
E       RuntimeError: TT_THROW @ /localdev/ddilbaz/tt-torch-2/third_party/tt-mlir/src/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:114: tt::exception
E       info:
E       Failed to generate binaries for tilize TT_THROW @ /localdev/ddilbaz/tt-torch-2/third_party/tt-mlir/src/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/jit_build/build.cpp:67: tt::exception
E       info:
E       trisc0 build failed. Log: /localdev/ddilbaz/tt-torch-2/install/tt-metal/runtime/sfpi/compiler/bin/../lib/gcc/riscv32-unknown-elf/12.4.0/../../../../riscv32-unknown-elf/bin/ld:/localdev/ddilbaz/tt-torch-2/install/tt-metal/runtime/h
w/toolchain/wormhole/kernel_trisc0.ld:18: non constant or forward reference address expression for section .text
E       collect2: error: ld returned 1 exit status
```
author_association: CONTRIBUTOR
user: login: kmabeeTT
created_at: 2025-05-13T18:48:40Z
body: > Even if not debugging, I just want to note down the exact reason why they're failing in the tests themselves. For example, when I reran SD3.5 medium, I got the following error this time:

op-by-op flow is useful here too... especially in this case where model is failing for many reasons (multiple depths in compiler, and also at runtime) I have been picking the assertion errors seen from runtime/tt-metal (or highest compile depth) since they typically have the longest path/time to fix (and the frontend/compiler errors end up getting fixed sooner). Try running the SD3.5 medium test in op by op flow locally and seeing if you get some real errors reported on in the generated .json files and XLS. 
author_association: CONTRIBUTOR
user: login: kmabeeTT
created_at: 2025-05-13T19:56:10Z
body: Follow up to my previous points 1,2 to 

1. model name cutoff, I don't see quick and easy and safe fix for this (modifying model_name to ModelTester wil impact reported models to superset which we don't want to do), it's purely XLS generation limited,  so I vote to leave it for now, revisit afterwards with fix for all models once discussed and some plan figured out to address .json/xls generation.
2. Your CI run for op-by-op flow at least had machine issues of some kind, since when I run SD3.5 med test locally, so far all ops are running fine (depth 7), certainly more than the 47 successful ops in the screenshot...

```
gron results/models/tests/models/stable_diffusion/test_stable_diffusion_3_5_py__test_stable_diffusion_3_5_op_by_op_torch_SD3_5_medium_eval___call__unique_ops.json | grep status | grep "= 7" | wc -l
132
```
author_association: CONTRIBUTOR
user: login: ddilbazTT
created_at: 2025-05-13T20:10:12Z
body: > 2. Your CI run for op-by-op flow at least had machine issues of some kind, since when I run SD3.5 med test locally, so far all ops are running fine (depth 7), certainly more than the 47 successful ops in the screenshot...


Yeah, this is why I'm rerunning locally and on CI. At the very least, I want to be sure I understand when exactly the models fail during e2e. So far, results are non-deterministic.
author_association: CONTRIBUTOR
user: login: kmabeeTT
created_at: 2025-05-14T17:14:57Z
body: > Yeah, this is why I'm rerunning locally and on CI. At the very least, I want to be sure I understand when exactly the models fail during e2e. So far, results are non-deterministic.

Locally is good idea, I agree - it worked great for me - don't know what is up with CI not showing actual errors (pytest cmd and logging did change a bit in past week, maybe related), but I think that might be seperate issue from this PR.

The op-by-op flow results are important to consider first though, because if model cannot compile all ops it generally doesn't make sense to try and run full model execute since compiling is prerequisitie to running in `full-eval` mode.  In this case, the op-by-op flow for SD3.5 (med and large) reports this compile error (on stablehlo -> ttir conversion) for 1 op (`aten::round.decimals`, all other ops pass execute in my local run from yesterday, see screenshot) and e2e compile (`TT_TORCH_COMPILE_DEPTH=TTNN_IR`) also reports the same error, pasted here:

```
error: failed to legalize operation 'stablehlo.round_nearest_even'
```

That would be the reason to use for what is blocking full model execute since it is blocking compile. Note that the original SD1.5 model we've had for a while is hitting that compiler error above too lately. I opened ticket for this here: https://github.com/tenstorrent/tt-torch/issues/769 for debug and reference it below.

So for SD3.5 med/large models, this should be the exact line (inspired by similar examples in other tests) you want to add to `test_stable_diffusion_3_5.py` for tracking purposes alongside adding them to`skip_tests_red_models` group in `.github/workflows/run-full-model-execution-tests-nightly.yml`, and tests added to op-by-op nightly list:

```
    skip_full_eval_test(
        record_property,
        cc,
        model_name,
        bringup_status="FAILED_TTMLIR_COMPILATION",
        reason="results/mlir_tests/stable_hlo/aten::round.decimals_0.mlir:3:10: error: failed to legalize operation 'stablehlo.round_nearest_even' - https://github.com/tenstorrent/tt-torch/issues/769",
        model_group=model_group,
    )
```

For reference, attaching new XLS [models_op_per_op_sd3.5_local.xlsx](https://github.com/user-attachments/files/20211298/models_op_per_op_sd3.5_local.xlsx) where I ran all 4 variants without any non-deterministic issues showing the real failures:

![image](https://github.com/user-attachments/assets/95cc3e19-62d6-47fd-b468-9ab2b7f8fcc2)
![image](https://github.com/user-attachments/assets/9db6d4dc-7de8-4a2d-a1ab-6b9d89ef8cb5)

All this said, I think we should cleanup this branch, apply changes above and merge today, what do you think?
author_association: CONTRIBUTOR
user: login: kmabeeTT
created_at: 2025-05-15T03:21:15Z
body: About model name issues... after more thought, I realized an additional problem: right now they are duplicated between the two different python tests which would clash during reporting to superset. So I put together a small change to fix both that problem and the "names are too long for XLS" issue at 15214541457f68868c5f1310f16273e6aab83007, you can cherry pick or adjust as needed to solve problem, please take a look.
author_association: CONTRIBUTOR
user: login: ddilbazTT
created_at: 2025-05-17T15:44:16Z
body: I am rebasing after Aleks' changes, will merge this weekend
author_association: CONTRIBUTOR