Repository: tenstorrent/vllm
Pull request #82: Support fabric_config
Author: login: ppetrovicTT
Created: 2025-04-23T14:50:47Z | Merged: 2025-04-24T16:28:10Z

* Add support for fabric_config to tt_worker
* Use override_tt_config in offline_inference_tt instead of separate arguments



Diff:
diff --git a/examples/offline_inference_tt.py b/examples/offline_inference_tt.py
index 7d543617cfe2..e36da43ee470 100644
--- a/examples/offline_inference_tt.py
+++ b/examples/offline_inference_tt.py
@@ -131,17 +131,10 @@ def run_inference(
     disable_async_output_proc=False,
     multi_modal=False,
     test_increasing_seq_lens=False,
-    sample_on_device_mode=None,
-    dispatch_core_axis=None,
+    override_tt_config=None
 ):
     check_tt_model_supported(model)
     
-    override_tt_config = {}
-    if sample_on_device_mode:
-        override_tt_config["sample_on_device_mode"] = sample_on_device_mode
-    if dispatch_core_axis:
-        override_tt_config["dispatch_core_axis"] = dispatch_core_axis.lower()
-    
     # LLM args
     engine_kw_args = {
         "model": model,
@@ -153,8 +146,13 @@ def run_inference(
         "log_global_stats": True if measure_perf else False,
         "num_scheduler_steps": num_scheduler_steps,
         "disable_async_output_proc": disable_async_output_proc,
-        "override_tt_config": override_tt_config,
     }
+
+    try:
+        if override_tt_config:
+            engine_kw_args["override_tt_config"] = json.loads(override_tt_config)
+    except json.JSONDecodeError as e:
+        raise ValueError(f"Invalid JSON string for override_tt_config: {e}")
     
     # Generation args
     ignore_eos = True if measure_perf else False
@@ -310,8 +308,8 @@ async def generate_tokens_async(llm : MQLLMEngineClient, prompts, sampling_param
     parser.add_argument("--num_scheduler_steps", type=int, default=10, help="Number of scheduler steps")
     parser.add_argument("--multi_modal", action="store_true", help="Run multi-modal inference with Llama3.2-11b")
     parser.add_argument("--test_increasing_seq_lens", action="store_true", help="Test generations of small to large sequences")
-    parser.add_argument("--sample_on_device_mode", type=str, choices=["all", "decode_only"], default=None, help="Enable sampling on device (during prefill and decode, or only during decode)")
-    parser.add_argument("--dispatch_core_axis", type=str, choices=["row", "col", None], default=None, help="Dispatch core axis [row, col]")
+    parser.add_argument("--override_tt_config", type=str, default=None, help="Custom TT options as Json string")
+    
     args = parser.parse_args()
 
     run_inference(
@@ -328,6 +326,5 @@ async def generate_tokens_async(llm : MQLLMEngineClient, prompts, sampling_param
         disable_async_output_proc=args.disable_async_output_proc,
         multi_modal=args.multi_modal,
         test_increasing_seq_lens=args.test_increasing_seq_lens,
-        sample_on_device_mode=args.sample_on_device_mode,
-        dispatch_core_axis=args.dispatch_core_axis,
+        override_tt_config=args.override_tt_config,
     )
diff --git a/tt_metal/README.md b/tt_metal/README.md
index 275a202b6387..f07eebde46a9 100644
--- a/tt_metal/README.md
+++ b/tt_metal/README.md
@@ -54,20 +54,29 @@ To run Meta-Llama-3.1/3.2, it is required to have access to the model on Hugging
 ### Llama-3.1/3.2 (1B, 3B, 8B, 70B) and Qwen-2.5 (7B, 72B) Text Models
 
 To generate tokens (Llama70B) for sample prompts (with batch size 32):
-```python
+```sh
 MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py
 ```
 
 To measure performance (Llama70B) for a single batch of 32 prompts (with the default prompt length of 128 tokens):
-```python
+```sh
 MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py --measure_perf
 ```
 
-**Note 1 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
+**Note 1**: Custom TT options can be set using `--override_tt_config` with a json string, e.g. `--override_tt_config '{"sample_on_device_mode": "all"}'`, however these shouldn't be used unless the model supports them (most currently do not). Supported parameters are:
+- `sample_on_device_mode`: ["all", "decode_only"]
+- `trace_region_size`" [default: 23887872]
+- `worker_l1_size`
+- `fabric_config`: ["DISABLED", "FABRIC_1D", "FABRIC_2D", "CUSTOM"]
+- `dispatch_core_axis`: ["row", "col"]
 
-**Note 2 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [Llama3 demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run with the [old Llama70B implemenentation](https://github.com/tenstorrent/tt-metal/tree/main/models/demos/t3000/llama3_70b), modify the `LlamaForCausalLM` model import in [offline_inference_tt.py](https://github.com/tenstorrent/vllm/blob/dev/examples/offline_inference_tt.py) to `from models.demos.t3000.llama2_70b.tt.generator_vllm import TtLlamaForCausalLM as LlamaForCausalLM`.
+**Note 2 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
 
-**Note 3 (Other Models)**: By default, the inference example will run with Llama-3.1-70B. To run with other Llama models, or Qwen-2.5, ensure that the apprioriate environment variables are set as per the [demo instructions](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers), then set `MESH_DEVICE=<device>` (valid options for `<device>` are `N150`, `N300`, `T3K`, or `TG`) and one of the following:
+**Note 3 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [tt_transformers demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run other implementations use the `TT_LLAMA_TEXT_VER` environment variable:
+- `"llama3_subdevices"` for the Llama TG implementation
+- `"llama2_70b"` for the old Llama implementation
+
+**Note 4 (Other Models)**: By default, the inference example will run with Llama-3.1-70B. To run with other Llama models, or Qwen-2.5, ensure that the apprioriate environment variables are set as per the [demo instructions](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers), then set `MESH_DEVICE=<device>` (valid options for `<device>` are `N150`, `N300`, `T3K`, or `TG`) and one of the following:
 - Llama-3.1-8B: `--model "meta-llama/Llama-3.1-8B"`
 - Llama-3.2-1B: `--model "meta-llama/Llama-3.2-1B"`
 - Llama-3.2-3B: `--model "meta-llama/Llama-3.2-3B"`
@@ -75,15 +84,20 @@ MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python exampl
 - Qwen-2.5-72B: `--model "Qwen/Qwen2.5-72B"` (currently only supported on T3K)
 - DeepSeek-R1-Distill-Llama-70B: `--model "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"`
 
+Command line to run Llama70B on TG is:
+```sh
+MESH_DEVICE=TG LLAMA_DIR=<path to weights> TT_LLAMA_TEXT_VER="llama3_subdevices" python examples/offline_inference_tt.py --model "meta-llama/Llama-3.1-70B-Instruct" --override_tt_config '{"dispatch_core_axis": "col", "sample_on_device_mode": "all", "fabric_config": "FABRIC_1D", "worker_l1_size": 1344544, "trace_region_size": 62000000}'
+```
+
 ### Llama-3.2 (11B and 90B) Vision models
 
 To generate tokens for sample prompts:
-```python
+```sh
 MESH_DEVICE=N300 WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py --model "meta-llama/Llama-3.2-11B-Vision-Instruct" --multi_modal --max_seqs_in_batch 16 --num_repeat_prompts 8
 ```
 
 To measure performance for a single batch (with the default prompt length of 128 tokens):
-```python
+```sh
 MESH_DEVICE=N300 WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py --model "meta-llama/Llama-3.2-11B-Vision-Instruct" --measure_perf --multi_modal --max_seqs_in_batch 16
 ```
 
@@ -91,20 +105,20 @@ MESH_DEVICE=N300 WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examp
 
 **Note**: Running `90B` model, set `MESH_DEVICE=T3K` and `--max_seqs_in_batch 4`:
 
-```python
+```sh
 MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py --model "meta-llama/Llama-3.2-90B-Vision-Instruct" --multi_modal --max_seqs_in_batch 4
 ```
 
 ## Running the server example
 
 To start up the server:
-```python
+```sh
 VLLM_RPC_TIMEOUT=100000 MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/server_example_tt.py
 ```
 
 **Note**: By default, the server will run with Llama-3.1-70B-Instruct. To run with other models, set `MESH_DEVICE` and `--model` as described in [Running the offline inference example](#running-the-offline-inference-example).
 
-**Note**: Custom TT options can be set using `--override_tt_config`, e.g. `--override_tt_config '{"sample_on_device_mode": "all"}'`, however these shouldn't be used unless the model supports them (most currently do not).
+**Note**: Custom TT options can be set using `--override_tt_config` as described in [Running the offline inference example](#running-the-offline-inference-example).
 
 To send a request to the server:
 ```sh
diff --git a/vllm/worker/tt_worker.py b/vllm/worker/tt_worker.py
index 2e9d9895e499..bccfee544013 100644
--- a/vllm/worker/tt_worker.py
+++ b/vllm/worker/tt_worker.py
@@ -398,6 +398,38 @@ def _get_dispatch_core_config(self, device_params):
 
         dispatch_core_config = ttnn.DispatchCoreConfig(dispatch_core_type, dispatch_core_axis)
         return dispatch_core_config
+    
+    # Set fabric config to passed in value
+    # Do nothing if not set
+    # Must be called before creating the mesh device
+    def _set_fabric(self):
+        override_tt_config = self.model_config.override_tt_config
+        if override_tt_config is not None and "fabric_config" in override_tt_config:
+            fabric_config_str = override_tt_config["fabric_config"]
+            fabric_config_map = {
+                "DISABLED": ttnn.FabricConfig.DISABLED,
+                "FABRIC_1D": ttnn.FabricConfig.FABRIC_1D,
+                "FABRIC_2D": ttnn.FabricConfig.FABRIC_2D,
+                "CUSTOM": ttnn.FabricConfig.CUSTOM,
+            }
+            fabric_config = fabric_config_map.get(fabric_config_str)
+            assert fabric_config is not None, f"Invalid fabric_config: {fabric_config_str}. Expected one of {list(fabric_config_map.keys())}."
+            ttnn.initialize_fabric_config(fabric_config)
+
+    def _device_params_from_override_tt_config(self):
+        override_tt_config = self.model_config.override_tt_config
+        device_params = {}
+
+        if self.trace_mode:
+            # Set the most common value as default, override later
+            device_params["trace_region_size"] = 23887872
+            if override_tt_config and "trace_region_size" in override_tt_config:
+                device_params["trace_region_size"] = override_tt_config["trace_region_size"]
+
+        if override_tt_config and "worker_l1_size" in override_tt_config:
+            device_params["worker_l1_size"] = override_tt_config["worker_l1_size"]
+
+        return device_params
 
     def _open_mesh_device(self):
         num_devices_available = len(ttnn.get_device_ids())
@@ -411,9 +443,9 @@ def _open_mesh_device(self):
         if mesh_grid[0] * mesh_grid[1] > num_devices_available:
             assert f"Requested mesh grid shape {mesh_grid} is larger than number of available devices {num_devices_available}"
 
-        device_params = {}
-        if self.trace_mode:
-            device_params["trace_region_size"] = 23887872  # TODO: make this configurable
+        device_params = self._device_params_from_override_tt_config()
+
+        self._set_fabric()
 
         mesh_device = ttnn.open_mesh_device(
             ttnn.MeshShape(*mesh_grid),


Reviews:
user: login: skhorasganiTT
state: CHANGES_REQUESTED
author_association: NONE
submitted_at: 2025-04-23T16:32:42Z
user: login: ppetrovicTT
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-23T16:48:20Z
user: login: ppetrovicTT
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-23T16:53:10Z
user: login: skhorasganiTT
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-23T16:55:54Z
user: login: skhorasganiTT
state: CHANGES_REQUESTED
author_association: NONE
submitted_at: 2025-04-23T16:58:57Z
user: login: ppetrovicTT
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-23T17:16:03Z
user: login: skhorasganiTT
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-23T17:21:40Z
user: login: skhorasganiTT
state: COMMENTED
author_association: NONE
submitted_at: 2025-04-24T15:19:25Z
user: login: skhorasganiTT
body: Lgtm, thanks @ppetrovicTT!
state: APPROVED
author_association: NONE
submitted_at: 2025-04-24T16:22:59Z

Review comments:
diff_hunk: @@ -409,6 +409,23 @@ def _get_dispatch_core_config(self, device_params):
 
         dispatch_core_config = ttnn.DispatchCoreConfig(dispatch_core_type, dispatch_core_axis)
         return dispatch_core_config
+    
+    # Set fabric config to passed in value
+    # Do nothing if not set
+    # Must be called before creating the mesh device
+    def _set_fabric(self):
+        override_tt_config = self.model_config.override_tt_config
+        if override_tt_config is not None and "fabric_config" in override_tt_config:
+            fabric_config_str = override_tt_config["fabric_config"]
path: vllm/worker/tt_worker.py
user: login: skhorasganiTT
body: Can `override_tt_config` and the if statement be moved outside the function, and pass in `fabric_config_str` to the function?
created_at: 2025-04-23T16:30:48Z
author_association: NONE
diff_hunk: @@ -426,6 +443,10 @@ def _open_mesh_device(self):
         if self.trace_mode:
             device_params["trace_region_size"] = 23887872  # TODO: make this configurable
 
+        device_params["worker_l1_size"] = 1344544
path: vllm/worker/tt_worker.py
user: login: skhorasganiTT
body: When should this be enabled? It shouldn't be set by default right?
created_at: 2025-04-23T16:31:29Z
author_association: NONE
diff_hunk: @@ -426,6 +443,10 @@ def _open_mesh_device(self):
         if self.trace_mode:
             device_params["trace_region_size"] = 23887872  # TODO: make this configurable
 
+        device_params["worker_l1_size"] = 1344544
path: vllm/worker/tt_worker.py
user: login: ppetrovicTT
body: Right! I've noticed it too late :/
I've followed up with full override of device_params with override_tt_config.
created_at: 2025-04-23T16:48:20Z
author_association: NONE
diff_hunk: @@ -409,6 +409,23 @@ def _get_dispatch_core_config(self, device_params):
 
         dispatch_core_config = ttnn.DispatchCoreConfig(dispatch_core_type, dispatch_core_axis)
         return dispatch_core_config
+    
+    # Set fabric config to passed in value
+    # Do nothing if not set
+    # Must be called before creating the mesh device
+    def _set_fabric(self):
+        override_tt_config = self.model_config.override_tt_config
+        if override_tt_config is not None and "fabric_config" in override_tt_config:
+            fabric_config_str = override_tt_config["fabric_config"]
path: vllm/worker/tt_worker.py
user: login: ppetrovicTT
body: If I understood well, fabric doesn't have a default config. At least in tt-metal, we don't initialize it unless the config is defined. That's why I decided to keep everything encapsulated in one function, and to follow the pattern from _get_dispatch_core_config where using override_tt_config is also encapsulated.

TL;DR I prefer to keep it this way, unless you see a functional issue with it? Thanks!
created_at: 2025-04-23T16:53:10Z
author_association: NONE
diff_hunk: @@ -409,6 +409,23 @@ def _get_dispatch_core_config(self, device_params):
 
         dispatch_core_config = ttnn.DispatchCoreConfig(dispatch_core_type, dispatch_core_axis)
         return dispatch_core_config
+    
+    # Set fabric config to passed in value
+    # Do nothing if not set
+    # Must be called before creating the mesh device
+    def _set_fabric(self):
+        override_tt_config = self.model_config.override_tt_config
+        if override_tt_config is not None and "fabric_config" in override_tt_config:
+            fabric_config_str = override_tt_config["fabric_config"]
path: vllm/worker/tt_worker.py
user: login: skhorasganiTT
body: Sounds good
created_at: 2025-04-23T16:55:54Z
author_association: NONE
diff_hunk: @@ -411,9 +438,9 @@ def _open_mesh_device(self):
         if mesh_grid[0] * mesh_grid[1] > num_devices_available:
             assert f"Requested mesh grid shape {mesh_grid} is larger than number of available devices {num_devices_available}"
 
-        device_params = {}
-        if self.trace_mode:
-            device_params["trace_region_size"] = 23887872  # TODO: make this configurable
path: vllm/worker/tt_worker.py
user: login: skhorasganiTT
body: Was this removed unintentionally? Can we keep the default trace region size since it's needed by the current models?
created_at: 2025-04-23T16:58:48Z
author_association: NONE
diff_hunk: @@ -411,9 +438,9 @@ def _open_mesh_device(self):
         if mesh_grid[0] * mesh_grid[1] > num_devices_available:
             assert f"Requested mesh grid shape {mesh_grid} is larger than number of available devices {num_devices_available}"
 
-        device_params = {}
-        if self.trace_mode:
-            device_params["trace_region_size"] = 23887872  # TODO: make this configurable
path: vllm/worker/tt_worker.py
user: login: ppetrovicTT
body: open_mesh_device uses ttnn._ttnn.device.DEFAULT_TRACE_REGION_SIZE if the argument is not provided (the value is 0). since it's now configurable, i thought it should be provided from outside.
for which models is 23887872 default? text_demo is using 62000000 now.
created_at: 2025-04-23T17:16:03Z
author_association: NONE
diff_hunk: @@ -411,9 +438,9 @@ def _open_mesh_device(self):
         if mesh_grid[0] * mesh_grid[1] > num_devices_available:
             assert f"Requested mesh grid shape {mesh_grid} is larger than number of available devices {num_devices_available}"
 
-        device_params = {}
-        if self.trace_mode:
-            device_params["trace_region_size"] = 23887872  # TODO: make this configurable
path: vllm/worker/tt_worker.py
user: login: skhorasganiTT
body: I think most, if not all, the supported models need that at the moment, but will need to check if any of them are different. Can we keep the default behaviour unaffected by this PR and address the default in a separate issue/PR?
created_at: 2025-04-23T17:21:40Z
author_association: NONE
diff_hunk: @@ -63,18 +63,32 @@ To measure performance (Llama70B) for a single batch of 32 prompts (with the def
 MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py --measure_perf
 ```
 
-**Note 1 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
+**Note 1**: Custom TT options can be set using `--override_tt_config` with a json string, e.g. `--override_tt_config '{"sample_on_device_mode": "all"}'`, however these shouldn't be used unless the model supports them (most currently do not). Supported parameters are:
+- `sample_on_device_mode`: ["all", "decode_only"]
+- `trace_region_size`" [default: 23887872]
+- `worker_l1_size`
+- `fabric_config`: ["DISABLED", "FABRIC_1D", "FABRIC_2D", "CUSTOM"]
+- `dispatch_core_axis`: ["row", "col"]
 
-**Note 2 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [Llama3 demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run with the [old Llama70B implemenentation](https://github.com/tenstorrent/tt-metal/tree/main/models/demos/t3000/llama3_70b), modify the `LlamaForCausalLM` model import in [offline_inference_tt.py](https://github.com/tenstorrent/vllm/blob/dev/examples/offline_inference_tt.py) to `from models.demos.t3000.llama2_70b.tt.generator_vllm import TtLlamaForCausalLM as LlamaForCausalLM`.
+**Note 2 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
 
-**Note 3 (Other Models)**: By default, the inference example will run with Llama-3.1-70B. To run with other Llama models, or Qwen-2.5, ensure that the apprioriate environment variables are set as per the [demo instructions](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers), then set `MESH_DEVICE=<device>` (valid options for `<device>` are `N150`, `N300`, `T3K`, or `TG`) and one of the following:
+**Note 3 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [tt_transformers demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run other implementations use `TT_LLAMA_TEXT_VER` environment variable:
path: tt_metal/README.md
user: login: skhorasganiTT
body: "use the `TT_LLAMA_TEXT_VER` environment variable"
created_at: 2025-04-24T15:17:19Z
author_association: NONE
diff_hunk: @@ -63,18 +63,32 @@ To measure performance (Llama70B) for a single batch of 32 prompts (with the def
 MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py --measure_perf
 ```
 
-**Note 1 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
+**Note 1**: Custom TT options can be set using `--override_tt_config` with a json string, e.g. `--override_tt_config '{"sample_on_device_mode": "all"}'`, however these shouldn't be used unless the model supports them (most currently do not). Supported parameters are:
+- `sample_on_device_mode`: ["all", "decode_only"]
+- `trace_region_size`" [default: 23887872]
+- `worker_l1_size`
+- `fabric_config`: ["DISABLED", "FABRIC_1D", "FABRIC_2D", "CUSTOM"]
+- `dispatch_core_axis`: ["row", "col"]
 
-**Note 2 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [Llama3 demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run with the [old Llama70B implemenentation](https://github.com/tenstorrent/tt-metal/tree/main/models/demos/t3000/llama3_70b), modify the `LlamaForCausalLM` model import in [offline_inference_tt.py](https://github.com/tenstorrent/vllm/blob/dev/examples/offline_inference_tt.py) to `from models.demos.t3000.llama2_70b.tt.generator_vllm import TtLlamaForCausalLM as LlamaForCausalLM`.
+**Note 2 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
 
-**Note 3 (Other Models)**: By default, the inference example will run with Llama-3.1-70B. To run with other Llama models, or Qwen-2.5, ensure that the apprioriate environment variables are set as per the [demo instructions](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers), then set `MESH_DEVICE=<device>` (valid options for `<device>` are `N150`, `N300`, `T3K`, or `TG`) and one of the following:
+**Note 3 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [tt_transformers demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run other implementations use `TT_LLAMA_TEXT_VER` environment variable:
+- "llama3_subdevices" for the Llama TG
path: tt_metal/README.md
user: login: skhorasganiTT
body: `"llama3_subdevices"` for the Llama TG implementation
created_at: 2025-04-24T15:17:34Z
author_association: NONE
diff_hunk: @@ -63,18 +63,32 @@ To measure performance (Llama70B) for a single batch of 32 prompts (with the def
 MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py --measure_perf
 ```
 
-**Note 1 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
+**Note 1**: Custom TT options can be set using `--override_tt_config` with a json string, e.g. `--override_tt_config '{"sample_on_device_mode": "all"}'`, however these shouldn't be used unless the model supports them (most currently do not). Supported parameters are:
+- `sample_on_device_mode`: ["all", "decode_only"]
+- `trace_region_size`" [default: 23887872]
+- `worker_l1_size`
+- `fabric_config`: ["DISABLED", "FABRIC_1D", "FABRIC_2D", "CUSTOM"]
+- `dispatch_core_axis`: ["row", "col"]
 
-**Note 2 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [Llama3 demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run with the [old Llama70B implemenentation](https://github.com/tenstorrent/tt-metal/tree/main/models/demos/t3000/llama3_70b), modify the `LlamaForCausalLM` model import in [offline_inference_tt.py](https://github.com/tenstorrent/vllm/blob/dev/examples/offline_inference_tt.py) to `from models.demos.t3000.llama2_70b.tt.generator_vllm import TtLlamaForCausalLM as LlamaForCausalLM`.
+**Note 2 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
 
-**Note 3 (Other Models)**: By default, the inference example will run with Llama-3.1-70B. To run with other Llama models, or Qwen-2.5, ensure that the apprioriate environment variables are set as per the [demo instructions](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers), then set `MESH_DEVICE=<device>` (valid options for `<device>` are `N150`, `N300`, `T3K`, or `TG`) and one of the following:
+**Note 3 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [tt_transformers demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run other implementations use `TT_LLAMA_TEXT_VER` environment variable:
+- "llama3_subdevices" for the Llama TG
+- "llama2_70b" for the old Llama implementation
path: tt_metal/README.md
user: login: skhorasganiTT
body: `"llama2_70b"`
created_at: 2025-04-24T15:17:43Z
author_association: NONE
diff_hunk: @@ -63,18 +63,32 @@ To measure performance (Llama70B) for a single batch of 32 prompts (with the def
 MESH_DEVICE=T3K WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml python examples/offline_inference_tt.py --measure_perf
 ```
 
-**Note 1 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
+**Note 1**: Custom TT options can be set using `--override_tt_config` with a json string, e.g. `--override_tt_config '{"sample_on_device_mode": "all"}'`, however these shouldn't be used unless the model supports them (most currently do not). Supported parameters are:
+- `sample_on_device_mode`: ["all", "decode_only"]
+- `trace_region_size`" [default: 23887872]
+- `worker_l1_size`
+- `fabric_config`: ["DISABLED", "FABRIC_1D", "FABRIC_2D", "CUSTOM"]
+- `dispatch_core_axis`: ["row", "col"]
 
-**Note 2 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [Llama3 demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run with the [old Llama70B implemenentation](https://github.com/tenstorrent/tt-metal/tree/main/models/demos/t3000/llama3_70b), modify the `LlamaForCausalLM` model import in [offline_inference_tt.py](https://github.com/tenstorrent/vllm/blob/dev/examples/offline_inference_tt.py) to `from models.demos.t3000.llama2_70b.tt.generator_vllm import TtLlamaForCausalLM as LlamaForCausalLM`.
+**Note 2 (Llama70B)**: To run Llama70B on Galaxy, set `MESH_DEVICE=TG` and do not set `WH_ARCH_YAML=...`.
 
-**Note 3 (Other Models)**: By default, the inference example will run with Llama-3.1-70B. To run with other Llama models, or Qwen-2.5, ensure that the apprioriate environment variables are set as per the [demo instructions](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers), then set `MESH_DEVICE=<device>` (valid options for `<device>` are `N150`, `N300`, `T3K`, or `TG`) and one of the following:
+**Note 3 (Llama70B)**: By default, this will run the newer tt-metal implementation of Llama70B from the [tt_transformers demo](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers). To run other implementations use `TT_LLAMA_TEXT_VER` environment variable:
+- "llama3_subdevices" for the Llama TG
+- "llama2_70b" for the old Llama implementation
+
+**Note 4 (Other Models)**: By default, the inference example will run with Llama-3.1-70B. To run with other Llama models, or Qwen-2.5, ensure that the apprioriate environment variables are set as per the [demo instructions](https://github.com/tenstorrent/tt-metal/tree/main/models/tt_transformers), then set `MESH_DEVICE=<device>` (valid options for `<device>` are `N150`, `N300`, `T3K`, or `TG`) and one of the following:
 - Llama-3.1-8B: `--model "meta-llama/Llama-3.1-8B"`
 - Llama-3.2-1B: `--model "meta-llama/Llama-3.2-1B"`
 - Llama-3.2-3B: `--model "meta-llama/Llama-3.2-3B"`
 - Qwen-2.5-7B: `--model "Qwen/Qwen2.5-7B"` (currently only supported on N300)
 - Qwen-2.5-72B: `--model "Qwen/Qwen2.5-72B"` (currently only supported on T3K)
 - DeepSeek-R1-Distill-Llama-70B: `--model "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"`
 
+Command line to run Llama70B on TG is:
+```python
path: tt_metal/README.md
user: login: skhorasganiTT
body: python -> sh. Could you please also change the other incorrect usages of python to sh?
created_at: 2025-04-24T15:18:16Z
author_association: NONE

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-23T14:50:58Z
body: 👋 Hi! Thank you for contributing to the vLLM project.
 Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org. 

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

 To run CI, PR reviewers can do one of these:
- Add `ready` label to the PR
- Enable auto-merge.

🚀
author_association: NONE
user: login: skhorasganiTT
created_at: 2025-04-23T17:04:29Z
body: Would you also be able to update the readme note about override tt config  (https://github.com/tenstorrent/vllm/tree/dev/tt_metal#running-the-server-example) in this PR to state all the possible new options?
author_association: NONE
user: login: tstescoTT
created_at: 2025-04-23T20:20:12Z
body: Is this supposed to support 4U TG or 6U Galaxy?

Running with tt-metal [a313c5db24](https://github.com/tenstorrent/tt-metal/commit/a313c5db246bb072e3398d8e9b3196667e7adcf4) I got:
```
2025-04-23 20:13:38.523 | INFO     | models.demos.llama3_subdevices.tt.generator:prefill_forward_text:81 - Prefilling User 1
                  Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 6,14. First unused index: 2. Kernels: writer_unary_interleaved_start_id_blocked, reader_unary_interleaved_ln_rm_gb_pre_allgather, layernorm_pre_allgather
TT_THROW @ /home/user/software/metal-versions/a313c5d/tt-metal/tt_metal/fabric/fabric_host_utils.cpp:36: tt::exception
info:
Unsupported fabric config for 1D: DISABLED
backtrace:
 --- /home/user/software/metal-versions/a313c5d/tt-metal/ttnn/ttnn/_ttnn.so(+0xa7f470) [0x7f153e228470]
 --- /home/user/software/metal-versions/a313c5d/tt-metal/build_Release/lib/libtt_metal.so(+0x40f3c4) [0x7f153d1a93c4]
 --- tt::tt_fabric::get_1d_fabric_config()
...
```
author_association: NONE
user: login: skhorasganiTT
created_at: 2025-04-24T15:20:23Z
body: @ppetrovicTT please see @tstescoTT's question above
author_association: NONE
user: login: ppetrovicTT
created_at: 2025-04-24T16:12:03Z
body: 


> Is this supposed to support 4U TG or 6U Galaxy?
> 
> Running with tt-metal [a313c5db24](https://github.com/tenstorrent/tt-metal/commit/a313c5db246bb072e3398d8e9b3196667e7adcf4) I got:
> 
> ```
> 2025-04-23 20:13:38.523 | INFO     | models.demos.llama3_subdevices.tt.generator:prefill_forward_text:81 - Prefilling User 1
>                   Metal | WARNING  | Circular buffer indices are not contiguous starting at 0. This will hurt dispatch performance. Non-contiguous indices: 6,14. First unused index: 2. Kernels: writer_unary_interleaved_start_id_blocked, reader_unary_interleaved_ln_rm_gb_pre_allgather, layernorm_pre_allgather
> TT_THROW @ /home/user/software/metal-versions/a313c5d/tt-metal/tt_metal/fabric/fabric_host_utils.cpp:36: tt::exception
> info:
> Unsupported fabric config for 1D: DISABLED
> backtrace:
>  --- /home/user/software/metal-versions/a313c5d/tt-metal/ttnn/ttnn/_ttnn.so(+0xa7f470) [0x7f153e228470]
>  --- /home/user/software/metal-versions/a313c5d/tt-metal/build_Release/lib/libtt_metal.so(+0x40f3c4) [0x7f153d1a93c4]
>  --- tt::tt_fabric::get_1d_fabric_config()
> ...
> ```

You need to use this config for Llama TG (you're missing fabric_config, hence the error):
`--override_tt_config '{"dispatch_core_axis": "col", "sample_on_device_mode": "all", "fabric_config": "FABRIC_1D", "worker_l1_size": 1344544, "trace_region_size": 62000000}'`
author_association: NONE