Repository: tenstorrent/tt-metal
Pull request #19857: #19790: Add an option to enable watcher in blackhole post-commit 
Author: login: williamlyTT
Created: 2025-03-28T17:16:46Z | Merged: 2025-04-14T20:39:09Z

### Ticket
Resolves #19790

### Problem description
Currently watcher is not enabled in BH post commit.
We want to enable it on workflow dispatch to debug ND behavior such as hangs.

### What's changed
Add an option `inputs.enable-watcher` to blackhole post commit workflows + impl reusables. This is defaulted to false (watcher not enabled).
Enabling sets the following env vars
```
TT_METAL_WATCHER=1 
TT_METAL_WATCHER_APPEND=1
TT_METAL_WATCHER_NOINLINE=1
```

### Checklist
- [x] [All post commit](https://github.com/tenstorrent/tt-metal/actions/workflows/all-post-commit-workflows.yaml) CI passes
https://github.com/tenstorrent/tt-metal/actions/runs/14175581294
- [x] [Blackhole Post commit](https://github.com/tenstorrent/tt-metal/actions/workflows/blackhole-post-commit.yaml) CI 
watcher enabled:  https://github.com/tenstorrent/tt-metal/actions/runs/14384446486

Linked issue #19790: Add an option to enable watcher for BH Post Commit
Watcher has a lot of features that allow it to catch various bugs.

Enabling it also sometimes changes timing slightly that helps debug various issues.

Therefore at the very least it would be good to have an option to enable it.

This request is specifically for BH Post Commit https://github.com/tenstorrent/tt-metal/actions/workflows/blackhole-post-commit.yaml hopefully to help with root causing an ND hang, although it might be useful in APC or other pipelines as well.

Diff:
diff --git a/.github/actions/setup-job/action.yml b/.github/actions/setup-job/action.yml
index 251c2c48bb1d..dc99a0672566 100644
--- a/.github/actions/setup-job/action.yml
+++ b/.github/actions/setup-job/action.yml
@@ -13,6 +13,10 @@ inputs:
     required: false
     description: Where to checkout
     default: docker-job
+  enable-watcher:
+    description: 'Enable watcher'
+    default: false
+    type: boolean
 runs:
   using: "composite"
   steps:
@@ -60,3 +64,13 @@ runs:
         echo "📦 Installing $WHEEL_FILENAME"
         pip3 install "$WHEEL_FILENAME"
       shell: bash
+
+    - name: Set up env vars for watcher
+      if: ${{ inputs.enable-watcher == 'true' }}
+      working-directory: ${{ inputs.path }}
+      run: |
+        echo "Enabling TT Metal Watcher"
+        echo "TT_METAL_WATCHER=1" >> $GITHUB_ENV
+        echo "TT_METAL_WATCHER_APPEND=1" >> $GITHUB_ENV
+        echo "TT_METAL_WATCHER_NOINLINE=1" >> $GITHUB_ENV
+      shell: bash
diff --git a/.github/workflows/blackhole-post-commit.yaml b/.github/workflows/blackhole-post-commit.yaml
index a8a562f4ce76..adc850f56444 100644
--- a/.github/workflows/blackhole-post-commit.yaml
+++ b/.github/workflows/blackhole-post-commit.yaml
@@ -8,6 +8,10 @@ on:
           required: false
           type: string
           default: 'BH'
+      enable-watcher:
+          description: 'Enable watcher in BH Post commit'
+          default: false
+          type: boolean
   workflow_dispatch:
     inputs:
       runner-label:
@@ -25,12 +29,18 @@ on:
           - RelWithDebInfo
           - ASan
           - TSan
+      enable-watcher:
+        description: 'Enable watcher in BH Post commit'
+        default: false
+        type: boolean
   schedule:
     - cron: "0 */4 * * *"
   # Pause this since not enough runners to support every commit to main
   # push:
   #  branches: ["main"]
 
+run-name: ${{ inputs.enable-watcher == true && 'Blackhole post-commit tests (watcher enabled) ' || 'Blackhole post-commit tests' }}
+
 permissions:
   actions: read
   contents: write
@@ -66,6 +76,7 @@ jobs:
       docker-image: ${{ needs.build-artifact-profiler.outputs.dev-docker-image }}
       build-artifact-name: ${{ needs.build-artifact-profiler.outputs.build-artifact-name }}
       wheel-artifact-name: ${{ needs.build-artifact-profiler.outputs.wheel-artifact-name }}
+      enable-watcher: ${{ inputs.enable-watcher }}
   umd-unit-tests:
     needs: build-artifact
     secrets: inherit
@@ -85,6 +96,7 @@ jobs:
       docker-image: ${{ needs.build-artifact.outputs.dev-docker-image }}
       build-artifact-name: ${{ needs.build-artifact.outputs.build-artifact-name }}
       wheel-artifact-name: ${{ needs.build-artifact.outputs.wheel-artifact-name }}
+      enable-watcher: ${{ inputs.enable-watcher }}
   fd-unit-tests:
     needs: build-artifact
     uses: ./.github/workflows/fast-dispatch-build-and-unit-tests.yaml
@@ -95,6 +107,7 @@ jobs:
       runner-label: ${{ inputs.runner-label || 'BH' }}
       docker-image: ${{ needs.build-artifact.outputs.dev-docker-image }}
       wheel-artifact-name: ${{ needs.build-artifact.outputs.wheel-artifact-name }}
+      enable-watcher: ${{ inputs.enable-watcher }}
   # FD C++ Unit Tests
   cpp-unit-tests:
     needs: build-artifact
@@ -107,6 +120,7 @@ jobs:
       docker-image: ${{ needs.build-artifact.outputs.dev-docker-image }}
       build-artifact-name: ${{ needs.build-artifact.outputs.build-artifact-name }}
       wheel-artifact-name: ${{ needs.build-artifact.outputs.wheel-artifact-name }}
+      enable-watcher: ${{ inputs.enable-watcher }}
   models-unit-tests:
     needs: build-artifact
     secrets: inherit
@@ -117,6 +131,7 @@ jobs:
       timeout: 20
       docker-image: ${{ needs.build-artifact.outputs.dev-docker-image }}
       wheel-artifact-name: ${{ needs.build-artifact.outputs.wheel-artifact-name }}
+      enable-watcher: ${{ inputs.enable-watcher }}
 
 #   profiler-regression:
 #     needs: build-artifact-profiler
diff --git a/.github/workflows/build-and-unit-tests.yaml b/.github/workflows/build-and-unit-tests.yaml
index 3c905a271cd7..f435cdd7a9c2 100644
--- a/.github/workflows/build-and-unit-tests.yaml
+++ b/.github/workflows/build-and-unit-tests.yaml
@@ -22,6 +22,10 @@ on:
       wheel-artifact-name:
         required: true
         type: string
+      enable-watcher:
+        description: 'Enable watcher'
+        default: false
+        type: boolean
 
 jobs:
   cpp-unit-tests-slow-dispatch:
@@ -74,6 +78,7 @@ jobs:
           build-artifact-name: ${{ inputs.build-artifact-name }}
           # Unsure why these C++ tests need the wheel...  a cleanup to do separately
           wheel-artifact-name: ${{ inputs.wheel-artifact-name }}
+          enable-watcher: ${{ inputs.enable-watcher }}
       - name: ${{ matrix.test-group.name }} tests
         timeout-minutes: ${{ inputs.timeout }}
         run: |
diff --git a/.github/workflows/cpp-post-commit.yaml b/.github/workflows/cpp-post-commit.yaml
index b3ff54c77d57..f5a78903aa82 100644
--- a/.github/workflows/cpp-post-commit.yaml
+++ b/.github/workflows/cpp-post-commit.yaml
@@ -22,6 +22,10 @@ on:
       wheel-artifact-name:
         required: true
         type: string
+      enable-watcher:
+        description: 'Enable watcher'
+        default: false
+        type: boolean
 
 jobs:
   cpp-unit-tests:
@@ -81,6 +85,7 @@ jobs:
           # Unsure why these C++ tests need the wheel...  a cleanup to do separately
           # https://github.com/tenstorrent/tt-metal/issues/20166
           wheel-artifact-name: ${{ inputs.wheel-artifact-name }}
+          enable-watcher: ${{ inputs.enable-watcher }}
       - name: ${{ matrix.test-group.name }} tests
         #GH Issue 16167
         if: ${{ !(inputs.runner-label == 'BH' && matrix.test-group.name == 'tools') }}
diff --git a/.github/workflows/fast-dispatch-build-and-unit-tests.yaml b/.github/workflows/fast-dispatch-build-and-unit-tests.yaml
index 62a4488e84c0..c39ef0ecd30a 100644
--- a/.github/workflows/fast-dispatch-build-and-unit-tests.yaml
+++ b/.github/workflows/fast-dispatch-build-and-unit-tests.yaml
@@ -19,6 +19,10 @@ on:
       wheel-artifact-name:
         required: true
         type: string
+      enable-watcher:
+        description: 'Enable watcher'
+        default: false
+        type: boolean
 
 jobs:
   fd-tests:
@@ -62,6 +66,7 @@ jobs:
         timeout-minutes: 10
         with:
           wheel-artifact-name: ${{ inputs.wheel-artifact-name }}
+          enable-watcher: ${{ inputs.enable-watcher }}
       - name: ${{ matrix.test-group.name }} tests
         timeout-minutes: ${{ inputs.timeout }}
         run: |
diff --git a/.github/workflows/models-post-commit.yaml b/.github/workflows/models-post-commit.yaml
index 493c79ed98aa..e66f74035d32 100644
--- a/.github/workflows/models-post-commit.yaml
+++ b/.github/workflows/models-post-commit.yaml
@@ -19,6 +19,10 @@ on:
       wheel-artifact-name:
         required: true
         type: string
+      enable-watcher:
+        description: 'Enable watcher'
+        default: false
+        type: boolean
 
 jobs:
   models:
@@ -55,6 +59,7 @@ jobs:
         timeout-minutes: 10
         with:
           wheel-artifact-name: ${{ inputs.wheel-artifact-name }}
+          enable-watcher: ${{ inputs.enable-watcher }}
       - name: ${{ matrix.test-group.name }} tests
         timeout-minutes: ${{ inputs.timeout }}
         run: |
@@ -106,6 +111,7 @@ jobs:
         timeout-minutes: 10
         with:
           wheel-artifact-name: ${{ inputs.wheel-artifact-name }}
+          enable-watcher: ${{ inputs.enable-watcher }}
       - name: ${{ matrix.test-group.name }} slow runtime mode tests
         timeout-minutes: ${{ inputs.timeout }}
         run: |
diff --git a/.github/workflows/run-profiler-regression.yaml b/.github/workflows/run-profiler-regression.yaml
index 643e994cf0a0..32eed3963ebe 100644
--- a/.github/workflows/run-profiler-regression.yaml
+++ b/.github/workflows/run-profiler-regression.yaml
@@ -22,6 +22,10 @@ on:
       docker-image:
         required: true
         type: string
+      enable-watcher:
+        description: 'Enable watcher'
+        default: false
+        type: boolean
   workflow_dispatch:
     inputs:
       arch:
@@ -50,6 +54,10 @@ on:
       docker-image:
         required: true
         type: string
+      enable-watcher:
+        description: 'Enable watcher'
+        default: false
+        type: boolean
 
 jobs:
   profiler-regression:
@@ -93,6 +101,7 @@ jobs:
         with:
           build-artifact-name: ${{ inputs.build-artifact-name }}
           wheel-artifact-name: ${{ inputs.wheel-artifact-name }}
+          enable-watcher: ${{ inputs.enable-watcher }}
       - name: Run profiler regression tests
         timeout-minutes: ${{ inputs.timeout }}
         run: |


Reviews:
user: login: bbradelTT
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-31T16:09:32Z
user: login: tt-rkim
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-09T16:08:23Z
user: login: williamlyTT
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T16:44:53Z
user: login: afuller-TT
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T19:43:13Z

Review comments:
diff_hunk: @@ -81,6 +85,12 @@ jobs:
           # Unsure why these C++ tests need the wheel...  a cleanup to do separately
           # https://github.com/tenstorrent/tt-metal/issues/20166
           wheel-artifact-name: ${{ inputs.wheel-artifact-name }}
+      - name: Set up env vars for watcher
+        if: inputs.enable-watcher == true
+        run: |
+          echo "TT_METAL_WATCHER=1" >> $GITHUB_ENV
path: .github/workflows/cpp-post-commit.yaml
user: login: tt-rkim
body: HRMMM can we hide this in `tenstorrent/tt-metal/.github/actions/setup-job` ? Although would be hard to test without changing the `@main`...
created_at: 2025-04-09T16:08:20Z
author_association: COLLABORATOR
diff_hunk: @@ -81,6 +85,12 @@ jobs:
           # Unsure why these C++ tests need the wheel...  a cleanup to do separately
           # https://github.com/tenstorrent/tt-metal/issues/20166
           wheel-artifact-name: ${{ inputs.wheel-artifact-name }}
+      - name: Set up env vars for watcher
+        if: inputs.enable-watcher == true
+        run: |
+          echo "TT_METAL_WATCHER=1" >> $GITHUB_ENV
path: .github/workflows/cpp-post-commit.yaml
user: login: williamlyTT
body: Yeah let me do that, once we move away from `docker-run` in the other 2 workflows we don't need the two different ways to set watcher env vars (`GITHUB_ENV` vs `docker-opts`) so it makes sense to just put it in one place.

It should be straightforward to test as long as I revert-me a commit hash to try it.
created_at: 2025-04-09T16:44:52Z
author_association: CONTRIBUTOR

PR comments:
user: login: williamlyTT
created_at: 2025-03-28T17:20:55Z
body: @bbradelTT are there any other watcher-specific env vars that you think would be good to have set?
author_association: CONTRIBUTOR
user: login: bbradelTT
created_at: 2025-03-28T19:39:26Z
body: You need to add
```
TT_METAL_WATCHER_NOINLINE=1
```
otherwise compiling kernels will fail due to taking too much space.
author_association: CONTRIBUTOR
user: login: williamlyTT
created_at: 2025-04-01T18:47:01Z
body: Seeing 3 failures on BH with watcher enabled:

Profiler regressions: https://github.com/tenstorrent/tt-metal/actions/runs/14175594951/job/39778488754 (appears to be same error as https://github.com/tenstorrent/tt-metal/pull/19456,  with watcher it is probably too large to fit in `MEM_BRISC_FIRMWARE_SIZE`
If this isn't a concern we can ignore it or choose not to enable watcher for profiler regression tests.

sd unit tests - api tests: https://github.com/tenstorrent/tt-metal/actions/runs/14175594951/job/39778491123
```
[ RUN      ] DeviceFixture.TensixIncrementStreamRegWrite
2025-04-01 14:34:00.079 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.31.0, IOMMU: disabled
                  Metal | INFO     | Initializing device 0. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 0 is:   1350 MHz
              LLRuntime | INFO     | Watcher log file: /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/generated/watcher/watcher.log
              LLRuntime | INFO     | Watcher attached device 0
              LLRuntime | INFO     | Watcher server initialized, disabled features: None
              LLRuntime | INFO     | Watcher checking device 0
                 Always | WARNING  | Watcher stopped the device due to tripped assert, see watcher log for more details
                 Always | WARNING  | Device 0 worker core(x= 0,y= 0) virtual(x= 1,y= 2): brisc tripped an assert on line 59. Current kernel: tests/tt_metal/tt_metal/test_kernels/dataflow/streams/stream_increment_reg_write.cpp. Note that file name reporting is not yet implemented, and the reported line number for the assert may be from a different file.
                 Always | INFO     | Last waypoint: NKFW,   W,   W,   W,   W 
                 Always | INFO     | While running kernels:
                 Always | INFO     |  brisc : tests/tt_metal/tt_metal/test_kernels/dataflow/streams/stream_increment_reg_write.cpp
                 Always | INFO     |  ncrisc: blank
                 Always | INFO     |  triscs: blank
terminate called after throwing an instance of 'std::runtime_error'
  what():  TT_THROW @ /work/tt_metal/impl/debug/watcher_device_reader.cpp:612: tt::exception
info:
Watcher detected tripped assert and stopped device.
backtrace:
 --- ./build/test/tt_metal/unit_tests_api_blackhole(+0xc7ce8) [0x5644505c6ce8]
 --- tt::watcher::WatcherDeviceReader::DumpAssertStatus(CoreDescriptor&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mailboxes_t const*)
 --- tt::watcher::WatcherDeviceReader::DumpCore(CoreDescriptor&, bool)
 --- tt::watcher::WatcherDeviceReader::Dump(_IO_FILE*)
 --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2bd09c) [0x7f8cc234609c]
 --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2bc792) [0x7f8cc2345792]
 --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f8cc197a253]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f8cc1600ac3]
 --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f8cc1691a04]
```

cpp unit tests - all c++ tests: https://github.com/tenstorrent/tt-metal/actions/runs/14175594951/job/39778492232
```
2025-04-01 14:37:46.051 | INFO     | __main__:run_single_programming_example:205 - RUNNING PROGRAMMING EXAMPLE - TestEntry(test_name='programming_examples/pad_multi_core', executable_name='programming_examples/pad_multi_core', extra_params='')
2025-04-01 14:37:46.054 | WARNING  | tests.scripts.common:build_executable_command_for_test:129 - tt-arch not implemented for programming_example-type tests
                 Device | INFO     | Opening user mode device driver
2025-04-01 14:37:46.063 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.31.0, IOMMU: disabled
2025-04-01 14:37:46.244 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.31.0, IOMMU: disabled
2025-04-01 14:37:46.424 | INFO     | SiliconDriver   - Harvesting mask for chip 0 is 0x0 (physical layout: 0x0, logical: 0x0, simulated harvesting mask: 0x0).
2025-04-01 14:37:46.433 | INFO     | SiliconDriver   - Opened PCI device 0; KMD version: 1.31.0, IOMMU: disabled
2025-04-01 14:37:46.535 | INFO     | SiliconDriver   - Detected PCI devices: [0]
2025-04-01 14:37:46.535 | INFO     | SiliconDriver   - Using local chip ids: {0} and remote chip ids {}
2025-04-01 14:37:46.882 | WARNING  | SiliconDriver   - hwloc_set_area_membind(): failed for physical_device_id: 0 on NodeSet: [0] with errno: Operation not permitted (pid: 4421 tid: 139824324783424)
2025-04-01 14:37:46.882 | WARNING  | SiliconDriver   - ---- ttSiliconDevice::init_hugepage: bind_area_to_memory_nodeset() failed (physical_device_id: 0 ch: 0). Hugepage allocation is not on NumaNode matching TT Device. Side-Effect is decreased Device->Host perf (Issue #893).
2025-04-01 14:37:46.885 | INFO     | SiliconDriver   - Device: 0 Mapped iATU region 0 from 0x0 to 0x3fffffff to 0xf80000000
                  Metal | INFO     | Initializing device 0. Program cache is NOT enabled
                  Metal | INFO     | AI CLK for device 0 is:   800 MHz
              LLRuntime | INFO     | Watcher log file: /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/generated/watcher/watcher.log
              LLRuntime | INFO     | Watcher attached device 0
              LLRuntime | INFO     | Watcher server initialized, disabled features: None
Padding tensor of shape (8, 4) to shape (8, 8) with pad value: 2
Original tensor with shape (8, 4):
1 2 3 4 
5 6 7 8 
9 10 11 12 
13 14 15 16 
17 18 19 20 
21 22 23 24 
25 26 27 28 
29 30 31 32 

              LLRuntime | INFO     | Watcher checking device 0
                 Always | WARNING  | Watcher detected NOC error and stopped device:
                 Always | WARNING  | Device 0 worker core(x= 0,y= 0) virtual(x= 1,y= 2): brisc using noc0 tried to unicast read 4 bytes to local L1[0x017c04] from DRAM core w/ virtual coords (x=0,y=11) DRAM[addr=0x00000040] (invalid address alignment in NOC transaction).
                 Always | INFO     | Last waypoint: NARW,CWFW,   W,   W,   W 
                 Always | INFO     | While running kernels:
                 Always | INFO     |  brisc : tt_metal/programming_examples/pad/kernels/pad_reader_dims_rm_interleaved.cpp
                 Always | INFO     |  ncrisc: tt_metal/programming_examples/pad/kernels/pad_writer_dims_rm_interleaved.cpp
                 Always | INFO     |  triscs: blank
terminate called after throwing an instance of 'std::runtime_error'
  what():  TT_THROW @ /work/tt_metal/impl/debug/watcher_device_reader.cpp:585: tt::exception
info:
Device 0 worker core(x= 0,y= 0) virtual(x= 1,y= 2): brisc using noc0 tried to unicast read 4 bytes to local L1[0x017c04] from DRAM core w/ virtual coords (x=0,y=11) DRAM[addr=0x00000040] (invalid address alignment in NOC transaction).
backtrace:
 --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x1e4319) [0x7f2b642c5319]
 --- tt::watcher::WatcherDeviceReader::DumpNocSanitizeStatus(CoreDescriptor&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mailboxes_t const*, int)
 --- tt::watcher::WatcherDeviceReader::DumpCore(CoreDescriptor&, bool)
 --- tt::watcher::WatcherDeviceReader::Dump(_IO_FILE*)
 --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2bd09c) [0x7f2b6439e09c]
 --- /home/ubuntu/actions-runner/_work/tt-metal/tt-metal/build/lib/libtt_metal.so(+0x2bc792) [0x7f2b6439d792]
 --- /lib/x86_64-linux-gnu/libstdc++.so.6(+0xdc253) [0x7f2b638eb253]
 --- /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7f2b63658ac3]
 --- /lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7f2b636e9a04]

timeout: the monitored command dumped core
Aborted
```
author_association: CONTRIBUTOR
user: login: bbradelTT
created_at: 2025-04-01T20:47:10Z
body: Created the following for the 3 failures
#20005 
#20006 
#20007 
author_association: CONTRIBUTOR
user: login: williamlyTT
created_at: 2025-04-10T18:54:00Z
body: FYI @bbradelTT I'm still seeing https://github.com/tenstorrent/tt-metal/issues/20006 on rebased main to April 9
https://github.com/tenstorrent/tt-metal/actions/runs/14384446486/job/40336590195#step:5:396
author_association: CONTRIBUTOR
user: login: bbradelTT
created_at: 2025-04-10T20:12:17Z
body: > FYI @bbradelTT I'm still seeing #20006 on rebased main to April 9
> https://github.com/tenstorrent/tt-metal/actions/runs/14384446486/job/40336590195#step:5:396

@jbaumanTT can you please take a look?
https://github.com/tenstorrent/tt-metal/issues/20006 might need to be re-opened, or a new issue created.
author_association: CONTRIBUTOR