Repository: tenstorrent/tt-torch
Pull request #799: Improve Docker image handling in CI workflows
Author: login: vmilosevic
Created: 2025-05-20T10:28:36Z | Merged: 2025-05-20T13:27:43Z

### Ticket
Closes: #787

### Problem description
Improve Docker image handling in CI workflows

### What's changed
- separate image building from "latest" tag management
- add "set-latest" job that runs on push to main branch
- replace latest tag with output from build job
- remove unnecessary git submodule cleanup step from build process

Cleanup
- revert changes from https://github.com/tenstorrent/tt-torch/pull/799 and remove submodule checkout and lfs

### Checklist
- [ ] New/Existing tests provide coverage for changes



Diff:
diff --git a/.github/build-docker-images.sh b/.github/build-docker-images.sh
index 30a064544..fc728b4ad 100755
--- a/.github/build-docker-images.sh
+++ b/.github/build-docker-images.sh
@@ -13,14 +13,9 @@ CI_IMAGE_NAME=ghcr.io/$REPO/tt-torch-ci-ubuntu-22-04
 DOCKER_TAG=$(./.github/get-docker-tag.sh)
 echo "Docker tag: $DOCKER_TAG"
 
-# Are we on main branch
-ON_MAIN=$(git branch --show-current | grep -q main && echo "true" || echo "false")
-
 build_and_push() {
     local image_name=$1
     local dockerfile=$2
-    local on_main=$3
-
 
     if docker manifest inspect $image_name:$DOCKER_TAG > /dev/null; then
         echo "Image $image_name:$DOCKER_TAG already exists"
@@ -30,22 +25,15 @@ build_and_push() {
             --progress=plain \
             --build-arg FROM_TAG=$DOCKER_TAG \
             -t $image_name:$DOCKER_TAG \
-            -t $image_name:latest \
             -f $dockerfile .
 
         echo "Pushing image $image_name:$DOCKER_TAG"
         docker push $image_name:$DOCKER_TAG
-
-        # If we are on main branch also push the latest tag
-        if [ "$on_main" = "true" ]; then
-            echo "Pushing image $image_name:latest"
-            docker push $image_name:latest
-        fi
     fi
 }
 
-build_and_push $BASE_IMAGE_NAME .github/Dockerfile.base $ON_MAIN
-build_and_push $CI_IMAGE_NAME .github/Dockerfile.ci $ON_MAIN
+build_and_push $BASE_IMAGE_NAME .github/Dockerfile.base
+build_and_push $CI_IMAGE_NAME .github/Dockerfile.ci
 
 echo "All images built and pushed successfully"
 echo "CI_IMAGE_NAME:"
diff --git a/.github/workflows/build-image.yml b/.github/workflows/build-image.yml
index e2dd4f833..4e45e61aa 100644
--- a/.github/workflows/build-image.yml
+++ b/.github/workflows/build-image.yml
@@ -35,7 +35,6 @@ jobs:
 
       - uses: actions/checkout@v4
         with:
-            submodules: recursive
             fetch-depth: 0 # Fetch all history and tags
 
       - name: Override tt-mlir SHA mlir_override is set
@@ -45,13 +44,6 @@ jobs:
             # Update the CMakeLists.txt file with the new SHA
             sed -i "s/set(TT_MLIR_VERSION \".*\")/set(TT_MLIR_VERSION \"${{ inputs.mlir_override }}\")/" third_party/CMakeLists.txt
 
-      # Clean everything from submodules (needed to avoid issues
-      # with cmake generated files leftover from previous builds)
-      - name: Cleanup submodules
-        run: |
-          git submodule foreach --recursive git clean -ffdx
-          git submodule foreach --recursive git reset --hard
-
       - name: Log in to GitHub Container Registry
         uses: docker/login-action@v3
         with:
@@ -69,3 +61,23 @@ jobs:
           DOCKER_CI_IMAGE=$(tail -n 1 docker.log)
           echo "DOCKER_CI_IMAGE $DOCKER_CI_IMAGE"
           echo "docker-image=$DOCKER_CI_IMAGE" >> "$GITHUB_OUTPUT"
+
+  set-latest:
+    # Set the latest tag on the CI image
+    runs-on: ubuntu-latest
+    needs: build-image
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+    steps:
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Set latest tag on the image
+        run: |
+          DOCKER_CI_IMAGE=${{ needs.build-image.outputs.docker-image }}
+          DOCKER_TAG=$(echo $DOCKER_CI_IMAGE | sed 's/^.*://')
+          IMAGE_NAME=$(echo $DOCKER_CI_IMAGE | sed 's/:.*//')
+          echo "Setting latest tag on the image $IMAGE_NAME:$DOCKER_TAG"
+          skopeo copy "docker://$IMAGE_NAME:$DOCKER_TAG" "docker://$IMAGE_NAME:latest"
diff --git a/.github/workflows/generate-benchmark-matrix.yml b/.github/workflows/generate-benchmark-matrix.yml
index 145c6bd3f..56b8739e9 100644
--- a/.github/workflows/generate-benchmark-matrix.yml
+++ b/.github/workflows/generate-benchmark-matrix.yml
@@ -12,7 +12,13 @@ on:
     types: [completed]
 
 jobs:
+
+  docker-build:
+    uses: ./.github/workflows/build-image.yml
+    secrets: inherit
+
   tests:
+    needs: docker-build
     outputs:
        matrix-json-splits: ${{ steps.generate_matrix.outputs.matrix-json-splits }}
 
@@ -24,7 +30,7 @@ jobs:
       - wormhole_b0
 
     container:
-      image: ghcr.io/tenstorrent/tt-torch/tt-torch-ci-ubuntu-22-04:latest
+      image: ${{ needs.docker-build.outputs.docker-image }}
       options: --user root --device /dev/tenstorrent/0
       volumes:
         - /dev/hugepages:/dev/hugepages
@@ -35,14 +41,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
-
-    - name: Install unzip package (Temporary, tt-torch issue 787)
-      shell: bash
-      run: |
-        apt-get update && apt-get install -y unzip
 
     - name: Set reusable strings
       id: strings
diff --git a/.github/workflows/generate-benchmark-report.yml b/.github/workflows/generate-benchmark-report.yml
index ee9c74807..c66ee79d2 100644
--- a/.github/workflows/generate-benchmark-report.yml
+++ b/.github/workflows/generate-benchmark-report.yml
@@ -3,7 +3,13 @@ name: Generate Benchmark Report
 on:
   workflow_call:
 jobs:
+
+  docker-build:
+    uses: ./.github/workflows/build-image.yml
+    secrets: inherit
+
   tests:
+    needs: docker-build
     timeout-minutes: 30
     strategy:
       fail-fast: false
@@ -12,7 +18,7 @@ jobs:
       - wormhole_b0
 
     container:
-      image: ghcr.io/tenstorrent/tt-torch/tt-torch-ci-ubuntu-22-04:latest
+      image: ${{ needs.docker-build.outputs.docker-image }}
       options: --user root --device /dev/tenstorrent/0
       volumes:
         - /dev/hugepages:/dev/hugepages
@@ -23,14 +29,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
-
-    - name: Install unzip package (Temporary, tt-torch issue 787)
-      shell: bash
-      run: |
-        apt-get update && apt-get install -y unzip
 
     - name: Set reusable strings
       id: strings
diff --git a/.github/workflows/generate-model-report.yml b/.github/workflows/generate-model-report.yml
index 55b928631..cb01d6c2a 100644
--- a/.github/workflows/generate-model-report.yml
+++ b/.github/workflows/generate-model-report.yml
@@ -12,7 +12,12 @@ on:
     types: [completed]
 
 jobs:
+  docker-build:
+    uses: ./.github/workflows/build-image.yml
+    secrets: inherit
+
   report-tests:
+    needs: docker-build
     outputs:
       spreadsheet_name: ${{ steps.git-info.outputs.spreadsheet_name }}
     timeout-minutes: 120
@@ -23,7 +28,7 @@ jobs:
       - wormhole_b0
 
     container:
-      image: ghcr.io/tenstorrent/tt-torch/tt-torch-ci-ubuntu-22-04:latest
+      image: ${{ needs.docker-build.outputs.docker-image }}
       options: --user root --device /dev/tenstorrent/0
       volumes:
         - /dev/hugepages:/dev/hugepages
@@ -34,14 +39,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
-
-    - name: Install unzip package (Temporary, tt-torch issue 787)
-      shell: bash
-      run: |
-        apt-get update && apt-get install -y unzip
 
     - name: Set reusable strings
       id: strings
diff --git a/.github/workflows/generate-ttnn-md.yml b/.github/workflows/generate-ttnn-md.yml
index 21d7c40a4..07329719d 100644
--- a/.github/workflows/generate-ttnn-md.yml
+++ b/.github/workflows/generate-ttnn-md.yml
@@ -19,21 +19,20 @@ on:
         type: string
 
 jobs:
+  docker-build:
+    uses: ./.github/workflows/build-image.yml
+    secrets: inherit
   generate_md:
+    needs: docker-build
     timeout-minutes: 120
     runs-on: ubuntu-latest
     container:
-      image: ghcr.io/tenstorrent/tt-torch/tt-torch-ci-ubuntu-22-04:latest
+      image: ${{ needs.docker-build.outputs.docker-image }}
       options: --user root
 
     steps:
       - uses: actions/checkout@v4
 
-      - name: Install unzip package (Temporary, tt-torch issue 787)
-        shell: bash
-        run: |
-          apt-get update && apt-get install -y unzip
-
       - name: Set reusable strings
         id: strings
         shell: bash
diff --git a/.github/workflows/nightly-uplift.yml b/.github/workflows/nightly-uplift.yml
index 9421cb5ad..c95895f24 100644
--- a/.github/workflows/nightly-uplift.yml
+++ b/.github/workflows/nightly-uplift.yml
@@ -20,7 +20,6 @@ jobs:
 
       - uses: actions/checkout@v4
         with:
-          submodules: recursive
           fetch-depth: 0 # Fetch all history and tags
           ref: main
 
diff --git a/.github/workflows/run-build.yml b/.github/workflows/run-build.yml
index 38f288a21..af9503fb6 100644
--- a/.github/workflows/run-build.yml
+++ b/.github/workflows/run-build.yml
@@ -46,9 +46,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
 
     - name: Override tt-mlir SHA mlir_override is set
       if: ${{ inputs.mlir_override }}
diff --git a/.github/workflows/run-depth-benchmark-tests.yml b/.github/workflows/run-depth-benchmark-tests.yml
index c82da6c3f..7f727cbae 100644
--- a/.github/workflows/run-depth-benchmark-tests.yml
+++ b/.github/workflows/run-depth-benchmark-tests.yml
@@ -46,9 +46,6 @@ jobs:
         - /mnt/dockercache:/mnt/dockercache
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
 
     - name: Fetch job id
       id: fetch-job-id
diff --git a/.github/workflows/run-e2e-compile-tests.yml b/.github/workflows/run-e2e-compile-tests.yml
index b59bce290..75cb456b6 100644
--- a/.github/workflows/run-e2e-compile-tests.yml
+++ b/.github/workflows/run-e2e-compile-tests.yml
@@ -73,9 +73,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
 
     - name: Fetch job id
       id: fetch-job-id
diff --git a/.github/workflows/run-full-model-execution-tests-nightly.yml b/.github/workflows/run-full-model-execution-tests-nightly.yml
index 0b16e9c20..31178ae75 100644
--- a/.github/workflows/run-full-model-execution-tests-nightly.yml
+++ b/.github/workflows/run-full-model-execution-tests-nightly.yml
@@ -196,9 +196,6 @@ jobs:
         - /mnt/dockercache:/mnt/dockercache
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
 
     - name: Fetch job id
       id: fetch-job-id
diff --git a/.github/workflows/run-full-model-execution-tests.yml b/.github/workflows/run-full-model-execution-tests.yml
index 17e18c501..21fb9cd6d 100644
--- a/.github/workflows/run-full-model-execution-tests.yml
+++ b/.github/workflows/run-full-model-execution-tests.yml
@@ -124,9 +124,6 @@ jobs:
         - /mnt/dockercache:/mnt/dockercache
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
 
     - name: Fetch job id
       id: fetch-job-id
diff --git a/.github/workflows/run-multidevice-tests.yml b/.github/workflows/run-multidevice-tests.yml
index 6b9ff5a6e..ceb38b19c 100644
--- a/.github/workflows/run-multidevice-tests.yml
+++ b/.github/workflows/run-multidevice-tests.yml
@@ -43,10 +43,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
-
     - name: Fetch job id
       id: fetch-job-id
       uses: tenstorrent/tt-github-actions/.github/actions/job_id@main
diff --git a/.github/workflows/run-op-by-op-model-tests-nightly.yml b/.github/workflows/run-op-by-op-model-tests-nightly.yml
index 7571174ca..b6df6974b 100644
--- a/.github/workflows/run-op-by-op-model-tests-nightly.yml
+++ b/.github/workflows/run-op-by-op-model-tests-nightly.yml
@@ -187,9 +187,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
 
     - name: Fetch job id
       id: fetch-job-id
diff --git a/.github/workflows/run-op-by-op-model-tests-weekly.yml b/.github/workflows/run-op-by-op-model-tests-weekly.yml
index d039ebd96..0182493a9 100644
--- a/.github/workflows/run-op-by-op-model-tests-weekly.yml
+++ b/.github/workflows/run-op-by-op-model-tests-weekly.yml
@@ -259,10 +259,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
-
     - name: Fetch job id
       id: fetch-job-id
       uses: tenstorrent/tt-github-actions/.github/actions/job_id@main
diff --git a/.github/workflows/run-tests.yml b/.github/workflows/run-tests.yml
index ef348b906..b4f9cbdce 100644
--- a/.github/workflows/run-tests.yml
+++ b/.github/workflows/run-tests.yml
@@ -44,9 +44,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
 
     - name: Fetch job id
       id: fetch-job-id
diff --git a/.github/workflows/run-tools-tests.yml b/.github/workflows/run-tools-tests.yml
index 3096d5467..01dcc9fe3 100644
--- a/.github/workflows/run-tools-tests.yml
+++ b/.github/workflows/run-tools-tests.yml
@@ -42,9 +42,6 @@ jobs:
 
     steps:
     - uses: actions/checkout@v4
-      with:
-        submodules: recursive
-        lfs: true
 
     - name: Fetch job id
       id: fetch-job-id
diff --git a/.github/workflows/upload-benchmark-file.yml b/.github/workflows/upload-benchmark-file.yml
index c34de12aa..b18ac9097 100644
--- a/.github/workflows/upload-benchmark-file.yml
+++ b/.github/workflows/upload-benchmark-file.yml
@@ -16,10 +16,6 @@ jobs:
         steps:
 
         - uses: actions/checkout@v4
-          with:
-            submodules: recursive
-            lfs: true
-
         - uses: actions/upload-artifact@v4
           with:
             name: ${{ inputs.benchmark_report_path }}


Reviews:
user: login: windsurf-bot[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-05-20T10:29:33Z
user: login: vmilosevic
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T10:34:54Z
user: login: jameszianxuTT
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-20T12:44:59Z
user: login: vvukomanTT
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T12:56:35Z
user: login: nsumrakTT
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T13:24:51Z

Review comments:
diff_hunk: @@ -69,3 +62,23 @@ jobs:
           DOCKER_CI_IMAGE=$(tail -n 1 docker.log)
           echo "DOCKER_CI_IMAGE $DOCKER_CI_IMAGE"
           echo "docker-image=$DOCKER_CI_IMAGE" >> "$GITHUB_OUTPUT"
+
+  set-latest:
+    # Set the latest tag on the CI image
+    runs-on: ubuntu-latest
+    needs: build-image
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+    steps:
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Set latest tag on the image
path: .github/workflows/build-image.yml
user: login: windsurf-bot[bot]
body: The `set-latest` job uses the `skopeo` command to copy Docker images, but there's no step to install this tool. If the `ubuntu-latest` runner doesn't have skopeo pre-installed, this job will fail. Consider adding an installation step:

```suggestion
      - name: Install skopeo
        run: |
          sudo apt-get update
          sudo apt-get install -y skopeo
      - name: Set latest tag on the image
```
created_at: 2025-05-20T10:29:32Z
author_association: NONE
diff_hunk: @@ -3,6 +3,11 @@ name: Generate Benchmark Report
 on:
   workflow_call:
 jobs:
+
+  docker-build:
+    uses: ./.github/workflows/build-image.yml
+    secrets: inherit
+
   tests:
     timeout-minutes: 30
path: .github/workflows/generate-benchmark-report.yml
user: login: windsurf-bot[bot]
body: The 'tests' job is using output from the 'docker-build' job (`needs.docker-build.outputs.docker-image`), but there's no explicit dependency declaration. Please add a 'needs' section to ensure the jobs run in the correct order:

```suggestion
  tests:
    needs: docker-build
    timeout-minutes: 30
```
created_at: 2025-05-20T10:29:32Z
author_association: NONE
diff_hunk: @@ -19,11 +19,14 @@ on:
         type: string
 
 jobs:
+  docker-build:
+    uses: ./.github/workflows/build-image.yml
+    secrets: inherit
   generate_md:
     timeout-minutes: 120
     runs-on: ubuntu-latest
path: .github/workflows/generate-ttnn-md.yml
user: login: windsurf-bot[bot]
body: The `generate_md` job is using the output from `docker-build` job, but there's no explicit dependency declared. You should add a `needs` field to ensure the jobs run in the correct order.

```suggestion
  generate_md:
    needs: docker-build
    timeout-minutes: 120
    runs-on: ubuntu-latest
    container:
```
created_at: 2025-05-20T10:29:32Z
author_association: NONE
diff_hunk: @@ -69,3 +62,23 @@ jobs:
           DOCKER_CI_IMAGE=$(tail -n 1 docker.log)
           echo "DOCKER_CI_IMAGE $DOCKER_CI_IMAGE"
           echo "docker-image=$DOCKER_CI_IMAGE" >> "$GITHUB_OUTPUT"
+
+  set-latest:
+    # Set the latest tag on the CI image
+    runs-on: ubuntu-latest
+    needs: build-image
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
+    steps:
+      - name: Log in to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.repository_owner }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Set latest tag on the image
path: .github/workflows/build-image.yml
user: login: vmilosevic
body: skopeo is pre-installed on ubuntu-latest
created_at: 2025-05-20T10:34:54Z
author_association: CONTRIBUTOR

PR comments:
user: login: codecov-commenter
created_at: 2025-05-20T11:14:45Z
body: ## [Codecov](https://app.codecov.io/gh/tenstorrent/tt-torch/pull/799?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 69.32%. Comparing base [(`374c34c`)](https://app.codecov.io/gh/tenstorrent/tt-torch/commit/374c34c73d26e84f3b7ac367c0b7f5b6854be3e1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent) to head [(`6f71edd`)](https://app.codecov.io/gh/tenstorrent/tt-torch/commit/6f71edd88708d182c9883128c187904b9cad1f1f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).

:white_check_mark: All tests successful. No failed tests found.

> :exclamation:  There is a different number of reports uploaded between BASE (374c34c) and HEAD (6f71edd). Click for more details.
> 
> <details><summary>HEAD has 14 uploads less than BASE</summary>
>
>| Flag | BASE (374c34c) | HEAD (6f71edd) |
>|------|------|------|
>||15|1|
></details>

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #799      +/-   ##
==========================================
- Coverage   75.31%   69.32%   -6.00%     
==========================================
  Files          18       18              
  Lines        2888     2888              
==========================================
- Hits         2175     2002     -173     
- Misses        713      886     +173     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/tenstorrent/tt-torch/pull/799?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=tenstorrent).

author_association: NONE