Repository: red-hat-data-services/notebooks
Pull request #514: RHOAIENG-21668: chore(gha): run GitHub Actions CI for our RHEL-based workbenches
Author: login: jiridanek
Labels: name: lgtm
name: approved
name: tide/merge-method-rebase
Created: 2025-03-03T13:02:25Z | Merged: 2025-03-18T09:20:15Z

https://issues.redhat.com/browse/RHOAIENG-21668

previously discussed on slack

* https://redhat-internal.slack.com/archives/C060A5FJEAD/p1741022594961149

depends on @andyatmiami 's rstudio work in

* https://github.com/red-hat-data-services/notebooks/pull/517

## How tested

* https://github.com/jiridanek/notebooks/actions/runs/13907732646
* https://github.com/jiridanek/notebooks/pull/10

## Notes

There does not seem a way to prevent the build to invalidate the stored secret by mistake, and then I get

```
[3/3] STEP 8/38: RUN subscription-manager status && touch /var/tmp/.subscription-initially-present
+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Consumer profile "4064ef48-aa03-49c2-b449-8dffc941015c" has been deleted from the server. You can use command clean or unregister to remove local profile.
Error: building at STEP "RUN subscription-manager status && touch /var/tmp/.subscription-initially-present": while running runtime: exit status 69
```

### docs

* https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html-single/builds_using_buildconfig/index#builds-create-imagestreamtag_running-entitled-builds

### subscription-manager usage in containers

* https://redhat-internal.slack.com/archives/C06NKAPNJRX/p1720635466075499?thread_ts=1720634314.342279&cid=C06NKAPNJRX
* https://redhat-internal.slack.com/archives/C060A5FJEAD/p1734972809129399?thread_ts=1734726658.813289&cid=C060A5FJEAD
* https://redhat-internal.slack.com/archives/C060A5FJEAD/p1734987478025129?thread_ts=1734726658.813289&cid=C060A5FJEAD

### UBI and RHEL subscription considerations discussions

* https://redhat-internal.slack.com/archives/C06B3AVCJ64/p1720079270728929
* https://redhat-internal.slack.com/archives/C05NXTEHLGY/p1720013666757209


Diff:
diff --git a/.gitattributes b/.gitattributes
new file mode 100644
index 0000000000..fda00e24c0
--- /dev/null
+++ b/.gitattributes
@@ -0,0 +1 @@
+ci/secrets/** filter=git-crypt diff=git-crypt
diff --git a/.github/workflows/build-notebooks-TEMPLATE.yaml b/.github/workflows/build-notebooks-TEMPLATE.yaml
index 8d8112321a..a268fc5c4e 100644
--- a/.github/workflows/build-notebooks-TEMPLATE.yaml
+++ b/.github/workflows/build-notebooks-TEMPLATE.yaml
@@ -15,6 +15,11 @@ name: Build & Publish Notebook Servers (TEMPLATE)
         required: true
         description: "top workflow's `github`"
         type: string
+      subscription:
+        required: false
+        default: false
+        description: "add RHEL subscription from github secret"
+        type: boolean
 
 jobs:
   build:
@@ -49,9 +54,40 @@ jobs:
           echo "CACHE=${CACHE,,}" >>${GITHUB_ENV}
 
       - uses: actions/checkout@v4
+        if: ${{ fromJson(inputs.github).event_name != 'pull_request_target' }}
+      # we need to checkout the pr branch, not pr target (the default for pull_request_target)
+      # user access check is done in calling workflow
+      - uses: actions/checkout@v4
+        if: ${{ fromJson(inputs.github).event_name == 'pull_request_target' }}
+        with:
+          ref: "refs/pull/${{ fromJson(inputs.github).event.number }}/merge"
 
       - run: mkdir -p $TMPDIR
 
+      # do this early because it's fast and why not
+      - name: Unlock encrypted secrets with git-crypt
+        if: ${{ inputs.subscription }}
+        run: |
+          sudo apt-get update
+          sudo apt-get install git-crypt
+          echo "${GIT_CRYPT_KEY}" | base64 --decode > ./git-crypt-key
+          git-crypt unlock ./git-crypt-key
+          rm ./git-crypt-key
+        env:
+          GIT_CRYPT_KEY: ${{ secrets.GIT_CRYPT_KEY }}
+
+      - name: Add subscriptions from GitHub secret
+        if: ${{ inputs.subscription }}
+        run: |
+          sudo mkdir -p /etc/pki/
+          sudo cp -R ${PWD}/ci/secrets/pki/* /etc/pki/
+          # https://access.redhat.com/solutions/5870841
+          # https://github.com/containers/common/issues/1735
+          printf "${PWD}/ci/secrets/run/secrets/rhsm:/etc/rhsm\n${PWD}/ci/secrets/run/secrets/etc-pki-entitlement:/etc/pki/entitlement\n${PWD}/ci/secrets/pki/consumer:/etc/pki/consumer\n" | sudo tee /usr/share/containers/mounts.conf
+
+          mkdir -p $HOME/.config/containers/
+          sudo cp ${PWD}/ci/secrets/pull-secret.json $HOME/.config/containers/auth.json
+
       # for bin/buildinputs in scripts/sandbox.py
       - uses: actions/setup-go@v5
         with:
@@ -257,7 +293,8 @@ jobs:
       - name: "pull_request: make ${{ inputs.target }}"
         run: |
           make ${{ inputs.target }}
-        if: "${{ fromJson(inputs.github).event_name == 'pull_request' }}"
+        if: "${{ fromJson(inputs.github).event_name == 'pull_request' ||
+          fromJson(inputs.github).event_name == 'pull_request_target' }}"
         env:
           IMAGE_TAG: "${{ steps.calculated_vars.outputs.IMAGE_TAG }}"
           CONTAINER_BUILD_CACHE_ARGS: "--cache-from ${{ env.CACHE }}"
diff --git a/.github/workflows/build-notebooks-pr-rhel.yaml b/.github/workflows/build-notebooks-pr-rhel.yaml
new file mode 100644
index 0000000000..aa6606d393
--- /dev/null
+++ b/.github/workflows/build-notebooks-pr-rhel.yaml
@@ -0,0 +1,69 @@
+---
+"name": "Build Notebooks (pr, RHEL images)"
+"on":
+  "pull_request_target":
+
+# BEWARE: This GitHub Actions workflow runs on pull_request_target, meaning it has access to our secrets
+# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-secrets
+# and https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
+
+permissions:
+  contents: read
+  packages: read
+
+env:
+  # language=json
+  contributors: |
+    ["atheo89", "andyatmiami", "caponetto", "daniellutz", "dibryant", "harshad16", "jesuino", "jiridanek", "jstourac", "paulovmr"]
+
+jobs:
+  gen:
+    name: Generate job matrix
+    runs-on: ubuntu-latest
+    outputs:
+      matrix: ${{ steps.gen.outputs.matrix }}
+      has_jobs: ${{ steps.gen.outputs.has_jobs }}
+    steps:
+
+      - name: Check permissions and deny untrusted users (this must be done FIRST, for security, before we checkout)
+        if: ${{ !contains(fromJSON(env.contributors), github.actor) }}
+        run: |
+          echo "GitHub user ${{ github.actor }} is not a registered project contributor, not allowed to run actions on RHEL!"
+          exit 1
+
+      # Here we are checking out the pull request, so that we can build from the new code
+      # We can do this because we already checked that the submitting user is a contributor
+      - uses: actions/checkout@v4
+        if: ${{ github.event_name == 'pull_request_target' }}
+        with:
+          ref: "refs/pull/${{ github.event.number }}/merge"
+      - uses: actions/checkout@v4
+        if: ${{ github.event_name != 'pull_request_target' }}
+
+      - name: Determine targets to build based on changed files
+        if: ${{ github.event_name == 'pull_request_target' }}
+        run: |
+          set -x
+          git fetch --no-tags origin 'pull/${{ github.event.pull_request.number }}/head:${{ github.event.pull_request.head.ref }}'
+          git fetch --no-tags origin '+refs/heads/${{ github.event.pull_request.base.ref }}:refs/remotes/origin/${{ github.event.pull_request.base.ref }}'
+          python3 ci/cached-builds/gen_gha_matrix_jobs.py \
+            --from-ref 'origin/${{ github.event.pull_request.base.ref }}' \
+            --to-ref '${{ github.event.pull_request.head.ref }}' \
+            --rhel-images include-only
+        id: gen
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        shell: bash
+
+  build:
+    needs: ["gen"]
+    strategy:
+      fail-fast: false
+      matrix: "${{ fromJson(needs.gen.outputs.matrix) }}"
+    uses: ./.github/workflows/build-notebooks-TEMPLATE.yaml
+    if: ${{ fromJson(needs.gen.outputs.has_jobs) }}
+    with:
+      target: "${{ matrix.target }}"
+      github: "${{ toJSON(github) }}"
+      subscription: "${{ matrix.subscription }}"
+    secrets: inherit
diff --git a/.github/workflows/build-notebooks-pr.yaml b/.github/workflows/build-notebooks-pr.yaml
index c479e29e53..f27a33ae70 100644
--- a/.github/workflows/build-notebooks-pr.yaml
+++ b/.github/workflows/build-notebooks-pr.yaml
@@ -33,7 +33,8 @@ jobs:
           git fetch --no-tags origin '+refs/heads/${{ github.event.pull_request.base.ref }}:refs/remotes/origin/${{ github.event.pull_request.base.ref }}'
           python3 ci/cached-builds/gen_gha_matrix_jobs.py \
             --from-ref 'origin/${{ github.event.pull_request.base.ref }}' \
-            --to-ref '${{ github.event.pull_request.head.ref }}'
+            --to-ref '${{ github.event.pull_request.head.ref }}' \
+            --rhel-images exclude
         id: gen
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -49,4 +50,5 @@ jobs:
     with:
       target: "${{ matrix.target }}"
       github: "${{ toJSON(github) }}"
+      subscription: "${{ matrix.subscription }}"
     secrets: inherit
diff --git a/.github/workflows/build-notebooks-push.yaml b/.github/workflows/build-notebooks-push.yaml
index 0cdbda57a0..b85c06b8db 100644
--- a/.github/workflows/build-notebooks-push.yaml
+++ b/.github/workflows/build-notebooks-push.yaml
@@ -39,4 +39,5 @@ jobs:
     with:
       target: "${{ matrix.target }}"
       github: "${{ toJSON(github) }}"
+      subscription: "${{ matrix.subscription }}"
     secrets: inherit
diff --git a/.github/workflows/code-quality.yaml b/.github/workflows/code-quality.yaml
index 082244d947..929f26d3d0 100644
--- a/.github/workflows/code-quality.yaml
+++ b/.github/workflows/code-quality.yaml
@@ -80,6 +80,9 @@ jobs:
     steps:
       - uses: actions/checkout@v4
 
+      - name: Do not check secrets, they are encrypted
+        run: rm -rf ./ci/secrets
+
       - name: Validate YAML files (best code practices check included)
         id: validate-yaml-files
         run: |
diff --git a/Makefile b/Makefile
index 173efb0e3f..3eb5ca6eca 100644
--- a/Makefile
+++ b/Makefile
@@ -172,6 +172,16 @@ rstudio-c9s-python-$(RELEASE_PYTHON_VERSION):
 cuda-rstudio-c9s-python-$(RELEASE_PYTHON_VERSION):
 	$(call image,$@,rstudio/c9s-python-$(RELEASE_PYTHON_VERSION)/Dockerfile.cuda)
 
+####################################### Buildchain for Python using rhel9 #######################################
+
+.PHONY: rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION)
+rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION):
+	$(call image,$@,rstudio/rhel9-python-$(RELEASE_PYTHON_VERSION)/Dockerfile.cpu)
+
+.PHONY: cuda-rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION)
+cuda-rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION):
+	$(call image,$@,rstudio/rhel9-python-$(RELEASE_PYTHON_VERSION)/Dockerfile.cuda)
+
 ####################################### Buildchain for AMD Python using UBI9 #######################################
 .PHONY: rocm-jupyter-minimal-ubi9-python-$(RELEASE_PYTHON_VERSION)
 rocm-jupyter-minimal-ubi9-python-$(RELEASE_PYTHON_VERSION):
@@ -438,6 +448,8 @@ all-images: jupyter-minimal-ubi9-python-$(RELEASE_PYTHON_VERSION) \
 	codeserver-ubi9-python-$(RELEASE_PYTHON_VERSION) \
 	rstudio-c9s-python-$(RELEASE_PYTHON_VERSION) \
 	cuda-rstudio-c9s-python-$(RELEASE_PYTHON_VERSION) \
+	rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION) \
+	cuda-rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION) \
 	rocm-jupyter-minimal-ubi9-python-$(RELEASE_PYTHON_VERSION) \
 	rocm-jupyter-tensorflow-ubi9-python-$(RELEASE_PYTHON_VERSION) \
 	rocm-jupyter-pytorch-ubi9-python-$(RELEASE_PYTHON_VERSION) \
diff --git a/ci/cached-builds/gen_gha_matrix_jobs.py b/ci/cached-builds/gen_gha_matrix_jobs.py
old mode 100644
new mode 100755
index 52c6fe179e..4276469e2e
--- a/ci/cached-builds/gen_gha_matrix_jobs.py
+++ b/ci/cached-builds/gen_gha_matrix_jobs.py
@@ -1,4 +1,7 @@
+#!/usr/bin/env python3
+
 import argparse
+import enum
 import json
 import logging
 import platform
@@ -6,7 +9,6 @@
 import os
 import pathlib
 import re
-import string
 import sys
 import unittest
 
@@ -21,7 +23,7 @@
 project_dir = pathlib.Path(__file__).parent.parent.parent.absolute()
 
 
-def parse_makefile(target: str, makefile_dir: str) -> str:
+def parse_makefile(target: str, makefile_dir: pathlib.Path | str) -> str:
     # Check if the operating system is macOS
     if platform.system() == 'Darwin':
         make_command = 'gmake'
@@ -30,7 +32,9 @@ def parse_makefile(target: str, makefile_dir: str) -> str:
 
     try:
         # Run the make (or gmake) command and capture the output
-        result = subprocess.run([make_command, '-nps', target], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=True, cwd=makefile_dir)
+        result = subprocess.run([make_command, '-nps', target],
+                                stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
+                                check=True, cwd=makefile_dir)
     except subprocess.CalledProcessError as e:
         # Handle errors if the make command fails
         print(f'{make_command} failed with return code: {e.returncode}:\n{e.stderr}', file=sys.stderr)
@@ -43,7 +47,7 @@ def parse_makefile(target: str, makefile_dir: str) -> str:
     return result.stdout
 
 
-def extract_image_targets(makefile_dir: str = os.getcwd()) -> list[str]:
+def extract_image_targets(makefile_dir: pathlib.Path | str = os.getcwd()) -> list[str]:
     makefile_all_target = 'all-images'
 
     output = parse_makefile(target=makefile_all_target, makefile_dir=makefile_dir)
@@ -60,6 +64,12 @@ def extract_image_targets(makefile_dir: str = os.getcwd()) -> list[str]:
     return all_images
 
 
+class RhelImages(enum.Enum):
+    EXCLUDE = "exclude"
+    INCLUDE = "include"
+    INCLUDE_ONLY = "include-only"
+
+
 def main() -> None:
     logging.basicConfig(level=logging.DEBUG, stream=sys.stderr)
 
@@ -68,9 +78,10 @@ def main() -> None:
                            help="Git ref of the base branch (to determine changed files)")
     argparser.add_argument("--to-ref", type=str, required=False,
                            help="Git ref of the PR branch (to determine changed files)")
+    argparser.add_argument("--rhel-images", type=RhelImages, required=False, default=RhelImages.INCLUDE, nargs='?',
+                           help="Whether to `include` rhel images or `exclude` them or `include-only` them")
     args = argparser.parse_args()
 
-
     targets = extract_image_targets()
 
     if args.from_ref:
@@ -78,9 +89,25 @@ def main() -> None:
         changed_files = gha_pr_changed_files.list_changed_files(args.from_ref, args.to_ref)
         targets = gha_pr_changed_files.filter_out_unchanged(targets, changed_files)
 
+    if args.rhel_images == RhelImages.INCLUDE:
+        pass
+    elif args.rhel_images == RhelImages.EXCLUDE:
+        targets = [target for target in targets if "rhel" not in target]
+    elif args.rhel_images == RhelImages.INCLUDE_ONLY:
+        targets = [target for target in targets if "rhel" in target]
+    else:
+        raise Exception(f"Unknown value for --rhel-images: {args.rhel_images}")
+
+    # https://stackoverflow.com/questions/66025220/paired-values-in-github-actions-matrix
     output = [
-        f"matrix={json.dumps({"target": targets}, separators=(',', ':'))}",
-        f"has_jobs={json.dumps(len(targets) > 0, separators=(',', ':'))}"
+        "matrix=" + json.dumps({
+            "include": [
+                {"target": target, "subscription": "rhel" in target} for target in targets
+            ],
+        }, separators=(',', ':')),
+        "has_jobs=" + json.dumps(
+            len(targets) > 0, separators=(',', ':')
+        ),
     ]
 
     print("targets", targets)
diff --git a/ci/cached-builds/mounts.conf b/ci/cached-builds/mounts.conf
new file mode 100644
index 0000000000..08e2d3aad0
--- /dev/null
+++ b/ci/cached-builds/mounts.conf
@@ -0,0 +1,5 @@
+# /usr/share/containers/mounts.conf
+# https://github.com/containers/common/issues/1735
+
+/etc/pki/consumer:/etc/pki/consumer
+/etc/pki/entitlement:/etc/pki/entitlement
diff --git a/ci/secrets/pki/consumer/cert.pem b/ci/secrets/pki/consumer/cert.pem
new file mode 100644
index 0000000000..b828a8f2e8
Binary files /dev/null and b/ci/secrets/pki/consumer/cert.pem differ
diff --git a/ci/secrets/pki/consumer/key.pem b/ci/secrets/pki/consumer/key.pem
new file mode 100644
index 0000000000..85a5335d7a
Binary files /dev/null and b/ci/secrets/pki/consumer/key.pem differ
diff --git a/ci/secrets/pull-secret.json b/ci/secrets/pull-secret.json
new file mode 100644
index 0000000000..53d945ab0f
Binary files /dev/null and b/ci/secrets/pull-secret.json differ
diff --git a/ci/secrets/rhsm/ca/redhat-entitlement-authority.pem b/ci/secrets/rhsm/ca/redhat-entitlement-authority.pem
new file mode 100644
index 0000000000..6330d5bd1d
Binary files /dev/null and b/ci/secrets/rhsm/ca/redhat-entitlement-authority.pem differ
diff --git a/ci/secrets/rhsm/ca/redhat-uep.pem b/ci/secrets/rhsm/ca/redhat-uep.pem
new file mode 100644
index 0000000000..23f9876f80
Binary files /dev/null and b/ci/secrets/rhsm/ca/redhat-uep.pem differ
diff --git a/ci/secrets/rhsm/rhsm.conf b/ci/secrets/rhsm/rhsm.conf
new file mode 100644
index 0000000000..bbded50d60
Binary files /dev/null and b/ci/secrets/rhsm/rhsm.conf differ
diff --git a/ci/secrets/rhsm/syspurpose/valid_fields.json b/ci/secrets/rhsm/syspurpose/valid_fields.json
new file mode 100644
index 0000000000..ae69a9b18a
Binary files /dev/null and b/ci/secrets/rhsm/syspurpose/valid_fields.json differ
diff --git a/ci/secrets/run/secrets/etc-pki-entitlement/1358292728734056503-key.pem b/ci/secrets/run/secrets/etc-pki-entitlement/1358292728734056503-key.pem
new file mode 100644
index 0000000000..ebad4dbce1
Binary files /dev/null and b/ci/secrets/run/secrets/etc-pki-entitlement/1358292728734056503-key.pem differ
diff --git a/ci/secrets/run/secrets/etc-pki-entitlement/1358292728734056503.pem b/ci/secrets/run/secrets/etc-pki-entitlement/1358292728734056503.pem
new file mode 100644
index 0000000000..f3cd9b40ce
Binary files /dev/null and b/ci/secrets/run/secrets/etc-pki-entitlement/1358292728734056503.pem differ
diff --git a/ci/secrets/run/secrets/redhat.repo b/ci/secrets/run/secrets/redhat.repo
new file mode 100644
index 0000000000..342d25c776
Binary files /dev/null and b/ci/secrets/run/secrets/redhat.repo differ
diff --git a/ci/secrets/run/secrets/rhsm/ca/redhat-entitlement-authority.pem b/ci/secrets/run/secrets/rhsm/ca/redhat-entitlement-authority.pem
new file mode 100644
index 0000000000..6330d5bd1d
Binary files /dev/null and b/ci/secrets/run/secrets/rhsm/ca/redhat-entitlement-authority.pem differ
diff --git a/ci/secrets/run/secrets/rhsm/ca/redhat-uep.pem b/ci/secrets/run/secrets/rhsm/ca/redhat-uep.pem
new file mode 100644
index 0000000000..23f9876f80
Binary files /dev/null and b/ci/secrets/run/secrets/rhsm/ca/redhat-uep.pem differ
diff --git a/ci/secrets/run/secrets/rhsm/rhsm.conf b/ci/secrets/run/secrets/rhsm/rhsm.conf
new file mode 100644
index 0000000000..269737ffc1
Binary files /dev/null and b/ci/secrets/run/secrets/rhsm/rhsm.conf differ
diff --git a/ci/secrets/run/secrets/rhsm/syspurpose/syspurpose.json b/ci/secrets/run/secrets/rhsm/syspurpose/syspurpose.json
new file mode 100644
index 0000000000..9cd79a49f5
Binary files /dev/null and b/ci/secrets/run/secrets/rhsm/syspurpose/syspurpose.json differ
diff --git a/ci/secrets/run/secrets/rhsm/syspurpose/valid_fields.json b/ci/secrets/run/secrets/rhsm/syspurpose/valid_fields.json
new file mode 100644
index 0000000000..ae69a9b18a
Binary files /dev/null and b/ci/secrets/run/secrets/rhsm/syspurpose/valid_fields.json differ
diff --git a/rstudio/rhel9-python-3.11/Dockerfile.cpu b/rstudio/rhel9-python-3.11/Dockerfile.cpu
index c1888c3532..3405b3a383 100644
--- a/rstudio/rhel9-python-3.11/Dockerfile.cpu
+++ b/rstudio/rhel9-python-3.11/Dockerfile.cpu
@@ -51,8 +51,9 @@ USER 0
 # uncomment the below line if you fall on this error: subscription-manager is disabled when running inside a container. Please refer to your host system for subscription management.
 #RUN sed -i 's/\(def in_container():\)/\1\n    return False/g' /usr/lib64/python*/*-packages/rhsm/config.py
 
-# Run the subscription manager command using the provided credentials. Only include --serverurl and --baseurl if they are provided
-RUN SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFAULT}) && \
+# If necessary, run the subscription manager command using the provided credentials. Only include --serverurl and --baseurl if they are provided
+RUN if [ -d "${SECRET_DIR}" ]; then \
+    SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFAULT}) && \
     BASEURL=$(cat ${SECRET_DIR}/BASEURL 2>/dev/null || echo ${BASEURL_DEFAULT}) && \
     USERNAME=$(cat ${SECRET_DIR}/USERNAME) && \
     PASSWORD=$(cat ${SECRET_DIR}/PASSWORD) && \
@@ -62,7 +63,8 @@ RUN SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFA
     --username=$USERNAME \
     --password=$PASSWORD \
     --force \
-    --auto-attach
+    --auto-attach; \
+  fi
 
 ENV R_VERSION=4.4.1
 
@@ -169,7 +171,9 @@ COPY ${RSTUDIO_SOURCE_CODE}/utils utils/
 COPY ${RSTUDIO_SOURCE_CODE}/run-rstudio.sh ${RSTUDIO_SOURCE_CODE}/setup_rstudio.py ${RSTUDIO_SOURCE_CODE}/rsession.sh ${RSTUDIO_SOURCE_CODE}/run-nginx.sh ./
 
 # Unregister the system
-RUN subscription-manager remove --all && subscription-manager unregister && subscription-manager clean
+RUN if [ -d "${SECRET_DIR}" ]; then \
+    subscription-manager remove --all && subscription-manager unregister && subscription-manager clean; \
+  fi
 
 USER 1001
 
diff --git a/rstudio/rhel9-python-3.11/Dockerfile.cuda b/rstudio/rhel9-python-3.11/Dockerfile.cuda
index 9f05abf202..1255f0e1a7 100644
--- a/rstudio/rhel9-python-3.11/Dockerfile.cuda
+++ b/rstudio/rhel9-python-3.11/Dockerfile.cuda
@@ -41,8 +41,9 @@ ARG BASEURL_DEFAULT=""
 USER 0
 WORKDIR /opt/app-root/bin
 
-# Run the subscription manager command using the provided credentials. Only include --serverurl and --baseurl if they are provided
-RUN SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFAULT}) && \
+# If necessary, run the subscription manager command using the provided credentials. Only include --serverurl and --baseurl if they are provided
+RUN if [ -d "${SECRET_DIR}" ]; then \
+    SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFAULT}) && \
     BASEURL=$(cat ${SECRET_DIR}/BASEURL 2>/dev/null || echo ${BASEURL_DEFAULT}) && \
     USERNAME=$(cat ${SECRET_DIR}/USERNAME) && \
     PASSWORD=$(cat ${SECRET_DIR}/PASSWORD) && \
@@ -52,7 +53,8 @@ RUN SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFA
     --username=$USERNAME \
     --password=$PASSWORD \
     --force \
-    --auto-attach
+    --auto-attach; \
+  fi
 
 ENV NVARCH=x86_64
 ENV NVIDIA_REQUIRE_CUDA="cuda>=12.4 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 brand=tesla,driver>=535,driver<536 brand=unknown,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=geforce,driver>=535,driver<536 brand=geforcertx,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=titan,driver>=535,driver<536 brand=titanrtx,driver>=535,driver<536"
@@ -165,7 +167,9 @@ RUN yum -y install cuda-toolkit-12-4 && \
     yum -y clean all --enablerepo="*"
 
 # Unregister the system
-RUN subscription-manager remove --all && subscription-manager unregister && subscription-manager clean
+RUN if [ -d "${SECRET_DIR}" ]; then \
+    subscription-manager remove --all && subscription-manager unregister && subscription-manager clean; \
+  fi
 
 # Restore notebook user workspace
 USER 1001
@@ -197,8 +201,9 @@ USER 0
 # uncomment the below line if you fall on this error: subscription-manager is disabled when running inside a container. Please refer to your host system for subscription management.
 #RUN sed -i 's/\(def in_container():\)/\1\n    return False/g' /usr/lib64/python*/*-packages/rhsm/config.py
 
-# Run the subscription manager command using the provided credentials. Only include --serverurl and --baseurl if they are provided
-RUN SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFAULT}) && \
+# If necessary, run the subscription manager command using the provided credentials. Only include --serverurl and --baseurl if they are provided
+RUN if [ -d "${SECRET_DIR}" ]; then \
+    SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFAULT}) && \
     BASEURL=$(cat ${SECRET_DIR}/BASEURL 2>/dev/null || echo ${BASEURL_DEFAULT}) && \
     USERNAME=$(cat ${SECRET_DIR}/USERNAME) && \
     PASSWORD=$(cat ${SECRET_DIR}/PASSWORD) && \
@@ -208,7 +213,8 @@ RUN SERVERURL=$(cat ${SECRET_DIR}/SERVERURL 2>/dev/null || echo ${SERVERURL_DEFA
     --username=$USERNAME \
     --password=$PASSWORD \
     --force \
-    --auto-attach
+    --auto-attach; \
+  fi
 
 ENV R_VERSION=4.4.1
 
@@ -315,7 +321,9 @@ COPY ${RSTUDIO_SOURCE_CODE}/utils utils/
 COPY ${RSTUDIO_SOURCE_CODE}/run-rstudio.sh ${RSTUDIO_SOURCE_CODE}/setup_rstudio.py ${RSTUDIO_SOURCE_CODE}/rsession.sh ${RSTUDIO_SOURCE_CODE}/run-nginx.sh ./
 
 # Unregister the system
-RUN subscription-manager remove --all && subscription-manager unregister && subscription-manager clean
+RUN if [ -d "${SECRET_DIR}" ]; then \
+    subscription-manager remove --all && subscription-manager unregister && subscription-manager clean; \
+  fi
 
 USER 1001
 


Reviews:
user: login: andyatmiami
body: ℹ️ 
will (likely/possibly) be "obvious" when it comes time to do the rebase...

but a lot of the "leaf" logic in the gen matrix script can be discarded - and instead the `all-images` `Makefile` target simply needs updated to add the targets we want to build..
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-07T14:27:19Z
user: login: atheo89
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-03-10T08:47:28Z
user: login: jstourac
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T16:43:20Z
user: login: jiridanek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T17:01:09Z
user: login: jstourac
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T10:28:34Z
user: login: jiridanek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T10:35:05Z
user: login: jstourac
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T07:40:27Z
user: login: jstourac
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T07:45:30Z
user: login: jiridanek
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-18T08:04:33Z

Review comments:
diff_hunk: @@ -0,0 +1,94 @@
+---
+"name": "Build Notebooks (pr, RHEL images)"
+"on":
+  "push":
+  "workflow_dispatch":
+  "pull_request_target":
+    "types": ["opened", "synchronize", "reopened", "edited"]
+
+# BEWARE: This GitHub Actions workflow runs on pull_request_target, meaning it has access to our secrets
+# see https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions#using-secrets
+# and https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
+
+permissions:
+  contents: read
+  # tmp, for checking pull request
+  packages: write
+
+env:
+  contributors: '["atheo89", "caponetto", "dibryant", "harshad16", "jiridanek", "jstourac", "paulovmr"]'
path: .github/workflows/build-notebooks-pr-rhel.yaml
user: login: atheo89
body: Let's add also: `andyatmiami`, `daniellutz` , `jesuino`
created_at: 2025-03-10T08:45:07Z
author_association: MEMBER
diff_hunk: @@ -33,7 +33,8 @@ jobs:
           git fetch --no-tags origin '+refs/heads/${{ github.event.pull_request.base.ref }}:refs/remotes/origin/${{ github.event.pull_request.base.ref }}'
           python3 ci/cached-builds/gen_gha_matrix_jobs.py \
             --from-ref 'origin/${{ github.event.pull_request.base.ref }}' \
-            --to-ref '${{ github.event.pull_request.head.ref }}'
+            --to-ref '${{ github.event.pull_request.head.ref }}' \
+            --leave-out-rhel '${{ github.event_name == 'pull_request' }}'
path: .github/workflows/build-notebooks-pr.yaml
user: login: jstourac
body: [based on the failure](https://github.com/red-hat-data-services/notebooks/actions/runs/13861154180/job/38789730801?pr=514), maybe this should be:
```suggestion
            --leave-out-rhel='${{ github.event_name == 'pull_request' }}'
```
created_at: 2025-03-14T16:43:20Z
author_association: MEMBER
diff_hunk: @@ -33,7 +33,8 @@ jobs:
           git fetch --no-tags origin '+refs/heads/${{ github.event.pull_request.base.ref }}:refs/remotes/origin/${{ github.event.pull_request.base.ref }}'
           python3 ci/cached-builds/gen_gha_matrix_jobs.py \
             --from-ref 'origin/${{ github.event.pull_request.base.ref }}' \
-            --to-ref '${{ github.event.pull_request.head.ref }}'
+            --to-ref '${{ github.event.pull_request.head.ref }}' \
+            --leave-out-rhel '${{ github.event_name == 'pull_request' }}'
path: .github/workflows/build-notebooks-pr.yaml
user: login: jiridanek
body: nope, that was not right either; hopefully I did fix it now, see last commit
created_at: 2025-03-14T17:01:09Z
author_association: MEMBER
diff_hunk: @@ -169,7 +173,7 @@ COPY ${RSTUDIO_SOURCE_CODE}/utils utils/
 COPY ${RSTUDIO_SOURCE_CODE}/run-rstudio.sh ${RSTUDIO_SOURCE_CODE}/setup_rstudio.py ${RSTUDIO_SOURCE_CODE}/rsession.sh ${RSTUDIO_SOURCE_CODE}/run-nginx.sh ./
 
 # Unregister the system
-RUN subscription-manager remove --all && subscription-manager unregister && subscription-manager clean
+RUN [ ! -f '/var/tmp/.subscription-initially-not-present' ] || (subscription-manager remove --all && subscription-manager unregister && subscription-manager clean)
path: rstudio/rhel9-python-3.11/Dockerfile.cpu
user: login: jstourac
body: Can you, please, enlighten me in which scenario we do have the subscription present which leads us not to remove it from the build machine? :thinking: you're thinking about the local build somehow?
created_at: 2025-03-17T10:28:34Z
author_association: MEMBER
diff_hunk: @@ -169,7 +173,7 @@ COPY ${RSTUDIO_SOURCE_CODE}/utils utils/
 COPY ${RSTUDIO_SOURCE_CODE}/run-rstudio.sh ${RSTUDIO_SOURCE_CODE}/setup_rstudio.py ${RSTUDIO_SOURCE_CODE}/rsession.sh ${RSTUDIO_SOURCE_CODE}/run-nginx.sh ./
 
 # Unregister the system
-RUN subscription-manager remove --all && subscription-manager unregister && subscription-manager clean
+RUN [ ! -f '/var/tmp/.subscription-initially-not-present' ] || (subscription-manager remove --all && subscription-manager unregister && subscription-manager clean)
path: rstudio/rhel9-python-3.11/Dockerfile.cpu
user: login: jiridanek
body: the github actions build added in this pr, it is subscribed by mounting the certs into the container, and therefore subscription-manager unregister should not be called at the end, because it would destroy the creds that are stored as repo secret
created_at: 2025-03-17T10:35:04Z
author_association: MEMBER
diff_hunk: @@ -169,7 +173,7 @@ COPY ${RSTUDIO_SOURCE_CODE}/utils utils/
 COPY ${RSTUDIO_SOURCE_CODE}/run-rstudio.sh ${RSTUDIO_SOURCE_CODE}/setup_rstudio.py ${RSTUDIO_SOURCE_CODE}/rsession.sh ${RSTUDIO_SOURCE_CODE}/run-nginx.sh ./
 
 # Unregister the system
-RUN subscription-manager remove --all && subscription-manager unregister && subscription-manager clean
+RUN [ ! -f '/var/tmp/.subscription-initially-not-present' ] || (subscription-manager remove --all && subscription-manager unregister && subscription-manager clean)
path: rstudio/rhel9-python-3.11/Dockerfile.cpu
user: login: jstourac
body: I see. One thing - maybe it would be slightly easier to understand if there isn't that many negations? E.g., let's create a file with the name `subscription-present-initially`. And remove the `!` in the first condition?

Just a proposal, I don't insist. It seems slightly more readable to me.
created_at: 2025-03-18T07:40:27Z
author_association: MEMBER
diff_hunk: @@ -169,7 +173,7 @@ COPY ${RSTUDIO_SOURCE_CODE}/utils utils/
 COPY ${RSTUDIO_SOURCE_CODE}/run-rstudio.sh ${RSTUDIO_SOURCE_CODE}/setup_rstudio.py ${RSTUDIO_SOURCE_CODE}/rsession.sh ${RSTUDIO_SOURCE_CODE}/run-nginx.sh ./
 
 # Unregister the system
-RUN subscription-manager remove --all && subscription-manager unregister && subscription-manager clean
+RUN [ ! -f '/var/tmp/.subscription-initially-not-present' ] || (subscription-manager remove --all && subscription-manager unregister && subscription-manager clean)
path: rstudio/rhel9-python-3.11/Dockerfile.cpu
user: login: jstourac
body: okay, I can see now, that this part of the code is outdated and you changed this already. Thanks!
created_at: 2025-03-18T07:45:30Z
author_association: MEMBER
diff_hunk: @@ -169,7 +173,7 @@ COPY ${RSTUDIO_SOURCE_CODE}/utils utils/
 COPY ${RSTUDIO_SOURCE_CODE}/run-rstudio.sh ${RSTUDIO_SOURCE_CODE}/setup_rstudio.py ${RSTUDIO_SOURCE_CODE}/rsession.sh ${RSTUDIO_SOURCE_CODE}/run-nginx.sh ./
 
 # Unregister the system
-RUN subscription-manager remove --all && subscription-manager unregister && subscription-manager clean
+RUN [ ! -f '/var/tmp/.subscription-initially-not-present' ] || (subscription-manager remove --all && subscription-manager unregister && subscription-manager clean)
path: rstudio/rhel9-python-3.11/Dockerfile.cpu
user: login: jiridanek
body: yes, I needed a negative so that i could use `[ ] ||`

later decided that it's worth redoing this
created_at: 2025-03-18T08:04:33Z
author_association: MEMBER

PR comments:
user: login: jiridanek
created_at: 2025-03-03T17:51:31Z
body: After andy's unchaining of build gets into RHDS, I'll update the pr; for now it builds, runs some tests, but deploy9 fails https://github.com/jiridanek/notebooks/actions/runs/13636395252/job/38116170177
author_association: MEMBER
user: login: jiridanek
created_at: 2025-03-03T17:52:09Z
body: The main magic there is
```
      - name: Unlock encrypted secrets with git-crypt
        if: ${{ inputs.subscription }}
        run: |
          sudo apt-get update
          sudo apt-get install git-crypt
          echo "${GIT_CRYPT_KEY}" | base64 --decode > ./git-crypt-key
          git-crypt unlock ./git-crypt-key
          rm ./git-crypt-key
        env:
          GIT_CRYPT_KEY: ${{ secrets.GIT_CRYPT_KEY }}
```
and then
```
      - name: Add subscriptions from GitHub secret
        if: ${{ inputs.subscription }}
        run: |
          sudo mkdir -p /etc/pki/
          sudo cp -R ${PWD}/ci/secrets/pki/* /etc/pki/
          printf "${PWD}/ci/secrets/pki/consumer:/etc/pki/consumer\n${PWD}/ci/secrets/pki/entitlement:/etc/pki/entitlement" | sudo tee /usr/share/containers/mounts.conf

          mkdir -p $HOME/.config/containers/
          sudo cp ${PWD}/ci/secrets/pull-secret.txt $HOME/.config/containers/auth.json
```
author_association: MEMBER
user: login: atheo89
created_at: 2025-03-07T13:33:52Z
body: Hey @jiridanek in case that we include that pr on main could you fix the commits before? 
author_association: MEMBER
user: login: jiridanek
created_at: 2025-03-07T14:16:36Z
body: I'll be taking a look later today as i do the necessary rebase
author_association: MEMBER
user: login: jiridanek
created_at: 2025-03-07T14:30:28Z
body: > but a lot of the "leaf" logic in the gen matrix script can be discarded - and instead the all-images Makefile target simply needs updated to add the targets we want to build..

all of it can be discarded and the gha "matrix" can build everything

now there is no reason to build pushes and prs differently, the way it used to be the case
author_association: MEMBER
user: login: jstourac
created_at: 2025-03-17T10:32:45Z
body: Let's wait until the work from Andy is merged successfully first. I put one question here but this LGTM in general. Thank you for this!

/lgtm
author_association: MEMBER
user: login: jstourac
created_at: 2025-03-18T07:45:58Z
body: /lgtm
author_association: MEMBER
user: login: jiridanek
created_at: 2025-03-18T09:18:21Z
body: Did one final check in cluster, rstudio workbench seems to be unharmed by these changes,

![image](https://github.com/user-attachments/assets/08e5b9ab-92fc-4618-826b-1da78e2aa3e0)

author_association: MEMBER
user: login: jiridanek
created_at: 2025-03-18T09:18:44Z
body: /approve

but that won't work since we don't have tide any more?
author_association: MEMBER
user: login: openshift-ci[bot]
created_at: 2025-03-18T09:18:49Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/red-hat-data-services/notebooks/pull/514#issuecomment-2732253547" title="Approved">jiridanek</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=red-hat-data-services%2Fnotebooks).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/red-hat-data-services/notebooks/blob/main/OWNERS)~~ [jiridanek]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: NONE
user: login: jiridanek
created_at: 2025-03-18T09:21:10Z
body: They key is in Bitwarden

![image](https://github.com/user-attachments/assets/92e23009-9281-4431-94da-b288f08b04e2)

author_association: MEMBER