Repository: opendatahub-io/ilab-on-ocp
Pull request #280: Add support for PDFs in SDG
Author: login: mprahl
Created: 2025-03-07T20:56:20Z | Merged: 2025-03-11T18:44:59Z

## Description

Taxonomy repos with PDFs require Docling in RHEL AI 1.3, which requires the Mixtral tokenizer. In RHEL AI 1.4, Docling will always be used so this is also required for that upgrade.

Relates:
https://issues.redhat.com/browse/RHOAIENG-21302

## How Has This Been Tested?

1. Using the latest ODH main branch
2. Forked https://github.com/juliadenham/Summit_knowledge.git and converted swifties.md to swifties.pdf
3. Forked https://github.com/instructlab/taxonomy and had it point to the knowledge fork and use swifties.pdf
4. Ran a pipeline pointing to the taxonomy fork
5. Confirmed the SDG step worked and didn't download Docling from HuggingFace

## Merge criteria:

- [x] The commits are squashed in a cohesive manner and have meaningful messages.
- [x] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
- [x] The developer has manually tested the changes and verified that the changes work



Diff:
diff --git a/Makefile b/Makefile
index df06b8a9..a111a763 100644
--- a/Makefile
+++ b/Makefile
@@ -4,7 +4,7 @@ PYTHON_IMAGE          ?= "quay.io/modh/odh-generic-data-science-notebook@sha256:
 TOOLBOX_IMAGE         ?= "registry.redhat.io/ubi9/toolbox@sha256:da31dee8904a535d12689346e65e5b00d11a6179abf1fa69b548dbd755fa2770"                     # v9.5
 OC_IMAGE              ?= "registry.redhat.io/openshift4/ose-cli@sha256:08bdbfae224dd39c81689ee73c183619d6b41eba7ac04f0dce7ee79f50531d0b"               # v4.15.0
 RHELAI_IMAGE          ?= "registry.redhat.io/rhelai1/instructlab-nvidia-rhel9@sha256:05cfba1fb13ed54b1de4d021da2a31dd78ba7d8cc48e10c7fe372815899a18ae" # v1.3.2
-RUNTIME_GENERIC_IMAGE ?= "quay.io/opendatahub/ds-pipelines-runtime-generic@sha256:a26aa185a77f0ea858eb12bc5c0ae1f9cd17d5b4f5fe3697bce34c958ee18b2f"    # main-4cd108e
+RUNTIME_GENERIC_IMAGE ?= "quay.io/opendatahub/ds-pipelines-runtime-generic@sha256:10fe95cd2e6b85c865249d26e99571fc9a9897a75070c02fa5dcfdb3252d3eb2"    # main-3dd8265
 
 standalone:
 	python3 pipeline.py gen-standalone
diff --git a/pipeline.py b/pipeline.py
index 384a8955..985362b1 100644
--- a/pipeline.py
+++ b/pipeline.py
@@ -37,7 +37,7 @@
     pvc_to_mt_bench_op,
     upload_model_op,
 )
-from utils.consts import RHELAI_IMAGE
+from utils.consts import RHELAI_IMAGE, RUNTIME_GENERIC_IMAGE
 
 TEACHER_CONFIG_MAP = "teacher-server"
 TEACHER_SECRET = "teacher-server"
@@ -188,6 +188,10 @@ def ilab_pipeline(
         storage_class_name=k8s_storage_class_name,
     )
 
+    model_tokenizer_source_task = dsl.importer(
+        artifact_uri=f"oci://{RUNTIME_GENERIC_IMAGE}", artifact_class=dsl.Model
+    )
+
     sdg_task = sdg_op(
         num_instructions_to_generate=sdg_scale_factor,
         pipeline=sdg_pipeline,
@@ -197,6 +201,7 @@ def ilab_pipeline(
         sdg_secret_name=sdg_teacher_secret,
         repo_url=sdg_repo_url,
         taxonomy_repo_secret=sdg_repo_secret,
+        tokenizer_model=model_tokenizer_source_task.output,
     )
     sdg_task.set_env_variable("HOME", "/tmp")
     sdg_task.set_env_variable("HF_HOME", "/tmp")
diff --git a/pipeline.yaml b/pipeline.yaml
index 0832ccb7..1454131d 100644
--- a/pipeline.yaml
+++ b/pipeline.yaml
@@ -303,6 +303,18 @@ components:
           artifactType:
             schemaTitle: system.Model
             schemaVersion: 0.0.1
+  comp-importer-2:
+    executorLabel: exec-importer-2
+    inputDefinitions:
+      parameters:
+        uri:
+          parameterType: STRING
+    outputDefinitions:
+      artifacts:
+        artifact:
+          artifactType:
+            schemaTitle: system.Model
+            schemaVersion: 0.0.1
   comp-knowledge-processed-data-to-artifact-op:
     executorLabel: exec-knowledge-processed-data-to-artifact-op
     inputDefinitions:
@@ -584,6 +596,11 @@ components:
   comp-sdg-op:
     executorLabel: exec-sdg-op
     inputDefinitions:
+      artifacts:
+        tokenizer_model:
+          artifactType:
+            schemaTitle: system.Model
+            schemaVersion: 0.0.1
       parameters:
         num_instructions_to_generate:
           parameterType: NUMBER_INTEGER
@@ -822,6 +839,13 @@ deploymentSpec:
           ]\n            )\n\n"
         image: quay.io/modh/odh-generic-data-science-notebook@sha256:72c1d095adbda216a1f1b4b6935e3e2c717cbc58964009464ccd36c0b98312b2
     exec-importer:
+      importer:
+        artifactUri:
+          constant: oci://quay.io/opendatahub/ds-pipelines-runtime-generic@sha256:10fe95cd2e6b85c865249d26e99571fc9a9897a75070c02fa5dcfdb3252d3eb2
+        typeSchema:
+          schemaTitle: system.Model
+          schemaVersion: 0.0.1
+    exec-importer-2:
       importer:
         artifactUri:
           runtimeParameter: uri
@@ -863,7 +887,7 @@ deploymentSpec:
           \ f)\n        print(f\"Copying {src} to {dest}\")\n        if os.path.isdir(src):\n\
           \            shutil.copytree(src, dest)\n        else:\n            shutil.copy(src,\
           \ dest)\n\n"
-        image: quay.io/opendatahub/ds-pipelines-runtime-generic@sha256:a26aa185a77f0ea858eb12bc5c0ae1f9cd17d5b4f5fe3697bce34c958ee18b2f
+        image: quay.io/opendatahub/ds-pipelines-runtime-generic@sha256:10fe95cd2e6b85c865249d26e99571fc9a9897a75070c02fa5dcfdb3252d3eb2
     exec-pvc-to-mmlu-branch-op:
       container:
         args:
@@ -1635,17 +1659,17 @@ deploymentSpec:
           '
         - "\nimport kfp\nfrom kfp import dsl\nfrom kfp.dsl import *\nfrom typing import\
           \ *\n\ndef sdg_op(\n    num_instructions_to_generate: int,\n    pipeline:\
-          \ str,\n    repo_branch: Optional[str],\n    repo_pr: Optional[int],\n \
-          \   taxonomy_path: str = \"/data/taxonomy\",\n    sdg_path: str = \"/data/sdg\"\
-          ,\n    sdg_sampling_size: float = 1.0,\n    sdg_secret_name: str = None,\n\
-          \    taxonomy_repo_secret: str = None,\n    repo_url: str = None,\n):\n\
-          \    import base64\n    import os\n    import re\n    import shutil\n  \
-          \  import subprocess\n    import tempfile\n    import urllib.parse\n\n \
-          \   import instructlab.sdg\n    import openai\n    import requests\n   \
-          \ import xdg_base_dirs\n    import yaml\n\n    REQUEST_TIMEOUT = 30  # seconds\n\
-          \n    def fetch_secret(secret_name, keys):\n        # Kubernetes API server\
-          \ inside the cluster\n        K8S_API_SERVER = \"https://kubernetes.default.svc\"\
-          \n        NAMESPACE_PATH = \"/var/run/secrets/kubernetes.io/serviceaccount/namespace\"\
+          \ str,\n    tokenizer_model: dsl.Input[dsl.Model],\n    repo_branch: Optional[str],\n\
+          \    repo_pr: Optional[int],\n    taxonomy_path: str = \"/data/taxonomy\"\
+          ,\n    sdg_path: str = \"/data/sdg\",\n    sdg_sampling_size: float = 1.0,\n\
+          \    sdg_secret_name: str = None,\n    taxonomy_repo_secret: str = None,\n\
+          \    repo_url: str = None,\n):\n    import base64\n    import os\n    import\
+          \ os.path\n    import re\n    import shutil\n    import subprocess\n   \
+          \ import tempfile\n    import urllib.parse\n\n    import instructlab.sdg\n\
+          \    import openai\n    import requests\n    import xdg_base_dirs\n    import\
+          \ yaml\n\n    REQUEST_TIMEOUT = 30  # seconds\n\n    def fetch_secret(secret_name,\
+          \ keys):\n        # Kubernetes API server inside the cluster\n        K8S_API_SERVER\
+          \ = \"https://kubernetes.default.svc\"\n        NAMESPACE_PATH = \"/var/run/secrets/kubernetes.io/serviceaccount/namespace\"\
           \n        TOKEN_PATH = \"/var/run/secrets/kubernetes.io/serviceaccount/token\"\
           \n\n        # Fetch namespace\n        try:\n            with open(NAMESPACE_PATH,\
           \ \"r\") as f:\n                namespace = f.read().strip()\n        except\
@@ -1683,21 +1707,28 @@ deploymentSpec:
           \ github.com, gitlab.com) from an SSH repository URL.\"\"\"\n        match\
           \ = re.match(r\"git@([\\w.-]+):\", repo_url)\n        if match:\n      \
           \      return match.group(1)  # Extracted host\n        raise ValueError(f\"\
-          Invalid SSH repository URL: {repo_url}\")\n\n    if not taxonomy_repo_secret:\n\
-          \        username = os.getenv(\"GIT_USERNAME\")\n        token = os.getenv(\"\
-          GIT_TOKEN\")\n        ssh_key = os.getenv(\"GIT_SSH_KEY\")\n    else:\n\
-          \        print(\"SDG Repo secret specified, fetching...\")\n        username,\
-          \ token, ssh_key = fetch_secret(\n            taxonomy_repo_secret, [\"\
-          GIT_USERNAME\", \"GIT_TOKEN\", \"GIT_SSH_KEY\"]\n        )\n        print(\"\
-          SDG Repo secret data retrieved.\")\n\n    # Whether not provided via env\
-          \ or secret\n    # Assume the repo is public\n    if not username or ssh_key:\n\
-          \        print(\"No credentials provided for taxonomy repo, assuming public\
-          \ repo...\")\n\n    ca_cert_path = os.getenv(\"SDG_CA_CERT_PATH\")\n   \
-          \ env = os.environ.copy()\n\n    # Todo(HumairAK): should be platform cert\n\
-          \    # Set custom CA certificate if provided\n    if ca_cert_path:\n   \
-          \     print(f\"SSL CA Info detected. ca cert path: {ca_cert_path}\")\n \
-          \       env[\"GIT_SSL_CAINFO\"] = ca_cert_path\n        # Consume this in\
-          \ the process execution environment\n        # This is required for read_taxonomy\
+          Invalid SSH repository URL: {repo_url}\")\n\n    tokenizer_model_path =\
+          \ tokenizer_model.path\n    if tokenizer_model_path.startswith(\"oci://\"\
+          ):\n        # Handle where the KFP SDK is <2.12.0.\n        escaped_uri\
+          \ = tokenizer_model_path[len(\"oci://\") :].replace(\"/\", \"\\\\/\")\n\
+          \        tokenizer_model_path = os.path.join(\"/oci\", escaped_uri, \"models\"\
+          )\n\n    # A hack because InstructLab assumes the value for model_name is\
+          \ a valid path and the name of the model.\n    os.symlink(tokenizer_model_path,\
+          \ os.path.join(tempfile.gettempdir(), \"mixtral\"))\n    os.chdir(tempfile.gettempdir())\n\
+          \n    if not taxonomy_repo_secret:\n        username = os.getenv(\"GIT_USERNAME\"\
+          )\n        token = os.getenv(\"GIT_TOKEN\")\n        ssh_key = os.getenv(\"\
+          GIT_SSH_KEY\")\n    else:\n        print(\"SDG Repo secret specified, fetching...\"\
+          )\n        username, token, ssh_key = fetch_secret(\n            taxonomy_repo_secret,\
+          \ [\"GIT_USERNAME\", \"GIT_TOKEN\", \"GIT_SSH_KEY\"]\n        )\n      \
+          \  print(\"SDG Repo secret data retrieved.\")\n\n    # Whether not provided\
+          \ via env or secret\n    # Assume the repo is public\n    if not username\
+          \ or ssh_key:\n        print(\"No credentials provided for taxonomy repo,\
+          \ assuming public repo...\")\n\n    ca_cert_path = os.getenv(\"SDG_CA_CERT_PATH\"\
+          )\n    env = os.environ.copy()\n\n    # Todo(HumairAK): should be platform\
+          \ cert\n    # Set custom CA certificate if provided\n    if ca_cert_path:\n\
+          \        print(f\"SSL CA Info detected. ca cert path: {ca_cert_path}\")\n\
+          \        env[\"GIT_SSL_CAINFO\"] = ca_cert_path\n        # Consume this\
+          \ in the process execution environment\n        # This is required for read_taxonomy\
           \ calls which\n        # Perform nested calls to git clones.\n        os.environ[\"\
           GIT_SSL_CAINFO\"] = ca_cert_path\n\n    git_credentials_path = \"\"\n  \
           \  ssh_key_path = \"\"\n    # Username/PAT takes precedence over ssh\n \
@@ -1750,13 +1781,12 @@ deploymentSpec:
           ],\n            cwd=taxonomy_path,\n            env=env,\n        )\n  \
           \      exec_cmd([\"git\", \"checkout\", repo_branch], cwd=taxonomy_path,\
           \ env=env)\n\n    if sdg_secret_name is None:\n        api_key = os.getenv(\"\
-          api_key\")\n        model = os.getenv(\"model\")\n        endpoint = os.getenv(\"\
-          endpoint\")\n    else:\n        print(\"SDG Teacher secret specified, fetching...\"\
-          )\n        api_key, model, endpoint = fetch_secret(\n            sdg_secret_name,\
-          \ [\"api_token\", \"model_name\", \"endpoint\"]\n        )\n        print(\"\
-          SDG Teacher secret data retrieved.\")\n\n    sdg_ca_cert_path = os.getenv(\"\
-          SDG_CA_CERT_PATH\")\n    use_tls = os.path.exists(sdg_ca_cert_path) and\
-          \ (\n        os.path.getsize(sdg_ca_cert_path) > 0\n    )\n    if use_tls:\n\
+          api_key\")\n        endpoint = os.getenv(\"endpoint\")\n    else:\n    \
+          \    print(\"SDG Teacher secret specified, fetching...\")\n        api_key,\
+          \ endpoint = fetch_secret(sdg_secret_name, [\"api_token\", \"endpoint\"\
+          ])\n        print(\"SDG Teacher secret data retrieved.\")\n\n    sdg_ca_cert_path\
+          \ = os.getenv(\"SDG_CA_CERT_PATH\")\n    use_tls = os.path.exists(sdg_ca_cert_path)\
+          \ and (\n        os.path.getsize(sdg_ca_cert_path) > 0\n    )\n    if use_tls:\n\
           \        import httpx\n\n        custom_http_client = httpx.Client(verify=sdg_ca_cert_path)\n\
           \        client = openai.OpenAI(\n            base_url=endpoint, api_key=api_key,\
           \ http_client=custom_http_client\n        )\n    else:\n        client =\
@@ -1772,9 +1802,9 @@ deploymentSpec:
           \        instructlab.sdg.generate_data(\n            client=client,\n  \
           \          num_instructions_to_generate=num_instructions_to_generate,\n\
           \            output_dir=sdg_path,\n            taxonomy=taxonomy_path,\n\
-          \            taxonomy_base=taxonomy_base,\n            model_name=model,\n\
-          \            pipeline=pipeline,\n            chunk_word_count=1000,\n  \
-          \          server_ctx_size=4096,\n        )\n    # Tweak precomputed skills\
+          \            taxonomy_base=taxonomy_base,\n            model_name=\"mixtral\"\
+          ,\n            pipeline=pipeline,\n            chunk_word_count=1000,\n\
+          \            server_ctx_size=4096,\n        )\n    # Tweak precomputed skills\
           \ data ratio if needed\n    else:\n        skills_recipe = \"/usr/share/instructlab/sdg/default_data_recipes/skills.yaml\"\
           \n\n        def set_precomputed_skills_data_ratio(sampling_size: float,\
           \ skills_recipe: str):\n            if os.path.exists(skills_recipe):\n\
@@ -1785,6 +1815,8 @@ deploymentSpec:
           \ file:\n                    yaml.dump(skills_yaml, file)\n\n        try:\n\
           \            set_precomputed_skills_data_ratio(\n                sampling_size=sdg_sampling_size,\
           \ skills_recipe=skills_recipe\n            )\n        except PermissionError:\n\
+          \            # TODO(mprahl): When upgrading to RHEL AI 1.4, replace all\
+          \ this with:\n            # https://github.com/instructlab/sdg/blob/v0.7.1/docs/examples/mix_datasets/example_mixing.py\n\
           \            print(\"Failed to set precomputed skills data ratio: Permission\
           \ denied\")\n            print(\"Attempting to move default data recipes\
           \ to temporary directory\")\n\n            # Create a temporary directory\n\
@@ -1800,21 +1832,31 @@ deploymentSpec:
           \               os.path.join(str(dir), \"instructlab\", \"sdg\")\n     \
           \               for dir in xdg_base_dirs.xdg_data_dirs()\n             \
           \   ]\n                temp_pipeline_dir = os.path.join(temp_dir, \"pipeline\"\
-          )\n                os.mkdir(temp_pipeline_dir)\n                for d in\
-          \ data_dirs:\n                    pipeline_path = os.path.join(d, \"pipelines\"\
-          , pipeline)\n                    if os.path.exists(pipeline_path):\n   \
-          \                     shutil.copytree(\n                            pipeline_path,\n\
+          )\n                os.mkdir(temp_pipeline_dir)\n                temp_models_dir\
+          \ = os.path.join(temp_dir, \"instructlab\", \"sdg\", \"models\")\n     \
+          \           os.makedirs(temp_models_dir, exist_ok=True)\n\n            \
+          \    for d in data_dirs:\n                    pipeline_path = os.path.join(d,\
+          \ \"pipelines\", pipeline)\n                    if os.path.exists(pipeline_path):\n\
+          \                        shutil.copytree(\n                            pipeline_path,\n\
           \                            temp_pipeline_dir,\n                      \
           \      dirs_exist_ok=True,\n                        )\n                \
-          \        break\n\n                # Build new skills.yaml path\n       \
-          \         new_skills_recipe = os.path.join(temp_dir, \"skills.yaml\")\n\
-          \                print(f\"New skills recipe path: {new_skills_recipe}\"\
-          )\n\n                # Override XDG_DATA_DIRS with the temporary directory\n\
-          \                # This allows SDG to read the new skills.yaml since it's\
-          \ looking into XDG_DATA_DIRS\n                # and looks for a default_data_recipes\
-          \ directory with a skills.yaml file\n                os.environ[\"XDG_DATA_DIRS\"\
-          ] = f\"{temp_dir}\"\n\n                # Try to set the precomputed skills\
-          \ data ratio again\n                try:\n                    set_precomputed_skills_data_ratio(\n\
+          \        break\n\n                # The docling SDG model is also needed,\
+          \ so just copy the models config which has paths to the\n              \
+          \  # default models.\n                for d in data_dirs:\n            \
+          \        models_config_path = os.path.join(d, \"models\", \"config.yaml\"\
+          )\n                    if os.path.exists(models_config_path):\n        \
+          \                shutil.copy(\n                            models_config_path,\n\
+          \                            os.path.join(temp_models_dir, \"config.yaml\"\
+          ),\n                        )\n                        break\n\n       \
+          \         # Build new skills.yaml path\n                new_skills_recipe\
+          \ = os.path.join(temp_dir, \"skills.yaml\")\n                print(f\"New\
+          \ skills recipe path: {new_skills_recipe}\")\n\n                # Override\
+          \ XDG_DATA_DIRS with the temporary directory\n                # This allows\
+          \ SDG to read the new skills.yaml since it's looking into XDG_DATA_DIRS\n\
+          \                # and looks for a default_data_recipes directory with a\
+          \ skills.yaml file\n                os.environ[\"XDG_DATA_DIRS\"] = f\"\
+          {temp_dir}\"\n\n                # Try to set the precomputed skills data\
+          \ ratio again\n                try:\n                    set_precomputed_skills_data_ratio(\n\
           \                        sampling_size=sdg_sampling_size, skills_recipe=new_skills_recipe\n\
           \                    )\n                    print(\n                   \
           \     f\"Successfully set precomputed skills data ratio to {sdg_sampling_size}\"\
@@ -1825,7 +1867,7 @@ deploymentSpec:
           \      client=client,\n                        num_instructions_to_generate=num_instructions_to_generate,\n\
           \                        output_dir=sdg_path,\n                        taxonomy=taxonomy_path,\n\
           \                        taxonomy_base=taxonomy_base,\n                \
-          \        model_name=model,\n                        pipeline=pipeline,\n\
+          \        model_name=\"mixtral\",\n                        pipeline=pipeline,\n\
           \                        chunk_word_count=1000,\n                      \
           \  server_ctx_size=4096,\n                    )\n                except\
           \ Exception as e:\n                    print(f\"Failed to set precomputed\
@@ -1999,7 +2041,7 @@ deploymentSpec:
           modelVersionName\"] = output_model_version\n    model.metadata[\"modelVersionId\"\
           ] = model_version_id\n    model.metadata[\"registeredModelId\"] = registered_model.id\n\
           \n"
-        image: quay.io/opendatahub/ds-pipelines-runtime-generic@sha256:a26aa185a77f0ea858eb12bc5c0ae1f9cd17d5b4f5fe3697bce34c958ee18b2f
+        image: quay.io/opendatahub/ds-pipelines-runtime-generic@sha256:10fe95cd2e6b85c865249d26e99571fc9a9897a75070c02fa5dcfdb3252d3eb2
 pipelineInfo:
   description: InstructLab pipeline
   displayName: InstructLab
@@ -2151,9 +2193,21 @@ root:
         inputs:
           parameters:
             uri:
-              componentInputParameter: sdg_base_model
+              runtimeValue:
+                constant: oci://quay.io/opendatahub/ds-pipelines-runtime-generic@sha256:10fe95cd2e6b85c865249d26e99571fc9a9897a75070c02fa5dcfdb3252d3eb2
         taskInfo:
           name: importer
+      importer-2:
+        cachingOptions:
+          enableCache: true
+        componentRef:
+          name: comp-importer-2
+        inputs:
+          parameters:
+            uri:
+              componentInputParameter: sdg_base_model
+        taskInfo:
+          name: importer-2
       knowledge-processed-data-to-artifact-op:
         cachingOptions: {}
         componentRef:
@@ -2169,13 +2223,13 @@ root:
           name: comp-model-to-pvc-op
         dependentTasks:
         - createpvc-2
-        - importer
+        - importer-2
         inputs:
           artifacts:
             model:
               taskOutputArtifact:
                 outputArtifactKey: artifact
-                producerTask: importer
+                producerTask: importer-2
         taskInfo:
           name: model-to-pvc-op
       pvc-to-mmlu-branch-op:
@@ -2416,7 +2470,13 @@ root:
           name: comp-sdg-op
         dependentTasks:
         - createpvc
+        - importer
         inputs:
+          artifacts:
+            tokenizer_model:
+              taskOutputArtifact:
+                outputArtifactKey: artifact
+                producerTask: importer
           parameters:
             num_instructions_to_generate:
               componentInputParameter: sdg_scale_factor
diff --git a/sdg/components.py b/sdg/components.py
index cc596e79..fc64fac7 100644
--- a/sdg/components.py
+++ b/sdg/components.py
@@ -53,6 +53,7 @@ def git_clone_op(
 def sdg_op(
     num_instructions_to_generate: int,
     pipeline: str,
+    tokenizer_model: dsl.Input[dsl.Model],
     repo_branch: Optional[str],
     repo_pr: Optional[int],
     taxonomy_path: str = "/data/taxonomy",
@@ -64,6 +65,7 @@ def sdg_op(
 ):
     import base64
     import os
+    import os.path
     import re
     import shutil
     import subprocess
@@ -151,6 +153,16 @@ def get_git_host(repo_url):
             return match.group(1)  # Extracted host
         raise ValueError(f"Invalid SSH repository URL: {repo_url}")
 
+    tokenizer_model_path = tokenizer_model.path
+    if tokenizer_model_path.startswith("oci://"):
+        # Handle where the KFP SDK is <2.12.0.
+        escaped_uri = tokenizer_model_path[len("oci://") :].replace("/", "\\/")
+        tokenizer_model_path = os.path.join("/oci", escaped_uri, "models")
+
+    # A hack because InstructLab assumes the value for model_name is a valid path and the name of the model.
+    os.symlink(tokenizer_model_path, os.path.join(tempfile.gettempdir(), "mixtral"))
+    os.chdir(tempfile.gettempdir())
+
     if not taxonomy_repo_secret:
         username = os.getenv("GIT_USERNAME")
         token = os.getenv("GIT_TOKEN")
@@ -277,13 +289,10 @@ def get_git_host(repo_url):
 
     if sdg_secret_name is None:
         api_key = os.getenv("api_key")
-        model = os.getenv("model")
         endpoint = os.getenv("endpoint")
     else:
         print("SDG Teacher secret specified, fetching...")
-        api_key, model, endpoint = fetch_secret(
-            sdg_secret_name, ["api_token", "model_name", "endpoint"]
-        )
+        api_key, endpoint = fetch_secret(sdg_secret_name, ["api_token", "endpoint"])
         print("SDG Teacher secret data retrieved.")
 
     sdg_ca_cert_path = os.getenv("SDG_CA_CERT_PATH")
@@ -322,7 +331,7 @@ def get_git_host(repo_url):
             output_dir=sdg_path,
             taxonomy=taxonomy_path,
             taxonomy_base=taxonomy_base,
-            model_name=model,
+            model_name="mixtral",
             pipeline=pipeline,
             chunk_word_count=1000,
             server_ctx_size=4096,
@@ -346,6 +355,8 @@ def set_precomputed_skills_data_ratio(sampling_size: float, skills_recipe: str):
                 sampling_size=sdg_sampling_size, skills_recipe=skills_recipe
             )
         except PermissionError:
+            # TODO(mprahl): When upgrading to RHEL AI 1.4, replace all this with:
+            # https://github.com/instructlab/sdg/blob/v0.7.1/docs/examples/mix_datasets/example_mixing.py
             print("Failed to set precomputed skills data ratio: Permission denied")
             print("Attempting to move default data recipes to temporary directory")
 
@@ -367,6 +378,9 @@ def set_precomputed_skills_data_ratio(sampling_size: float, skills_recipe: str):
                 ]
                 temp_pipeline_dir = os.path.join(temp_dir, "pipeline")
                 os.mkdir(temp_pipeline_dir)
+                temp_models_dir = os.path.join(temp_dir, "instructlab", "sdg", "models")
+                os.makedirs(temp_models_dir, exist_ok=True)
+
                 for d in data_dirs:
                     pipeline_path = os.path.join(d, "pipelines", pipeline)
                     if os.path.exists(pipeline_path):
@@ -377,6 +391,17 @@ def set_precomputed_skills_data_ratio(sampling_size: float, skills_recipe: str):
                         )
                         break
 
+                # The docling SDG model is also needed, so just copy the models config which has paths to the
+                # default models.
+                for d in data_dirs:
+                    models_config_path = os.path.join(d, "models", "config.yaml")
+                    if os.path.exists(models_config_path):
+                        shutil.copy(
+                            models_config_path,
+                            os.path.join(temp_models_dir, "config.yaml"),
+                        )
+                        break
+
                 # Build new skills.yaml path
                 new_skills_recipe = os.path.join(temp_dir, "skills.yaml")
                 print(f"New skills recipe path: {new_skills_recipe}")
@@ -404,7 +429,7 @@ def set_precomputed_skills_data_ratio(sampling_size: float, skills_recipe: str):
                         output_dir=sdg_path,
                         taxonomy=taxonomy_path,
                         taxonomy_base=taxonomy_base,
-                        model_name=model,
+                        model_name="mixtral",
                         pipeline=pipeline,
                         chunk_word_count=1000,
                         server_ctx_size=4096,


Reviews:
user: login: rimolive
body: I added some comments.
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-07T22:47:30Z
user: login: mprahl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T13:12:19Z
user: login: mprahl
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T13:14:02Z
user: login: gmfrasca
body: /lgtm
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T18:28:25Z

Review comments:
diff_hunk: @@ -404,7 +429,7 @@ def set_precomputed_skills_data_ratio(sampling_size: float, skills_recipe: str):
                         output_dir=sdg_path,
                         taxonomy=taxonomy_path,
                         taxonomy_base=taxonomy_base,
-                        model_name=model,
+                        model_name="mixtral",
path: sdg/components.py
user: login: rimolive
body: Should this be hard-coded?
created_at: 2025-03-07T22:43:14Z
author_association: NONE
diff_hunk: @@ -367,6 +378,9 @@ def set_precomputed_skills_data_ratio(sampling_size: float, skills_recipe: str):
                 ]
                 temp_pipeline_dir = os.path.join(temp_dir, "pipeline")
                 os.mkdir(temp_pipeline_dir)
+                temp_models_dir = os.path.join(temp_dir, "instructlab", "sdg", "models")
+                os.makedirs(temp_models_dir, exist_ok=True)
path: sdg/components.py
user: login: rimolive
body: Optional: It would be good to set proper permissions to avoid potential Permission Denied errors (assuming this path is stored in a PVC).
created_at: 2025-03-07T22:45:26Z
author_association: NONE
diff_hunk: @@ -322,7 +331,7 @@ def get_git_host(repo_url):
             output_dir=sdg_path,
             taxonomy=taxonomy_path,
             taxonomy_base=taxonomy_base,
-            model_name=model,
+            model_name="mixtral",
path: sdg/components.py
user: login: rimolive
body: Should this be hard-coded?
created_at: 2025-03-07T22:45:39Z
author_association: NONE
diff_hunk: @@ -367,6 +378,9 @@ def set_precomputed_skills_data_ratio(sampling_size: float, skills_recipe: str):
                 ]
                 temp_pipeline_dir = os.path.join(temp_dir, "pipeline")
                 os.mkdir(temp_pipeline_dir)
+                temp_models_dir = os.path.join(temp_dir, "instructlab", "sdg", "models")
+                os.makedirs(temp_models_dir, exist_ok=True)
path: sdg/components.py
user: login: mprahl
body: This is coming from the `tempfile.TemporaryDirectory` context manager so it's just in a directory in `/tmp`, so the current user will have write access and we don't have to worry about other users/pods having access since the data is deleted after the context manager is done executing.
created_at: 2025-03-10T13:12:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -404,7 +429,7 @@ def set_precomputed_skills_data_ratio(sampling_size: float, skills_recipe: str):
                         output_dir=sdg_path,
                         taxonomy=taxonomy_path,
                         taxonomy_base=taxonomy_base,
-                        model_name=model,
+                        model_name="mixtral",
path: sdg/components.py
user: login: mprahl
body: Yes, because the embedded tokenizer is Mixtral and that is the only one that RHEL AI supports right now from my understanding. If we add support for more in the future, we'll need to make this a parameter and have RHEL AI distribute tokenizers in a better way (not included in an ~80GB container image). The latter is tracked in:
https://issues.redhat.com/browse/RHELAIRFE-10
created_at: 2025-03-10T13:14:02Z
author_association: CONTRIBUTOR

PR comments:
user: login: mprahl
created_at: 2025-03-07T20:57:10Z
body: /cc @gmfrasca @rimolive 
author_association: CONTRIBUTOR
user: login: rimolive
created_at: 2025-03-11T18:34:23Z
body: /lgtm
author_association: NONE