Repository: kaito-project/kaito
Pull request #950: feat: reimplement pull and push of images with skopeo and oras
Author: login: t0rr3sp3dr0
Created: 2025-03-25T17:48:35Z | Merged: 2025-04-04T17:50:03Z

**Reason for Change**:
This patch allows us to get rid of DinD, which currently runs in a privileged container. Skopeo will be used to pull images, mimicking the exact pull behavior of [K8s Image Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/). While ORAS is used to build the image in a way compatible with the implementation of [K8s Image Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/) in `containerd` and `cri-o`.

**Requirements**
- [ ] added unit tests and e2e tests (if applicable).

**Issue Fixed**:
Fixes #901

**Notes for Reviewers**:
After this patch, images produced with newer versions of Kaito won't run on older versions of Kaito. However, images produced by older versions of Kaito will run on newer versions of Kaito. We probably want to make this clear on the release notes.

Linked issue #901: Use skopeo as the image management tool to manage adapters
**Is your feature request related to a problem? Please describe.**

Currently, we use docker to push adapter images using busybox base image. It introduces a few problems:
1) Busybox base image can potentially contain CVEs, which may pollute user container registries;
2) We are not building adapter images with mutli-arch support, so it does not work in ARM GPU;
3) Docker may hit issues when used in rootless containers. 

**Describe the solution you'd like**

We will change the way we load adapters in the init-containers.
- The adapter image will NOT be the image of the init-container
- We use base image of tools like "skopeo" as the init-container image, which will be used to download the adapter image locally (completely avoid the multi-arch issue) and extract the data layer out. 
- We use skopeo in the sidecar container to push container images without using busybox base image (using distroless or other safe base images)



Diff:
diff --git a/api/v1alpha1/workspace_validation.go b/api/v1alpha1/workspace_validation.go
index 1a8dec1f44..304b7257fb 100644
--- a/api/v1alpha1/workspace_validation.go
+++ b/api/v1alpha1/workspace_validation.go
@@ -8,10 +8,10 @@ import (
 	"fmt"
 	"os"
 	"reflect"
-	"regexp"
 	"strconv"
 	"strings"
 
+	"github.com/distribution/reference"
 	admissionregistrationv1 "k8s.io/api/admissionregistration/v1"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/errors"
@@ -207,18 +207,11 @@ func (r *DataSource) validateCreate() (errs *apis.FieldError) {
 		errs = errs.Also(apis.ErrInvalidValue("Volume support is not implemented yet", "Volume"))
 		sourcesSpecified++
 	}
-	// Regex checks for a / and a colon followed by a tag
-	if r.Image != "" {
-		re := regexp.MustCompile(`^(.+/[^:/]+):([^:/]+)$`)
-		if !re.MatchString(r.Image) {
-			errs = errs.Also(apis.ErrInvalidValue("Invalid image format, require full input image URL", "Image"))
-		} else {
-			// Executes if image is of correct format
-			err := utils.ExtractAndValidateRepoName(r.Image)
-			if err != nil {
-				errs = errs.Also(apis.ErrInvalidValue(err.Error(), "Image"))
-			}
+	if image := r.Image; image != "" {
+		if _, err := reference.ParseDockerRef(image); err != nil {
+			errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("Unable to parse input image reference: %s", err), "Image"))
 		}
+
 		sourcesSpecified++
 	}
 
@@ -237,6 +230,11 @@ func (r *DataSource) validateUpdate(old *DataSource, isTuning bool) (errs *apis.
 	if r.Volume != nil {
 		errs = errs.Also(apis.ErrInvalidValue("Volume support is not implemented yet", "Volume"))
 	}
+	if image := r.Image; image != "" {
+		if _, err := reference.ParseDockerRef(image); err != nil {
+			errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("Unable to parse input image reference: %s", err), "Image"))
+		}
+	}
 
 	return errs
 }
@@ -248,22 +246,16 @@ func (r *DataDestination) validateCreate() (errs *apis.FieldError) {
 		errs = errs.Also(apis.ErrInvalidValue("Volume support is not implemented yet", "Volume"))
 		destinationsSpecified++
 	}
-	if r.Image != "" {
-		// Regex checks for a / and a colon followed by a tag
-		re := regexp.MustCompile(`^(.+/[^:/]+):([^:/]+)$`)
-		if !re.MatchString(r.Image) {
-			errs = errs.Also(apis.ErrInvalidValue("Invalid image format, require full output image URL", "Image"))
-		} else {
-			// Executes if image is of correct format
-			err := utils.ExtractAndValidateRepoName(r.Image)
-			if err != nil {
-				errs = errs.Also(apis.ErrInvalidValue(err.Error(), "Image"))
-			}
+	if image := r.Image; image != "" {
+		if _, err := reference.ParseDockerRef(image); err != nil {
+			errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("Unable to parse output image reference: %s", err), "Image"))
 		}
+
 		// Cloud Provider requires credentials to push image
 		if r.ImagePushSecret == "" {
 			errs = errs.Also(apis.ErrMissingField("Must specify imagePushSecret with destination image"))
 		}
+
 		destinationsSpecified++
 	}
 
@@ -279,6 +271,11 @@ func (r *DataDestination) validateUpdate() (errs *apis.FieldError) {
 	if r.Volume != nil {
 		errs = errs.Also(apis.ErrInvalidValue("Volume support is not implemented yet", "Volume"))
 	}
+	if image := r.Image; image != "" {
+		if _, err := reference.ParseDockerRef(image); err != nil {
+			errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("Unable to parse output image reference: %s", err), "Image"))
+		}
+	}
 
 	return errs
 }
diff --git a/api/v1alpha1/workspace_validation_test.go b/api/v1alpha1/workspace_validation_test.go
index a49648af8f..cdc2ab4cb8 100644
--- a/api/v1alpha1/workspace_validation_test.go
+++ b/api/v1alpha1/workspace_validation_test.go
@@ -1167,8 +1167,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "All fields valid",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/test:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/test:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: "secret"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1178,8 +1178,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Verify QLoRA Config",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/test:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/test:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: "secret"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodQLora,
 			},
@@ -1189,7 +1189,7 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Missing Input",
 			tuningSpec: &TuningSpec{
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: ""},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: ""},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1210,7 +1210,7 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 			name: "Missing Preset",
 			tuningSpec: &TuningSpec{
 				Input:  &DataSource{Name: "valid-input"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: ""},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: ""},
 				Method: TuningMethodLora,
 			},
 			wantErr:   true,
@@ -1220,7 +1220,7 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 			name: "Invalid Preset",
 			tuningSpec: &TuningSpec{
 				Input:  &DataSource{Name: "valid-input"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: ""},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: ""},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("invalid-preset")}},
 				Method: TuningMethodLora,
 			},
@@ -1231,7 +1231,7 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 			name: "Invalid Method",
 			tuningSpec: &TuningSpec{
 				Input:  &DataSource{Name: "valid-input"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: ""},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: ""},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: "invalid-method",
 			},
@@ -1241,8 +1241,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Invalid Input Source Casing",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/INPUT:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/output:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/INPUT:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/output:0.0.0", ImagePushSecret: "secret"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1252,8 +1252,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Invalid Output Destination Casing",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/input:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/OUTPUT:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/input:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/OUTPUT:0.0.0", ImagePushSecret: "secret"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1295,13 +1295,13 @@ func TestTuningSpecValidateUpdate(t *testing.T) {
 			name: "No changes",
 			oldTuning: &TuningSpec{
 				Input:  &DataSource{Name: "input1"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
 			newTuning: &TuningSpec{
 				Input:  &DataSource{Name: "input1"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1368,7 +1368,7 @@ func TestDataSourceValidateCreate(t *testing.T) {
 		{
 			name: "Volume specified only",
 			dataSource: &DataSource{
-				Image: "AZURE_ACR.azurecr.io/test:0.0.0",
+				Image: "kaito.azurecr.io/test:0.0.0",
 			},
 			wantErr: false,
 		},
@@ -1385,7 +1385,7 @@ func TestDataSourceValidateCreate(t *testing.T) {
 				Image:            "data-image:latest",
 				ImagePullSecrets: []string{"imagePushSecret"},
 			},
-			wantErr: true,
+			wantErr: false,
 		},
 		{
 			name: "Image without Tag Specified",
@@ -1393,7 +1393,16 @@ func TestDataSourceValidateCreate(t *testing.T) {
 				Image:            "aimodels.azurecr.io/data-image",
 				ImagePullSecrets: []string{"imagePushSecret"},
 			},
-			wantErr: true,
+			wantErr: false,
+		},
+		{
+			name: "Invalid image",
+			dataSource: &DataSource{
+				Image:            "!ValidImage",
+				ImagePullSecrets: []string{"imagePushSecret"},
+			},
+			wantErr:  true,
+			errField: "invalid reference format",
 		},
 		{
 			name:       "None specified",
@@ -1472,6 +1481,16 @@ func TestDataSourceValidateUpdate(t *testing.T) {
 			wantErr:   true,
 			errFields: []string{"Name"},
 		},
+		{
+			name:      "Invalid image",
+			oldSource: &DataSource{},
+			newSource: &DataSource{
+				Image:            "!ValidImage",
+				ImagePullSecrets: []string{"imagePushSecret"},
+			},
+			wantErr:   true,
+			errFields: []string{"invalid reference format"},
+		},
 	}
 
 	for _, tt := range tests {
@@ -1528,7 +1547,7 @@ func TestDataDestinationValidateCreate(t *testing.T) {
 				Image:           "data-image:latest",
 				ImagePushSecret: "imagePushSecret",
 			},
-			wantErr: true,
+			wantErr: false,
 		},
 		{
 			name: "Image without Tag Specified",
@@ -1536,7 +1555,16 @@ func TestDataDestinationValidateCreate(t *testing.T) {
 				Image:           "aimodels.azurecr.io/data-image",
 				ImagePushSecret: "imagePushSecret",
 			},
-			wantErr: true,
+			wantErr: false,
+		},
+		{
+			name: "Invalid image",
+			dataDestination: &DataDestination{
+				Image:           "!ValidImage",
+				ImagePushSecret: "imagePushSecret",
+			},
+			wantErr:  true,
+			errField: "invalid reference format",
 		},
 		// {
 		// 	name: "Both fields specified",
@@ -1587,6 +1615,16 @@ func TestDataDestinationValidateUpdate(t *testing.T) {
 			},
 			wantErr: false,
 		},
+		{
+			name:    "Invalid image",
+			oldDest: &DataDestination{},
+			newDest: &DataDestination{
+				Image:           "!ValidImage",
+				ImagePushSecret: "imagePushSecret",
+			},
+			wantErr:   true,
+			errFields: []string{"invalid reference format"},
+		},
 	}
 
 	for _, tt := range tests {
diff --git a/api/v1beta1/workspace_validation.go b/api/v1beta1/workspace_validation.go
index b2b4653c08..76636140fa 100644
--- a/api/v1beta1/workspace_validation.go
+++ b/api/v1beta1/workspace_validation.go
@@ -8,10 +8,10 @@ import (
 	"fmt"
 	"os"
 	"reflect"
-	"regexp"
 	"strconv"
 	"strings"
 
+	"github.com/distribution/reference"
 	admissionregistrationv1 "k8s.io/api/admissionregistration/v1"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/errors"
@@ -207,18 +207,11 @@ func (r *DataSource) validateCreate() (errs *apis.FieldError) {
 		errs = errs.Also(apis.ErrInvalidValue("Volume support is not implemented yet", "Volume"))
 		sourcesSpecified++
 	}
-	// Regex checks for a / and a colon followed by a tag
-	if r.Image != "" {
-		re := regexp.MustCompile(`^(.+/[^:/]+):([^:/]+)$`)
-		if !re.MatchString(r.Image) {
-			errs = errs.Also(apis.ErrInvalidValue("Invalid image format, require full input image URL", "Image"))
-		} else {
-			// Executes if image is of correct format
-			err := utils.ExtractAndValidateRepoName(r.Image)
-			if err != nil {
-				errs = errs.Also(apis.ErrInvalidValue(err.Error(), "Image"))
-			}
+	if image := r.Image; image != "" {
+		if _, err := reference.ParseDockerRef(image); err != nil {
+			errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("Unable to parse input image reference: %s", err), "Image"))
 		}
+
 		sourcesSpecified++
 	}
 
@@ -237,6 +230,11 @@ func (r *DataSource) validateUpdate(old *DataSource, isTuning bool) (errs *apis.
 	if r.Volume != nil {
 		errs = errs.Also(apis.ErrInvalidValue("Volume support is not implemented yet", "Volume"))
 	}
+	if image := r.Image; image != "" {
+		if _, err := reference.ParseDockerRef(image); err != nil {
+			errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("Unable to parse input image reference: %s", err), "Image"))
+		}
+	}
 
 	return errs
 }
@@ -248,22 +246,16 @@ func (r *DataDestination) validateCreate() (errs *apis.FieldError) {
 		errs = errs.Also(apis.ErrInvalidValue("Volume support is not implemented yet", "Volume"))
 		destinationsSpecified++
 	}
-	if r.Image != "" {
-		// Regex checks for a / and a colon followed by a tag
-		re := regexp.MustCompile(`^(.+/[^:/]+):([^:/]+)$`)
-		if !re.MatchString(r.Image) {
-			errs = errs.Also(apis.ErrInvalidValue("Invalid image format, require full output image URL", "Image"))
-		} else {
-			// Executes if image is of correct format
-			err := utils.ExtractAndValidateRepoName(r.Image)
-			if err != nil {
-				errs = errs.Also(apis.ErrInvalidValue(err.Error(), "Image"))
-			}
+	if image := r.Image; image != "" {
+		if _, err := reference.ParseDockerRef(image); err != nil {
+			errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("Unable to parse output image reference: %s", err), "Image"))
 		}
+
 		// Cloud Provider requires credentials to push image
 		if r.ImagePushSecret == "" {
 			errs = errs.Also(apis.ErrMissingField("Must specify imagePushSecret with destination image"))
 		}
+
 		destinationsSpecified++
 	}
 
@@ -279,6 +271,11 @@ func (r *DataDestination) validateUpdate() (errs *apis.FieldError) {
 	if r.Volume != nil {
 		errs = errs.Also(apis.ErrInvalidValue("Volume support is not implemented yet", "Volume"))
 	}
+	if image := r.Image; image != "" {
+		if _, err := reference.ParseDockerRef(image); err != nil {
+			errs = errs.Also(apis.ErrInvalidValue(fmt.Sprintf("Unable to parse output image reference: %s", err), "Image"))
+		}
+	}
 
 	return errs
 }
diff --git a/api/v1beta1/workspace_validation_test.go b/api/v1beta1/workspace_validation_test.go
index 101e43de27..ec71c9709a 100644
--- a/api/v1beta1/workspace_validation_test.go
+++ b/api/v1beta1/workspace_validation_test.go
@@ -1167,8 +1167,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "All fields valid",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/test:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/test:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: "secret"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1178,8 +1178,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Verify QLoRA Config",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/test:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/test:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: "secret"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodQLora,
 			},
@@ -1189,7 +1189,7 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Missing Input",
 			tuningSpec: &TuningSpec{
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: ""},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: ""},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1210,7 +1210,7 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 			name: "Missing Preset",
 			tuningSpec: &TuningSpec{
 				Input:  &DataSource{Name: "valid-input"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: ""},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: ""},
 				Method: TuningMethodLora,
 			},
 			wantErr:   true,
@@ -1220,7 +1220,7 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 			name: "Invalid Preset",
 			tuningSpec: &TuningSpec{
 				Input:  &DataSource{Name: "valid-input"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: ""},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: ""},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("invalid-preset")}},
 				Method: TuningMethodLora,
 			},
@@ -1231,7 +1231,7 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 			name: "Invalid Method",
 			tuningSpec: &TuningSpec{
 				Input:  &DataSource{Name: "valid-input"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0", ImagePushSecret: ""},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0", ImagePushSecret: ""},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: "invalid-method",
 			},
@@ -1241,8 +1241,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Invalid Input Source Casing",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/INPUT:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/output:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/INPUT:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/output:0.0.0", ImagePushSecret: "secret"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1252,8 +1252,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Invalid Output Destination Casing",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/input:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/OUTPUT:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/input:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/OUTPUT:0.0.0", ImagePushSecret: "secret"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1295,13 +1295,13 @@ func TestTuningSpecValidateUpdate(t *testing.T) {
 			name: "No changes",
 			oldTuning: &TuningSpec{
 				Input:  &DataSource{Name: "input1"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
 			newTuning: &TuningSpec{
 				Input:  &DataSource{Name: "input1"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/test:0.0.0"},
+				Output: &DataDestination{Image: "kaito.azurecr.io/test:0.0.0"},
 				Preset: &PresetSpec{PresetMeta: PresetMeta{Name: ModelName("test-validation")}},
 				Method: TuningMethodLora,
 			},
@@ -1368,7 +1368,7 @@ func TestDataSourceValidateCreate(t *testing.T) {
 		{
 			name: "Volume specified only",
 			dataSource: &DataSource{
-				Image: "AZURE_ACR.azurecr.io/test:0.0.0",
+				Image: "kaito.azurecr.io/test:0.0.0",
 			},
 			wantErr: false,
 		},
@@ -1385,7 +1385,7 @@ func TestDataSourceValidateCreate(t *testing.T) {
 				Image:            "data-image:latest",
 				ImagePullSecrets: []string{"imagePushSecret"},
 			},
-			wantErr: true,
+			wantErr: false,
 		},
 		{
 			name: "Image without Tag Specified",
@@ -1393,7 +1393,16 @@ func TestDataSourceValidateCreate(t *testing.T) {
 				Image:            "aimodels.azurecr.io/data-image",
 				ImagePullSecrets: []string{"imagePushSecret"},
 			},
-			wantErr: true,
+			wantErr: false,
+		},
+		{
+			name: "Invalid image",
+			dataSource: &DataSource{
+				Image:            "!ValidImage",
+				ImagePullSecrets: []string{"imagePushSecret"},
+			},
+			wantErr:  true,
+			errField: "invalid reference format",
 		},
 		{
 			name:       "None specified",
@@ -1472,6 +1481,16 @@ func TestDataSourceValidateUpdate(t *testing.T) {
 			wantErr:   true,
 			errFields: []string{"Name"},
 		},
+		{
+			name:      "Invalid image",
+			oldSource: &DataSource{},
+			newSource: &DataSource{
+				Image:            "!ValidImage",
+				ImagePullSecrets: []string{"imagePushSecret"},
+			},
+			wantErr:   true,
+			errFields: []string{"invalid reference format"},
+		},
 	}
 
 	for _, tt := range tests {
@@ -1528,7 +1547,7 @@ func TestDataDestinationValidateCreate(t *testing.T) {
 				Image:           "data-image:latest",
 				ImagePushSecret: "imagePushSecret",
 			},
-			wantErr: true,
+			wantErr: false,
 		},
 		{
 			name: "Image without Tag Specified",
@@ -1536,7 +1555,16 @@ func TestDataDestinationValidateCreate(t *testing.T) {
 				Image:           "aimodels.azurecr.io/data-image",
 				ImagePushSecret: "imagePushSecret",
 			},
-			wantErr: true,
+			wantErr: false,
+		},
+		{
+			name: "Invalid image",
+			dataDestination: &DataDestination{
+				Image:           "!ValidImage",
+				ImagePushSecret: "imagePushSecret",
+			},
+			wantErr:  true,
+			errField: "invalid reference format",
 		},
 		// {
 		// 	name: "Both fields specified",
@@ -1587,6 +1615,16 @@ func TestDataDestinationValidateUpdate(t *testing.T) {
 			},
 			wantErr: false,
 		},
+		{
+			name:    "Invalid image",
+			oldDest: &DataDestination{},
+			newDest: &DataDestination{
+				Image:           "!ValidImage",
+				ImagePushSecret: "imagePushSecret",
+			},
+			wantErr:   true,
+			errFields: []string{"invalid reference format"},
+		},
 	}
 
 	for _, tt := range tests {
diff --git a/go.mod b/go.mod
index 65a0070ee8..7bcd88a6c0 100644
--- a/go.mod
+++ b/go.mod
@@ -8,6 +8,7 @@ require (
 	github.com/Azure/karpenter-provider-azure v0.7.0
 	github.com/aws/karpenter-provider-aws v1.0.6
 	github.com/awslabs/operatorpkg v0.0.0-20240805231134-67d0acfb6306
+	github.com/distribution/reference v0.5.0
 	github.com/go-logr/logr v1.4.2
 	github.com/onsi/ginkgo/v2 v2.22.2
 	github.com/onsi/gomega v1.36.2
@@ -72,6 +73,7 @@ require (
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
 	github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect
+	github.com/opencontainers/go-digest v1.0.0 // indirect
 	github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
 	github.com/pkg/errors v0.9.1 // indirect
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
diff --git a/go.sum b/go.sum
index a8d12e679d..ca02508b30 100644
--- a/go.sum
+++ b/go.sum
@@ -148,6 +148,8 @@ github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSs
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/distribution/reference v0.5.0 h1:/FUIFXtfc/x2gpa5/VGfiGLuOIdYa1t65IKK2OFGvA0=
+github.com/distribution/reference v0.5.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
 github.com/emicklei/go-restful/v3 v3.12.1 h1:PJMDIM/ak7btuL8Ex0iYET9hxM3CI2sjZtzpL63nKAU=
 github.com/emicklei/go-restful/v3 v3.12.1/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
 github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
@@ -371,6 +373,8 @@ github.com/onsi/ginkgo/v2 v2.22.2 h1:/3X8Panh8/WwhU/3Ssa6rCKqPLuAkVY2I0RoyDLySlU
 github.com/onsi/ginkgo/v2 v2.22.2/go.mod h1:oeMosUL+8LtarXBHu/c0bx2D/K9zyQ6uX3cTyztHwsk=
 github.com/onsi/gomega v1.36.2 h1:koNYke6TVk6ZmnyHrCXba/T/MoLBXFjeC1PtvYgw0A8=
 github.com/onsi/gomega v1.36.2/go.mod h1:DdwyADRjrc825LhMEkD76cHR5+pUnjhUN8GlHlRPHzY=
+github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
+github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
 github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs=
 github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc=
 github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
diff --git a/pkg/utils/common-preset.go b/pkg/utils/common-preset.go
index c9f5f8dd06..10fc21e4a6 100644
--- a/pkg/utils/common-preset.go
+++ b/pkg/utils/common-preset.go
@@ -3,6 +3,8 @@
 package utils
 
 import (
+	"fmt"
+
 	corev1 "k8s.io/api/core/v1"
 )
 
@@ -27,6 +29,45 @@ func ConfigResultsVolume(outputPath string) (corev1.Volume, corev1.VolumeMount)
 	return sharedWorkspaceVolume, sharedVolumeMount
 }
 
+func ConfigImagePullSecretVolume(nameSuffix string, imagePullSecrets []string) (corev1.Volume, corev1.VolumeMount) {
+	name := fmt.Sprintf("docker-config-%s", nameSuffix)
+
+	sources := make([]corev1.VolumeProjection, 0, len(imagePullSecrets))
+	for _, imagePullSecret := range imagePullSecrets {
+		volumeProjection := corev1.VolumeProjection{
+			Secret: &corev1.SecretProjection{
+				LocalObjectReference: corev1.LocalObjectReference{
+					Name: imagePullSecret,
+				},
+				Items: []corev1.KeyToPath{
+					{
+						Key:  ".dockerconfigjson",
+						Path: fmt.Sprintf("%s.json", imagePullSecret),
+					},
+				},
+			},
+		}
+
+		sources = append(sources, volumeProjection)
+	}
+
+	volume := corev1.Volume{
+		Name: name,
+		VolumeSource: corev1.VolumeSource{
+			Projected: &corev1.ProjectedVolumeSource{
+				Sources: sources,
+			},
+		},
+	}
+
+	volumeMount := corev1.VolumeMount{
+		Name:      name,
+		MountPath: fmt.Sprintf("/root/.docker/config.d/%s", nameSuffix),
+	}
+
+	return volume, volumeMount
+}
+
 func ConfigImagePushSecretVolume(imagePushSecret string) (corev1.Volume, corev1.VolumeMount) {
 	volume := corev1.Volume{
 		Name: "docker-config",
@@ -53,7 +94,7 @@ func ConfigImagePushSecretVolume(imagePushSecret string) (corev1.Volume, corev1.
 
 	volumeMount := corev1.VolumeMount{
 		Name:      "docker-config",
-		MountPath: "/tmp/.docker/config",
+		MountPath: "/root/.docker",
 	}
 
 	return volume, volumeMount
diff --git a/pkg/workspace/controllers/workspace_controller.go b/pkg/workspace/controllers/workspace_controller.go
index 09659d66c3..895a6d6ddc 100644
--- a/pkg/workspace/controllers/workspace_controller.go
+++ b/pkg/workspace/controllers/workspace_controller.go
@@ -691,17 +691,17 @@ func (c *WorkspaceReconciler) applyInference(ctx context.Context, wObj *kaitov1b
 							volumes = append(volumes, adapterVolume)
 							volumeMounts = append(volumeMounts, adapterVolumeMount)
 						}
-						initContainers, envs := manifests.GenerateInitContainers(wObj, volumeMounts)
-						spec := &deployment.Spec
 
-						spec.Template.Spec.InitContainers = initContainers
-						spec.Template.Spec.Containers[0].Env = envs
-						spec.Template.Spec.Containers[0].VolumeMounts = volumeMounts
-						deployment.Annotations[kaitov1beta1.WorkspaceRevisionAnnotation] = revisionStr
-						spec.Template.Spec.Volumes = volumes
+						pullerContainers, pullerEnvVars, pullerVolumes := manifests.GeneratePullerContainers(wObj, volumeMounts)
+						volumes = append(volumes, pullerVolumes...)
+
+						spec := &deployment.Spec.Template.Spec
+						spec.Containers[0].Env = pullerEnvVars
+						spec.Containers[0].VolumeMounts = volumeMounts
+						spec.InitContainers = pullerContainers
+						spec.Volumes = volumes
 
-						_, imagePullSecrets := inference.GetInferenceImageInfo(ctx, wObj, inferenceParam)
-						deployment.Spec.Template.Spec.ImagePullSecrets = imagePullSecrets
+						deployment.Annotations[kaitov1beta1.WorkspaceRevisionAnnotation] = revisionStr
 
 						if err := c.Update(ctx, deployment); err != nil {
 							return
diff --git a/pkg/workspace/image/README.md b/pkg/workspace/image/README.md
new file mode 100644
index 0000000000..8a40f6dd35
--- /dev/null
+++ b/pkg/workspace/image/README.md
@@ -0,0 +1,31 @@
+# Kaito Adapter Image Format Specification
+
+This document describes the specification for Kaito-compatible adapter images. It assumes the reader is familiar with the [OCI Image Format Specification](https://github.com/opencontainers/image-spec).
+
+## Image Layout
+
+A Kaito-compatible adapter image consists of any OCI image with the file structure below. Files added to the image but not specified here are simply ignored. The image can use any base and have as many layers as needed.
+```
+/
+└── data
+    ├── adapter_config.json
+    └── adapter_model.safetensors
+```
+
+A minimal and conformant image can be created by building the following Dockerfile:
+
+```dockerfile
+FROM scratch
+COPY adapter_config.json       /data/
+COPY adapter_model.safetensors /data/
+```
+
+## Building Platform-Independent Images
+
+To build a platform-independent image compatible with Kaito, you can leverage [pusher.sh](pusher.sh). Simply run `./pusher.sh ${PATH_TO_DATA_DIRECTORY} ${IMAGE_REFERENCE}` to build and push a platform-independent image containing the files in the specified directory.
+
+The following example pushes `~/Documents/kaito/adapter/data` to `ghcr.io/kaito-project/kaito/adapter` with tag `1.2.3`:
+
+```bash
+./pusher.sh ~/Documents/kaito/adapter/data ghcr.io/kaito-project/kaito/adapter:1.2.3
+```
diff --git a/pkg/workspace/image/puller.go b/pkg/workspace/image/puller.go
new file mode 100644
index 0000000000..2cf6e7fad3
--- /dev/null
+++ b/pkg/workspace/image/puller.go
@@ -0,0 +1,66 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+package image
+
+import (
+	"bytes"
+	_ "embed"
+	"log"
+	"text/template"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed puller.sh
+	pullerSHTextData string
+
+	pullerSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("puller.sh").Option("missingkey=zero").Parse(pullerSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pullerSHTemplate = t
+}
+
+func renderPullerSH(imgRef string, volDir string) string {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
+	if err != nil {
+		log.Printf("failed to normalize image reference `%s`: %v", imgRef, err)
+	}
+	if normalizedImgRef != nil {
+		imgRef = normalizedImgRef.String()
+	}
+
+	data := map[string]string{
+		"imgRef": imgRef,
+		"volDir": volDir,
+	}
+
+	var buf bytes.Buffer
+	if err := pullerSHTemplate.Execute(&buf, data); err != nil {
+		panic(err)
+	}
+
+	return buf.String()
+}
+
+func NewPullerContainer(inputImage string, outputDirectory string) *corev1.Container {
+	return &corev1.Container{
+		Name:  "puller",
+		Image: "quay.io/skopeo/stable:v1.18.0-immutable",
+		Command: []string{
+			"/bin/sh",
+			"-c",
+		},
+		Args: []string{
+			renderPullerSH(inputImage, outputDirectory),
+		},
+	}
+}
diff --git a/pkg/workspace/image/puller.sh b/pkg/workspace/image/puller.sh
new file mode 100755
index 0000000000..095a1a84d4
--- /dev/null
+++ b/pkg/workspace/image/puller.sh
@@ -0,0 +1,58 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+IMG_REF="${1}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+VOL_DIR="${2}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+#{{`
+
+pull() {
+    LAYOUT_DIR="${TMPDIR}/layout"
+
+    local DOCKER_CONFIG_D="${HOME}/.docker/config.d"
+    [ ! -e "${DOCKER_CONFIG_D}" ] || local REGISTRY_AUTH_FILES="$(find -L "${DOCKER_CONFIG_D}" -type 'f' -printf '%p ')"
+
+    for REGISTRY_AUTH_FILE in '' ${REGISTRY_AUTH_FILES}
+    do
+        local SKOPEO_ARGS='--dest-decompress'
+        [ -z "${REGISTRY_AUTH_FILE}" ] || SKOPEO_ARGS="${SKOPEO_ARGS} --authfile=${REGISTRY_AUTH_FILE}"
+
+        skopeo copy ${SKOPEO_ARGS} "docker://${IMG_REF}" "dir:${LAYOUT_DIR}" || continue
+
+        local OK='1'
+        break
+    done
+
+    [ "${OK}" = '1' ]
+}
+
+expand() {
+    ROOTFS_DIR="${TMPDIR}/rootfs"
+    mkdir -p "${ROOTFS_DIR}"
+
+    local LAYER_DIFFS="$(skopeo inspect --format '{{ range .Layers }}{{ slice . 7 }} {{ end }}' "dir:${LAYOUT_DIR}")"
+    for LAYER_DIFF in ${LAYER_DIFFS}
+    do
+        local LAYER_PATH="${LAYOUT_DIR}/${LAYER_DIFF}"
+        tar x -f "${LAYER_PATH}" -C "${ROOTFS_DIR}"
+    done
+}
+
+relocate() {
+    local DATA_PATH="${ROOTFS_DIR}/data"
+    mv -f "${DATA_PATH}" "${VOL_DIR}"
+}
+
+#`}}
+
+pull
+expand
+relocate
diff --git a/pkg/workspace/image/puller_test.go b/pkg/workspace/image/puller_test.go
new file mode 100644
index 0000000000..8cb1769bf1
--- /dev/null
+++ b/pkg/workspace/image/puller_test.go
@@ -0,0 +1,101 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+package image
+
+import (
+	"text/template"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"k8s.io/apimachinery/pkg/util/rand"
+)
+
+var _ = Describe("Puller", func() {
+	Context("embed", func() {
+		It("initializes the puller text data", func() {
+			Expect(pullerSHTextData).NotTo(BeEmpty())
+		})
+
+		It("uses sh as the interpreter", func() {
+			Expect(pullerSHTextData).To(HavePrefix("#!/bin/sh\n"))
+		})
+	})
+
+	Context("init", func() {
+		It("instantiates the puller template", func() {
+			Expect(pusherSHTemplate).NotTo(BeNil())
+		})
+	})
+
+	Context("renderPullerSH", func() {
+		It("renders the puller script", func() {
+			var (
+				imgRef = "docker.io/library/scratch:" + rand.String(8)
+				volDir = "/tmp/" + rand.String(8)
+			)
+
+			ret := renderPullerSH(imgRef, volDir)
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z "${IMG_REF}" ] || IMG_REF='` + imgRef + `'` + "\n"))
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z "${VOL_DIR}" ] || VOL_DIR='` + volDir + `'` + "\n"))
+		})
+
+		It("normalizes the image reference", func() {
+			var (
+				imgRef = "scratch-" + rand.String(8)
+				volDir = "/tmp/" + rand.String(8)
+			)
+
+			ret := renderPullerSH(imgRef, volDir)
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z "${IMG_REF}" ] || IMG_REF='docker.io/library/` + imgRef + `:latest'` + "\n"))
+		})
+
+		It("falls back to original image reference when unable to normalize", func() {
+			var (
+				imgRef = "!" + rand.String(8)
+				volDir = "/tmp/" + rand.String(8)
+			)
+
+			ret := renderPullerSH(imgRef, volDir)
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z "${IMG_REF}" ] || IMG_REF='` + imgRef + `'` + "\n"))
+		})
+
+		It("panics when the template is bad", func() {
+			var (
+				imgRef = "docker.io/library/scratch:" + rand.String(8)
+				volDir = "/tmp/" + rand.String(8)
+			)
+
+			pullerSHTemplate0 := pullerSHTemplate
+			defer func() {
+				pullerSHTemplate = pullerSHTemplate0
+			}()
+
+			pullerSHTemplate = &template.Template{}
+
+			f := func() {
+				ret := renderPullerSH(imgRef, volDir)
+				Expect(ret).To(BeEmpty())
+			}
+			Expect(f).To(PanicWith(MatchError(`template: : "" is an incomplete or empty template`)))
+		})
+	})
+
+	Context("NewPullerContainer", func() {
+		It("returns the expected container", func() {
+			var (
+				imgRef = "docker.io/library/scratch:" + rand.String(8)
+				volDir = "/tmp/" + rand.String(8)
+			)
+
+			pullerSH := renderPullerSH(imgRef, volDir)
+
+			ret := NewPullerContainer(imgRef, volDir)
+			Expect(ret).NotTo(BeNil())
+			Expect(ret.Name).To(Equal("puller"))
+			Expect(ret.Image).To(Equal("quay.io/skopeo/stable:v1.18.0-immutable"))
+			Expect(ret.Command).To(Equal([]string{"/bin/sh", "-c"}))
+			Expect(ret.Args).To(Equal([]string{pullerSH}))
+		})
+	})
+})
diff --git a/pkg/workspace/image/pusher.go b/pkg/workspace/image/pusher.go
new file mode 100644
index 0000000000..ab1f8b386f
--- /dev/null
+++ b/pkg/workspace/image/pusher.go
@@ -0,0 +1,83 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+package image
+
+import (
+	"bytes"
+	_ "embed"
+	"encoding/json"
+	"log"
+	"path"
+	"text/template"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed pusher.sh
+	pusherSHTextData string
+
+	pusherSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("pusher.sh").Option("missingkey=zero").Parse(pusherSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pusherSHTemplate = t
+}
+
+func renderPusherSH(volDir string, imgRef string, annotationsData map[string]map[string]string, sentinelPath *string) string {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
+	if err != nil {
+		log.Printf("failed to normalize image reference `%s`: %v", imgRef, err)
+	}
+	if normalizedImgRef != nil {
+		imgRef = normalizedImgRef.String()
+	}
+
+	data := map[string]string{
+		"volDir":          volDir,
+		"imgRef":          imgRef,
+		"annotationsData": "{}",
+		"sentinelPath":    path.Join(volDir, "fine_tuning_completed.txt"),
+	}
+
+	if annotationsData != nil {
+		annotationsData, err := json.Marshal(annotationsData)
+		if err != nil {
+			panic(err)
+		}
+
+		data["annotationsData"] = string(annotationsData)
+	}
+
+	if sentinelPath != nil {
+		data["sentinelPath"] = *sentinelPath
+	}
+
+	var buf bytes.Buffer
+	if err := pusherSHTemplate.Execute(&buf, data); err != nil {
+		panic(err)
+	}
+
+	return buf.String()
+}
+
+func NewPusherContainer(inputDirectory string, outputImage string, annotationsData map[string]map[string]string, sentinelPath *string) *corev1.Container {
+	return &corev1.Container{
+		Name:  "pusher",
+		Image: "ghcr.io/oras-project/oras:v1.2.2",
+		Command: []string{
+			"/bin/sh",
+			"-c",
+		},
+		Args: []string{
+			renderPusherSH(inputDirectory, outputImage, annotationsData, sentinelPath),
+		},
+	}
+}
diff --git a/pkg/workspace/image/pusher.sh b/pkg/workspace/image/pusher.sh
new file mode 100755
index 0000000000..75820f5779
--- /dev/null
+++ b/pkg/workspace/image/pusher.sh
@@ -0,0 +1,85 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+[ ! -z '{{ "" }}' ] || ANNOTATIONS_DATA='{{ .annotationsData }}'
+ANNOTATIONS_DATA="${ANNOTATIONS_DATA:-{}}"
+
+[ ! -z '{{ "" }}' ] || SENTINEL_PATH='{{ .sentinelPath }}'
+SENTINEL_PATH="${SENTINEL_PATH:-.}"
+
+#{{`
+
+wait() {
+    until [ -e "${SENTINEL_PATH}" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mkannotations() {
+    ANNOTATIONS_PATH="${TMPDIR}/annotations.json"
+
+    printf '%s' "${ANNOTATIONS_DATA}" > "${ANNOTATIONS_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --annotation-file "${ANNOTATIONS_PATH}" --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
+    cd -
+}
+
+push() {
+    oras cp --from-oci-layout "${LAYOUT_REF}" "${IMG_REF}"
+}
+
+resume() {
+    killall -SIGCHLD 'pause' 0</dev/null 1>&0 2>&0 || true
+}
+
+#`}}
+
+wait
+mklayer
+mkconfig
+mkannotations
+mklayout
+push
+resume
diff --git a/pkg/workspace/image/pusher_test.go b/pkg/workspace/image/pusher_test.go
new file mode 100644
index 0000000000..34c2a695db
--- /dev/null
+++ b/pkg/workspace/image/pusher_test.go
@@ -0,0 +1,109 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+package image
+
+import (
+	"encoding/json"
+	"text/template"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"k8s.io/apimachinery/pkg/util/rand"
+)
+
+var _ = Describe("Pusher", func() {
+	Context("embed", func() {
+		It("initializes the pusher text data", func() {
+			Expect(pusherSHTextData).NotTo(BeEmpty())
+		})
+
+		It("uses sh as the interpreter", func() {
+			Expect(pusherSHTextData).To(HavePrefix("#!/bin/sh\n"))
+		})
+	})
+
+	Context("init", func() {
+		It("instantiates the pusher template", func() {
+			Expect(pusherSHTemplate).NotTo(BeNil())
+		})
+	})
+
+	Context("renderPusherSH", func() {
+		It("renders the pusher script", func() {
+			var (
+				volDir          = "/tmp/" + rand.String(8)
+				imgRef          = "docker.io/library/scratch:" + rand.String(8)
+				annotationsData = map[string]map[string]string{"$config": {"hello": "world"}, "$manifest": {"foo": "bar"}}
+				sentinelPath    = "/tmp/" + rand.String(8)
+			)
+
+			bytes, err := json.Marshal(annotationsData)
+			Expect(err).NotTo(HaveOccurred())
+
+			ret := renderPusherSH(volDir, imgRef, annotationsData, &sentinelPath)
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z "${VOL_DIR}" ] || VOL_DIR='` + volDir + `'` + "\n"))
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z "${IMG_REF}" ] || IMG_REF='` + imgRef + `'` + "\n"))
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z '' ] || ANNOTATIONS_DATA='` + string(bytes) + `'` + "\n"))
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z '' ] || SENTINEL_PATH='` + sentinelPath + `'` + "\n"))
+		})
+
+		It("normalizes the image reference", func() {
+			var (
+				volDir = "/tmp/" + rand.String(8)
+				imgRef = "scratch-" + rand.String(8)
+			)
+
+			ret := renderPusherSH(volDir, imgRef, nil, nil)
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z "${IMG_REF}" ] || IMG_REF='docker.io/library/` + imgRef + `:latest'` + "\n"))
+		})
+
+		It("falls back to original image reference when unable to normalize", func() {
+			var (
+				volDir = "/tmp/" + rand.String(8)
+				imgRef = "^" + rand.String(8)
+			)
+
+			ret := renderPusherSH(volDir, imgRef, nil, nil)
+			Expect(ret).To(ContainSubstring("\n" + `[ ! -z "${IMG_REF}" ] || IMG_REF='` + imgRef + `'` + "\n"))
+		})
+
+		It("panics when the template is bad", func() {
+			var (
+				volDir = "/tmp/" + rand.String(8)
+				imgRef = "docker.io/library/scratch:" + rand.String(8)
+			)
+
+			pusherSHTemplate0 := pusherSHTemplate
+			defer func() {
+				pusherSHTemplate = pusherSHTemplate0
+			}()
+
+			pusherSHTemplate = &template.Template{}
+
+			f := func() {
+				ret := renderPusherSH(volDir, imgRef, nil, nil)
+				Expect(ret).To(BeEmpty())
+			}
+			Expect(f).To(PanicWith(MatchError(`template: : "" is an incomplete or empty template`)))
+		})
+	})
+
+	Context("NewPusherContainer", func() {
+		It("returns the expected container", func() {
+			var (
+				volDir = "/tmp/" + rand.String(8)
+				imgRef = "docker.io/library/scratch:" + rand.String(8)
+			)
+
+			pusherSH := renderPusherSH(volDir, imgRef, nil, nil)
+
+			ret := NewPusherContainer(volDir, imgRef, nil, nil)
+			Expect(ret).NotTo(BeNil())
+			Expect(ret.Name).To(Equal("pusher"))
+			Expect(ret.Image).To(Equal("ghcr.io/oras-project/oras:v1.2.2"))
+			Expect(ret.Command).To(Equal([]string{"/bin/sh", "-c"}))
+			Expect(ret.Args).To(Equal([]string{pusherSH}))
+		})
+	})
+})
diff --git a/pkg/workspace/image/suite_test.go b/pkg/workspace/image/suite_test.go
new file mode 100644
index 0000000000..f60133808b
--- /dev/null
+++ b/pkg/workspace/image/suite_test.go
@@ -0,0 +1,16 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+package image
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestImage(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "Image Suite")
+}
diff --git a/pkg/workspace/manifests/manifests.go b/pkg/workspace/manifests/manifests.go
index e3c0962d0e..350d4d5c40 100644
--- a/pkg/workspace/manifests/manifests.go
+++ b/pkg/workspace/manifests/manifests.go
@@ -6,6 +6,7 @@ package manifests
 import (
 	"context"
 	"fmt"
+	"path"
 
 	"github.com/samber/lo"
 	appsv1 "k8s.io/api/apps/v1"
@@ -16,6 +17,8 @@ import (
 	"k8s.io/utils/ptr"
 
 	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+	"github.com/kaito-project/kaito/pkg/utils"
+	"github.com/kaito-project/kaito/pkg/workspace/image"
 )
 
 var controller = true
@@ -202,7 +205,12 @@ func GenerateTuningJobManifest(ctx context.Context, wObj *kaitov1beta1.Workspace
 		kaitov1beta1.LabelWorkspaceName: wObj.Name,
 	}
 
-	// Add volume mounts to sidecar containers
+	// TODO: make containers only mount the volumes they need
+
+	for i := range initContainers {
+		initContainers[i].VolumeMounts = append(initContainers[i].VolumeMounts, volumeMounts...)
+	}
+
 	for i := range sidecarContainers {
 		sidecarContainers[i].VolumeMounts = append(sidecarContainers[i].VolumeMounts, volumeMounts...)
 	}
@@ -252,12 +260,13 @@ func GenerateTuningJobManifest(ctx context.Context, wObj *kaitov1beta1.Workspace
 					Labels: labels,
 				},
 				Spec: corev1.PodSpec{
-					InitContainers:   initContainers,
-					Containers:       containers,
-					RestartPolicy:    corev1.RestartPolicyNever,
-					Volumes:          volumes,
-					Tolerations:      tolerations,
-					ImagePullSecrets: imagePullSecretRefs,
+					InitContainers:        initContainers,
+					Containers:            containers,
+					RestartPolicy:         corev1.RestartPolicyNever,
+					ShareProcessNamespace: ptr.To(true),
+					Volumes:               volumes,
+					Tolerations:           tolerations,
+					ImagePullSecrets:      imagePullSecretRefs,
 				},
 			},
 		},
@@ -291,10 +300,9 @@ func GenerateDeploymentManifest(ctx context.Context, workspaceObj *kaitov1beta1.
 		Value: "expandable_segments:True",
 	})
 
-	initContainers := []corev1.Container{}
-	if len(workspaceObj.Inference.Adapters) > 0 {
-		initContainers, envs = GenerateInitContainers(workspaceObj, volumeMount)
-	}
+	pullerContainers, pullerEnvVars, pullerVolumes := GeneratePullerContainers(workspaceObj, volumeMount)
+	envs = append(envs, pullerEnvVars...)
+	volumes = append(volumes, pullerVolumes...)
 
 	return &appsv1.Deployment{
 		ObjectMeta: v1.ObjectMeta{
@@ -346,7 +354,7 @@ func GenerateDeploymentManifest(ctx context.Context, workspaceObj *kaitov1beta1.
 							},
 						},
 					},
-					InitContainers: initContainers,
+					InitContainers: pullerContainers,
 					Containers: []corev1.Container{
 						{
 							Name:           workspaceObj.Name,
@@ -368,28 +376,34 @@ func GenerateDeploymentManifest(ctx context.Context, workspaceObj *kaitov1beta1.
 	}
 }
 
-func GenerateInitContainers(wObj *kaitov1beta1.Workspace, volumeMount []corev1.VolumeMount) ([]corev1.Container, []corev1.EnvVar) {
-	var initContainers []corev1.Container
-	var envs []corev1.EnvVar
-	if len(wObj.Inference.Adapters) > 0 {
-		for _, adapter := range wObj.Inference.Adapters {
-			initContainer := corev1.Container{
-				Name:            adapter.Source.Name,
-				Image:           adapter.Source.Image,
-				Command:         []string{"/bin/sh", "-c", fmt.Sprintf("mkdir -p /mnt/adapter/%s && cp -r /data/* /mnt/adapter/%s", adapter.Source.Name, adapter.Source.Name)},
-				VolumeMounts:    volumeMount,
-				ImagePullPolicy: corev1.PullAlways,
-			}
-			initContainers = append(initContainers, initContainer)
-			env := corev1.EnvVar{
-				Name:  adapter.Source.Name,
-				Value: *adapter.Strength,
-			}
-			envs = append(envs, env)
+func GeneratePullerContainers(wObj *kaitov1beta1.Workspace, volumeMounts []corev1.VolumeMount) ([]corev1.Container, []corev1.EnvVar, []corev1.Volume) {
+	size := len(wObj.Inference.Adapters)
+
+	initContainers := make([]corev1.Container, 0, size)
+	envVars := make([]corev1.EnvVar, 0, size)
+	volumes := make([]corev1.Volume, 0, size)
+
+	for _, adapter := range wObj.Inference.Adapters {
+		source := adapter.Source
+		sourceName := source.Name
+
+		volume, volumeMount := utils.ConfigImagePullSecretVolume(sourceName+"-inference-adapter", source.ImagePullSecrets)
+		volumes = append(volumes, volume)
+
+		envVar := corev1.EnvVar{
+			Name:  sourceName,
+			Value: *adapter.Strength,
 		}
+		envVars = append(envVars, envVar)
 
+		outputDirectory := path.Join("/mnt/adapter", sourceName)
+		pullerContainer := image.NewPullerContainer(source.Image, outputDirectory)
+		pullerContainer.Name += "-" + sourceName
+		pullerContainer.VolumeMounts = append(volumeMounts, volumeMount)
+		initContainers = append(initContainers, *pullerContainer)
 	}
-	return initContainers, envs
+
+	return initContainers, envVars, volumes
 }
 
 func GenerateDeploymentManifestWithPodTemplate(ctx context.Context, workspaceObj *kaitov1beta1.Workspace, tolerations []corev1.Toleration) *appsv1.Deployment {
diff --git a/pkg/workspace/tuning/preset-tuning.go b/pkg/workspace/tuning/preset-tuning.go
index 9dc46b47e0..ead8cbc912 100644
--- a/pkg/workspace/tuning/preset-tuning.go
+++ b/pkg/workspace/tuning/preset-tuning.go
@@ -11,7 +11,6 @@ import (
 	"k8s.io/apimachinery/pkg/api/resource"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/klog/v2"
-	"k8s.io/utils/ptr"
 	"knative.dev/pkg/apis"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
@@ -20,6 +19,7 @@ import (
 	"github.com/kaito-project/kaito/pkg/utils"
 	"github.com/kaito-project/kaito/pkg/utils/consts"
 	"github.com/kaito-project/kaito/pkg/utils/resources"
+	"github.com/kaito-project/kaito/pkg/workspace/image"
 	"github.com/kaito-project/kaito/pkg/workspace/manifests"
 )
 
@@ -90,66 +90,6 @@ func GetDataSrcImageInfo(ctx context.Context, wObj *kaitov1beta1.Workspace) (str
 	return wObj.Tuning.Input.Image, imagePullSecretRefs
 }
 
-func dockerSidecarScriptPushImage(outputDir, image string) string {
-	return fmt.Sprintf(`
-# Start the Docker daemon in the background with specific options for DinD
-dockerd &
-# Wait for the Docker daemon to be ready
-while ! docker info > /dev/null 2>&1; do
-  echo "Waiting for Docker daemon to start..."
-  sleep 1
-done
-echo 'Docker daemon started'
-
-PUSH_SUCCEEDED=false
-
-while true; do
-  FILE_PATH=$(find %s -name 'fine_tuning_completed.txt')
-  if [ ! -z "$FILE_PATH" ]; then
-    if [ "$PUSH_SUCCEEDED" = false ]; then
-      echo "FOUND TRAINING COMPLETED FILE at $FILE_PATH"
-
-      PARENT_DIR=$(dirname "$FILE_PATH")
-      echo "Parent directory is $PARENT_DIR"
-
-      TEMP_CONTEXT=$(mktemp -d)
-      cp "$PARENT_DIR/adapter_config.json" "$TEMP_CONTEXT/adapter_config.json"
-      cp -r "$PARENT_DIR/adapter_model.safetensors" "$TEMP_CONTEXT/adapter_model.safetensors"
-
-      # Create a minimal Dockerfile
-      echo 'FROM busybox:latest
-      RUN mkdir -p /data
-      ADD adapter_config.json /data/
-      ADD adapter_model.safetensors /data/' > "$TEMP_CONTEXT/Dockerfile"
-
-	  # Add symbolic link to read-only mounted config.json
-      mkdir -p /root/.docker
-	  ln -s /tmp/.docker/config/config.json /root/.docker/config.json
-
-      docker build -t %s "$TEMP_CONTEXT"
-      
-      while true; do
-        if docker push %s; then
-          echo "Upload complete"
-          # Cleanup: Remove the temporary directory
-          rm -rf "$TEMP_CONTEXT"
-          # Remove the file to prevent repeated builds
-          rm "$FILE_PATH"
-          PUSH_SUCCEEDED=true
-          # Signal completion
-          touch /tmp/upload_complete
-          exit 0
-        else
-          echo "Push failed, retrying in 30 seconds..."
-          sleep 30
-        fi
-      done
-    fi
-  fi
-  sleep 10  # Check every 10 seconds
-done`, outputDir, image, image)
-}
-
 // PrepareOutputDir ensures the output directory is within the base directory.
 func PrepareOutputDir(outputDir string) (string, error) {
 	if outputDir == "" {
@@ -251,28 +191,25 @@ func CreatePresetTuning(ctx context.Context, workspaceObj *kaitov1beta1.Workspac
 		return nil, err
 	}
 
+	var imagePullSecrets []corev1.LocalObjectReference
 	var initContainers, sidecarContainers []corev1.Container
 	volumes, volumeMounts := setupDefaultSharedVolumes(workspaceObj, configVolume.Name)
 
+	// TODO: make containers only mount the volumes they need
+
 	// Add shared volume for training output
 	trainingOutputVolume, trainingOutputVolumeMount, outputDir := SetupTrainingOutputVolume(ctx, configVolume)
 	volumes = append(volumes, trainingOutputVolume)
 	volumeMounts = append(volumeMounts, trainingOutputVolumeMount)
 
-	initContainer, imagePullSecrets, dataSourceVolume, dataSourceVolumeMount, err := prepareDataSource(ctx, workspaceObj)
-	if err != nil {
-		return nil, err
-	}
-	volumes = append(volumes, dataSourceVolume)
-	volumeMounts = append(volumeMounts, dataSourceVolumeMount)
+	initContainer, dataSourceVolumes, dataSourceVolumeMounts := prepareDataSource(ctx, workspaceObj)
+	volumes = append(volumes, dataSourceVolumes...)
+	volumeMounts = append(volumeMounts, dataSourceVolumeMounts...)
 	if initContainer.Name != "" {
 		initContainers = append(initContainers, *initContainer)
 	}
 
-	sidecarContainer, imagePushSecret, dataDestVolume, dataDestVolumeMount, err := prepareDataDestination(ctx, workspaceObj, outputDir)
-	if err != nil {
-		return nil, err
-	}
+	sidecarContainer, imagePushSecret, dataDestVolume, dataDestVolumeMount := prepareDataDestination(ctx, workspaceObj, outputDir)
 	volumes = append(volumes, dataDestVolume)
 	volumeMounts = append(volumeMounts, dataDestVolumeMount)
 	if sidecarContainer != nil {
@@ -282,6 +219,13 @@ func CreatePresetTuning(ctx context.Context, workspaceObj *kaitov1beta1.Workspac
 		imagePullSecrets = append(imagePullSecrets, *imagePushSecret)
 	}
 
+	pauseContainer := corev1.Container{
+		Name:            "pause",
+		Image:           "registry.k8s.io/pause:latest",
+		ImagePullPolicy: corev1.PullAlways,
+	}
+	sidecarContainers = append(sidecarContainers, pauseContainer)
+
 	modelCommand, err := prepareModelRunParameters(ctx, tuningObj)
 	if err != nil {
 		return nil, err
@@ -324,72 +268,53 @@ func CreatePresetTuning(ctx context.Context, workspaceObj *kaitov1beta1.Workspac
 }
 
 // Now there are two options for data destination 1. HostPath - 2. Image
-func prepareDataDestination(ctx context.Context, workspaceObj *kaitov1beta1.Workspace, outputDir string) (*corev1.Container, *corev1.LocalObjectReference, corev1.Volume, corev1.VolumeMount, error) {
-	var sidecarContainer *corev1.Container
-	var volume corev1.Volume
-	var volumeMount corev1.VolumeMount
-	var imagePushSecret *corev1.LocalObjectReference
-	switch {
-	case workspaceObj.Tuning.Output.Image != "":
-		image, secret := workspaceObj.Tuning.Output.Image, workspaceObj.Tuning.Output.ImagePushSecret
-		imagePushSecret = &corev1.LocalObjectReference{Name: secret}
-		sidecarContainer, volume, volumeMount = handleImageDataDestination(ctx, outputDir, image, secret)
-		// TODO: Future PR include
-		//case workspaceObj.Tuning.Output.Volume != nil:
+func prepareDataDestination(ctx context.Context, workspaceObj *kaitov1beta1.Workspace, inputDirectory string) (*corev1.Container, *corev1.LocalObjectReference, corev1.Volume, corev1.VolumeMount) {
+	tuning := workspaceObj.Tuning
+	output := tuning.Output
+
+	outputImage := output.Image
+	if outputImage == "" {
+		return nil, nil, corev1.Volume{}, corev1.VolumeMount{}
 	}
-	return sidecarContainer, imagePushSecret, volume, volumeMount, nil
-}
 
-func handleImageDataDestination(ctx context.Context, outputDir, image, imagePushSecret string) (*corev1.Container, corev1.Volume, corev1.VolumeMount) {
-	sidecarContainer := &corev1.Container{
-		Name:  "docker-sidecar",
-		Image: "docker:dind",
-		SecurityContext: &corev1.SecurityContext{
-			Privileged: ptr.To(true),
-		},
-		Command: []string{"/bin/sh", "-c"},
-		Args:    []string{dockerSidecarScriptPushImage(outputDir, image)},
+	var annotationsData map[string]map[string]string
+	if preset := tuning.Preset; preset != nil {
+		annotationsData = map[string]map[string]string{
+			"$manifest": {
+				"sh.kaito.model.name": string(preset.Name),
+			},
+		}
 	}
-	volume, volumeMount := utils.ConfigImagePushSecretVolume(imagePushSecret)
-	return sidecarContainer, volume, volumeMount
-}
 
-// Now there are three options for DataSource: 1. URL - 2. HostPath - 3. Image
-func prepareDataSource(ctx context.Context, workspaceObj *kaitov1beta1.Workspace) (*corev1.Container, []corev1.LocalObjectReference, corev1.Volume, corev1.VolumeMount, error) {
-	var initContainer *corev1.Container
-	var volume corev1.Volume
-	var volumeMount corev1.VolumeMount
-	var imagePullSecrets []corev1.LocalObjectReference
-	switch {
-	case workspaceObj.Tuning.Input.Image != "":
-		var image string
-		image, imagePullSecrets = GetDataSrcImageInfo(ctx, workspaceObj)
-		initContainer, volume, volumeMount = handleImageDataSource(ctx, image)
-	case len(workspaceObj.Tuning.Input.URLs) > 0:
-		initContainer, volume, volumeMount = handleURLDataSource(ctx, workspaceObj)
-		// TODO: Future PR include
-		// case workspaceObj.Tuning.Input.Volume != nil:
+	pusherContainer := image.NewPusherContainer(inputDirectory, outputImage, annotationsData, nil)
+
+	imagePushSecretRef := corev1.LocalObjectReference{
+		Name: output.ImagePushSecret,
 	}
-	return initContainer, imagePullSecrets, volume, volumeMount, nil
+
+	volume, volumeMount := utils.ConfigImagePushSecretVolume(imagePushSecretRef.Name)
+
+	return pusherContainer, &imagePushSecretRef, volume, volumeMount
 }
 
-func handleImageDataSource(ctx context.Context, image string) (*corev1.Container, corev1.Volume, corev1.VolumeMount) {
-	// Constructing a multistep command that lists, copies, and then lists the destination
-	command := "ls -la /data && cp -r /data/* " + utils.DefaultDataVolumePath + " && ls -la " + utils.DefaultDataVolumePath
-	initContainer := &corev1.Container{
-		Name:    "data-extractor",
-		Image:   image,
-		Command: []string{"sh", "-c", command},
-		VolumeMounts: []corev1.VolumeMount{
-			{
-				Name:      "data-volume",
-				MountPath: utils.DefaultDataVolumePath,
-			},
-		},
-	}
+// Now there are three options for DataSource: 1. URL - 2. HostPath - 3. Image
+func prepareDataSource(ctx context.Context, workspaceObj *kaitov1beta1.Workspace) (*corev1.Container, []corev1.Volume, []corev1.VolumeMount) {
+	input := workspaceObj.Tuning.Input
 
-	volume, volumeMount := utils.ConfigDataVolume(nil)
-	return initContainer, volume, volumeMount
+	switch {
+	case input.Image != "":
+		dataVolume, dataVolumeMount := utils.ConfigDataVolume(nil)
+		imagePullSecretVolume, imagePullSecretVolumeMount := utils.ConfigImagePullSecretVolume(input.Name+"-tuning-input", input.ImagePullSecrets)
+		pullerContainer := image.NewPullerContainer(input.Image, utils.DefaultDataVolumePath)
+		return pullerContainer, []corev1.Volume{imagePullSecretVolume, dataVolume}, []corev1.VolumeMount{imagePullSecretVolumeMount, dataVolumeMount}
+
+	case len(input.URLs) > 0:
+		initContainer, volume, volumeMount := handleURLDataSource(ctx, workspaceObj)
+		return initContainer, []corev1.Volume{volume}, []corev1.VolumeMount{volumeMount}
+
+	default:
+		return nil, nil, nil
+	}
 }
 
 func handleURLDataSource(ctx context.Context, workspaceObj *kaitov1beta1.Workspace) (*corev1.Container, corev1.Volume, corev1.VolumeMount) {
@@ -425,12 +350,6 @@ func handleURLDataSource(ctx context.Context, workspaceObj *kaitov1beta1.Workspa
 			done
 			echo "All downloads completed successfully"
 		`},
-		VolumeMounts: []corev1.VolumeMount{
-			{
-				Name:      "data-volume",
-				MountPath: utils.DefaultDataVolumePath,
-			},
-		},
 		Env: []corev1.EnvVar{
 			{
 				Name:  "DATA_URLS",
diff --git a/pkg/workspace/tuning/preset-tuning_test.go b/pkg/workspace/tuning/preset-tuning_test.go
index 47637de508..bffa67b899 100644
--- a/pkg/workspace/tuning/preset-tuning_test.go
+++ b/pkg/workspace/tuning/preset-tuning_test.go
@@ -11,12 +11,12 @@ import (
 	"github.com/stretchr/testify/assert"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
-	"k8s.io/utils/ptr"
 
 	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
 	"github.com/kaito-project/kaito/pkg/model"
 	"github.com/kaito-project/kaito/pkg/utils"
 	"github.com/kaito-project/kaito/pkg/utils/consts"
+	"github.com/kaito-project/kaito/pkg/workspace/image"
 )
 
 func normalize(s string) string {
@@ -220,45 +220,6 @@ training_config:
 	}
 }
 
-func TestHandleImageDataSource(t *testing.T) {
-	testcases := map[string]struct {
-		workspaceObj              *kaitov1beta1.Workspace
-		expectedInitContainerName string
-		expectedVolumeName        string
-		expectedVolumeMountPath   string
-	}{
-		"Handle Image Data Source": {
-			workspaceObj: &kaitov1beta1.Workspace{
-				Resource: kaitov1beta1.ResourceSpec{
-					Count: ptr.To(1),
-				},
-				Tuning: &kaitov1beta1.TuningSpec{
-					Input: &kaitov1beta1.DataSource{
-						Image: "data-image",
-					},
-				},
-			},
-			expectedInitContainerName: "data-extractor",
-			expectedVolumeName:        "data-volume",
-			expectedVolumeMountPath:   "/mnt/data",
-		},
-	}
-
-	for name, tc := range testcases {
-		t.Run(name, func(t *testing.T) {
-			initContainer, volume, volumeMount := handleImageDataSource(context.Background(), tc.workspaceObj.Tuning.Input.Image)
-
-			assert.Equal(t, tc.expectedInitContainerName, initContainer.Name)
-			assert.Equal(t, tc.workspaceObj.Tuning.Input.Image, initContainer.Image)
-			assert.Contains(t, initContainer.Command[2], "cp -r /data/* /mnt/data")
-
-			assert.Equal(t, tc.expectedVolumeName, volume.Name)
-
-			assert.Equal(t, tc.expectedVolumeMountPath, volumeMount.MountPath)
-		})
-	}
-}
-
 func TestHandleURLDataSource(t *testing.T) {
 	testcases := map[string]struct {
 		workspaceObj              *kaitov1beta1.Workspace
@@ -351,40 +312,101 @@ func TestPrepareTuningParameters(t *testing.T) {
 	}
 }
 
+func TestPrepareDataDestination_ImageDestination(t *testing.T) {
+	ctx := context.TODO()
+
+	workspaceObj := &kaitov1beta1.Workspace{
+		Tuning: &kaitov1beta1.TuningSpec{
+			Output: &kaitov1beta1.DataDestination{
+				Image:           "custom/data-loader-image",
+				ImagePushSecret: "image-push-secret",
+			},
+		},
+	}
+
+	expectedImagePushSecret := corev1.LocalObjectReference{
+		Name: workspaceObj.Tuning.Output.ImagePushSecret,
+	}
+
+	expectedVolume, expectedVolumeMount := utils.ConfigImagePushSecretVolume(expectedImagePushSecret.Name)
+
+	outDir := "/mnt/results"
+
+	expectedSidecarContainer := image.NewPusherContainer(outDir, workspaceObj.Tuning.Output.Image, nil, nil)
+
+	sidecarContainer, imagePushSecret, volume, volumeMount := prepareDataDestination(ctx, workspaceObj, outDir)
+
+	// Assertions
+	assert.Equal(t, expectedSidecarContainer, sidecarContainer)
+	assert.Equal(t, expectedVolume, volume)
+	assert.Equal(t, expectedVolumeMount, volumeMount)
+	assert.Equal(t, expectedImagePushSecret, *imagePushSecret)
+}
+
 func TestPrepareDataSource_ImageSource(t *testing.T) {
 	ctx := context.TODO()
 
 	workspaceObj := &kaitov1beta1.Workspace{
 		Tuning: &kaitov1beta1.TuningSpec{
 			Input: &kaitov1beta1.DataSource{
+				Name:  "some-name",
 				Image: "custom/data-loader-image",
+				ImagePullSecrets: []string{
+					"image-pull-secret-name",
+				},
 			},
 		},
 	}
 
 	// Expected outputs from mocked functions
-	expectedVolume := corev1.Volume{
-		Name: "data-volume",
-		VolumeSource: corev1.VolumeSource{
-			EmptyDir: &corev1.EmptyDirVolumeSource{}, // Assume we expect an EmptyDir
+	expectedInitContainer := image.NewPullerContainer(workspaceObj.Tuning.Input.Image, "/mnt/data")
+
+	expectedVolumes := []corev1.Volume{
+		{
+			Name: "docker-config-some-name-tuning-input",
+			VolumeSource: corev1.VolumeSource{
+				Projected: &corev1.ProjectedVolumeSource{
+					Sources: []corev1.VolumeProjection{
+						{
+							Secret: &corev1.SecretProjection{
+								LocalObjectReference: corev1.LocalObjectReference{
+									Name: "image-pull-secret-name",
+								},
+								Items: []corev1.KeyToPath{
+									{
+										Key:  ".dockerconfigjson",
+										Path: "image-pull-secret-name.json",
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+		{
+			Name: "data-volume",
+			VolumeSource: corev1.VolumeSource{
+				EmptyDir: &corev1.EmptyDirVolumeSource{},
+			},
 		},
 	}
 
-	expectedVolumeMount := corev1.VolumeMount{Name: "data-volume", MountPath: "/mnt/data"}
-	expectedImagePullSecrets := []corev1.LocalObjectReference{}
-	expectedInitContainer := &corev1.Container{
-		Name:         "data-extractor",
-		Image:        "custom/data-loader-image",
-		Command:      []string{"sh", "-c", "ls -la /data && cp -r /data/* /mnt/data && ls -la /mnt/data"},
-		VolumeMounts: []corev1.VolumeMount{expectedVolumeMount},
+	expectedVolumeMounts := []corev1.VolumeMount{
+		{
+			Name:      "docker-config-some-name-tuning-input",
+			MountPath: "/root/.docker/config.d/some-name-tuning-input",
+		},
+		{
+			Name:      "data-volume",
+			MountPath: "/mnt/data",
+		},
 	}
 
-	initContainer, imagePullSecrets, volume, volumeMount, err := prepareDataSource(ctx, workspaceObj)
+	initContainer, volumes, volumeMounts := prepareDataSource(ctx, workspaceObj)
 
 	// Assertions
-	assert.NoError(t, err)
 	assert.Equal(t, expectedInitContainer, initContainer)
-	assert.Equal(t, expectedVolume, volume)
-	assert.Equal(t, expectedVolumeMount, volumeMount)
-	assert.Equal(t, expectedImagePullSecrets, imagePullSecrets)
+	assert.Equal(t, expectedVolumes, volumes)
+	assert.Equal(t, expectedVolumeMounts, volumeMounts)
 }
diff --git a/test/e2e/e2e_test.go b/test/e2e/e2e_test.go
index 0facad607d..0fb754b3dc 100644
--- a/test/e2e/e2e_test.go
+++ b/test/e2e/e2e_test.go
@@ -94,6 +94,14 @@ var _ = BeforeSuite(func() {
 		},
 	})
 	Expect(err).NotTo(HaveOccurred())
+
+	loadTestEnvVars()
+
+	err = copySecretToNamespace(aiModelsRegistrySecret, namespaceName)
+	Expect(err).NotTo(HaveOccurred())
+
+	err = copySecretToNamespace(e2eACRSecret, namespaceName)
+	Expect(err).NotTo(HaveOccurred())
 })
 
 var _ = AfterSuite(func() {
diff --git a/test/e2e/inference_with_adapters_test.go b/test/e2e/inference_with_adapters_test.go
index 61c4c3ed3c..7fd33f4be5 100644
--- a/test/e2e/inference_with_adapters_test.go
+++ b/test/e2e/inference_with_adapters_test.go
@@ -17,6 +17,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
 	kaitov1beta1 "github.com/kaito-project/kaito/api/v1beta1"
+	"github.com/kaito-project/kaito/pkg/workspace/image"
 	"github.com/kaito-project/kaito/test/e2e/utils"
 )
 
@@ -33,7 +34,7 @@ var validAdapters1 = []kaitov1beta1.AdapterSpec{
 			Name:  imageName1,
 			Image: fullImageName1,
 			ImagePullSecrets: []string{
-				utils.GetEnv("AI_MODELS_REGISTRY_SECRET"),
+				utils.GetEnv("E2E_ACR_REGISTRY_SECRET"),
 			},
 		},
 		Strength: &DefaultStrength,
@@ -53,17 +54,19 @@ var validAdapters2 = []kaitov1beta1.AdapterSpec{
 	},
 }
 
+var baseInitContainer = image.NewPullerContainer("", "")
+
 var expectedInitContainers1 = []corev1.Container{
 	{
-		Name:  imageName1,
-		Image: fullImageName1,
+		Name:  baseInitContainer.Name + "-" + imageName1,
+		Image: baseInitContainer.Image,
 	},
 }
 
 var expectedInitContainers2 = []corev1.Container{
 	{
-		Name:  imageName2,
-		Image: fullImageName2,
+		Name:  baseInitContainer.Name + "-" + imageName2,
+		Image: baseInitContainer.Image,
 	},
 }
 
diff --git a/test/e2e/preset_test.go b/test/e2e/preset_test.go
index a47987ce62..48d2b9c598 100644
--- a/test/e2e/preset_test.go
+++ b/test/e2e/preset_test.go
@@ -6,7 +6,6 @@ package e2e
 import (
 	"context"
 	"fmt"
-	"log"
 	"math/rand"
 	"os"
 	"strconv"
@@ -526,36 +525,48 @@ func validateTuningResource(workspaceObj *kaitov1beta1.Workspace) {
 	})
 }
 
-func validateTuningJobInputOutput(workspaceObj *kaitov1beta1.Workspace, inputImage string, output string) {
+func validateTuningJobInputOutput(workspaceObj *kaitov1beta1.Workspace, inputImage string, outputImage string) {
 	By("Checking the tuning input and output", func() {
 		Eventually(func() bool {
-			var err error
-
-			job := &batchv1.Job{}
-			err = utils.TestingCluster.KubeClient.Get(ctx, client.ObjectKey{
-				Namespace: workspaceObj.Namespace,
-				Name:      workspaceObj.Name,
-			}, job)
+			var job batchv1.Job
+			if err := utils.TestingCluster.KubeClient.Get(ctx, client.ObjectKeyFromObject(workspaceObj), &job); err != nil {
+				if client.IgnoreNotFound(err) == nil {
+					GinkgoWriter.Printf("Job not found: %v\n", err)
+					return false
+				}
 
-			if err != nil {
-				GinkgoWriter.Printf("Error fetching resource: %v\n", err)
-				return false
+				Expect(err).NotTo(HaveOccurred())
 			}
 
-			image := job.Spec.Template.Spec.InitContainers[0].Image
-
-			expectedString1 := "docker build -t " + output
-			expectedString2 := "if docker push " + output + "; then"
+			var pullerContainer *v1.Container
+			for _, container := range job.Spec.Template.Spec.InitContainers {
+				if strings.HasPrefix(container.Name, "puller") {
+					pullerContainer = &container
+					break
+				}
+			}
 
-			var sidecarContainer v1.Container
+			pullerSH := pullerContainer.Args[0]
+			if !strings.Contains(pullerSH, fmt.Sprintf("\n[ ! -z \"${IMG_REF}\" ] || IMG_REF='%s'\n", inputImage)) {
+				GinkgoWriter.Printf("Unexpected pullerSH: %s\n", pullerSH)
+				return false
+			}
 
+			var pusherContainer *v1.Container
 			for _, container := range job.Spec.Template.Spec.Containers {
-				if container.Name == "docker-sidecar" {
-					sidecarContainer = container
+				if strings.HasPrefix(container.Name, "pusher") {
+					pusherContainer = &container
 					break
 				}
 			}
-			return image == inputImage && strings.Contains(sidecarContainer.Args[0], expectedString1) && strings.Contains(sidecarContainer.Args[0], expectedString2)
+
+			pusherSH := pusherContainer.Args[0]
+			if !strings.Contains(pusherSH, fmt.Sprintf("\n[ ! -z \"${IMG_REF}\" ] || IMG_REF='%s'\n", outputImage)) {
+				GinkgoWriter.Printf("Unexpected pusherSH: %s\n", pullerSH)
+				return false
+			}
+
+			return true
 		}, 10*time.Minute, utils.PollInterval).Should(BeTrue(), "Failed to wait for Tuning resource to be ready")
 	})
 }
@@ -563,28 +574,27 @@ func validateTuningJobInputOutput(workspaceObj *kaitov1beta1.Workspace, inputIma
 func validateACRTuningResultsUploaded(workspaceObj *kaitov1beta1.Workspace, jobName string) {
 	coreClient, err := utils.GetK8sClientset()
 	if err != nil {
-		log.Fatalf("Failed to create core client: %v", err)
-	}
-	namespace := workspaceObj.Namespace
-	podName, err := utils.GetPodNameForJob(coreClient, namespace, jobName)
-	if err != nil {
-		log.Fatalf("Failed to get pod name for job %s: %v", jobName, err)
+		Fail(fmt.Sprintf("Failed to create core client: %v", err))
 	}
 
 	for {
-		logs, err := utils.GetPodLogs(coreClient, namespace, podName, "docker-sidecar")
+		job, err := coreClient.BatchV1().Jobs(workspaceObj.Namespace).Get(ctx, jobName, metav1.GetOptions{})
 		if err != nil {
-			log.Printf("Failed to get logs from pod %s: %v", podName, err)
-			time.Sleep(10 * time.Second)
+			Fail(fmt.Sprintf("Failed to get job %s: %v", jobName, err))
+		}
+
+		if job.Status.CompletionTime.IsZero() {
+			time.Sleep(10 * time.Second) // Poll every 10 seconds
 			continue
 		}
 
-		if strings.Contains(logs, "Upload complete") {
-			fmt.Println("Upload complete")
+		if job.Status.Succeeded == 0 {
+			Fail("Job did not succeed")
 			break
 		}
 
-		time.Sleep(10 * time.Second) // Poll every 10 seconds
+		GinkgoWriter.Println("Upload complete")
+		break
 	}
 }
 
@@ -912,10 +922,6 @@ var _ = Describe("Workspace Preset", func() {
 
 	It("should create a workspace for tuning successfully, and update the workspace with another dataset and output image", utils.GinkgoLabelFastCheck, func() {
 		numOfNode := 1
-		err := copySecretToNamespace(e2eACRSecret, namespaceName)
-		if err != nil {
-			log.Fatalf("Error copying secret: %v", err)
-		}
 		configMap := createCustomTuningConfigMapForE2E()
 		workspaceObj, jobName, outputRegistryUrl1 := createPhi3TuningWorkspaceWithPresetPublicMode(configMap.Name, numOfNode)
 
diff --git a/test/e2e/utils/utils.go b/test/e2e/utils/utils.go
index e92da6958a..493c658a6c 100644
--- a/test/e2e/utils/utils.go
+++ b/test/e2e/utils/utils.go
@@ -380,7 +380,8 @@ func GenerateE2ETuningWorkspaceManifest(name, namespace, imageName, datasetImage
 	workspace.Tuning = &workspaceTuning
 	workspace.Tuning.Method = kaitov1beta1.TuningMethodQLora
 	workspace.Tuning.Input = &kaitov1beta1.DataSource{
-		Image: datasetImageName,
+		Image:            datasetImageName,
+		ImagePullSecrets: imagePullSecret,
 	}
 	workspace.Tuning.Output = &kaitov1beta1.DataDestination{
 		Image:           outputRegistry,


Reviews:
user: login: Fei-Guo
body: Add license header for all new files please.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T20:33:06Z
user: login: Fei-Guo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T20:40:54Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T21:29:03Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T21:39:43Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-25T21:42:03Z
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T02:12:44Z
user: login: zhuangqh
body: Cool! Thanks for your investigation. Daemonless pull/push is good in this case.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T02:41:23Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T03:02:45Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T03:15:11Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T03:20:43Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T03:20:54Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T03:23:50Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T03:26:41Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T03:27:56Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T03:40:26Z
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T04:02:45Z
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T04:04:45Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T18:06:18Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T18:08:23Z
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T23:01:02Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-27T23:27:47Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-28T01:11:01Z
user: login: Fei-Guo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-28T23:03:11Z
user: login: Fei-Guo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-28T23:13:55Z
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-30T22:54:50Z
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-30T22:56:05Z
user: login: zhuangqh
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-30T22:56:54Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-01T22:50:26Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-01T22:50:35Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-01T22:50:48Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-01T22:50:56Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-01T22:51:03Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-01T22:51:08Z
user: login: ishaansehgal99
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T21:44:24Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T22:14:07Z
user: login: zhuangqh
body: Good job! Please make sure e2e test is always good before merging.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-03T22:28:23Z
user: login: t0rr3sp3dr0
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-04T17:05:25Z
user: login: Fei-Guo
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-04T17:49:47Z

Review comments:
diff_hunk: @@ -0,0 +1,66 @@
+package image
+
+import (
+	"bytes"
+	"text/template"
+
+	_ "embed"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed puller.sh
+	pullerSHTextData string
+
+	pullerSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("puller.sh").Parse(pullerSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pullerSHTemplate = t
+}
+
+func renderPullerSH(imgRef string, volDir string) (string, error) {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
path: pkg/workspace/image/puller.go
user: login: Fei-Guo
body: Should we do this check in validation webhook? I was thinking to avoid any err path in this method. 
created_at: 2025-03-25T20:40:37Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,66 @@
+package image
+
+import (
+	"bytes"
+	"text/template"
+
+	_ "embed"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed puller.sh
+	pullerSHTextData string
+
+	pullerSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("puller.sh").Parse(pullerSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pullerSHTemplate = t
+}
+
+func renderPullerSH(imgRef string, volDir string) (string, error) {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
path: pkg/workspace/image/puller.go
user: login: t0rr3sp3dr0
body: Yeah, it makes sense for us to have that check in the webhook. I'll implement that!

But what should we do if we don't have a valid image reference at this point? I know it's the user's fault if they disable the webhook and create an invalid workspace, but I need to do something in code if that function fails. Should I just panic? Or should I continue with the original invalid value? The latter will eventually cause pull/push to fail.
created_at: 2025-03-25T21:29:03Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,66 @@
+package image
+
+import (
+	"bytes"
+	"text/template"
+
+	_ "embed"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed puller.sh
+	pullerSHTextData string
+
+	pullerSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("puller.sh").Parse(pullerSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pullerSHTemplate = t
+}
+
+func renderPullerSH(imgRef string, volDir string) (string, error) {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
path: pkg/workspace/image/puller.go
user: login: t0rr3sp3dr0
body: Also, the rendering of the Go template can technically fail. But it won't, as long as we have good coverage on that method. So, I can safely make that a panic. It's not user input dependent.
created_at: 2025-03-25T21:39:42Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,66 @@
+package image
+
+import (
+	"bytes"
+	"text/template"
+
+	_ "embed"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed puller.sh
+	pullerSHTextData string
+
+	pullerSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("puller.sh").Parse(pullerSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pullerSHTemplate = t
+}
+
+func renderPullerSH(imgRef string, volDir string) (string, error) {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
path: pkg/workspace/image/puller.go
user: login: t0rr3sp3dr0
body: I think panic would be the best option for both situations if we don't want to have error handling on that path
created_at: 2025-03-25T21:42:03Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,66 @@
+package image
+
+import (
+	"bytes"
+	"text/template"
+
+	_ "embed"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed puller.sh
+	pullerSHTextData string
+
+	pullerSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("puller.sh").Parse(pullerSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pullerSHTemplate = t
+}
+
+func renderPullerSH(imgRef string, volDir string) (string, error) {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
path: pkg/workspace/image/puller.go
user: login: zhuangqh
body: If cannot normalize, just use the original `imgRef`.
later steps will give users the error. 
created_at: 2025-03-26T02:12:44Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,58 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+IMG_REF="${1}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+VOL_DIR="${2}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+#{{`
+
+pull() {
+    LAYOUT_DIR="${TMPDIR}/layout"
+
+    local DOCKER_CONFIG_D="${HOME}/.docker/config.d"
+    [ ! -e "${DOCKER_CONFIG_D}" ] || local REGISTRY_AUTH_FILES="$(find "${DOCKER_CONFIG_D}" -printf '%p ' -type 'f')"
+
+    for REGISTRY_AUTH_FILE in '' ${REGISTRY_AUTH_FILES}
+    do
+        local SKOPEO_ARGS='--dest-decompress'
+        [ -z "${REGISTRY_AUTH_FILE}" ] || SKOPEO_ARGS="${SKOPEO_ARGS} --authfile=${REGISTRY_AUTH_FILE}"
+
+        skopeo copy ${SKOPEO_ARGS} "docker://${IMG_REF}" "dir:${LAYOUT_DIR}" || continue
path: pkg/workspace/image/puller.sh
user: login: zhuangqh
body: I can see that oras supports image pulling too. Can we use it for pulling to avoid introduce two technical context?
created_at: 2025-03-26T02:22:19Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
path: pkg/workspace/image/pusher.sh
user: login: zhuangqh
body: construct an oci image from scratch. It's hard for somebody who are not familar with oci spec to maintain this script.
Can you add this document https://github.com/opencontainers/image-spec/blob/main/manifest.md#example-image-manifest as a comment.
created_at: 2025-03-26T02:26:00Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
path: pkg/workspace/image/pusher.sh
user: login: zhuangqh
body: how about pass this sentinel file name through args? Determine it by upper control plane.
This script is expected to be simple and rarely updated.
created_at: 2025-03-26T02:27:50Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
path: pkg/workspace/image/pusher.sh
user: login: zhuangqh
body: Do oras add some metadata to the image manifest? e.g. creation timestamp.
We can add some metadata to this image to make it easier to maintain.
e.g. maintainer: kaito.dev.xxx;   target model name this adapter files can be used for.
created_at: 2025-03-26T02:40:01Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,66 @@
+package image
+
+import (
+	"bytes"
+	"text/template"
+
+	_ "embed"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed puller.sh
+	pullerSHTextData string
+
+	pullerSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("puller.sh").Parse(pullerSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pullerSHTemplate = t
+}
+
+func renderPullerSH(imgRef string, volDir string) (string, error) {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
path: pkg/workspace/image/puller.go
user: login: t0rr3sp3dr0
body: Okay, I've made the changes here. The methods in this package no longer return errors.
created_at: 2025-03-26T03:02:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,58 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+IMG_REF="${1}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+VOL_DIR="${2}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+#{{`
+
+pull() {
+    LAYOUT_DIR="${TMPDIR}/layout"
+
+    local DOCKER_CONFIG_D="${HOME}/.docker/config.d"
+    [ ! -e "${DOCKER_CONFIG_D}" ] || local REGISTRY_AUTH_FILES="$(find "${DOCKER_CONFIG_D}" -printf '%p ' -type 'f')"
+
+    for REGISTRY_AUTH_FILE in '' ${REGISTRY_AUTH_FILES}
+    do
+        local SKOPEO_ARGS='--dest-decompress'
+        [ -z "${REGISTRY_AUTH_FILE}" ] || SKOPEO_ARGS="${SKOPEO_ARGS} --authfile=${REGISTRY_AUTH_FILE}"
+
+        skopeo copy ${SKOPEO_ARGS} "docker://${IMG_REF}" "dir:${LAYOUT_DIR}" || continue
path: pkg/workspace/image/puller.sh
user: login: t0rr3sp3dr0
body: I discussed this briefly with Fei, and the reason for using Skopeo here is to keep the pull behavior compatible with the [K8s Image Volume](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/) implementation. The idea is that when this feature becomes GA and we no longer support older K8s versions, we will remove the custom puller and simply mount the image as a volume in the inference pod.

I've run some tests with both `containerd` and `cri-o`, and Skopeo pulls images in a way that is compatible with both container runtimes and supports images built by older versions of Kaito. These projects actually share code that is used in the pull process. While ORAS has its own implementation that differs from them, not allowing a transparent transition to K8s Image Volumes.
created_at: 2025-03-26T03:15:11Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: By default, it only sets `org.opencontainers.image.created`.

![image](https://github.com/user-attachments/assets/82deb2fa-0b98-4838-98a8-282c90f68bfb)
created_at: 2025-03-26T03:20:43Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: We can add more stuff if we would like
created_at: 2025-03-26T03:20:54Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: Yeah, sure!
created_at: 2025-03-26T03:23:50Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: One extra thing to consider here, and unfortunately, there's no documentation for it, is that I'm building this image in a way that is architecture-independent and compatible with the K8s Image Volume implementations of `containerd` and `cri-o`. I got here by trial and error, tweaking stuff until both implementations accepted my images.
created_at: 2025-03-26T03:26:41Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: I can write a README in this package, documenting my findings
created_at: 2025-03-26T03:27:56Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: This was hard-coded in the previous script, so I thought it would be okay to do the same here.

Since I've written this in a way that allows local execution of the script, I can rewrite it in a way that uses the hard-coded values by default but allows these values to be changed when the script is rendered.
created_at: 2025-03-26T03:40:26Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,58 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+IMG_REF="${1}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+VOL_DIR="${2}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+#{{`
+
+pull() {
+    LAYOUT_DIR="${TMPDIR}/layout"
+
+    local DOCKER_CONFIG_D="${HOME}/.docker/config.d"
+    [ ! -e "${DOCKER_CONFIG_D}" ] || local REGISTRY_AUTH_FILES="$(find "${DOCKER_CONFIG_D}" -printf '%p ' -type 'f')"
+
+    for REGISTRY_AUTH_FILE in '' ${REGISTRY_AUTH_FILES}
+    do
+        local SKOPEO_ARGS='--dest-decompress'
+        [ -z "${REGISTRY_AUTH_FILE}" ] || SKOPEO_ARGS="${SKOPEO_ARGS} --authfile=${REGISTRY_AUTH_FILE}"
+
+        skopeo copy ${SKOPEO_ARGS} "docker://${IMG_REF}" "dir:${LAYOUT_DIR}" || continue
path: pkg/workspace/image/puller.sh
user: login: zhuangqh
body: What do you mean "transparent transition to K8s Image Volumes"? I think only built and push image will break this.

Can ORAS pull older versions adapter images of Kaito successfully?
I thought ORAS just download what you built in the pusher. In pulling phase, just download image using ORAS, and copy files from layer like what skopeo do.
created_at: 2025-03-26T04:02:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
path: pkg/workspace/image/pusher.sh
user: login: zhuangqh
body: Can you get the model's name in controller code simply? If so, we can add a model name annotation here.
This is the most useful information for this image.
created_at: 2025-03-26T04:04:45Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,58 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+IMG_REF="${1}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+VOL_DIR="${2}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+#{{`
+
+pull() {
+    LAYOUT_DIR="${TMPDIR}/layout"
+
+    local DOCKER_CONFIG_D="${HOME}/.docker/config.d"
+    [ ! -e "${DOCKER_CONFIG_D}" ] || local REGISTRY_AUTH_FILES="$(find "${DOCKER_CONFIG_D}" -printf '%p ' -type 'f')"
+
+    for REGISTRY_AUTH_FILE in '' ${REGISTRY_AUTH_FILES}
+    do
+        local SKOPEO_ARGS='--dest-decompress'
+        [ -z "${REGISTRY_AUTH_FILE}" ] || SKOPEO_ARGS="${SKOPEO_ARGS} --authfile=${REGISTRY_AUTH_FILE}"
+
+        skopeo copy ${SKOPEO_ARGS} "docker://${IMG_REF}" "dir:${LAYOUT_DIR}" || continue
path: pkg/workspace/image/puller.sh
user: login: t0rr3sp3dr0
body: Transparent would mean that we would be able to swap our custom puller with an image volume, and there would be no behavior change at all. The most crucial factors here are how authentication and image selection based on the host architecture are handled.

I was having problems making ORAS deal with the new arch-independent image, but it turns out that it can handle the final version of the image I came up with. Another good indication that the final design of the image is widely compatible with existing tools and won't cause problems in the future.

But the main problem I have with ORAS right now is that it's not made to deal with regular Docker images, like the old ones produced by Kaito. I can make it fetch the image configuration, but I'll have to process it manually to find out the layers. I can fetch the layers with it, but I'll have to deal with decompression myself. In the end, we would have to install extra tools on the fly or build a new image with those tools and write more code to accomplish what Skopeo does by itself. Fei would like to avoid that and keep this simple.
created_at: 2025-03-26T18:06:18Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: What do you think the label key should be? `sh.kaito.model`? `sh.kaito.model.name`? `sh.kaito.model-name`?
created_at: 2025-03-26T18:08:23Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
path: pkg/workspace/image/pusher.sh
user: login: zhuangqh
body: `sh.kaito.model.name` this one works for me.
created_at: 2025-03-26T23:01:02Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: Done!
created_at: 2025-03-27T23:27:47Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,67 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+#{{`
+
+wait() {
+    until [ -e "${VOL_DIR}/fine_tuning_completed.txt" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: Done!
created_at: 2025-03-28T01:11:01Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,80 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+package image
+
+import (
+	"bytes"
+	_ "embed"
+	"encoding/json"
+	"log"
+	"text/template"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed pusher.sh
+	pusherSHTextData string
+
+	pusherSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("pusher.sh").Option("missingkey=zero").Parse(pusherSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pusherSHTemplate = t
+}
+
+func renderPusherSH(volDir string, imgRef string, annotationsData map[string]map[string]string, waitPath *string) string {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
+	if err != nil {
+		log.Printf("failed to normalize image reference `%s`: %v", imgRef, err)
+	}
+	if normalizedImgRef != nil {
+		imgRef = normalizedImgRef.String()
+	}
+
+	data := map[string]string{
+		"volDir": volDir,
+		"imgRef": imgRef,
+	}
+
+	if annotationsData != nil {
+		annotationsData, err := json.Marshal(annotationsData)
+		if err != nil {
+			panic(err)
+		}
+
+		data["annotationsData"] = string(annotationsData)
+	}
+
+	if waitPath != nil {
path: pkg/workspace/image/pusher.go
user: login: Fei-Guo
body: Rename to "sentinelfilepath"
created_at: 2025-03-28T22:37:16Z
author_association: COLLABORATOR
diff_hunk: @@ -194,7 +197,10 @@ func GenerateTuningJobManifest(ctx context.Context, wObj *kaitov1beta1.Workspace
 		kaitov1beta1.LabelWorkspaceName: wObj.Name,
 	}
 
-	// Add volume mounts to sidecar containers
+	for i := range initContainers {
path: pkg/workspace/manifests/manifests.go
user: login: Fei-Guo
body: Add a TODO to clean up the volume mount need by different containers. 
created_at: 2025-03-28T22:47:26Z
author_association: COLLABORATOR
diff_hunk: @@ -259,20 +200,14 @@ func CreatePresetTuning(ctx context.Context, workspaceObj *kaitov1beta1.Workspac
 	volumes = append(volumes, trainingOutputVolume)
 	volumeMounts = append(volumeMounts, trainingOutputVolumeMount)
 
-	initContainer, imagePullSecrets, dataSourceVolume, dataSourceVolumeMount, err := prepareDataSource(ctx, workspaceObj)
-	if err != nil {
-		return nil, err
-	}
-	volumes = append(volumes, dataSourceVolume)
-	volumeMounts = append(volumeMounts, dataSourceVolumeMount)
+	initContainer, dataSourceVolumes, dataSourceVolumeMounts := prepareDataSource(ctx, workspaceObj)
+	volumes = append(volumes, dataSourceVolumes...)
+	volumeMounts = append(volumeMounts, dataSourceVolumeMounts...)
path: pkg/workspace/tuning/preset-tuning.go
user: login: Fei-Guo
body: Add a TODO: make sure to only mount the needed mounts. 
created_at: 2025-03-28T22:50:28Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,80 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+[ ! -z '{{ "" }}' ] || ANNOTATIONS_DATA='{{ .annotationsData }}'
+ANNOTATIONS_DATA="${ANNOTATIONS_DATA:-{}}"
+
+[ ! -z '{{ "" }}' ] || WAIT_PATH='{{ .waitPath }}'
+WAIT_PATH="${WAIT_PATH:-${VOL_DIR}/fine_tuning_completed.txt}"
+
+#{{`
+
+wait() {
+    until [ -e "${WAIT_PATH}" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mkannotations() {
+    ANNOTATIONS_PATH="${TMPDIR}/annotations.json"
+
+    printf '%s' "${ANNOTATIONS_DATA}" > "${ANNOTATIONS_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --annotation-file "${ANNOTATIONS_PATH}" --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
+    cd -
+}
+
+push() {
+    oras cp --from-oci-layout "${LAYOUT_REF}" "${IMG_REF}"
+}
+
+#`}}
+
+wait
+mklayer
+mkconfig
+mkannotations
+mklayout
+push
path: pkg/workspace/image/pusher.sh
user: login: Fei-Guo
body: Make sure if the push step fails, we have a way to copy the tuning outcome out of the container. 
created_at: 2025-03-28T23:03:06Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,42 @@
+# Kaito Adapter Image Format Specification
+
+This document describes the specification for Kaito-compatible adapter images. It assumes the reader is familiar with the [OCI Image Format Specification](https://github.com/opencontainers/image-spec).
+
+## Image Layout
+
+A Kaito-compatible adapter image consists of any OCI image with the file structure below. Files added to the image but not specified here are simply ignored. The image can use any base and have as many layers as needed.
+```
+/
+└── data
+    ├── adapter_config.json
+    └── adapter_model.safetensors
+```
+
+A minimal and conformant image can be created by building the following Dockerfile:
+
+```dockerfile
+FROM scratch
+COPY adapter_config.json       /data/
+COPY adapter_model.safetensors /data/
+```
+
+## Image Platform
+
+The adapter artifacts are not dependent on the operating system or architecture. Therefore, it is recommended that they be held in a platform-independent image to ensure interoperability between systems. Kaito can handle platform-specific images, but they must match the platform of the host system. As a rule of thumb, Kaito will be able to read any image that you can mount in your cluster using [K8s Image Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/#create-pod).
+
+## Building Platform-Independent Images
+
+Kaito builds platform-independent images in a few steps described at a high level below. The implementation leverages [ORAS](https://github.com/oras-project/oras) to simplify the process, and its source code can be found in [pusher.sh](./pusher.sh). 
+
+1. Create the image layer by archiving `/data/adapter_config.json` and `/data/adapter_model.safetensors`.
+2. Compute the diff ID by hashing the image layer.
+3. Optionally compress the image layer.
+4. Create the image configuration with the diff ID.
+5. Optionally create the annotations file.
+6. Build the image layout with the image layer, image configuration, and annotations file.
+
path: pkg/workspace/image/README.md
user: login: Fei-Guo
body: We can keep the above, but we need to provide turn-key solution script and point to that script here. 
created_at: 2025-03-28T23:13:51Z
author_association: COLLABORATOR
diff_hunk: @@ -351,6 +376,11 @@ func validateResourceStatus(workspaceObj *kaitov1beta1.Workspace) {
 				return false
 			}
 
+			workspaceJSON, err := json.Marshal(workspaceObj)
path: test/e2e/preset_test.go
user: login: zhuangqh
body: This line is inside a `Eventually` function. It means it will be printed multiple times.
I guess you only want the spec definition for debugging. Make sure it will be printed only once.
Otherwise, there are too much verbose logs.
created_at: 2025-03-30T22:54:50Z
author_association: COLLABORATOR
diff_hunk: @@ -351,6 +376,11 @@ func validateResourceStatus(workspaceObj *kaitov1beta1.Workspace) {
 				return false
 			}
 
+			workspaceJSON, err := json.Marshal(workspaceObj)
path: test/e2e/preset_test.go
user: login: zhuangqh
body: One coding suggestion, save it and print it after `Eventually`
created_at: 2025-03-30T22:56:05Z
author_association: COLLABORATOR
diff_hunk: @@ -452,6 +496,11 @@ func validateRevision(workspaceObj *kaitov1beta1.Workspace, revisionStr string)
 					return false
 				}
 				isWorkloadAnnotationCorrect = dep.Annotations[WorkspaceRevisionAnnotation] == revisionStr
+
+				deploymentJSON, err := json.Marshal(dep)
+				Expect(err).NotTo(HaveOccurred())
+
+				GinkgoWriter.Printf("Deployment definition: %s\n", string(deploymentJSON))
path: test/e2e/preset_test.go
user: login: zhuangqh
body: same here. Please check all of them.
created_at: 2025-03-30T22:56:53Z
author_association: COLLABORATOR
diff_hunk: @@ -351,6 +376,11 @@ func validateResourceStatus(workspaceObj *kaitov1beta1.Workspace) {
 				return false
 			}
 
+			workspaceJSON, err := json.Marshal(workspaceObj)
path: test/e2e/preset_test.go
user: login: t0rr3sp3dr0
body: Fei asked me to extract those changes to another PR
created_at: 2025-04-01T22:50:26Z
author_association: COLLABORATOR
diff_hunk: @@ -452,6 +496,11 @@ func validateRevision(workspaceObj *kaitov1beta1.Workspace, revisionStr string)
 					return false
 				}
 				isWorkloadAnnotationCorrect = dep.Annotations[WorkspaceRevisionAnnotation] == revisionStr
+
+				deploymentJSON, err := json.Marshal(dep)
+				Expect(err).NotTo(HaveOccurred())
+
+				GinkgoWriter.Printf("Deployment definition: %s\n", string(deploymentJSON))
path: test/e2e/preset_test.go
user: login: t0rr3sp3dr0
body: Removed from this PR
created_at: 2025-04-01T22:50:35Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,42 @@
+# Kaito Adapter Image Format Specification
+
+This document describes the specification for Kaito-compatible adapter images. It assumes the reader is familiar with the [OCI Image Format Specification](https://github.com/opencontainers/image-spec).
+
+## Image Layout
+
+A Kaito-compatible adapter image consists of any OCI image with the file structure below. Files added to the image but not specified here are simply ignored. The image can use any base and have as many layers as needed.
+```
+/
+└── data
+    ├── adapter_config.json
+    └── adapter_model.safetensors
+```
+
+A minimal and conformant image can be created by building the following Dockerfile:
+
+```dockerfile
+FROM scratch
+COPY adapter_config.json       /data/
+COPY adapter_model.safetensors /data/
+```
+
+## Image Platform
+
+The adapter artifacts are not dependent on the operating system or architecture. Therefore, it is recommended that they be held in a platform-independent image to ensure interoperability between systems. Kaito can handle platform-specific images, but they must match the platform of the host system. As a rule of thumb, Kaito will be able to read any image that you can mount in your cluster using [K8s Image Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/#create-pod).
+
+## Building Platform-Independent Images
+
+Kaito builds platform-independent images in a few steps described at a high level below. The implementation leverages [ORAS](https://github.com/oras-project/oras) to simplify the process, and its source code can be found in [pusher.sh](./pusher.sh). 
+
+1. Create the image layer by archiving `/data/adapter_config.json` and `/data/adapter_model.safetensors`.
+2. Compute the diff ID by hashing the image layer.
+3. Optionally compress the image layer.
+4. Create the image configuration with the diff ID.
+5. Optionally create the annotations file.
+6. Build the image layout with the image layer, image configuration, and annotations file.
+
path: pkg/workspace/image/README.md
user: login: t0rr3sp3dr0
body: Done
created_at: 2025-04-01T22:50:48Z
author_association: COLLABORATOR
diff_hunk: @@ -259,20 +200,14 @@ func CreatePresetTuning(ctx context.Context, workspaceObj *kaitov1beta1.Workspac
 	volumes = append(volumes, trainingOutputVolume)
 	volumeMounts = append(volumeMounts, trainingOutputVolumeMount)
 
-	initContainer, imagePullSecrets, dataSourceVolume, dataSourceVolumeMount, err := prepareDataSource(ctx, workspaceObj)
-	if err != nil {
-		return nil, err
-	}
-	volumes = append(volumes, dataSourceVolume)
-	volumeMounts = append(volumeMounts, dataSourceVolumeMount)
+	initContainer, dataSourceVolumes, dataSourceVolumeMounts := prepareDataSource(ctx, workspaceObj)
+	volumes = append(volumes, dataSourceVolumes...)
+	volumeMounts = append(volumeMounts, dataSourceVolumeMounts...)
path: pkg/workspace/tuning/preset-tuning.go
user: login: t0rr3sp3dr0
body: Done
created_at: 2025-04-01T22:50:56Z
author_association: COLLABORATOR
diff_hunk: @@ -194,7 +197,10 @@ func GenerateTuningJobManifest(ctx context.Context, wObj *kaitov1beta1.Workspace
 		kaitov1beta1.LabelWorkspaceName: wObj.Name,
 	}
 
-	// Add volume mounts to sidecar containers
+	for i := range initContainers {
path: pkg/workspace/manifests/manifests.go
user: login: t0rr3sp3dr0
body: Done
created_at: 2025-04-01T22:51:03Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,80 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT license.
+
+package image
+
+import (
+	"bytes"
+	_ "embed"
+	"encoding/json"
+	"log"
+	"text/template"
+
+	"github.com/distribution/reference"
+	corev1 "k8s.io/api/core/v1"
+)
+
+var (
+	//go:embed pusher.sh
+	pusherSHTextData string
+
+	pusherSHTemplate *template.Template
+)
+
+func init() {
+	t, err := template.New("pusher.sh").Option("missingkey=zero").Parse(pusherSHTextData)
+	if err != nil {
+		panic(err)
+	}
+
+	pusherSHTemplate = t
+}
+
+func renderPusherSH(volDir string, imgRef string, annotationsData map[string]map[string]string, waitPath *string) string {
+	normalizedImgRef, err := reference.ParseDockerRef(imgRef)
+	if err != nil {
+		log.Printf("failed to normalize image reference `%s`: %v", imgRef, err)
+	}
+	if normalizedImgRef != nil {
+		imgRef = normalizedImgRef.String()
+	}
+
+	data := map[string]string{
+		"volDir": volDir,
+		"imgRef": imgRef,
+	}
+
+	if annotationsData != nil {
+		annotationsData, err := json.Marshal(annotationsData)
+		if err != nil {
+			panic(err)
+		}
+
+		data["annotationsData"] = string(annotationsData)
+	}
+
+	if waitPath != nil {
path: pkg/workspace/image/pusher.go
user: login: t0rr3sp3dr0
body: Done
created_at: 2025-04-01T22:51:08Z
author_association: COLLABORATOR
diff_hunk: @@ -1252,8 +1252,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Invalid Output Destination Casing",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/input:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/OUTPUT:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/input:0.0.0"},
path: api/v1beta1/workspace_validation_test.go
user: login: ishaansehgal99
body: why rename
created_at: 2025-04-03T21:44:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1252,8 +1252,8 @@ func TestTuningSpecValidateCreate(t *testing.T) {
 		{
 			name: "Invalid Output Destination Casing",
 			tuningSpec: &TuningSpec{
-				Input:  &DataSource{Name: "valid-input", Image: "AZURE_ACR.azurecr.io/input:0.0.0"},
-				Output: &DataDestination{Image: "AZURE_ACR.azurecr.io/OUTPUT:0.0.0", ImagePushSecret: "secret"},
+				Input:  &DataSource{Name: "valid-input", Image: "kaito.azurecr.io/input:0.0.0"},
path: api/v1beta1/workspace_validation_test.go
user: login: t0rr3sp3dr0
body: `AZURE_ACR.azurecr.io` is not a valid domain. You cannot have uppercase characters or underscores in a domain name. With the new validation method for image names, this was causing errors.
created_at: 2025-04-03T22:14:07Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,80 @@
+#!/bin/sh
+set -ex
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+TMPDIR="$(mktemp -d)"
+
+VOL_DIR="${1}"
+[ ! -z "${VOL_DIR}" ] || VOL_DIR='{{ .volDir }}'
+
+IMG_REF="${2}"
+[ ! -z "${IMG_REF}" ] || IMG_REF='{{ .imgRef }}'
+
+[ ! -z '{{ "" }}' ] || ANNOTATIONS_DATA='{{ .annotationsData }}'
+ANNOTATIONS_DATA="${ANNOTATIONS_DATA:-{}}"
+
+[ ! -z '{{ "" }}' ] || WAIT_PATH='{{ .waitPath }}'
+WAIT_PATH="${WAIT_PATH:-${VOL_DIR}/fine_tuning_completed.txt}"
+
+#{{`
+
+wait() {
+    until [ -e "${WAIT_PATH}" ]
+    do
+        sleep 1
+    done
+}
+
+mklayer() {
+    local DATA_DIR="${TMPDIR}/data"
+    mkdir -p "${DATA_DIR}"
+
+    cp -R "${VOL_DIR}/adapter_config.json" "${VOL_DIR}/adapter_model.safetensors" "${DATA_DIR}"
+
+    local TAR_LAYER_PATH="${TMPDIR}/layer.tar"
+
+    tar c -f "${TAR_LAYER_PATH}" -C "$(dirname "${DATA_DIR}")" "$(basename "${DATA_DIR}")"
+
+    TAR_LAYER_DIFF="$(sha256sum "${TAR_LAYER_PATH}" | cut -d ' ' -f '1')"
+
+    gzip -9 "${TAR_LAYER_PATH}"
+
+    TGZ_LAYER_MIME='application/vnd.oci.image.layer.v1.tar+gzip'
+    TGZ_LAYER_PATH="${TAR_LAYER_PATH}.gz"
+}
+
+mkconfig() {
+    CONFIG_MIME='application/vnd.oci.image.config.v1+json'
+    CONFIG_PATH="${TMPDIR}/config.json"
+
+    printf '{"rootfs":{"diff_ids":["sha256:%s"]}}' "${TAR_LAYER_DIFF}" > "${CONFIG_PATH}"
+}
+
+mkannotations() {
+    ANNOTATIONS_PATH="${TMPDIR}/annotations.json"
+
+    printf '%s' "${ANNOTATIONS_DATA}" > "${ANNOTATIONS_PATH}"
+}
+
+mklayout() {
+    LAYOUT_REF="${TMPDIR}/layout:latest"
+
+    cd "$(dirname "${TGZ_LAYER_PATH}")"
+    oras push --disable-path-validation --annotation-file "${ANNOTATIONS_PATH}" --config "${CONFIG_PATH}:${CONFIG_MIME}" --oci-layout "${LAYOUT_REF}" "$(basename "${TGZ_LAYER_PATH}"):${TGZ_LAYER_MIME}"
+    cd -
+}
+
+push() {
+    oras cp --from-oci-layout "${LAYOUT_REF}" "${IMG_REF}"
+}
+
+#`}}
+
+wait
+mklayer
+mkconfig
+mkannotations
+mklayout
+push
path: pkg/workspace/image/pusher.sh
user: login: t0rr3sp3dr0
body: Done! It's possible to attach an ephemeral container to the pod and access the tuning results.

![image](https://github.com/user-attachments/assets/91da9193-f94b-479c-86c8-8e9912fbaffa)

created_at: 2025-04-04T17:05:25Z
author_association: COLLABORATOR

PR comments:
user: login: kaito-pr-agent[bot]
created_at: 2025-03-25T17:49:57Z
body: Failed to generate code suggestions for PR
author_association: CONTRIBUTOR
user: login: codecov[bot]
created_at: 2025-03-25T18:04:55Z
body: ## [Codecov](https://app.codecov.io/gh/kaito-project/kaito/pull/950?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kaito-project) Report
Attention: Patch coverage is `59.33610%` with `98 lines` in your changes missing coverage. Please review.

```diff
@@            Coverage Diff             @@
##             main     #950      +/-   ##
==========================================
- Coverage   60.05%   57.43%   -2.62%     
==========================================
  Files          64       66       +2     
  Lines        6784     6863      +79     
==========================================
- Hits         4074     3942     -132     
- Misses       2502     2719     +217     
+ Partials      208      202       -6     
```

| [Components](https://app.codecov.io/gh/kaito-project/kaito/pull/950/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kaito-project) | Coverage Δ | |
|---|---|---|
| [workspace](https://app.codecov.io/gh/kaito-project/kaito/pull/950/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kaito-project) | `51.86% <50.98%> (-4.81%)` | :arrow_down: |
| [presets](https://app.codecov.io/gh/kaito-project/kaito/pull/950/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kaito-project) | `86.58% <ø> (ø)` | |
| [main](https://app.codecov.io/gh/kaito-project/kaito/pull/950/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kaito-project) | `∅ <ø> (∅)` | |

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: kaito-pr-agent[bot]
created_at: 2025-04-01T22:48:09Z
body: ## Title

#### (Describe updated until commit https://github.com/kaito-project/kaito/commit/88ca5a0e233b234eab7c0f9e3f479f3c526992c7)


feat: reimplement pull and push of images with skopeo and oras

___
### **User description**
**Reason for Change**:
This patch allows us to get rid of DinD, which currently runs in a privileged container. Skopeo will be used to pull images, mimicking the exact pull behavior of [K8s Image Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/). While ORAS is used to build the image in a way compatible with the implementation of [K8s Image Volumes](https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/) in `containerd` and `cri-o`.

**Requirements**
- [ ] added unit tests and e2e tests (if applicable).

**Issue Fixed**:
Fixes #901

**Notes for Reviewers**:
After this patch, images produced with newer versions of Kaito won't run on older versions of Kaito. However, images produced by older versions of Kaito will run on newer versions of Kaito. We probably want to make this clear on the release notes.


___

### **PR Type**
Enhancement


___

### **Description**
- Replaced image validation logic using regular expressions with `github.com/distribution/reference` for better accuracy and security.

- Introduced new files for managing image pulling and pushing operations using Skopeo and ORAS.

- Updated existing test cases to reflect the new image validation and management mechanisms.

- Added new scripts (`puller.sh` and `pusher.sh`) for handling image pulling and pushing tasks.

- Refactored container generation logic to utilize the new image management utilities.


___



### **Changes walkthrough** 📝
<table><thead><tr><th></th><th align="left">Relevant files</th></tr></thead><tbody><tr><td><strong>Enhancement</strong></td><td><details><summary>11 files</summary><table>
<tr>
  <td><strong>workspace_validation.go</strong><dd><code>Update image validation logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-5aa8d95205b91001e903b72a7a9dfb992d1acbeae62204197db42d7c684d811e">+20/-23</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>workspace_validation.go</strong><dd><code>Update image validation logic</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-ff810030efa090347e63f5a61e4344f94c6f3167ace34bfcc8f602249acd1959">+20/-23</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>common-preset.go</strong><dd><code>Add function to configure image pull secrets volume</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-92b0992bed5759464758e045db1af5d4724bfa636b5acbb18644256654a186a2">+42/-1</a>&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>workspace_controller.go</strong><dd><code>Use new image puller containers</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-48b5462b98109860e3cfdedcf776300516aa4d25678b4e008e829482b034a43e">+9/-9</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>puller.go</strong><dd><code>Add new image puller container implementation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-7134602adec3ec095fd1474b29baceedf0a01a62e6447c2e4ebb71b1a3c3a2e6">+66/-0</a>&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>pusher.go</strong><dd><code>Add new image pusher container implementation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-7dfe884df7295fe10a8cf12205db41b3723f870b125a689f5fce50729be298fa">+83/-0</a>&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>manifests.go</strong><dd><code>Update manifest generation to use new image puller containers</code></dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-5857e0b2f1941c47dd88c752fab691c28690d8ca2c3a497dbc78a7ed4b23d4b4">+45/-31</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>preset-tuning.go</strong><dd><code>Update data source and destination preparation to use new image </code><br><code>management</code></dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-0223c3e6955ed7cbef3a6fb52fd223333948df376025ff4d7b32220ecd59221e">+54/-135</a></td>

</tr>

<tr>
  <td><strong>utils.go</strong><dd><code>Update workspace creation to use new image management</code>&nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-2547c5a5595d73ed4720c414c96d15d919e53abf7441f91f1b2c7250196a2606">+2/-1</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>puller.sh</strong><dd><code>Add script for pulling images</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-614ef3ab0b2248179953f6bf3e377b7d9a1d642c9f719416d2808a5223bc0fcb">+58/-0</a>&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>pusher.sh</strong><dd><code>Add script for pushing images</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-a650f8cbaf34d071ea16f6d390e0d37e960bae1dad5f3325366f7b5de00d666a">+85/-0</a>&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Tests</strong></td><td><details><summary>9 files</summary><table>
<tr>
  <td><strong>workspace_validation_test.go</strong><dd><code>Update test cases for image validation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-40361cf72c705b9ed5e31c558e6ef9fa52e32b75118ae5d14d875513614fd686">+57/-19</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>workspace_validation_test.go</strong><dd><code>Update test cases for image validation</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-f7f7c3213c1c9eace61d77a0dcd6d091f3ea17c0b81ba8cb32ad416bdf09f888">+57/-19</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>puller_test.go</strong><dd><code>Add tests for image puller container</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-413d77be7b664a95dfee69d12ad38e7de99f7807444a44dda58a87856de03b16">+101/-0</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>pusher_test.go</strong><dd><code>Add tests for image pusher container</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-836b25185b731575daff0faf6e4792f7e6348b8417899454ec010f67aec10507">+109/-0</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>suite_test.go</strong><dd><code>Add suite test for image package</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-6dac5833b8b1d077f20ee9c09adcb9ba8d13891919c5874bb171074be72f6404">+16/-0</a>&nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>preset-tuning_test.go</strong><dd><code>Update tests for data source and destination preparation</code>&nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-0aa47707713f17820c71c5dbb7b492e3efcf86a4d5efc53c13c7a2dba6bdf376">+78/-56</a>&nbsp; </td>

</tr>

<tr>
  <td><strong>e2e_test.go</strong><dd><code>Copy necessary secrets for E2E tests</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-d5d12b6ea07b0a78c9ad1987e1486c565ecf16839f77a93e91b09475cf6c0e99">+8/-0</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>inference_with_adapters_test.go</strong><dd><code>Update test cases to use new image management</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-f0da6759781436985c4933dca4fc9820e2355b2cc446b08b8dfc71db6663c575">+8/-5</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>preset_test.go</strong><dd><code>Update test cases to use new image management</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-32f83f64c1acaee851fb38181dabd901f539301bcc5e3f10d7f792cbac0bf2a8">+42/-36</a>&nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Dependencies</strong></td><td><details><summary>2 files</summary><table>
<tr>
  <td><strong>go.mod</strong><dd><code>Add dependency for image reference parsing</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-33ef32bf6c23acb95f5902d7097b7a1d5128ca061167ec0716715b0b9eeaa5f6">+2/-0</a>&nbsp; &nbsp; &nbsp; </td>

</tr>

<tr>
  <td><strong>go.sum</strong><dd><code>Update go.sum for new dependency</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-3295df7234525439d778f1b282d146a4f1ff6b415248aaac074e8042d9f42d63">+4/-0</a>&nbsp; &nbsp; &nbsp; </td>

</tr>
</table></details></td></tr><tr><td><strong>Documentation</strong></td><td><details><summary>1 files</summary><table>
<tr>
  <td><strong>README.md</strong><dd><code>Document new image management features</code>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; </dd></td>
  <td><a href="https://github.com/kaito-project/kaito/pull/950/files#diff-deab3600e61d6ef28b5f3e8879035f23be386257987ee3c79b56663e93c714a0">+31/-0</a>&nbsp; &nbsp; </td>

</tr>
</table></details></td></tr></tr></tbody></table>

___

> <details> <summary>  Need help?</summary><li>Type <code>/help how to ...</code> in the comments thread for any questions about PR-Agent usage.</li><li>Check out the <a href="https://qodo-merge-docs.qodo.ai/usage-guide/">documentation</a> for more information.</li></details>
author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-03T21:25:32Z
body: ## PR Reviewer Guide 🔍

#### (Review updated until commit https://github.com/kaito-project/kaito/commit/88ca5a0e233b234eab7c0f9e3f479f3c526992c7)


Here are some key observations to aid the review process:

<table>
<tr><td>

**🎫 Ticket compliance analysis 🔶**



**[901](https://github.com/kaito-project/kaito/issues/901) - Partially compliant**

Compliant requirements:

- Added unit tests and e2e tests (if applicable).
- Replaced Docker with Skopeo for image management.
- Ensured compatibility with multi-architecture images.

Non-compliant requirements:

- None identified.

Requires further human verification:

- None identified.



</td></tr>
<tr><td>&nbsp;<strong>Estimated effort to review</strong>: 3 🔵🔵🔵⚪⚪</td></tr>
<tr><td>🧪&nbsp;<strong>PR contains tests</strong></td></tr>
<tr><td>🔒&nbsp;<strong>No security concerns identified</strong></td></tr>
<tr><td>⚡&nbsp;<strong>Recommended focus areas for review</strong><br><br>

<details><summary><a href='https://github.com/kaito-project/kaito/pull/950/files#diff-40361cf72c705b9ed5e31c558e6ef9fa52e32b75118ae5d14d875513614fd686R210-R213'><strong>Potential Issue with Image Reference Parsing</strong></a>

The new code adds validation for image references using `reference.ParseDockerRef`. Ensure that this function correctly handles various edge cases and invalid inputs.
</summary>

```go
  train_test_split: 1

DataCollator:
  mlm: true`,
```

</details>

<details><summary><a href='https://github.com/kaito-project/kaito/pull/950/files#diff-7dfe884df7295fe10a8cf12205db41b3723f870b125a689f5fce50729be298faR36-R39'><strong>Missing Error Handling in Pusher Script</strong></a>

The `renderPusherSH` function in `pusher.go` does not handle errors returned by `reference.ParseDockerRef`. Consider adding appropriate error handling to ensure robustness.
</summary>

```go
if err != nil {
	log.Printf("failed to normalize image reference `%s`: %v", imgRef, err)
}
if normalizedImgRef != nil {
```

</details>

<details><summary><a href='https://github.com/kaito-project/kaito/pull/950/files#diff-7134602adec3ec095fd1474b29baceedf0a01a62e6447c2e4ebb71b1a3c3a2e6R36-R39'><strong>Potential Security Risk in Puller Script</strong></a>

The `renderPullerSH` function in `puller.go` constructs shell commands directly from user-provided inputs. This could lead to command injection vulnerabilities. Ensure that user inputs are properly sanitized before being used in shell commands.
</summary>

```go
}
if normalizedImgRef != nil {
	imgRef = normalizedImgRef.String()
}
```

</details>

</td></tr>
</table>

author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-03T21:29:48Z
body: **[Persistent review](https://github.com/kaito-project/kaito/pull/950#issuecomment-2776951885)** updated to latest commit https://github.com/kaito-project/kaito/commit/48f2e4ed38c05aa411e27a9fe5a8575d62a0049b
author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-03T21:36:20Z
body: **[Persistent review](https://github.com/kaito-project/kaito/pull/950#issuecomment-2776951885)** updated to latest commit https://github.com/kaito-project/kaito/commit/ee41f2cc1b29bda77dcb687136d37c719f5cca25
author_association: CONTRIBUTOR
user: login: ishaansehgal99
created_at: 2025-04-03T21:46:07Z
body: If you see ways, please try to split up PR.
author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-04T06:05:17Z
body: **[Persistent review](https://github.com/kaito-project/kaito/pull/950#issuecomment-2776951885)** updated to latest commit https://github.com/kaito-project/kaito/commit/e677d012bec9e9d9ac1617f9ff86052e76fac6a2
author_association: CONTRIBUTOR
user: login: kaito-pr-agent[bot]
created_at: 2025-04-04T06:06:46Z
body: **[Persistent review](https://github.com/kaito-project/kaito/pull/950#issuecomment-2776951885)** updated to latest commit https://github.com/kaito-project/kaito/commit/88ca5a0e233b234eab7c0f9e3f479f3c526992c7
author_association: CONTRIBUTOR