Repository: GoogleContainerTools/skaffold
Pull request #8695 (2023-04-20 18:10:29)

feat: add wait group in docker and k8s job logger to avoid race condition
Fixes: #8672

**Description**
This PR adds a wait group in the k8s and docker logger to avoid a race condition when:
1. We have the loggers running, and
2. The execution of the containers/jobs finish, and 
3. We trigger the loggers stop to start draining their content
 
Before, we were draining the logs in the stop step without waiting for the logs triggered during execution to finish.

codecov[bot]: ## [Codecov](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) Report
> Merging [#8695](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (53b4bde) into [main](https://codecov.io/gh/GoogleContainerTools/skaffold/commit/290280e8c6c19231e9ed5b5c027f2c01d22f75be?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (290280e) will **decrease** coverage by `5.99%`.
> The diff coverage is `54.27%`.

```diff
@@            Coverage Diff             @@
##             main    #8695      +/-   ##
==========================================
- Coverage   70.48%   64.49%   -5.99%     
==========================================
  Files         515      616     +101     
  Lines       23150    30982    +7832     
==========================================
+ Hits        16317    19982    +3665     
- Misses       5776     9501    +3725     
- Partials     1057     1499     +442     
```


| [Impacted Files](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | Coverage Δ | |
|---|---|---|
| [cmd/skaffold/app/cmd/completion.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29tcGxldGlvbi5nbw==) | `13.04% <0.00%> (-1.25%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/config/list.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL2xpc3QuZ28=) | `65.21% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/config/set.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL3NldC5nbw==) | `88.72% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/config/util.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL3V0aWwuZ28=) | `54.28% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/credits.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY3JlZGl0cy5nbw==) | `100.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/credits/export.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY3JlZGl0cy9leHBvcnQuZ28=) | `0.00% <0.00%> (ø)` | |
| [cmd/skaffold/app/cmd/deploy.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZGVwbG95Lmdv) | `40.90% <0.00%> (-12.94%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/generate\_pipeline.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZ2VuZXJhdGVfcGlwZWxpbmUuZ28=) | `60.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/inspect\_modules.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvaW5zcGVjdF9tb2R1bGVzLmdv) | `65.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/inspect\_profiles.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvaW5zcGVjdF9wcm9maWxlcy5nbw==) | `66.66% <ø> (ø)` | |
| ... and [39 more](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | |

... and [407 files with indirect coverage changes](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8695/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools)

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools)


Patch:
diff --git a/docs-v2/content/en/schemas/v4beta5.json b/docs-v2/content/en/schemas/v4beta5.json
index c55ac59e6..c29997e22 100755
--- a/docs-v2/content/en/schemas/v4beta5.json
+++ b/docs-v2/content/en/schemas/v4beta5.json
@@ -2606,12 +2606,6 @@
           "x-intellij-html-description": "to add timestamps to log format.",
           "default": "false"
         },
-        "noPush": {
-          "type": "boolean",
-          "description": "if you only want to build the image, without pushing to a registry.",
-          "x-intellij-html-description": "if you only want to build the image, without pushing to a registry.",
-          "default": "false"
-        },
         "ociLayoutPath": {
           "type": "string",
           "description": "to specify a directory in the container where the OCI image layout of a built image will be placed. This can be used to automatically track the exact image built by kaniko.",
@@ -2719,7 +2713,6 @@
         "cleanup",
         "insecure",
         "insecurePull",
-        "noPush",
         "force",
         "logTimestamp",
         "reproducible",
diff --git a/pkg/skaffold/build/kaniko/args.go b/pkg/skaffold/build/kaniko/args.go
index 05ded7ebe..4d2bdf67a 100644
--- a/pkg/skaffold/build/kaniko/args.go
+++ b/pkg/skaffold/build/kaniko/args.go
@@ -101,10 +101,6 @@ func Args(artifact *latest.KanikoArtifact, tag, context string) ([]string, error
 		args = append(args, LogTimestampFlag)
 	}
 
-	if artifact.NoPush {
-		args = append(args, NoPushFlag)
-	}
-
 	if artifact.OCILayoutPath != "" {
 		args = append(args, OCILayoutFlag, artifact.OCILayoutPath)
 	}
diff --git a/pkg/skaffold/build/kaniko/types.go b/pkg/skaffold/build/kaniko/types.go
index c47251881..21291e9b7 100644
--- a/pkg/skaffold/build/kaniko/types.go
+++ b/pkg/skaffold/build/kaniko/types.go
@@ -53,8 +53,6 @@ const (
 	LogFormatFlag = "--log-format"
 	// LogTimestampFlag additional flag
 	LogTimestampFlag = "--log-timestamp"
-	// NoPushFlag additional flag
-	NoPushFlag = "--no-push"
 	// OCILayoutFlag additional flag
 	OCILayoutFlag = "--oci-layout-path"
 	// RegistryCertificateFlag additional flag
diff --git a/pkg/skaffold/docker/logger/log.go b/pkg/skaffold/docker/logger/log.go
index 93b4e65c8..0beaab80e 100644
--- a/pkg/skaffold/docker/logger/log.go
+++ b/pkg/skaffold/docker/logger/log.go
@@ -35,6 +35,7 @@ import (
 )
 
 type Logger struct {
+	wg                  sync.WaitGroup
 	out                 io.Writer
 	tracker             *tracker.ContainerTracker
 	colorPicker         output.ColorPicker
@@ -126,6 +127,8 @@ func NewStatusBackoff() *wait.Backoff {
 
 func (l *Logger) streamLogsFromContainer(ctx context.Context, id string, force bool) {
 	tr, tw := io.Pipe()
+	l.wg.Add(1)
+	defer l.wg.Done()
 	go func() {
 		var err error
 		backoff := NewStatusBackoff()
@@ -166,6 +169,7 @@ func (l *Logger) Stop() {
 		return
 	}
 	l.childThreadEmitLogs.Set(false)
+	l.wg.Wait()
 
 	l.hadLogsOutput.Range(func(key, value interface{}) bool {
 		if !value.(bool) {
diff --git a/pkg/skaffold/inspect/executionModes/list.go b/pkg/skaffold/inspect/executionModes/list.go
index 011129644..49f5ce79d 100644
--- a/pkg/skaffold/inspect/executionModes/list.go
+++ b/pkg/skaffold/inspect/executionModes/list.go
@@ -48,6 +48,8 @@ func PrintExecutionModesList(ctx context.Context, out io.Writer, opts inspect.Op
 		CustomActionsExecutionModes: map[string]string{},
 	}
 
+	allActions := map[string]string{}
+
 	for _, c := range cfgs {
 		for _, tc := range c.Verify {
 			if tc.ExecutionMode.KubernetesClusterExecutionMode != nil {
@@ -56,8 +58,21 @@ func PrintExecutionModesList(ctx context.Context, out io.Writer, opts inspect.Op
 				l.VerifyExecutionModes[tc.Name] = "local"
 			}
 		}
+
+		for _, a := range c.CustomActions {
+			if a.ExecutionModeConfig.KubernetesClusterExecutionMode != nil {
+				allActions[a.Name] = "kubernetesCluster"
+			} else {
+				allActions[a.Name] = "local"
+			}
+		}
+	}
+
+	for _, a := range customActions {
+		if execMode, found := allActions[a]; found {
+			l.CustomActionsExecutionModes[a] = execMode
+		}
 	}
-	// TODO(#8572) add similar logic for customAction schema fields when they are complete
 
 	return formatter.Write(l)
 }
diff --git a/pkg/skaffold/k8sjob/logger/log.go b/pkg/skaffold/k8sjob/logger/log.go
index 13047fb91..9ca377451 100644
--- a/pkg/skaffold/k8sjob/logger/log.go
+++ b/pkg/skaffold/k8sjob/logger/log.go
@@ -39,12 +39,12 @@ import (
 )
 
 type Logger struct {
+	wg                  sync.WaitGroup
 	out                 io.Writer
 	tracker             *tracker.JobTracker
 	labeller            *label.DefaultLabeller
 	colorPicker         output.ColorPicker
 	hadLogsOutput       sync.Map
-	lastLogOutput       sync.Map
 	childThreadEmitLogs AtomicBool
 	muted               int32
 	kubeContext         string
@@ -126,20 +126,23 @@ func (l *Logger) Start(ctx context.Context, out io.Writer) error {
 				return
 			case info := <-l.tracker.Notifier():
 				id, namespace := info[0], info[1]
-				go l.streamLogsFromKubernetesJob(ctx, id, namespace, false, metav1.Time{}, false)
+				go l.streamLogsFromKubernetesJob(ctx, id, namespace, false)
 			}
 		}
 	}()
 	return nil
 }
 
-func (l *Logger) streamLogsFromKubernetesJob(ctx context.Context, id, namespace string, force bool, sinceTime metav1.Time, useSincetime bool) {
+func (l *Logger) streamLogsFromKubernetesJob(ctx context.Context, id, namespace string, force bool) {
 	clientset, err := kubernetesclient.Client(l.kubeContext)
 	if err != nil {
 		log.Entry(ctx).Warn(err)
 	}
 
 	tr, tw := io.Pipe()
+	l.wg.Add(1)
+	defer l.wg.Done()
+
 	go func() {
 		var err error
 		backoff := NewStatusBackoff()
@@ -170,11 +173,8 @@ func (l *Logger) streamLogsFromKubernetesJob(ctx context.Context, id, namespace
 				}
 			}
 
-			podLogOptions := &corev1.PodLogOptions{}
-			if useSincetime {
-				podLogOptions = &corev1.PodLogOptions{
-					SinceTime: &sinceTime,
-				}
+			podLogOptions := &corev1.PodLogOptions{
+				Follow: true,
 			}
 
 			// Stream the logs
@@ -186,7 +186,6 @@ func (l *Logger) streamLogsFromKubernetesJob(ctx context.Context, id, namespace
 			defer podLogs.Close()
 			io.Copy(tw, podLogs)
 			l.hadLogsOutput.Store(id, true)
-			l.lastLogOutput.Store(id, metav1.Now())
 			return true, nil
 		}); waitErr != nil {
 			// Don't print errors if the user interrupted the logs
@@ -212,14 +211,7 @@ func (l *Logger) Stop() {
 	l.hadLogsOutput.Range(func(key, value interface{}) bool {
 		if !value.(bool) {
 			l.streamLogsFromKubernetesJob(context.TODO(),
-				key.(string), l.tracker.DeployedJobs()[key.(string)].Namespace, true, metav1.Time{}, false)
-		} else {
-			value, _ = l.lastLogOutput.Load(key)
-			if val, ok := value.(metav1.Time); ok {
-				l.streamLogsFromKubernetesJob(context.TODO(),
-					key.(string), l.tracker.DeployedJobs()[key.(string)].Namespace, true, val, true)
-			}
-			return true
+				key.(string), l.tracker.DeployedJobs()[key.(string)].Namespace, true)
 		}
 		return true
 	})
diff --git a/pkg/skaffold/verify/k8sjob/verify.go b/pkg/skaffold/verify/k8sjob/verify.go
index e746e6884..b7c4976e2 100644
--- a/pkg/skaffold/verify/k8sjob/verify.go
+++ b/pkg/skaffold/verify/k8sjob/verify.go
@@ -51,6 +51,7 @@ import (
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/log"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/schema/latest"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/status"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/util"
 )
 
 // Verifier verifies deployments using kubernetes libs/CLI.
@@ -329,6 +330,7 @@ func (v *Verifier) createJob(jobName string, container latest.VerifyContainer) *
 			Namespace: v.defaultNamespace,
 		},
 		Spec: batchv1.JobSpec{
+			BackoffLimit: util.Ptr[int32](0),
 			Template: corev1.PodTemplateSpec{
 				ObjectMeta: metav1.ObjectMeta{
 					Labels: map[string]string{


Test patch:
diff --git a/integration/testdata/verify-fail-k8s/skaffold.yaml b/integration/testdata/verify-fail-k8s/skaffold.yaml
index fe6e68418..c5828dded 100644
--- a/integration/testdata/verify-fail-k8s/skaffold.yaml
+++ b/integration/testdata/verify-fail-k8s/skaffold.yaml
@@ -15,3 +15,24 @@ verify:
       args: ["-c", "echo $FOO;exit 1"]
     executionMode:
       kubernetesCluster: {}
+
+profiles:
+  - name: no-duplicated-logs
+    verify:
+      - name: alpine-1
+        executionMode:
+          kubernetesCluster: {}
+        container:
+          name: alpine-1
+          image: alpine:3.15.4
+          command: ["/bin/sh"]
+          args: ["-c", "echo alpine-1; sleep 1; echo bye alpine-1"]
+      
+      - name: alpine-2
+        executionMode:
+          kubernetesCluster: {}
+        container:
+          name: alpine-2
+          image: alpine:3.15.4
+          command: ["/bin/sh"]
+          args: ["-c", "echo alpine-2; echo bye alpine-2; exit 1"]
\ No newline at end of file
diff --git a/integration/testdata/verify-fail/skaffold.yaml b/integration/testdata/verify-fail/skaffold.yaml
index 6c423a935..eea1e2ab7 100644
--- a/integration/testdata/verify-fail/skaffold.yaml
+++ b/integration/testdata/verify-fail/skaffold.yaml
@@ -11,3 +11,20 @@ verify:
     image: alpine:3.15.4
     command: ["/bin/sh"]
     args: ["-c", "echo $FOO;exit 1"]
+
+profiles:
+  - name: no-duplicated-logs
+    verify:
+      - name: alpine-1
+        container:
+          name: alpine-1
+          image: alpine:3.15.4
+          command: ["/bin/sh"]
+          args: ["-c", "echo alpine-1; sleep 2; echo bye alpine-1"]
+      
+      - name: alpine-2
+        container:
+          name: alpine-2
+          image: alpine:3.15.4
+          command: ["/bin/sh"]
+          args: ["-c", "echo alpine-2; echo bye alpine-2; exit 1"]
diff --git a/integration/testdata/verify-succeed-k8s/skaffold.yaml b/integration/testdata/verify-succeed-k8s/skaffold.yaml
index 85da37e86..9f216021a 100644
--- a/integration/testdata/verify-succeed-k8s/skaffold.yaml
+++ b/integration/testdata/verify-succeed-k8s/skaffold.yaml
@@ -34,4 +34,16 @@ verify:
       command: ["/bin/sh"]
       args: ["-c", "echo $FOO; sleep 10; echo bye"]
     executionMode:
-      kubernetesCluster: {}
\ No newline at end of file
+      kubernetesCluster: {}
+
+profiles:
+  - name: no-duplicated-logs
+    verify:
+      - name: alpine-1
+        executionMode:
+          kubernetesCluster: {}
+        container:
+          name: alpine-1
+          image: alpine:3.15.4
+          command: ["/bin/sh"]
+          args: ["-c", "echo alpine-1; sleep 1; echo bye alpine-1"]
\ No newline at end of file
diff --git a/integration/testdata/verify-succeed/skaffold.yaml b/integration/testdata/verify-succeed/skaffold.yaml
index 4d4854069..5613c374f 100644
--- a/integration/testdata/verify-succeed/skaffold.yaml
+++ b/integration/testdata/verify-succeed/skaffold.yaml
@@ -20,4 +20,14 @@ verify:
     name: alpine-2
     image: alpine:3.15.4
     command: ["/bin/sh"]
-    args: ["-c", "echo $FOO; sleep 10; echo bye"]
\ No newline at end of file
+    args: ["-c", "echo $FOO; sleep 10; echo bye"]
+
+profiles:
+  - name: no-duplicated-logs
+    verify:
+      - name: alpine-1
+        container:
+          name: alpine-1
+          image: alpine:3.15.4
+          command: ["/bin/sh"]
+          args: ["-c", "echo alpine-1; sleep 2; echo bye alpine-1"]
\ No newline at end of file
diff --git a/integration/verify_test.go b/integration/verify_test.go
index ac514ba3e..58f538077 100644
--- a/integration/verify_test.go
+++ b/integration/verify_test.go
@@ -19,6 +19,7 @@ package integration
 import (
 	"os"
 	"path/filepath"
+	"strings"
 	"testing"
 
 	"github.com/google/uuid"
@@ -156,3 +157,104 @@ func TestKubernetesJobVerifyOneTestFailsWithEnvVar(t *testing.T) {
 
 	// TODO(aaron-prindle) verify that FAILED event is found where expected
 }
+
+func TestNoDuplicateLogsLocal(t *testing.T) {
+	tests := []struct {
+		description        string
+		dir                string
+		profile            string
+		shouldErr          bool
+		expectedUniqueLogs []string
+	}{
+		{
+			description: "no duplicated logs in docker actions, success execution",
+			dir:         "testdata/verify-succeed",
+			profile:     "no-duplicated-logs",
+			expectedUniqueLogs: []string{
+				"[alpine-1] alpine-1",
+				"[alpine-1] bye alpine-1",
+			},
+		},
+		{
+			description: "no duplicated logs in docker actions, fail execution",
+			dir:         "testdata/verify-fail",
+			profile:     "no-duplicated-logs",
+			shouldErr:   true,
+			expectedUniqueLogs: []string{
+				"[alpine-1] alpine-1",
+				"[alpine-1] bye alpine-1",
+				"[alpine-2] alpine-2",
+				"[alpine-2] bye alpine-2",
+			},
+		},
+	}
+
+	for _, test := range tests {
+		testutil.Run(t, test.description, func(t *testutil.T) {
+			MarkIntegrationTest(t.T, CanRunWithoutGcp)
+
+			args := []string{"-p", test.profile}
+			out, err := skaffold.Verify(args...).InDir(test.dir).RunWithCombinedOutput(t.T)
+
+			t.CheckError(test.shouldErr, err)
+
+			logs := string(out)
+			checkUniqueLogs(t, logs, test.expectedUniqueLogs)
+		})
+	}
+}
+
+func TestNoDuplicateLogsK8SJobs(t *testing.T) {
+	tests := []struct {
+		description        string
+		dir                string
+		profile            string
+		shouldErr          bool
+		expectedUniqueLogs []string
+	}{
+		{
+			description: "no duplicated logs in k8s actions, success execution",
+			dir:         "testdata/verify-succeed-k8s",
+			profile:     "no-duplicated-logs",
+			expectedUniqueLogs: []string{
+				"[alpine-1] alpine-1",
+				"[alpine-1] bye alpine-1",
+			},
+		},
+		{
+			description: "no duplicated logs in k8s actions, fail execution",
+			dir:         "testdata/verify-fail-k8s",
+			profile:     "no-duplicated-logs",
+			shouldErr:   true,
+			expectedUniqueLogs: []string{
+				"[alpine-1] alpine-1",
+				"[alpine-1] bye alpine-1",
+				"[alpine-2] alpine-2",
+				"[alpine-2] bye alpine-2",
+			},
+		},
+	}
+
+	for _, test := range tests {
+		testutil.Run(t, test.description, func(t *testutil.T) {
+			MarkIntegrationTest(t.T, NeedsGcp)
+
+			args := []string{"-p", test.profile}
+			out, err := skaffold.Verify(args...).InDir(test.dir).RunWithCombinedOutput(t.T)
+
+			t.CheckError(test.shouldErr, err)
+
+			logs := string(out)
+			checkUniqueLogs(t, logs, test.expectedUniqueLogs)
+		})
+	}
+}
+
+func checkUniqueLogs(t *testutil.T, logs string, expectedUniqueLogs []string) {
+	for _, uniqueLog := range expectedUniqueLogs {
+		timesFound := strings.Count(logs, uniqueLog)
+		if timesFound != 1 {
+			t.Fatalf(`Log message "%v" found %v times, expected exactly 1 time`, uniqueLog, timesFound)
+		}
+	}
+}
diff --git a/pkg/skaffold/build/gcb/kaniko_test.go b/pkg/skaffold/build/gcb/kaniko_test.go
index d9c8d0e6d..0f8e88fe1 100644
--- a/pkg/skaffold/build/gcb/kaniko_test.go
+++ b/pkg/skaffold/build/gcb/kaniko_test.go
@@ -194,16 +194,6 @@ func TestKanikoBuildSpec(t *testing.T) {
 				kaniko.LogTimestampFlag,
 			},
 		},
-		{
-			description: "with NoPush",
-			artifact: &latest.KanikoArtifact{
-				DockerfilePath: "Dockerfile",
-				NoPush:         true,
-			},
-			expectedArgs: []string{
-				kaniko.NoPushFlag,
-			},
-		},
 		{
 			description: "with OCILayoutPath",
 			artifact: &latest.KanikoArtifact{
diff --git a/pkg/skaffold/build/kaniko/args_test.go b/pkg/skaffold/build/kaniko/args_test.go
index f00c65de3..b66f308d1 100644
--- a/pkg/skaffold/build/kaniko/args_test.go
+++ b/pkg/skaffold/build/kaniko/args_test.go
@@ -187,17 +187,6 @@ func TestArgs(t *testing.T) {
 			},
 			wantErr: false,
 		},
-		{
-			description: "with NoPush",
-			artifact: &latest.KanikoArtifact{
-				DockerfilePath: "dir/Dockerfile",
-				NoPush:         true,
-			},
-			expectedArgs: []string{
-				NoPushFlag,
-			},
-			wantErr: false,
-		},
 		{
 			description: "with OCILayoutPath",
 			artifact: &latest.KanikoArtifact{
diff --git a/pkg/skaffold/inspect/executionModes/list_test.go b/pkg/skaffold/inspect/executionModes/list_test.go
index a0c924a81..d192d8fe1 100644
--- a/pkg/skaffold/inspect/executionModes/list_test.go
+++ b/pkg/skaffold/inspect/executionModes/list_test.go
@@ -67,6 +67,24 @@ func TestPrintExecutionModesList(t *testing.T) {
 			err:         errors.New("some error occurred"),
 			expected:    `{"errorCode":"INSPECT_UNKNOWN_ERR","errorMessage":"some error occurred"}` + "\n",
 		},
+		{
+			description:   "print executionModes for k8s custom action",
+			expected:      `{"verifyExecutionModes":{},"customActionsExecutionModes":{"action1":"kubernetesCluster"}}` + "\n",
+			customActions: []string{"action1"},
+			module:        []string{"cfg-with-customActions"},
+		},
+		{
+			description:   "print executionModes for local custom action",
+			expected:      `{"verifyExecutionModes":{},"customActionsExecutionModes":{"action2":"local"}}` + "\n",
+			customActions: []string{"action2"},
+			module:        []string{"cfg-with-customActions"},
+		},
+		{
+			description:   "print executionModes for custom action and verify",
+			expected:      `{"verifyExecutionModes":{"bar":"kubernetesCluster"},"customActionsExecutionModes":{"action1":"kubernetesCluster"}}` + "\n",
+			customActions: []string{"action1"},
+			module:        []string{"cfg-with-customActions-and-verify"},
+		},
 	}
 
 	for _, test := range tests {
@@ -115,6 +133,73 @@ func TestPrintExecutionModesList(t *testing.T) {
 						},
 					},
 				}, SourceFile: "path/to/cfg-with-default-namespace"},
+
+				&parser.SkaffoldConfigEntry{SkaffoldConfig: &latest.SkaffoldConfig{
+					Metadata: latest.Metadata{Name: "cfg-with-customActions"},
+					Pipeline: latest.Pipeline{
+						CustomActions: []latest.Action{
+							{
+								Name: "action1",
+								ExecutionModeConfig: latest.ActionExecutionModeConfig{
+									VerifyExecutionModeType: latest.VerifyExecutionModeType{
+										KubernetesClusterExecutionMode: &latest.KubernetesClusterVerifier{},
+									},
+								},
+								Containers: []latest.VerifyContainer{
+									{
+										Name:  "task1",
+										Image: "task1-image",
+									},
+								},
+							},
+							{
+								Name: "action2",
+								Containers: []latest.VerifyContainer{
+									{
+										Name:  "task2",
+										Image: "task2-image",
+									},
+								},
+							},
+						},
+					},
+				}, SourceFile: "path/to/cfg-with-customActions"},
+
+				&parser.SkaffoldConfigEntry{SkaffoldConfig: &latest.SkaffoldConfig{
+					Metadata: latest.Metadata{Name: "cfg-with-customActions-and-verify"},
+					Pipeline: latest.Pipeline{
+						Verify: []*latest.VerifyTestCase{
+							{
+								Name: "bar",
+								Container: latest.VerifyContainer{
+									Name:  "bar",
+									Image: "bar",
+								},
+								ExecutionMode: latest.VerifyExecutionModeConfig{
+									VerifyExecutionModeType: latest.VerifyExecutionModeType{
+										KubernetesClusterExecutionMode: &latest.KubernetesClusterVerifier{},
+									},
+								},
+							},
+						},
+						CustomActions: []latest.Action{
+							{
+								Name: "action1",
+								ExecutionModeConfig: latest.ActionExecutionModeConfig{
+									VerifyExecutionModeType: latest.VerifyExecutionModeType{
+										KubernetesClusterExecutionMode: &latest.KubernetesClusterVerifier{},
+									},
+								},
+								Containers: []latest.VerifyContainer{
+									{
+										Name:  "task1",
+										Image: "task1-image",
+									},
+								},
+							},
+						},
+					},
+				}, SourceFile: "path/to/cfg-with-customActions-and-verify"},
 			}
 			t.Override(&inspect.GetConfigSet, func(_ context.Context, opts config.SkaffoldOptions) (parser.SkaffoldConfigSet, error) {
 				// mock profile activation
diff --git a/pkg/skaffold/schema/latest/config.go b/pkg/skaffold/schema/latest/config.go
index 5749f1a6e..24cec608e 100644
--- a/pkg/skaffold/schema/latest/config.go
+++ b/pkg/skaffold/schema/latest/config.go
@@ -1344,9 +1344,6 @@ type KanikoArtifact struct {
 	// InsecurePull if you want to pull images from a plain HTTP registry.
 	InsecurePull bool `yaml:"insecurePull,omitempty"`
 
-	// NoPush if you only want to build the image, without pushing to a registry.
-	NoPush bool `yaml:"noPush,omitempty"`
-
 	// Force building outside of a container.
 	Force bool `yaml:"force,omitempty"`