Repository: GoogleContainerTools/skaffold
Pull request #8340 (2023-01-19 14:56:15)

feat: add ingore-path kaniko flag support
**Related**:  https://github.com/GoogleContainerTools/kaniko/issues/2164

**Description**
For local development we are using skaffold. Also, as we are from Ukraine and there could be no elictricty or no internet we need to be able to working offline. For such purposes we  have created a script which ups the kind cluster and we can continue working on.
On mac based systems we don't have any issues with kind and kaniko. 
But unfortunately on linux machines we have a similar issue as in linked kaniko issue.
So we need to add such flag to kaniko.

**User facing changes**

Ability to add an ingore paths for kaniko.


google-cla[bot]: Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this [failed invocation](https://github.com/GoogleContainerTools/skaffold/pull/8340/checks?check_run_id=10755311444) of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.
codecov[bot]: # [Codecov](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) Report
> Merging [#8340](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (34f9f5c) into [main](https://codecov.io/gh/GoogleContainerTools/skaffold/commit/290280e8c6c19231e9ed5b5c027f2c01d22f75be?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (290280e) will **decrease** coverage by `4.40%`.
> The diff coverage is `54.85%`.

```diff
@@            Coverage Diff             @@
##             main    #8340      +/-   ##
==========================================
- Coverage   70.48%   66.09%   -4.40%     
==========================================
  Files         515      605      +90     
  Lines       23150    29658    +6508     
==========================================
+ Hits        16317    19601    +3284     
- Misses       5776     8588    +2812     
- Partials     1057     1469     +412     
```


| [Impacted Files](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | Coverage Δ | |
|---|---|---|
| [cmd/skaffold/app/cmd/completion.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29tcGxldGlvbi5nbw==) | `13.04% <0.00%> (-1.25%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/config/list.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL2xpc3QuZ28=) | `65.21% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/config/set.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL3NldC5nbw==) | `88.72% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/config/util.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL3V0aWwuZ28=) | `54.28% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/credits.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY3JlZGl0cy5nbw==) | `100.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/credits/export.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY3JlZGl0cy9leHBvcnQuZ28=) | `0.00% <0.00%> (ø)` | |
| [cmd/skaffold/app/cmd/deploy.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZGVwbG95Lmdv) | `40.90% <0.00%> (-12.94%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/generate\_pipeline.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZ2VuZXJhdGVfcGlwZWxpbmUuZ28=) | `60.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/inspect\_modules.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvaW5zcGVjdF9tb2R1bGVzLmdv) | `65.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/inspect\_profiles.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvaW5zcGVjdF9wcm9maWxlcy5nbw==) | `66.66% <ø> (ø)` | |
| ... and [415 more](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8340?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools)


Patch:
diff --git a/cmd/skaffold/app/cmd/flags.go b/cmd/skaffold/app/cmd/flags.go
index 0b3c526b3..99161eae3 100644
--- a/cmd/skaffold/app/cmd/flags.go
+++ b/cmd/skaffold/app/cmd/flags.go
@@ -738,6 +738,15 @@ The build result from a previous 'skaffold build --file-output' run can be used
 		FlagAddMethod: "StringVar",
 		DefinedOn:     []string{"dev", "run", "debug", "deploy", "apply", "delete"},
 	},
+	{
+		Name:          "keep-running-on-failure",
+		Shorthand:     "",
+		Usage:         "If true, the session will be suspended instead of ending if any errors occur, the user can fix the errors during the session suspension, the session can be restored and continued by pressing any key. ",
+		Value:         &opts.KeepRunningOnFailure,
+		DefValue:      false,
+		FlagAddMethod: "BoolVar",
+		DefinedOn:     []string{"dev", "debug"},
+	},
 }
 
 func methodNameByType(v reflect.Value) string {
diff --git a/docs-v2/content/en/docs/pipeline-stages/deployers/cloudrun.md b/docs-v2/content/en/docs/pipeline-stages/deployers/cloudrun.md
index 13d791a81..5907ea3d0 100644
--- a/docs-v2/content/en/docs/pipeline-stages/deployers/cloudrun.md
+++ b/docs-v2/content/en/docs/pipeline-stages/deployers/cloudrun.md
@@ -19,6 +19,10 @@ See the [Cloud Run YAML reference](https://cloud.google.com/run/docs/reference/y
 
 This deployer will use the [application default credentials](https://cloud.google.com/docs/authentication/production#automatically) to deploy.  You can configure this to use your user credentials by running `gcloud auth application-default login`.
 
+# Features
+- As of Skaffold `v2.1.0`, Skaffold's Cloud Run Deployer now supports log streaming from the deployed Cloud Run Service(s)/Job(s).  NOTE: this requires installing the following `gcloud components`: `alpha`, `beta`, `cloud-run-proxy`, and `log-streaming`
+- Supports deploying Cloud Run `Service` and `Job` objects
+
 ## Configuring Cloud Run
 
 To deploy to Cloud Run, use the `cloudrun` type in the `deploy` section of `skaffold.yaml`.
@@ -27,6 +31,7 @@ The `cloudrun` type offers the following options:
 
 {{< schema root="CloudRunDeploy" >}}
 
+
 ### Example
 
 The following `deploy` section instructs Skaffold to deploy
@@ -39,3 +44,4 @@ Images listed to be deployed with the Cloud Run deployer must be present in Goog
 Registry or Google Container Registry. If you are using Skaffold to build the images, ensure `push` is 
 set to true.
 {{< /alert >}}
+
diff --git a/docs-v2/content/en/docs/references/cli/_index.md b/docs-v2/content/en/docs/references/cli/_index.md
index 76fd8f2ef..183f0d960 100644
--- a/docs-v2/content/en/docs/references/cli/_index.md
+++ b/docs-v2/content/en/docs/references/cli/_index.md
@@ -435,6 +435,7 @@ Options:
       --hydration-dir='.kpt-pipeline': The directory to where the (kpt) hydration takes place. Default to a hidden directory .kpt-pipeline.
       --insecure-registry=[]: Target registries for built images which are not secure
       --iterative-status-check=true: Run `status-check` iteratively after each deploy step, instead of all-together at the end of all deploys (default).
+      --keep-running-on-failure=false: If true, the session will be suspended instead of ending if any errors occur, the user can fix the errors during the session suspension, the session can be restored and continued by pressing any key. 
       --kube-context='': Deploy to this Kubernetes context
       --kubeconfig='': Path to the kubeconfig file to use for CLI requests.
   -l, --label=[]: Add custom labels to deployed objects. Set multiple times for multiple labels
@@ -500,6 +501,7 @@ Env vars:
 * `SKAFFOLD_HYDRATION_DIR` (same as `--hydration-dir`)
 * `SKAFFOLD_INSECURE_REGISTRY` (same as `--insecure-registry`)
 * `SKAFFOLD_ITERATIVE_STATUS_CHECK` (same as `--iterative-status-check`)
+* `SKAFFOLD_KEEP_RUNNING_ON_FAILURE` (same as `--keep-running-on-failure`)
 * `SKAFFOLD_KUBE_CONTEXT` (same as `--kube-context`)
 * `SKAFFOLD_KUBECONFIG` (same as `--kubeconfig`)
 * `SKAFFOLD_LABEL` (same as `--label`)
@@ -740,6 +742,7 @@ Options:
       --hydration-dir='.kpt-pipeline': The directory to where the (kpt) hydration takes place. Default to a hidden directory .kpt-pipeline.
       --insecure-registry=[]: Target registries for built images which are not secure
       --iterative-status-check=true: Run `status-check` iteratively after each deploy step, instead of all-together at the end of all deploys (default).
+      --keep-running-on-failure=false: If true, the session will be suspended instead of ending if any errors occur, the user can fix the errors during the session suspension, the session can be restored and continued by pressing any key. 
       --kube-context='': Deploy to this Kubernetes context
       --kubeconfig='': Path to the kubeconfig file to use for CLI requests.
   -l, --label=[]: Add custom labels to deployed objects. Set multiple times for multiple labels
@@ -805,6 +808,7 @@ Env vars:
 * `SKAFFOLD_HYDRATION_DIR` (same as `--hydration-dir`)
 * `SKAFFOLD_INSECURE_REGISTRY` (same as `--insecure-registry`)
 * `SKAFFOLD_ITERATIVE_STATUS_CHECK` (same as `--iterative-status-check`)
+* `SKAFFOLD_KEEP_RUNNING_ON_FAILURE` (same as `--keep-running-on-failure`)
 * `SKAFFOLD_KUBE_CONTEXT` (same as `--kube-context`)
 * `SKAFFOLD_KUBECONFIG` (same as `--kubeconfig`)
 * `SKAFFOLD_LABEL` (same as `--label`)
diff --git a/docs-v2/content/en/schemas/v4beta2.json b/docs-v2/content/en/schemas/v4beta2.json
index 6ef426471..037357e21 100755
--- a/docs-v2/content/en/schemas/v4beta2.json
+++ b/docs-v2/content/en/schemas/v4beta2.json
@@ -2523,6 +2523,15 @@
           "x-intellij-html-description": "building outside of a container.",
           "default": "false"
         },
+        "ignorePaths": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array",
+          "description": "a list of ignored paths when making an image snapshot.",
+          "x-intellij-html-description": "a list of ignored paths when making an image snapshot.",
+          "default": "[]"
+        },
         "image": {
           "type": "string",
           "description": "Docker image used by the Kaniko pod. Defaults to the latest released version of `gcr.io/kaniko-project/executor`.",
@@ -2729,7 +2738,8 @@
         "label",
         "buildArgs",
         "volumeMounts",
-        "contextSubPath"
+        "contextSubPath",
+        "ignorePaths"
       ],
       "additionalProperties": false,
       "type": "object",
diff --git a/pkg/skaffold/build/kaniko/args.go b/pkg/skaffold/build/kaniko/args.go
index 8653e05a4..05ded7ebe 100644
--- a/pkg/skaffold/build/kaniko/args.go
+++ b/pkg/skaffold/build/kaniko/args.go
@@ -174,6 +174,12 @@ func Args(artifact *latest.KanikoArtifact, tag, context string) ([]string, error
 	}
 	args = append(args, sRegArgs...)
 
+	var ignPathArgs []string
+	for _, r := range artifact.IgnorePaths {
+		ignPathArgs = append(ignPathArgs, IgnorePathFlag, r)
+	}
+	args = append(args, ignPathArgs...)
+
 	registryCertificate, err := util.MapToFlag(artifact.RegistryCertificate, RegistryCertificateFlag)
 	if err != nil {
 		return args, err
diff --git a/pkg/skaffold/build/kaniko/types.go b/pkg/skaffold/build/kaniko/types.go
index 63b4996a2..c47251881 100644
--- a/pkg/skaffold/build/kaniko/types.go
+++ b/pkg/skaffold/build/kaniko/types.go
@@ -107,4 +107,6 @@ const (
 	DefaultDockerConfigPath = "/kaniko/.docker"
 	// DefaultSecretMountPath for kaniko pod
 	DefaultSecretMountPath = "/secret"
+	// IgnorePath additional flag
+	IgnorePathFlag = "--ignore-path"
 )
diff --git a/pkg/skaffold/config/options.go b/pkg/skaffold/config/options.go
index 78025402c..85d3b84bf 100644
--- a/pkg/skaffold/config/options.go
+++ b/pkg/skaffold/config/options.go
@@ -48,6 +48,7 @@ type SkaffoldOptions struct {
 	ForceLoadImages             bool
 	IterativeStatusCheck        bool
 	FastFailStatusCheck         bool
+	KeepRunningOnFailure        bool
 	TolerateFailuresStatusCheck bool
 	Notification                bool
 	NoPrune                     bool
diff --git a/pkg/skaffold/deploy/kubectl/cli.go b/pkg/skaffold/deploy/kubectl/cli.go
index de93b7b58..1e6e1e407 100644
--- a/pkg/skaffold/deploy/kubectl/cli.go
+++ b/pkg/skaffold/deploy/kubectl/cli.go
@@ -95,7 +95,6 @@ func (c *CLI) Apply(ctx context.Context, out io.Writer, manifests manifest.Manif
 	// TODO(dgageot): should we delete a manifest that was deployed and is not anymore?
 	updated := c.previousApply.Diff(manifests)
 	log.Entry(ctx).Debugf("%d manifests to deploy. %d are updated or new", len(manifests), len(updated))
-	c.previousApply = manifests
 	if len(updated) == 0 {
 		return nil
 	}
@@ -113,6 +112,7 @@ func (c *CLI) Apply(ctx context.Context, out io.Writer, manifests manifest.Manif
 		endTrace(instrumentation.TraceEndError(err))
 		return userErr(fmt.Errorf("kubectl apply: %w", err))
 	}
+	c.previousApply = manifests
 
 	return nil
 }
diff --git a/pkg/skaffold/kubernetes/status/resource/deployment.go b/pkg/skaffold/kubernetes/status/resource/deployment.go
index 4fe299de4..a4e66fa70 100644
--- a/pkg/skaffold/kubernetes/status/resource/deployment.go
+++ b/pkg/skaffold/kubernetes/status/resource/deployment.go
@@ -51,8 +51,9 @@ type Type string
 var (
 	statefulsetRolloutSuccess = regexp.MustCompile("(roll out|rolling update) complete")
 
-	msgKubectlKilled     = "kubectl rollout status command interrupted\n"
-	MsgKubectlConnection = "kubectl connection error\n"
+	msgKubectlKilled            = "kubectl rollout status command interrupted\n"
+	MsgKubectlConnection        = "kubectl connection error\n"
+	msgStrategyTypeNotSupported = "rollout status is only available for RollingUpdate strategy type"
 
 	nonRetryContainerErrors = map[proto.StatusCode]struct{}{
 		proto.StatusCode_STATUSCHECK_IMAGE_PULL_ERR:       {},
@@ -373,6 +374,13 @@ func parseKubectlRolloutError(details string, deadline time.Duration, err error)
 			ErrCode: proto.StatusCode_STATUSCHECK_KUBECTL_PID_KILLED,
 			Message: fmt.Sprintf("received Ctrl-C or deployments could not stabilize within %v: %s", deadline, msgKubectlKilled),
 		}
+	// statefulset rollouts that use OnDelete strategy type don't support monitoring rollout, treat it as
+	// if the deployment just completed successfully
+	case strings.Contains(err.Error(), msgStrategyTypeNotSupported):
+		return &proto.ActionableErr{
+			ErrCode: proto.StatusCode_STATUSCHECK_SUCCESS,
+			Message: details,
+		}
 	default:
 		return &proto.ActionableErr{
 			ErrCode: proto.StatusCode_STATUSCHECK_UNKNOWN,
diff --git a/pkg/skaffold/runner/dev.go b/pkg/skaffold/runner/dev.go
index ad786a48c..59048d7b1 100644
--- a/pkg/skaffold/runner/dev.go
+++ b/pkg/skaffold/runner/dev.go
@@ -33,6 +33,7 @@ import (
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/output/log"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/schema/latest"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/sync"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/util/term"
 	timeutil "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/util/time"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
 )
@@ -224,6 +225,102 @@ func (r *SkaffoldRunner) Dev(ctx context.Context, out io.Writer, artifacts []*la
 		"devIteration": strconv.Itoa(r.devIteration),
 	})
 
+	// First build
+	var err error
+	bRes, err := r.Build(ctx, out, artifacts)
+	for ; err != nil && r.runCtx.Opts.KeepRunningOnFailure; bRes, err = r.Build(ctx, out, artifacts) {
+		log.Entry(ctx).Warnf("Failed to build artifacts: %v, please fix the error and press any key to continue.", err)
+		errT := term.WaitForKeyPress()
+		if errT != nil {
+			return errT
+		}
+	}
+
+	if err != nil {
+		event.DevLoopFailedInPhase(r.devIteration, constants.Build, err)
+		eventV2.TaskFailed(constants.DevLoop, err)
+		endTrace()
+		return fmt.Errorf("exiting dev mode because first build failed: %w", err)
+	}
+	// First test
+	if r.runCtx.IsTestPhaseActive() {
+		err = r.Test(ctx, out, bRes)
+		for ; err != nil && r.runCtx.Opts.KeepRunningOnFailure; err = r.Test(ctx, out, bRes) {
+			log.Entry(ctx).Warnf("Failed to run tests :%v, please fix the error and press any key to continue.", err)
+			errT := term.WaitForKeyPress()
+			if errT != nil {
+				return errT
+			}
+		}
+		if err != nil {
+			event.DevLoopFailedInPhase(r.devIteration, constants.Build, err)
+			eventV2.TaskFailed(constants.DevLoop, err)
+			endTrace()
+			return fmt.Errorf("exiting dev mode because test failed after first build: %w", err)
+		}
+	}
+
+	defer r.deployer.GetLogger().Stop()
+	defer r.deployer.GetDebugger().Stop()
+
+	// Logs should be retrieved up to just before the deploy
+	r.deployer.GetLogger().SetSince(time.Now())
+
+	// First render
+	manifests, err := r.Render(ctx, out, r.Builds, false)
+	for ; err != nil && r.runCtx.Opts.KeepRunningOnFailure; manifests, err = r.Render(ctx, out, r.Builds, false) {
+		log.Entry(ctx).Warnf("Failed to render :%v, please fix the error and press any key to continue.", err)
+		errT := term.WaitForKeyPress()
+		if errT != nil {
+			return errT
+		}
+	}
+	if err != nil {
+		event.DevLoopFailedInPhase(r.devIteration, constants.Render, err)
+		eventV2.TaskFailed(constants.DevLoop, err)
+		endTrace()
+		return fmt.Errorf("exiting dev mode because first render failed: %w", err)
+	}
+
+	// First deploy
+	err = r.Deploy(ctx, out, r.Builds, manifests)
+	for ; err != nil && r.runCtx.Opts.KeepRunningOnFailure; err = r.Deploy(ctx, out, r.Builds, manifests) {
+		log.Entry(ctx).Warnf("Failed to deploy :%v, please fix the error and press any key to continue.", err)
+		errT := term.WaitForKeyPress()
+		if errT != nil {
+			return errT
+		}
+		// The previous Render Stage could succeed even for kubernetes resource with unknown fields, this will lead to failure in Deploy Stage
+		// users need to fix the problems in their manifests, and skaffold needs to re-render them before re-running Deploy Stage in this case.
+		for manifests, err = r.Render(ctx, out, r.Builds, false); err != nil; manifests, err = r.Render(ctx, out, r.Builds, false) {
+			log.Entry(ctx).Warnf("Failed to Render, please fix the error and press any key to continue. %v", err)
+			errT := term.WaitForKeyPress()
+			if errT != nil {
+				return errT
+			}
+		}
+	}
+	if err != nil {
+		event.DevLoopFailedInPhase(r.devIteration, constants.Deploy, err)
+		eventV2.TaskFailed(constants.DevLoop, err)
+		endTrace()
+		return fmt.Errorf("exiting dev mode because first deploy failed: %w", err)
+	}
+	r.deployManifests = manifests
+
+	defer r.deployer.GetAccessor().Stop()
+
+	if err := r.deployer.GetAccessor().Start(ctx, out); err != nil {
+		log.Entry(ctx).Warn("Error starting resource accessor:", err)
+	}
+	if err := r.deployer.GetDebugger().Start(ctx); err != nil {
+		log.Entry(ctx).Warn("Error starting debug container notification:", err)
+	}
+	// Start printing the logs after deploy is finished
+	if err := r.deployer.GetLogger().Start(ctx, out); err != nil {
+		return fmt.Errorf("starting logger: %w", err)
+	}
+
 	g := getTransposeGraph(artifacts)
 	// Watch artifacts
 	start := time.Now()
@@ -327,61 +424,6 @@ func (r *SkaffoldRunner) Dev(ctx context.Context, out io.Writer, artifacts []*la
 		return fmt.Errorf("exiting dev mode because initializing sync state failed: %w", err)
 	}
 
-	// First build
-	bRes, err := r.Build(ctx, out, artifacts)
-	if err != nil {
-		event.DevLoopFailedInPhase(r.devIteration, constants.Build, err)
-		eventV2.TaskFailed(constants.DevLoop, err)
-		endTrace()
-		return fmt.Errorf("exiting dev mode because first build failed: %w", err)
-	}
-	// First test
-	if r.runCtx.IsTestPhaseActive() {
-		if err = r.Test(ctx, out, bRes); err != nil {
-			event.DevLoopFailedInPhase(r.devIteration, constants.Build, err)
-			eventV2.TaskFailed(constants.DevLoop, err)
-			endTrace()
-			return fmt.Errorf("exiting dev mode because test failed after first build: %w", err)
-		}
-	}
-
-	defer r.deployer.GetLogger().Stop()
-	defer r.deployer.GetDebugger().Stop()
-
-	// Logs should be retrieved up to just before the deploy
-	r.deployer.GetLogger().SetSince(time.Now())
-
-	// First render
-	manifests, err := r.Render(ctx, out, r.Builds, false)
-	r.deployManifests = manifests
-	if err != nil {
-		event.DevLoopFailedInPhase(r.devIteration, constants.Render, err)
-		eventV2.TaskFailed(constants.DevLoop, err)
-		endTrace()
-		return fmt.Errorf("exiting dev mode because first render failed: %w", err)
-	}
-
-	// First deploy
-	if err := r.Deploy(ctx, out, r.Builds, manifests); err != nil {
-		event.DevLoopFailedInPhase(r.devIteration, constants.Deploy, err)
-		eventV2.TaskFailed(constants.DevLoop, err)
-		endTrace()
-		return fmt.Errorf("exiting dev mode because first deploy failed: %w", err)
-	}
-
-	defer r.deployer.GetAccessor().Stop()
-
-	if err := r.deployer.GetAccessor().Start(ctx, out); err != nil {
-		log.Entry(ctx).Warn("Error starting resource accessor:", err)
-	}
-	if err := r.deployer.GetDebugger().Start(ctx); err != nil {
-		log.Entry(ctx).Warn("Error starting debug container notification:", err)
-	}
-	// Start printing the logs after deploy is finished
-	if err := r.deployer.GetLogger().Start(ctx, out); err != nil {
-		return fmt.Errorf("starting logger: %w", err)
-	}
-
 	output.Yellow.Fprintln(out, "Press Ctrl+C to exit")
 
 	event.DevLoopComplete(r.devIteration)
diff --git a/pkg/skaffold/schema/v2beta29/upgrade.go b/pkg/skaffold/schema/v2beta29/upgrade.go
index 1bda6f15b..506374cc9 100644
--- a/pkg/skaffold/schema/v2beta29/upgrade.go
+++ b/pkg/skaffold/schema/v2beta29/upgrade.go
@@ -20,9 +20,12 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"regexp"
+	"strconv"
 	"strings"
 
 	"github.com/pkg/errors"
+	"gopkg.in/yaml.v2"
 
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/output/log"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/schema/util"
@@ -30,6 +33,8 @@ import (
 	pkgutil "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/util"
 )
 
+var artifactOverridesRegexp = regexp.MustCompile("/deploy/helm/releases/[0-9]+/artifactOverrides/image")
+
 var migrations = map[string]string{
 	"/deploy/kubectl":             "/manifests/rawYaml",
 	"/deploy/kustomize/paths":     "/manifests/kustomize/paths",
@@ -45,28 +50,57 @@ func (c *SkaffoldConfig) Upgrade() (util.VersionedConfig, error) {
 	pkgutil.CloneThroughJSON(c, &newConfig)
 	newConfig.APIVersion = next.Version
 
-	err := util.UpgradePipelines(c, &newConfig, upgradeOnePipeline)
+	var nProfiles []next.Profile
+	// seed with existing Profiles
+	if c.Profiles != nil {
+		pkgutil.CloneThroughJSON(newConfig.Profiles, &nProfiles)
+	}
+
+	oldPatchesMap := map[string][]JSONPatch{}
+	newPatchesMap := map[string][]next.JSONPatch{}
+	for i, p := range c.Profiles {
+		oldPatchesMap[p.Name], newPatchesMap[p.Name] = p.Patches, nProfiles[i].Patches
+	}
+	onePipelineUpgrader := NewOnePipelineUpgrader(oldPatchesMap, newPatchesMap)
+
+	err := util.UpgradePipelines(c, &newConfig, onePipelineUpgrader.upgradeOnePipeline)
 	if err != nil {
 		return &newConfig, err
 	}
-
 	var newProfiles []next.Profile
 	// seed with existing Profiles
 	if c.Profiles != nil {
 		pkgutil.CloneThroughJSON(newConfig.Profiles, &newProfiles)
 	}
 
+	for i := range newProfiles {
+		if _, ok := newPatchesMap[newProfiles[i].Name]; ok {
+			newProfiles[i].Patches = newPatchesMap[newProfiles[i].Name]
+		}
+	}
+
 	// Update profiles patches
 	for i, p := range c.Profiles {
 		upgradePatches(p.Patches, newProfiles[i].Patches)
 	}
 
 	newConfig.Profiles = newProfiles
-
 	return &newConfig, nil
 }
 
-func upgradeOnePipeline(oldPipeline, newPipeline interface{}) error {
+type OnePipelineUpgrader struct {
+	oldPatchesMap map[string][]JSONPatch
+	newPatchesMap map[string][]next.JSONPatch
+}
+
+func NewOnePipelineUpgrader(oldPatchesMap map[string][]JSONPatch, newPatchesMap map[string][]next.JSONPatch) OnePipelineUpgrader {
+	return OnePipelineUpgrader{
+		oldPatchesMap: oldPatchesMap,
+		newPatchesMap: newPatchesMap,
+	}
+}
+
+func (opu *OnePipelineUpgrader) upgradeOnePipeline(oldPipeline, newPipeline interface{}) error {
 	oldPL := oldPipeline.(*Pipeline)
 	newPL := newPipeline.(*next.Pipeline)
 
@@ -151,12 +185,112 @@ func upgradeOnePipeline(oldPipeline, newPipeline interface{}) error {
 		newPL.Deploy.LegacyHelmDeploy = &next.LegacyHelmDeploy{}
 		newPL.Deploy.LegacyHelmDeploy.LifecycleHooks = newHelm.LifecycleHooks
 	}
+
+	err := upgradeArtifactOverridesPatches(opu.oldPatchesMap, opu.newPatchesMap, oldPL)
+	if err != nil {
+		return fmt.Errorf("error converting helm artifactOverrides during version upgrade: %w", err)
+	}
+
+	return nil
+}
+
+func upgradeArtifactOverridesPatches(oldsMap map[string][]JSONPatch, newsMap map[string][]next.JSONPatch, oldPL *Pipeline) error {
+	for k := range oldsMap {
+		for i, old := range oldsMap[k] {
+			if artifactOverridesRegexp.Match([]byte(old.Path)) {
+				idx, err := strconv.Atoi(strings.Split(old.Path, "/")[4])
+				if err != nil {
+					return err
+				}
+				newPathModified := strings.ReplaceAll(strings.ReplaceAll(old.Path, "/deploy/helm", "/manifests/helm"),
+					"artifactOverrides", "setValueTemplates")
+				if oldPL.Deploy.HelmDeploy != nil {
+					validV := pkgutil.SanitizeHelmTemplateValue(old.Value.Node.Value().(string))
+					n := &util.YamlpatchNode{}
+
+					if oldPL.Deploy.HelmDeploy.Releases[idx].ImageStrategy.HelmConventionConfig != nil {
+						err = yaml.Unmarshal([]byte(fmt.Sprintf("\"{{.IMAGE_TAG_%s}}@{{.IMAGE_DIGEST_%s}}\"", validV, validV)), n)
+						if err != nil {
+							return err
+						}
+
+						newsMap[k][i] = next.JSONPatch{
+							Op:    newsMap[k][i].Op,
+							Path:  newPathModified + ".tag",
+							From:  newsMap[k][i].From,
+							Value: n,
+						}
+
+						if oldPL.Deploy.HelmDeploy.Releases[idx].ImageStrategy.HelmConventionConfig.ExplicitRegistry {
+							// is 'helm' imageStrategy + explicitRegistry
+							n := &util.YamlpatchNode{}
+							err = yaml.Unmarshal([]byte(fmt.Sprintf("\"{{.IMAGE_DOMAIN_%s}}\"", validV)), n)
+							if err != nil {
+								return err
+							}
+
+							newsMap[k] = append(newsMap[k], next.JSONPatch{
+								Op:    newsMap[k][i].Op,
+								Path:  newPathModified + ".registry",
+								From:  newsMap[k][i].From,
+								Value: n,
+							})
+
+							n = &util.YamlpatchNode{}
+							err := yaml.Unmarshal([]byte(fmt.Sprintf("\"{{.IMAGE_REPO_NO_DOMAIN_%s}}\"", validV)), n)
+							if err != nil {
+								return err
+							}
+
+							newsMap[k] = append(newsMap[k], next.JSONPatch{
+								Op:    newsMap[k][i].Op,
+								Path:  newPathModified + ".repository",
+								From:  newsMap[k][i].From,
+								Value: n,
+							})
+						} else {
+							// is 'helm' imageStrategy
+							n = &util.YamlpatchNode{}
+							err := yaml.Unmarshal([]byte(fmt.Sprintf("\"{{.IMAGE_REPO_%s}}\"", validV)), n)
+							if err != nil {
+								return err
+							}
+
+							newsMap[k] = append(newsMap[k], next.JSONPatch{
+
+								Op:    newsMap[k][i].Op,
+								Path:  newPathModified + ".repository",
+								From:  newsMap[k][i].From,
+								Value: n,
+							})
+						}
+					} else {
+						// is 'fqn' imageStrategy
+						// replace commonly used image name chars that are illegal helm template chars "/" & "-" with "_"
+						newValue := fmt.Sprintf("\"{{.IMAGE_FULLY_QUALIFIED_%s}}\"", validV)
+
+						n := &util.YamlpatchNode{}
+						err := yaml.Unmarshal([]byte(newValue), n)
+						if err != nil {
+							return err
+						}
+						newsMap[k][i].Value = n
+						newsMap[k][i].Path = newPathModified
+					}
+				}
+			}
+		}
+	}
 	return nil
 }
 
 func upgradePatches(olds []JSONPatch, news []next.JSONPatch) {
 	for i, old := range olds {
 		for str, repStr := range migrations {
+			if artifactOverridesRegexp.Match([]byte(old.Path)) {
+				// this is handled by upgradeArtifactOverridesPatches
+				continue
+			}
 			if strings.Contains(old.Path, str) {
 				news[i].Path = strings.ReplaceAll(old.Path, str, repStr)
 			}
diff --git a/pkg/skaffold/util/term/term.go b/pkg/skaffold/util/term/term.go
index 22d549ca7..b9f5a3653 100644
--- a/pkg/skaffold/util/term/term.go
+++ b/pkg/skaffold/util/term/term.go
@@ -17,9 +17,11 @@ limitations under the License.
 package term
 
 import (
+	"bufio"
 	"context"
 	"fmt"
 	"io"
+	"os"
 	"os/exec"
 	"runtime"
 	"strconv"
@@ -63,3 +65,23 @@ func SupportsColor(ctx context.Context) (bool, error) {
 
 	return numColors > 0, nil
 }
+
+func WaitForKeyPress() error {
+	// use rawMode so that we can read without the user to hit enter key.
+	previousState, err := term.MakeRaw(int(os.Stdin.Fd()))
+	if err != nil {
+		return err
+	}
+	defer term.Restore(int(os.Stdin.Fd()), previousState)
+
+	reader := bufio.NewReader(os.Stdin)
+	r, _, err := reader.ReadRune()
+	if err != nil {
+		return err
+	}
+	// control + c
+	if r == 3 {
+		return context.Canceled
+	}
+	return nil
+}


Test patch:
diff --git a/integration/control_api_test.go b/integration/control_api_test.go
index c0a8ef127..8299a406e 100644
--- a/integration/control_api_test.go
+++ b/integration/control_api_test.go
@@ -25,6 +25,7 @@ import (
 	"time"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
 )
 
@@ -47,6 +48,11 @@ func TestControlAPIManualTriggers(t *testing.T) {
 
 	dep := client.GetDeployment(testDev)
 
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
+
 	// Make a change to foo
 	Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
 
diff --git a/integration/custom_test.go b/integration/custom_test.go
index 882cceabd..3ee840824 100644
--- a/integration/custom_test.go
+++ b/integration/custom_test.go
@@ -24,6 +24,8 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
+	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
 )
 
 func TestCustomTest(t *testing.T) {
@@ -44,7 +46,16 @@ func TestCustomTest(t *testing.T) {
 
 	ns, client := SetupNamespace(t)
 
-	skaffold.Dev().InDir(testDir).WithConfig(config).InNs(ns.Name).RunBackground(t)
+	rpcAddr := randomPort()
+	skaffold.Dev("--rpc-port", rpcAddr).InDir(testDir).WithConfig(config).InNs(ns.Name).RunBackground(t)
+
+	_, entries := apiEvents(t, rpcAddr)
+
+	// Wait for the first devloop to register target files to the monitor before running command to change target files
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
 
 	client.WaitForPodsReady("custom-test-example")
 	os.WriteFile(depFile, []byte("foo"), 0644)
diff --git a/integration/dev_dependencies_test.go b/integration/dev_dependencies_test.go
index 4e6c6f5fa..085b37c3f 100644
--- a/integration/dev_dependencies_test.go
+++ b/integration/dev_dependencies_test.go
@@ -23,6 +23,9 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
+	V2proto "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 )
 
 func TestDev_WithDependencies(t *testing.T) {
@@ -30,9 +33,17 @@ func TestDev_WithDependencies(t *testing.T) {
 		MarkIntegrationTest(t, CanRunWithoutGcp)
 		ns, client := SetupNamespace(t)
 
-		skaffold.Dev().InDir("testdata/build-dependencies").InNs(ns.Name).RunBackground(t)
+		rpcAddr := randomPort()
+		skaffold.Dev("--rpc-port", rpcAddr).InDir("testdata/build-dependencies").InNs(ns.Name).RunBackground(t)
 		client.waitForDeploymentsToStabilizeWithTimeout(3*time.Minute, "app1", "app2", "app3", "app4")
 
+		_, entries := v2apiEvents(t, rpcAddr)
+
+		failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+			taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+			return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+		}))
+
 		dep1 := client.GetDeployment("app1")
 		dep2 := client.GetDeployment("app2")
 		dep3 := client.GetDeployment("app3")
diff --git a/integration/dev_test.go b/integration/dev_test.go
index a0e8e3671..9e4ffd0d8 100644
--- a/integration/dev_test.go
+++ b/integration/dev_test.go
@@ -35,7 +35,9 @@ import (
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/config"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
+	V2proto "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/testutil"
 )
 
@@ -64,10 +66,19 @@ func TestDevNotification(t *testing.T) {
 
 			ns, client := SetupNamespace(t)
 
-			skaffold.Dev("--trigger", test.trigger).InDir("testdata/dev").InNs(ns.Name).RunBackground(t)
+			rpcAddr := randomPort()
+			skaffold.Dev("--rpc-port", rpcAddr, "--trigger", test.trigger).InDir("testdata/dev").InNs(ns.Name).RunBackground(t)
 
 			dep := client.GetDeployment(testDev)
 
+			_, entries := v2apiEvents(t, rpcAddr)
+
+			// Wait for the first devloop to register target files to the monitor before running command to change target files
+			failNowIfError(t, waitForV2Event(100*time.Second, entries, func(e *V2proto.Event) bool {
+				taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+				return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+			}))
+
 			// Make a change to foo so that dev is forced to delete the Deployment and redeploy
 			Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
 
@@ -136,9 +147,8 @@ func TestDevGracefulCancel(t *testing.T) {
 	}
 }
 
-func TestDevAPITriggers(t *testing.T) {
+func TestDevAPIBuildTrigger(t *testing.T) {
 	MarkIntegrationTest(t, CanRunWithoutGcp)
-	t.Skip("failing since go 1.19.1")
 
 	Run(t, "testdata/dev", "sh", "-c", "echo foo > foo")
 	defer Run(t, "testdata/dev", "rm", "foo")
@@ -146,13 +156,18 @@ func TestDevAPITriggers(t *testing.T) {
 	// Run skaffold build first to fail quickly on a build failure
 	skaffold.Build().InDir("testdata/dev").RunOrFail(t)
 
-	ns, client := SetupNamespace(t)
+	ns, _ := SetupNamespace(t)
 
 	rpcAddr := randomPort()
 	skaffold.Dev("--auto-build=false", "--auto-sync=false", "--auto-deploy=false", "--rpc-port", rpcAddr, "--cache-artifacts=false").InDir("testdata/dev").InNs(ns.Name).RunBackground(t)
 
 	rpcClient, entries := apiEvents(t, rpcAddr)
-	dep := client.GetDeployment(testDev)
+
+	// Wait for the first devloop to register target files to the monitor before running command to change target files
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
 
 	// Make a change to foo
 	Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
@@ -169,6 +184,33 @@ func TestDevAPITriggers(t *testing.T) {
 		return e.GetEvent().GetBuildEvent().GetArtifact() == testDev
 	})
 	failNowIfError(t, err)
+}
+
+func TestDevApiDeployTrigger(t *testing.T) {
+	MarkIntegrationTest(t, CanRunWithoutGcp)
+
+	Run(t, "testdata/dev", "sh", "-c", "echo foo > foo")
+	defer Run(t, "testdata/dev", "rm", "foo")
+
+	// Run skaffold build first to fail quickly on a build failure
+	skaffold.Build().InDir("testdata/dev").RunOrFail(t)
+
+	ns, client := SetupNamespace(t)
+
+	rpcAddr := randomPort()
+	skaffold.Dev("--auto-deploy=false", "--rpc-port", rpcAddr, "--cache-artifacts=false").InDir("testdata/dev").InNs(ns.Name).RunBackground(t)
+
+	rpcClient, entries := apiEvents(t, rpcAddr)
+	dep := client.GetDeployment(testDev)
+
+	// Wait for the first devloop to register target files to the monitor before running command to change target files
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
+
+	// Make a change to foo
+	Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
 
 	// Issue a deploy trigger
 	rpcClient.Execute(context.Background(), &proto.UserIntentRequest{
@@ -197,6 +239,12 @@ func TestDevAPIAutoTriggers(t *testing.T) {
 	rpcClient, entries := apiEvents(t, rpcAddr)
 	dep := client.GetDeployment(testDev)
 
+	// Wait for the first devloop to register target files to the monitor before running command to change target files
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
+
 	// Make a change to foo
 	Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
 
diff --git a/integration/fix_test.go b/integration/fix_test.go
index e968d151e..7e45cf109 100644
--- a/integration/fix_test.go
+++ b/integration/fix_test.go
@@ -45,6 +45,17 @@ func TestFixStdout(t *testing.T) {
 	skaffold.Run().WithConfig("-").InDir("testdata/fix").InNs(ns.Name).WithStdin(out).RunOrFail(t)
 }
 
+func TestFixV2Beta29ToV3Alpha1PatchProfiles(t *testing.T) {
+	MarkIntegrationTest(t, CanRunWithoutGcp)
+
+	out, err := skaffold.Fix().InDir("testdata/fix-v2beta29").RunWithCombinedOutput(t)
+	testutil.CheckError(t, false, err)
+	testutil.CheckContains(t, "/manifests/helm/releases/0/setValueTemplates/image.repository", string(out))
+	testutil.CheckContains(t, "{{.IMAGE_REPO_skaffold_helm_v2}}", string(out))
+	testutil.CheckContains(t, "/manifests/helm/releases/0/setValueTemplates/image.tag", string(out))
+	testutil.CheckContains(t, "{{.IMAGE_TAG_skaffold_helm_v2}}@{{.IMAGE_DIGEST_skaffold_helm_v2}}", string(out))
+}
+
 func TestFixOutputFile(t *testing.T) {
 	MarkIntegrationTest(t, CanRunWithoutGcp)
 
diff --git a/integration/modules_test.go b/integration/modules_test.go
index b5cf7eb36..da6806ed9 100644
--- a/integration/modules_test.go
+++ b/integration/modules_test.go
@@ -23,20 +23,30 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
+	V2proto "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 )
 
 func TestModules_BuildDependency(t *testing.T) {
 	t.Run("build dependency between artifacts of different modules", func(t *testing.T) {
 		MarkIntegrationTest(t, CanRunWithoutGcp)
 		ns, client := SetupNamespace(t)
-
-		skaffold.Dev().InDir("testdata/modules").InNs(ns.Name).RunBackground(t)
+		rpcAddr := randomPort()
+		skaffold.Dev("--rpc-port", rpcAddr).InDir("testdata/modules").InNs(ns.Name).RunBackground(t)
 		client.waitForDeploymentsToStabilizeWithTimeout(3*time.Minute, "app1", "app2", "app3")
 
 		dep1 := client.GetDeployment("app1")
 		dep2 := client.GetDeployment("app2")
 		dep3 := client.GetDeployment("app3")
 
+		_, entries := v2apiEvents(t, rpcAddr)
+
+		failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+			taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+			return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+		}))
+
 		// Make a change to app3/foo so that dev is forced to delete the Deployment and redeploy app1, app2 and app3,
 		// since app2 depends on app3 and app1 depends on app2
 		Run(t, "testdata/modules/app3", "sh", "-c", "echo bar > foo")
diff --git a/integration/rpc_test.go b/integration/rpc_test.go
index 951466f03..db41cd4fc 100644
--- a/integration/rpc_test.go
+++ b/integration/rpc_test.go
@@ -32,10 +32,12 @@ import (
 	"github.com/golang/protobuf/jsonpb"
 	"github.com/golang/protobuf/ptypes/empty"
 	"google.golang.org/grpc"
+	"google.golang.org/protobuf/types/known/emptypb"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
 	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
+	protoV2 "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/testutil"
 )
 
@@ -454,3 +456,87 @@ func setupRPCClient(t *testing.T, port string) proto.SkaffoldServiceClient {
 
 	return client
 }
+
+func setupV2RPCClient(t *testing.T, port string) protoV2.SkaffoldV2ServiceClient {
+	// start a grpc client
+	var (
+		conn   *grpc.ClientConn
+		err    error
+		client protoV2.SkaffoldV2ServiceClient
+	)
+
+	// connect to the skaffold grpc server
+	for i := 0; i < connectionRetries; i++ {
+		conn, err = grpc.Dial(fmt.Sprintf(":%s", port), grpc.WithInsecure(), grpc.WithBackoffMaxDelay(10*time.Second))
+		if err != nil {
+			t.Logf("unable to establish skaffold grpc connection: retrying...")
+			time.Sleep(waitTime)
+			continue
+		}
+
+		client = protoV2.NewSkaffoldV2ServiceClient(conn)
+		break
+	}
+
+	if client == nil {
+		t.Fatalf("error establishing skaffold grpc connection")
+	}
+
+	t.Cleanup(func() { conn.Close() })
+
+	return client
+}
+
+func readV2EventAPIStream(client protoV2.SkaffoldV2ServiceClient, t *testing.T, retries int) (protoV2.SkaffoldV2Service_EventsClient, error) {
+	t.Helper()
+	// read the event log stream from the skaffold grpc server
+	var stream protoV2.SkaffoldV2Service_EventsClient
+	var err error
+	var protoReq emptypb.Empty
+	for i := 0; i < retries; i++ {
+		stream, err = client.Events(context.Background(), &protoReq, grpc.WaitForReady(true))
+		if err == nil {
+			break
+		}
+		t.Logf("waiting for connection...")
+		time.Sleep(waitTime)
+	}
+	return stream, err
+}
+
+func v2apiEvents(t *testing.T, rpcAddr string) (protoV2.SkaffoldV2ServiceClient, chan *protoV2.Event) { // nolint
+	client := setupV2RPCClient(t, rpcAddr)
+
+	stream, err := readV2EventAPIStream(client, t, readRetries)
+	if stream == nil {
+		t.Fatalf("error retrieving event log: %v\n", err)
+	}
+
+	// read entries from the log
+	entries := make(chan *protoV2.Event)
+	go func() {
+		for {
+			entry, _ := stream.Recv()
+			if entry != nil {
+				entries <- entry
+			}
+		}
+	}()
+
+	return client, entries
+}
+
+func waitForV2Event(timeout time.Duration, entries chan *protoV2.Event, condition func(event2 *protoV2.Event) bool) error {
+	ctx, cancelTimeout := context.WithTimeout(context.Background(), timeout)
+	defer cancelTimeout()
+	for {
+		select {
+		case <-ctx.Done():
+			return fmt.Errorf("timed out waiting for condition on log entry")
+		case ev := <-entries:
+			if condition(ev) {
+				return nil
+			}
+		}
+	}
+}
diff --git a/integration/sync_test.go b/integration/sync_test.go
index f4d7b9b08..c620dbd04 100644
--- a/integration/sync_test.go
+++ b/integration/sync_test.go
@@ -30,7 +30,10 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
+	V2proto "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 )
 
 // TODO: remove nolint once we've reenabled integration tests
@@ -65,10 +68,19 @@ func TestDevSync(t *testing.T) {
 
 			ns, client := SetupNamespace(t)
 
-			skaffold.Dev("--trigger", test.trigger).InDir("testdata/file-sync").WithConfig(test.config).InNs(ns.Name).RunBackground(t)
+			rpcAddr := randomPort()
+
+			skaffold.Dev("--rpc-port", rpcAddr, "--trigger", test.trigger).InDir("testdata/file-sync").WithConfig(test.config).InNs(ns.Name).RunBackground(t)
 
 			client.WaitForPodsReady("test-file-sync")
 
+			_, entries := v2apiEvents(t, rpcAddr)
+
+			failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+				taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+				return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+			}))
+
 			os.WriteFile("testdata/file-sync/foo", []byte("foo"), 0644)
 			defer func() { os.Truncate("testdata/file-sync/foo", 0) }()
 
@@ -110,11 +122,20 @@ func TestDevSyncDefaultNamespace(t *testing.T) {
 
 			_, client := DefaultNamespace(t)
 
-			skaffold.Dev("--trigger", test.trigger).InDir("testdata/file-sync").WithConfig(test.config).RunBackground(t)
+			rpcAddr := randomPort()
+
+			skaffold.Dev("--rpc-port", rpcAddr, "--trigger", test.trigger).InDir("testdata/file-sync").WithConfig(test.config).RunBackground(t)
+
 			defer skaffold.Delete().InDir("testdata/file-sync").WithConfig(test.config).Run(t)
 
 			client.WaitForPodsReady(id)
 
+			_, entries := v2apiEvents(t, rpcAddr)
+			failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+				taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+				return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+			}))
+
 			os.WriteFile("testdata/file-sync/foo", []byte("foo"), 0644)
 			defer func() { os.Truncate("testdata/file-sync/foo", 0) }()
 
@@ -153,7 +174,8 @@ func TestDevAutoSync(t *testing.T) {
 
 			ns, client := SetupNamespace(t)
 
-			output := skaffold.Dev("--trigger", "notify").WithConfig(test.configFile).InDir(dir).InNs(ns.Name).RunLive(t)
+			rpcAddr := randomPort()
+			output := skaffold.Dev("--trigger", "notify", "--rpc-port", rpcAddr).WithConfig(test.configFile).InDir(dir).InNs(ns.Name).RunLive(t)
 
 			client.WaitForPodsReady("test-file-sync")
 
@@ -172,6 +194,13 @@ func TestDevAutoSync(t *testing.T) {
 				}
 			}
 
+			_, entries := v2apiEvents(t, rpcAddr)
+
+			failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+				taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+				return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+			}))
+
 			// direct file sync (this file is an existing file checked in for this testdata)
 			directFile := "direct-file"
 			directFilePath := dir + "src/main/jib/" + directFile
@@ -222,6 +251,10 @@ func TestDevSyncAPITrigger(t *testing.T) {
 
 	rpcClient, entries := apiEvents(t, rpcAddr)
 	client.WaitForPodsReady("test-file-sync")
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
 
 	os.WriteFile("testdata/file-sync/foo", []byte("foo"), 0644)
 	defer func() { os.Truncate("testdata/file-sync/foo", 0) }()
@@ -253,6 +286,11 @@ func TestDevAutoSyncAPITrigger(t *testing.T) {
 
 	client.WaitForPodsReady("test-file-sync")
 
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
+
 	os.WriteFile("testdata/file-sync/foo", []byte("foo"), 0644)
 	defer func() { os.Truncate("testdata/file-sync/foo", 0) }()
 
diff --git a/integration/testdata/fix-v2beta29/skaffold.yaml b/integration/testdata/fix-v2beta29/skaffold.yaml
new file mode 100644
index 000000000..4d54d946f
--- /dev/null
+++ b/integration/testdata/fix-v2beta29/skaffold.yaml
@@ -0,0 +1,24 @@
+apiVersion: skaffold/v2beta29
+kind: Config
+build:
+  artifacts:
+  - image: skaffold-helm
+deploy:
+ helm:
+   releases:
+     - name: skaffold-helm
+       chartPath: charts
+       namespace: repro-ns
+       recreatePods: false
+       useHelmSecrets: false
+       wait: false
+       artifactOverrides:
+         image: skaffold-helm
+       imageStrategy:
+         helm: {}
+profiles:
+ - name: skaffold-helm-v2
+   patches:
+      - op: replace
+        path: /deploy/helm/releases/0/artifactOverrides/image
+        value: skaffold-helm-v2
\ No newline at end of file
diff --git a/pkg/skaffold/build/gcb/kaniko_test.go b/pkg/skaffold/build/gcb/kaniko_test.go
index 57f30a3b4..d9c8d0e6d 100644
--- a/pkg/skaffold/build/gcb/kaniko_test.go
+++ b/pkg/skaffold/build/gcb/kaniko_test.go
@@ -397,6 +397,17 @@ func TestKanikoBuildSpec(t *testing.T) {
 				kaniko.LabelFlag, "label2",
 			},
 		},
+		{
+			description: "with IgnorePaths",
+			artifact: &latest.KanikoArtifact{
+				DockerfilePath: "Dockerfile",
+				IgnorePaths:    []string{"/var", "/tmp"},
+			},
+			expectedArgs: []string{
+				kaniko.IgnorePathFlag, "/var",
+				kaniko.IgnorePathFlag, "/tmp",
+			},
+		},
 	}
 	store := mockArtifactStore{
 		"img2": "img2:tag",
diff --git a/pkg/skaffold/build/kaniko/args_test.go b/pkg/skaffold/build/kaniko/args_test.go
index fd9874c09..f00c65de3 100644
--- a/pkg/skaffold/build/kaniko/args_test.go
+++ b/pkg/skaffold/build/kaniko/args_test.go
@@ -407,6 +407,18 @@ func TestArgs(t *testing.T) {
 			},
 			wantErr: false,
 		},
+		{
+			description: "with IgnorePaths",
+			artifact: &latest.KanikoArtifact{
+				DockerfilePath: "dir/Dockerfile",
+				IgnorePaths:    []string{"/tmp", "/proc"},
+			},
+			expectedArgs: []string{
+				IgnorePathFlag, "/tmp",
+				IgnorePathFlag, "/proc",
+			},
+			wantErr: false,
+		},
 	}
 
 	defaultExpectedArgs := []string{
diff --git a/pkg/skaffold/schema/latest/config.go b/pkg/skaffold/schema/latest/config.go
index 10266ccd6..b87e0d958 100644
--- a/pkg/skaffold/schema/latest/config.go
+++ b/pkg/skaffold/schema/latest/config.go
@@ -1387,6 +1387,9 @@ type KanikoArtifact struct {
 
 	// ContextSubPath is to specify a sub path within the context.
 	ContextSubPath string `yaml:"contextSubPath,omitempty" skaffold:"filepath"`
+
+	// IgnorePaths is a list of ignored paths when making an image snapshot.
+	IgnorePaths []string `yaml:"ignorePaths,omitempty"`
 }
 
 // DockerArtifact describes an artifact built from a Dockerfile,