Repository: GoogleContainerTools/skaffold
Pull request #8337 (2023-01-19 03:59:11)

docs: add status check documentation for new tolerateFailuresUntilDeadline config field

codecov[bot]: # [Codecov](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) Report
> Merging [#8337](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (f0d346f) into [main](https://codecov.io/gh/GoogleContainerTools/skaffold/commit/290280e8c6c19231e9ed5b5c027f2c01d22f75be?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (290280e) will **decrease** coverage by `4.41%`.
> The diff coverage is `54.85%`.

```diff
@@            Coverage Diff             @@
##             main    #8337      +/-   ##
==========================================
- Coverage   70.48%   66.08%   -4.41%     
==========================================
  Files         515      605      +90     
  Lines       23150    29654    +6504     
==========================================
+ Hits        16317    19596    +3279     
- Misses       5776     8590    +2814     
- Partials     1057     1468     +411     
```


| [Impacted Files](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | Coverage Δ | |
|---|---|---|
| [cmd/skaffold/app/cmd/completion.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29tcGxldGlvbi5nbw==) | `13.04% <0.00%> (-1.25%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/config/list.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL2xpc3QuZ28=) | `65.21% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/config/set.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL3NldC5nbw==) | `88.72% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/config/util.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY29uZmlnL3V0aWwuZ28=) | `54.28% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/credits.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY3JlZGl0cy5nbw==) | `100.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/credits/export.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY3JlZGl0cy9leHBvcnQuZ28=) | `0.00% <0.00%> (ø)` | |
| [cmd/skaffold/app/cmd/deploy.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZGVwbG95Lmdv) | `40.90% <0.00%> (-12.94%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/generate\_pipeline.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZ2VuZXJhdGVfcGlwZWxpbmUuZ28=) | `60.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/inspect\_modules.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvaW5zcGVjdF9tb2R1bGVzLmdv) | `65.00% <ø> (ø)` | |
| [cmd/skaffold/app/cmd/inspect\_profiles.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvaW5zcGVjdF9wcm9maWxlcy5nbw==) | `66.66% <ø> (ø)` | |
| ... and [416 more](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/8337?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools)


Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 02ffaf606..84ba5ca9b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,3 +1,48 @@
+# v2.0.5 Release - 1/20/2023
+**Linux amd64**
+`curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v2.0.5/skaffold-linux-amd64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
+
+**Linux arm64**
+`curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v2.0.5/skaffold-linux-arm64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
+
+**macOS amd64**
+`curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v2.0.5/skaffold-darwin-amd64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
+
+**macOS arm64**
+`curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v2.0.5/skaffold-darwin-arm64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
+
+**Windows**
+https://storage.googleapis.com/skaffold/releases/v2.0.5/skaffold-windows-amd64.exe
+
+**Docker image**
+`gcr.io/k8s-skaffold/skaffold:v2.0.5`
+
+Fixes:
+* fix: use release namespace in render when specified [#8259](https://github.com/GoogleContainerTools/skaffold/pull/8259)
+* fix: add proper artifactOverrides->setValueTemplates conversion when upgrading from v2beta29 [#8335](https://github.com/GoogleContainerTools/skaffold/pull/8335)
+
+# v1.39.5 Release - 1/20/2022
+**Linux amd64**
+`curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v1.39.5/skaffold-linux-amd64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
+
+**Linux arm64**
+`curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v1.39.5/skaffold-linux-arm64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
+
+**macOS amd64**
+`curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v1.39.5/skaffold-darwin-amd64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
+
+**macOS arm64**
+`curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v1.39.5/skaffold-darwin-arm64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
+
+**Windows**
+https://storage.googleapis.com/skaffold/releases/v1.39.5/skaffold-windows-amd64.exe
+
+**Docker image**
+`gcr.io/k8s-skaffold/skaffold:v1.39.5`
+
+Fixes:
+* fix: backport, divide stdout and stderr from helm to not create corrupted outputs [#8333](https://github.com/GoogleContainerTools/skaffold/pull/8333)
+
 # v2.0.4 Release - 12/21/2022
 **Linux amd64**
 `curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/v2.0.4/skaffold-linux-amd64 && chmod +x skaffold && sudo mv skaffold /usr/local/bin`
diff --git a/cmd/skaffold/app/cmd/flags.go b/cmd/skaffold/app/cmd/flags.go
index 0b3c526b3..99161eae3 100644
--- a/cmd/skaffold/app/cmd/flags.go
+++ b/cmd/skaffold/app/cmd/flags.go
@@ -738,6 +738,15 @@ The build result from a previous 'skaffold build --file-output' run can be used
 		FlagAddMethod: "StringVar",
 		DefinedOn:     []string{"dev", "run", "debug", "deploy", "apply", "delete"},
 	},
+	{
+		Name:          "keep-running-on-failure",
+		Shorthand:     "",
+		Usage:         "If true, the session will be suspended instead of ending if any errors occur, the user can fix the errors during the session suspension, the session can be restored and continued by pressing any key. ",
+		Value:         &opts.KeepRunningOnFailure,
+		DefValue:      false,
+		FlagAddMethod: "BoolVar",
+		DefinedOn:     []string{"dev", "debug"},
+	},
 }
 
 func methodNameByType(v reflect.Value) string {
diff --git a/docs-v2/content/en/docs/pipeline-stages/deployers/cloudrun.md b/docs-v2/content/en/docs/pipeline-stages/deployers/cloudrun.md
index 13d791a81..453f27a5d 100644
--- a/docs-v2/content/en/docs/pipeline-stages/deployers/cloudrun.md
+++ b/docs-v2/content/en/docs/pipeline-stages/deployers/cloudrun.md
@@ -6,7 +6,7 @@ featureId: deploy.cloudrun
 ---
 
 {{< alert title="Note" >}}
-This feature is currently experimental and subject to change. Not all Skaffold features e.g. log tailing, debugging are supported.
+This feature is currently experimental and subject to change. Not all Skaffold features are supported, for example `debug` is currently not supported in Cloud Run (but is on our roadmap).
 {{< /alert >}}
 
 Cloud Run is a managed compute platform on Google Cloud that allows you to run containers on Google's infrastructure.
@@ -19,6 +19,10 @@ See the [Cloud Run YAML reference](https://cloud.google.com/run/docs/reference/y
 
 This deployer will use the [application default credentials](https://cloud.google.com/docs/authentication/production#automatically) to deploy.  You can configure this to use your user credentials by running `gcloud auth application-default login`.
 
+# Features
+- As of Skaffold `v2.1.0`, Skaffold's Cloud Run Deployer now supports log streaming from the deployed Cloud Run Service(s)/Job(s).  NOTE: this requires installing the following `gcloud components`: `alpha`, `beta`, `cloud-run-proxy`, and `log-streaming`
+- Supports deploying Cloud Run `Service` and `Job` objects
+
 ## Configuring Cloud Run
 
 To deploy to Cloud Run, use the `cloudrun` type in the `deploy` section of `skaffold.yaml`.
@@ -27,6 +31,7 @@ The `cloudrun` type offers the following options:
 
 {{< schema root="CloudRunDeploy" >}}
 
+
 ### Example
 
 The following `deploy` section instructs Skaffold to deploy
@@ -39,3 +44,4 @@ Images listed to be deployed with the Cloud Run deployer must be present in Goog
 Registry or Google Container Registry. If you are using Skaffold to build the images, ensure `push` is 
 set to true.
 {{< /alert >}}
+
diff --git a/docs-v2/content/en/docs/pipeline-stages/status-check.md b/docs-v2/content/en/docs/pipeline-stages/status-check.md
index a81408b42..2753c193e 100644
--- a/docs-v2/content/en/docs/pipeline-stages/status-check.md
+++ b/docs-v2/content/en/docs/pipeline-stages/status-check.md
@@ -100,6 +100,16 @@ Waiting for deployments to stabilize
 FATA[0006] 1/1 deployment(s) failed
 ```
 
+
+### Configuring failure behavior for `status-check`
+You can also configure status checking's failure tolerance with the `tolerateFailuresUntilDeadline` config field in the `skaffold.yaml` as well as the flag `--tolerate-failures-until-deadline`.
+
+The `tolerateFailuresUntilDeadline` modifies the status check to no longer exit when a single deployment fails (desired for local dev) but to instead tolerate failures
+until the status check deadline is reached (either default 10 minute deadline or specified via `statusCheckDeadlineSeconds`). As such it should normally be used with the `statusCheckDeadlineSeconds` option so that the deadline is known/set by the user.  This is useful in CI/CD use cases where deployments may fail/flap for a time period while different services initialize but eventually are healthy and stable.  Using this command essentially makes it so that skaffold waits for all deployed resources to be successful or times out, not exiting on any single deployment failure (which might go away) as the status check does by default.
+
+For example, to configure deployments to stabilize within 5 minutes AND TO NOT FAIL UNTIL the time period is reached:
+{{% readfile file="samples/deployers/status-check-tolerateFailuresUntilDeadline.yaml" %}}
+
 ### Configuring `status-check` for multiple deployers or multiple modules
 
 If you define multiple deployers, say `kubectl`, `helm`, and `kustomize`, all in the same skaffold config, or compose a multi-config project by importing other configs as dependencies, then the `status-check` can be run in one of two ways:
diff --git a/docs-v2/content/en/docs/references/cli/_index.md b/docs-v2/content/en/docs/references/cli/_index.md
index 76fd8f2ef..183f0d960 100644
--- a/docs-v2/content/en/docs/references/cli/_index.md
+++ b/docs-v2/content/en/docs/references/cli/_index.md
@@ -435,6 +435,7 @@ Options:
       --hydration-dir='.kpt-pipeline': The directory to where the (kpt) hydration takes place. Default to a hidden directory .kpt-pipeline.
       --insecure-registry=[]: Target registries for built images which are not secure
       --iterative-status-check=true: Run `status-check` iteratively after each deploy step, instead of all-together at the end of all deploys (default).
+      --keep-running-on-failure=false: If true, the session will be suspended instead of ending if any errors occur, the user can fix the errors during the session suspension, the session can be restored and continued by pressing any key. 
       --kube-context='': Deploy to this Kubernetes context
       --kubeconfig='': Path to the kubeconfig file to use for CLI requests.
   -l, --label=[]: Add custom labels to deployed objects. Set multiple times for multiple labels
@@ -500,6 +501,7 @@ Env vars:
 * `SKAFFOLD_HYDRATION_DIR` (same as `--hydration-dir`)
 * `SKAFFOLD_INSECURE_REGISTRY` (same as `--insecure-registry`)
 * `SKAFFOLD_ITERATIVE_STATUS_CHECK` (same as `--iterative-status-check`)
+* `SKAFFOLD_KEEP_RUNNING_ON_FAILURE` (same as `--keep-running-on-failure`)
 * `SKAFFOLD_KUBE_CONTEXT` (same as `--kube-context`)
 * `SKAFFOLD_KUBECONFIG` (same as `--kubeconfig`)
 * `SKAFFOLD_LABEL` (same as `--label`)
@@ -740,6 +742,7 @@ Options:
       --hydration-dir='.kpt-pipeline': The directory to where the (kpt) hydration takes place. Default to a hidden directory .kpt-pipeline.
       --insecure-registry=[]: Target registries for built images which are not secure
       --iterative-status-check=true: Run `status-check` iteratively after each deploy step, instead of all-together at the end of all deploys (default).
+      --keep-running-on-failure=false: If true, the session will be suspended instead of ending if any errors occur, the user can fix the errors during the session suspension, the session can be restored and continued by pressing any key. 
       --kube-context='': Deploy to this Kubernetes context
       --kubeconfig='': Path to the kubeconfig file to use for CLI requests.
   -l, --label=[]: Add custom labels to deployed objects. Set multiple times for multiple labels
@@ -805,6 +808,7 @@ Env vars:
 * `SKAFFOLD_HYDRATION_DIR` (same as `--hydration-dir`)
 * `SKAFFOLD_INSECURE_REGISTRY` (same as `--insecure-registry`)
 * `SKAFFOLD_ITERATIVE_STATUS_CHECK` (same as `--iterative-status-check`)
+* `SKAFFOLD_KEEP_RUNNING_ON_FAILURE` (same as `--keep-running-on-failure`)
 * `SKAFFOLD_KUBE_CONTEXT` (same as `--kube-context`)
 * `SKAFFOLD_KUBECONFIG` (same as `--kubeconfig`)
 * `SKAFFOLD_LABEL` (same as `--label`)
diff --git a/docs-v2/content/en/samples/deployers/status-check-tolerateFailuresUntilDeadline.yaml b/docs-v2/content/en/samples/deployers/status-check-tolerateFailuresUntilDeadline.yaml
new file mode 100644
index 000000000..32ac0fef7
--- /dev/null
+++ b/docs-v2/content/en/samples/deployers/status-check-tolerateFailuresUntilDeadline.yaml
@@ -0,0 +1,4 @@
+deploy:
+  statusCheckDeadlineSeconds: 300
+  tolerateFailuresUntilDeadline: true
+  kubectl: {}
\ No newline at end of file
diff --git a/docs-v2/content/en/schemas/v4beta2.json b/docs-v2/content/en/schemas/v4beta2.json
index 6ef426471..037357e21 100755
--- a/docs-v2/content/en/schemas/v4beta2.json
+++ b/docs-v2/content/en/schemas/v4beta2.json
@@ -2523,6 +2523,15 @@
           "x-intellij-html-description": "building outside of a container.",
           "default": "false"
         },
+        "ignorePaths": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array",
+          "description": "a list of ignored paths when making an image snapshot.",
+          "x-intellij-html-description": "a list of ignored paths when making an image snapshot.",
+          "default": "[]"
+        },
         "image": {
           "type": "string",
           "description": "Docker image used by the Kaniko pod. Defaults to the latest released version of `gcr.io/kaniko-project/executor`.",
@@ -2729,7 +2738,8 @@
         "label",
         "buildArgs",
         "volumeMounts",
-        "contextSubPath"
+        "contextSubPath",
+        "ignorePaths"
       ],
       "additionalProperties": false,
       "type": "object",
diff --git a/pkg/skaffold/build/kaniko/args.go b/pkg/skaffold/build/kaniko/args.go
index 8653e05a4..05ded7ebe 100644
--- a/pkg/skaffold/build/kaniko/args.go
+++ b/pkg/skaffold/build/kaniko/args.go
@@ -174,6 +174,12 @@ func Args(artifact *latest.KanikoArtifact, tag, context string) ([]string, error
 	}
 	args = append(args, sRegArgs...)
 
+	var ignPathArgs []string
+	for _, r := range artifact.IgnorePaths {
+		ignPathArgs = append(ignPathArgs, IgnorePathFlag, r)
+	}
+	args = append(args, ignPathArgs...)
+
 	registryCertificate, err := util.MapToFlag(artifact.RegistryCertificate, RegistryCertificateFlag)
 	if err != nil {
 		return args, err
diff --git a/pkg/skaffold/build/kaniko/types.go b/pkg/skaffold/build/kaniko/types.go
index 63b4996a2..c47251881 100644
--- a/pkg/skaffold/build/kaniko/types.go
+++ b/pkg/skaffold/build/kaniko/types.go
@@ -107,4 +107,6 @@ const (
 	DefaultDockerConfigPath = "/kaniko/.docker"
 	// DefaultSecretMountPath for kaniko pod
 	DefaultSecretMountPath = "/secret"
+	// IgnorePath additional flag
+	IgnorePathFlag = "--ignore-path"
 )
diff --git a/pkg/skaffold/config/options.go b/pkg/skaffold/config/options.go
index 78025402c..85d3b84bf 100644
--- a/pkg/skaffold/config/options.go
+++ b/pkg/skaffold/config/options.go
@@ -48,6 +48,7 @@ type SkaffoldOptions struct {
 	ForceLoadImages             bool
 	IterativeStatusCheck        bool
 	FastFailStatusCheck         bool
+	KeepRunningOnFailure        bool
 	TolerateFailuresStatusCheck bool
 	Notification                bool
 	NoPrune                     bool
diff --git a/pkg/skaffold/deploy/kubectl/cli.go b/pkg/skaffold/deploy/kubectl/cli.go
index de93b7b58..1e6e1e407 100644
--- a/pkg/skaffold/deploy/kubectl/cli.go
+++ b/pkg/skaffold/deploy/kubectl/cli.go
@@ -95,7 +95,6 @@ func (c *CLI) Apply(ctx context.Context, out io.Writer, manifests manifest.Manif
 	// TODO(dgageot): should we delete a manifest that was deployed and is not anymore?
 	updated := c.previousApply.Diff(manifests)
 	log.Entry(ctx).Debugf("%d manifests to deploy. %d are updated or new", len(manifests), len(updated))
-	c.previousApply = manifests
 	if len(updated) == 0 {
 		return nil
 	}
@@ -113,6 +112,7 @@ func (c *CLI) Apply(ctx context.Context, out io.Writer, manifests manifest.Manif
 		endTrace(instrumentation.TraceEndError(err))
 		return userErr(fmt.Errorf("kubectl apply: %w", err))
 	}
+	c.previousApply = manifests
 
 	return nil
 }
diff --git a/pkg/skaffold/kubernetes/status/resource/deployment.go b/pkg/skaffold/kubernetes/status/resource/deployment.go
index 4fe299de4..a4e66fa70 100644
--- a/pkg/skaffold/kubernetes/status/resource/deployment.go
+++ b/pkg/skaffold/kubernetes/status/resource/deployment.go
@@ -51,8 +51,9 @@ type Type string
 var (
 	statefulsetRolloutSuccess = regexp.MustCompile("(roll out|rolling update) complete")
 
-	msgKubectlKilled     = "kubectl rollout status command interrupted\n"
-	MsgKubectlConnection = "kubectl connection error\n"
+	msgKubectlKilled            = "kubectl rollout status command interrupted\n"
+	MsgKubectlConnection        = "kubectl connection error\n"
+	msgStrategyTypeNotSupported = "rollout status is only available for RollingUpdate strategy type"
 
 	nonRetryContainerErrors = map[proto.StatusCode]struct{}{
 		proto.StatusCode_STATUSCHECK_IMAGE_PULL_ERR:       {},
@@ -373,6 +374,13 @@ func parseKubectlRolloutError(details string, deadline time.Duration, err error)
 			ErrCode: proto.StatusCode_STATUSCHECK_KUBECTL_PID_KILLED,
 			Message: fmt.Sprintf("received Ctrl-C or deployments could not stabilize within %v: %s", deadline, msgKubectlKilled),
 		}
+	// statefulset rollouts that use OnDelete strategy type don't support monitoring rollout, treat it as
+	// if the deployment just completed successfully
+	case strings.Contains(err.Error(), msgStrategyTypeNotSupported):
+		return &proto.ActionableErr{
+			ErrCode: proto.StatusCode_STATUSCHECK_SUCCESS,
+			Message: details,
+		}
 	default:
 		return &proto.ActionableErr{
 			ErrCode: proto.StatusCode_STATUSCHECK_UNKNOWN,
diff --git a/pkg/skaffold/runner/dev.go b/pkg/skaffold/runner/dev.go
index ad786a48c..59048d7b1 100644
--- a/pkg/skaffold/runner/dev.go
+++ b/pkg/skaffold/runner/dev.go
@@ -33,6 +33,7 @@ import (
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/output/log"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/schema/latest"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/sync"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/util/term"
 	timeutil "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/util/time"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
 )
@@ -224,6 +225,102 @@ func (r *SkaffoldRunner) Dev(ctx context.Context, out io.Writer, artifacts []*la
 		"devIteration": strconv.Itoa(r.devIteration),
 	})
 
+	// First build
+	var err error
+	bRes, err := r.Build(ctx, out, artifacts)
+	for ; err != nil && r.runCtx.Opts.KeepRunningOnFailure; bRes, err = r.Build(ctx, out, artifacts) {
+		log.Entry(ctx).Warnf("Failed to build artifacts: %v, please fix the error and press any key to continue.", err)
+		errT := term.WaitForKeyPress()
+		if errT != nil {
+			return errT
+		}
+	}
+
+	if err != nil {
+		event.DevLoopFailedInPhase(r.devIteration, constants.Build, err)
+		eventV2.TaskFailed(constants.DevLoop, err)
+		endTrace()
+		return fmt.Errorf("exiting dev mode because first build failed: %w", err)
+	}
+	// First test
+	if r.runCtx.IsTestPhaseActive() {
+		err = r.Test(ctx, out, bRes)
+		for ; err != nil && r.runCtx.Opts.KeepRunningOnFailure; err = r.Test(ctx, out, bRes) {
+			log.Entry(ctx).Warnf("Failed to run tests :%v, please fix the error and press any key to continue.", err)
+			errT := term.WaitForKeyPress()
+			if errT != nil {
+				return errT
+			}
+		}
+		if err != nil {
+			event.DevLoopFailedInPhase(r.devIteration, constants.Build, err)
+			eventV2.TaskFailed(constants.DevLoop, err)
+			endTrace()
+			return fmt.Errorf("exiting dev mode because test failed after first build: %w", err)
+		}
+	}
+
+	defer r.deployer.GetLogger().Stop()
+	defer r.deployer.GetDebugger().Stop()
+
+	// Logs should be retrieved up to just before the deploy
+	r.deployer.GetLogger().SetSince(time.Now())
+
+	// First render
+	manifests, err := r.Render(ctx, out, r.Builds, false)
+	for ; err != nil && r.runCtx.Opts.KeepRunningOnFailure; manifests, err = r.Render(ctx, out, r.Builds, false) {
+		log.Entry(ctx).Warnf("Failed to render :%v, please fix the error and press any key to continue.", err)
+		errT := term.WaitForKeyPress()
+		if errT != nil {
+			return errT
+		}
+	}
+	if err != nil {
+		event.DevLoopFailedInPhase(r.devIteration, constants.Render, err)
+		eventV2.TaskFailed(constants.DevLoop, err)
+		endTrace()
+		return fmt.Errorf("exiting dev mode because first render failed: %w", err)
+	}
+
+	// First deploy
+	err = r.Deploy(ctx, out, r.Builds, manifests)
+	for ; err != nil && r.runCtx.Opts.KeepRunningOnFailure; err = r.Deploy(ctx, out, r.Builds, manifests) {
+		log.Entry(ctx).Warnf("Failed to deploy :%v, please fix the error and press any key to continue.", err)
+		errT := term.WaitForKeyPress()
+		if errT != nil {
+			return errT
+		}
+		// The previous Render Stage could succeed even for kubernetes resource with unknown fields, this will lead to failure in Deploy Stage
+		// users need to fix the problems in their manifests, and skaffold needs to re-render them before re-running Deploy Stage in this case.
+		for manifests, err = r.Render(ctx, out, r.Builds, false); err != nil; manifests, err = r.Render(ctx, out, r.Builds, false) {
+			log.Entry(ctx).Warnf("Failed to Render, please fix the error and press any key to continue. %v", err)
+			errT := term.WaitForKeyPress()
+			if errT != nil {
+				return errT
+			}
+		}
+	}
+	if err != nil {
+		event.DevLoopFailedInPhase(r.devIteration, constants.Deploy, err)
+		eventV2.TaskFailed(constants.DevLoop, err)
+		endTrace()
+		return fmt.Errorf("exiting dev mode because first deploy failed: %w", err)
+	}
+	r.deployManifests = manifests
+
+	defer r.deployer.GetAccessor().Stop()
+
+	if err := r.deployer.GetAccessor().Start(ctx, out); err != nil {
+		log.Entry(ctx).Warn("Error starting resource accessor:", err)
+	}
+	if err := r.deployer.GetDebugger().Start(ctx); err != nil {
+		log.Entry(ctx).Warn("Error starting debug container notification:", err)
+	}
+	// Start printing the logs after deploy is finished
+	if err := r.deployer.GetLogger().Start(ctx, out); err != nil {
+		return fmt.Errorf("starting logger: %w", err)
+	}
+
 	g := getTransposeGraph(artifacts)
 	// Watch artifacts
 	start := time.Now()
@@ -327,61 +424,6 @@ func (r *SkaffoldRunner) Dev(ctx context.Context, out io.Writer, artifacts []*la
 		return fmt.Errorf("exiting dev mode because initializing sync state failed: %w", err)
 	}
 
-	// First build
-	bRes, err := r.Build(ctx, out, artifacts)
-	if err != nil {
-		event.DevLoopFailedInPhase(r.devIteration, constants.Build, err)
-		eventV2.TaskFailed(constants.DevLoop, err)
-		endTrace()
-		return fmt.Errorf("exiting dev mode because first build failed: %w", err)
-	}
-	// First test
-	if r.runCtx.IsTestPhaseActive() {
-		if err = r.Test(ctx, out, bRes); err != nil {
-			event.DevLoopFailedInPhase(r.devIteration, constants.Build, err)
-			eventV2.TaskFailed(constants.DevLoop, err)
-			endTrace()
-			return fmt.Errorf("exiting dev mode because test failed after first build: %w", err)
-		}
-	}
-
-	defer r.deployer.GetLogger().Stop()
-	defer r.deployer.GetDebugger().Stop()
-
-	// Logs should be retrieved up to just before the deploy
-	r.deployer.GetLogger().SetSince(time.Now())
-
-	// First render
-	manifests, err := r.Render(ctx, out, r.Builds, false)
-	r.deployManifests = manifests
-	if err != nil {
-		event.DevLoopFailedInPhase(r.devIteration, constants.Render, err)
-		eventV2.TaskFailed(constants.DevLoop, err)
-		endTrace()
-		return fmt.Errorf("exiting dev mode because first render failed: %w", err)
-	}
-
-	// First deploy
-	if err := r.Deploy(ctx, out, r.Builds, manifests); err != nil {
-		event.DevLoopFailedInPhase(r.devIteration, constants.Deploy, err)
-		eventV2.TaskFailed(constants.DevLoop, err)
-		endTrace()
-		return fmt.Errorf("exiting dev mode because first deploy failed: %w", err)
-	}
-
-	defer r.deployer.GetAccessor().Stop()
-
-	if err := r.deployer.GetAccessor().Start(ctx, out); err != nil {
-		log.Entry(ctx).Warn("Error starting resource accessor:", err)
-	}
-	if err := r.deployer.GetDebugger().Start(ctx); err != nil {
-		log.Entry(ctx).Warn("Error starting debug container notification:", err)
-	}
-	// Start printing the logs after deploy is finished
-	if err := r.deployer.GetLogger().Start(ctx, out); err != nil {
-		return fmt.Errorf("starting logger: %w", err)
-	}
-
 	output.Yellow.Fprintln(out, "Press Ctrl+C to exit")
 
 	event.DevLoopComplete(r.devIteration)
diff --git a/pkg/skaffold/schema/v2beta29/upgrade.go b/pkg/skaffold/schema/v2beta29/upgrade.go
index 1bda6f15b..506374cc9 100644
--- a/pkg/skaffold/schema/v2beta29/upgrade.go
+++ b/pkg/skaffold/schema/v2beta29/upgrade.go
@@ -20,9 +20,12 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"regexp"
+	"strconv"
 	"strings"
 
 	"github.com/pkg/errors"
+	"gopkg.in/yaml.v2"
 
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/output/log"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/schema/util"
@@ -30,6 +33,8 @@ import (
 	pkgutil "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/util"
 )
 
+var artifactOverridesRegexp = regexp.MustCompile("/deploy/helm/releases/[0-9]+/artifactOverrides/image")
+
 var migrations = map[string]string{
 	"/deploy/kubectl":             "/manifests/rawYaml",
 	"/deploy/kustomize/paths":     "/manifests/kustomize/paths",
@@ -45,28 +50,57 @@ func (c *SkaffoldConfig) Upgrade() (util.VersionedConfig, error) {
 	pkgutil.CloneThroughJSON(c, &newConfig)
 	newConfig.APIVersion = next.Version
 
-	err := util.UpgradePipelines(c, &newConfig, upgradeOnePipeline)
+	var nProfiles []next.Profile
+	// seed with existing Profiles
+	if c.Profiles != nil {
+		pkgutil.CloneThroughJSON(newConfig.Profiles, &nProfiles)
+	}
+
+	oldPatchesMap := map[string][]JSONPatch{}
+	newPatchesMap := map[string][]next.JSONPatch{}
+	for i, p := range c.Profiles {
+		oldPatchesMap[p.Name], newPatchesMap[p.Name] = p.Patches, nProfiles[i].Patches
+	}
+	onePipelineUpgrader := NewOnePipelineUpgrader(oldPatchesMap, newPatchesMap)
+
+	err := util.UpgradePipelines(c, &newConfig, onePipelineUpgrader.upgradeOnePipeline)
 	if err != nil {
 		return &newConfig, err
 	}
-
 	var newProfiles []next.Profile
 	// seed with existing Profiles
 	if c.Profiles != nil {
 		pkgutil.CloneThroughJSON(newConfig.Profiles, &newProfiles)
 	}
 
+	for i := range newProfiles {
+		if _, ok := newPatchesMap[newProfiles[i].Name]; ok {
+			newProfiles[i].Patches = newPatchesMap[newProfiles[i].Name]
+		}
+	}
+
 	// Update profiles patches
 	for i, p := range c.Profiles {
 		upgradePatches(p.Patches, newProfiles[i].Patches)
 	}
 
 	newConfig.Profiles = newProfiles
-
 	return &newConfig, nil
 }
 
-func upgradeOnePipeline(oldPipeline, newPipeline interface{}) error {
+type OnePipelineUpgrader struct {
+	oldPatchesMap map[string][]JSONPatch
+	newPatchesMap map[string][]next.JSONPatch
+}
+
+func NewOnePipelineUpgrader(oldPatchesMap map[string][]JSONPatch, newPatchesMap map[string][]next.JSONPatch) OnePipelineUpgrader {
+	return OnePipelineUpgrader{
+		oldPatchesMap: oldPatchesMap,
+		newPatchesMap: newPatchesMap,
+	}
+}
+
+func (opu *OnePipelineUpgrader) upgradeOnePipeline(oldPipeline, newPipeline interface{}) error {
 	oldPL := oldPipeline.(*Pipeline)
 	newPL := newPipeline.(*next.Pipeline)
 
@@ -151,12 +185,112 @@ func upgradeOnePipeline(oldPipeline, newPipeline interface{}) error {
 		newPL.Deploy.LegacyHelmDeploy = &next.LegacyHelmDeploy{}
 		newPL.Deploy.LegacyHelmDeploy.LifecycleHooks = newHelm.LifecycleHooks
 	}
+
+	err := upgradeArtifactOverridesPatches(opu.oldPatchesMap, opu.newPatchesMap, oldPL)
+	if err != nil {
+		return fmt.Errorf("error converting helm artifactOverrides during version upgrade: %w", err)
+	}
+
+	return nil
+}
+
+func upgradeArtifactOverridesPatches(oldsMap map[string][]JSONPatch, newsMap map[string][]next.JSONPatch, oldPL *Pipeline) error {
+	for k := range oldsMap {
+		for i, old := range oldsMap[k] {
+			if artifactOverridesRegexp.Match([]byte(old.Path)) {
+				idx, err := strconv.Atoi(strings.Split(old.Path, "/")[4])
+				if err != nil {
+					return err
+				}
+				newPathModified := strings.ReplaceAll(strings.ReplaceAll(old.Path, "/deploy/helm", "/manifests/helm"),
+					"artifactOverrides", "setValueTemplates")
+				if oldPL.Deploy.HelmDeploy != nil {
+					validV := pkgutil.SanitizeHelmTemplateValue(old.Value.Node.Value().(string))
+					n := &util.YamlpatchNode{}
+
+					if oldPL.Deploy.HelmDeploy.Releases[idx].ImageStrategy.HelmConventionConfig != nil {
+						err = yaml.Unmarshal([]byte(fmt.Sprintf("\"{{.IMAGE_TAG_%s}}@{{.IMAGE_DIGEST_%s}}\"", validV, validV)), n)
+						if err != nil {
+							return err
+						}
+
+						newsMap[k][i] = next.JSONPatch{
+							Op:    newsMap[k][i].Op,
+							Path:  newPathModified + ".tag",
+							From:  newsMap[k][i].From,
+							Value: n,
+						}
+
+						if oldPL.Deploy.HelmDeploy.Releases[idx].ImageStrategy.HelmConventionConfig.ExplicitRegistry {
+							// is 'helm' imageStrategy + explicitRegistry
+							n := &util.YamlpatchNode{}
+							err = yaml.Unmarshal([]byte(fmt.Sprintf("\"{{.IMAGE_DOMAIN_%s}}\"", validV)), n)
+							if err != nil {
+								return err
+							}
+
+							newsMap[k] = append(newsMap[k], next.JSONPatch{
+								Op:    newsMap[k][i].Op,
+								Path:  newPathModified + ".registry",
+								From:  newsMap[k][i].From,
+								Value: n,
+							})
+
+							n = &util.YamlpatchNode{}
+							err := yaml.Unmarshal([]byte(fmt.Sprintf("\"{{.IMAGE_REPO_NO_DOMAIN_%s}}\"", validV)), n)
+							if err != nil {
+								return err
+							}
+
+							newsMap[k] = append(newsMap[k], next.JSONPatch{
+								Op:    newsMap[k][i].Op,
+								Path:  newPathModified + ".repository",
+								From:  newsMap[k][i].From,
+								Value: n,
+							})
+						} else {
+							// is 'helm' imageStrategy
+							n = &util.YamlpatchNode{}
+							err := yaml.Unmarshal([]byte(fmt.Sprintf("\"{{.IMAGE_REPO_%s}}\"", validV)), n)
+							if err != nil {
+								return err
+							}
+
+							newsMap[k] = append(newsMap[k], next.JSONPatch{
+
+								Op:    newsMap[k][i].Op,
+								Path:  newPathModified + ".repository",
+								From:  newsMap[k][i].From,
+								Value: n,
+							})
+						}
+					} else {
+						// is 'fqn' imageStrategy
+						// replace commonly used image name chars that are illegal helm template chars "/" & "-" with "_"
+						newValue := fmt.Sprintf("\"{{.IMAGE_FULLY_QUALIFIED_%s}}\"", validV)
+
+						n := &util.YamlpatchNode{}
+						err := yaml.Unmarshal([]byte(newValue), n)
+						if err != nil {
+							return err
+						}
+						newsMap[k][i].Value = n
+						newsMap[k][i].Path = newPathModified
+					}
+				}
+			}
+		}
+	}
 	return nil
 }
 
 func upgradePatches(olds []JSONPatch, news []next.JSONPatch) {
 	for i, old := range olds {
 		for str, repStr := range migrations {
+			if artifactOverridesRegexp.Match([]byte(old.Path)) {
+				// this is handled by upgradeArtifactOverridesPatches
+				continue
+			}
 			if strings.Contains(old.Path, str) {
 				news[i].Path = strings.ReplaceAll(old.Path, str, repStr)
 			}
diff --git a/pkg/skaffold/util/term/term.go b/pkg/skaffold/util/term/term.go
index 22d549ca7..b9f5a3653 100644
--- a/pkg/skaffold/util/term/term.go
+++ b/pkg/skaffold/util/term/term.go
@@ -17,9 +17,11 @@ limitations under the License.
 package term
 
 import (
+	"bufio"
 	"context"
 	"fmt"
 	"io"
+	"os"
 	"os/exec"
 	"runtime"
 	"strconv"
@@ -63,3 +65,23 @@ func SupportsColor(ctx context.Context) (bool, error) {
 
 	return numColors > 0, nil
 }
+
+func WaitForKeyPress() error {
+	// use rawMode so that we can read without the user to hit enter key.
+	previousState, err := term.MakeRaw(int(os.Stdin.Fd()))
+	if err != nil {
+		return err
+	}
+	defer term.Restore(int(os.Stdin.Fd()), previousState)
+
+	reader := bufio.NewReader(os.Stdin)
+	r, _, err := reader.ReadRune()
+	if err != nil {
+		return err
+	}
+	// control + c
+	if r == 3 {
+		return context.Canceled
+	}
+	return nil
+}


Test patch:
diff --git a/integration/control_api_test.go b/integration/control_api_test.go
index c0a8ef127..8299a406e 100644
--- a/integration/control_api_test.go
+++ b/integration/control_api_test.go
@@ -25,6 +25,7 @@ import (
 	"time"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
 )
 
@@ -47,6 +48,11 @@ func TestControlAPIManualTriggers(t *testing.T) {
 
 	dep := client.GetDeployment(testDev)
 
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
+
 	// Make a change to foo
 	Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
 
diff --git a/integration/custom_test.go b/integration/custom_test.go
index 882cceabd..3ee840824 100644
--- a/integration/custom_test.go
+++ b/integration/custom_test.go
@@ -24,6 +24,8 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
+	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
 )
 
 func TestCustomTest(t *testing.T) {
@@ -44,7 +46,16 @@ func TestCustomTest(t *testing.T) {
 
 	ns, client := SetupNamespace(t)
 
-	skaffold.Dev().InDir(testDir).WithConfig(config).InNs(ns.Name).RunBackground(t)
+	rpcAddr := randomPort()
+	skaffold.Dev("--rpc-port", rpcAddr).InDir(testDir).WithConfig(config).InNs(ns.Name).RunBackground(t)
+
+	_, entries := apiEvents(t, rpcAddr)
+
+	// Wait for the first devloop to register target files to the monitor before running command to change target files
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
 
 	client.WaitForPodsReady("custom-test-example")
 	os.WriteFile(depFile, []byte("foo"), 0644)
diff --git a/integration/dev_dependencies_test.go b/integration/dev_dependencies_test.go
index 4e6c6f5fa..085b37c3f 100644
--- a/integration/dev_dependencies_test.go
+++ b/integration/dev_dependencies_test.go
@@ -23,6 +23,9 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
+	V2proto "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 )
 
 func TestDev_WithDependencies(t *testing.T) {
@@ -30,9 +33,17 @@ func TestDev_WithDependencies(t *testing.T) {
 		MarkIntegrationTest(t, CanRunWithoutGcp)
 		ns, client := SetupNamespace(t)
 
-		skaffold.Dev().InDir("testdata/build-dependencies").InNs(ns.Name).RunBackground(t)
+		rpcAddr := randomPort()
+		skaffold.Dev("--rpc-port", rpcAddr).InDir("testdata/build-dependencies").InNs(ns.Name).RunBackground(t)
 		client.waitForDeploymentsToStabilizeWithTimeout(3*time.Minute, "app1", "app2", "app3", "app4")
 
+		_, entries := v2apiEvents(t, rpcAddr)
+
+		failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+			taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+			return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+		}))
+
 		dep1 := client.GetDeployment("app1")
 		dep2 := client.GetDeployment("app2")
 		dep3 := client.GetDeployment("app3")
diff --git a/integration/dev_test.go b/integration/dev_test.go
index a0e8e3671..9e4ffd0d8 100644
--- a/integration/dev_test.go
+++ b/integration/dev_test.go
@@ -35,7 +35,9 @@ import (
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/config"
 	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
+	V2proto "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/testutil"
 )
 
@@ -64,10 +66,19 @@ func TestDevNotification(t *testing.T) {
 
 			ns, client := SetupNamespace(t)
 
-			skaffold.Dev("--trigger", test.trigger).InDir("testdata/dev").InNs(ns.Name).RunBackground(t)
+			rpcAddr := randomPort()
+			skaffold.Dev("--rpc-port", rpcAddr, "--trigger", test.trigger).InDir("testdata/dev").InNs(ns.Name).RunBackground(t)
 
 			dep := client.GetDeployment(testDev)
 
+			_, entries := v2apiEvents(t, rpcAddr)
+
+			// Wait for the first devloop to register target files to the monitor before running command to change target files
+			failNowIfError(t, waitForV2Event(100*time.Second, entries, func(e *V2proto.Event) bool {
+				taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+				return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+			}))
+
 			// Make a change to foo so that dev is forced to delete the Deployment and redeploy
 			Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
 
@@ -136,9 +147,8 @@ func TestDevGracefulCancel(t *testing.T) {
 	}
 }
 
-func TestDevAPITriggers(t *testing.T) {
+func TestDevAPIBuildTrigger(t *testing.T) {
 	MarkIntegrationTest(t, CanRunWithoutGcp)
-	t.Skip("failing since go 1.19.1")
 
 	Run(t, "testdata/dev", "sh", "-c", "echo foo > foo")
 	defer Run(t, "testdata/dev", "rm", "foo")
@@ -146,13 +156,18 @@ func TestDevAPITriggers(t *testing.T) {
 	// Run skaffold build first to fail quickly on a build failure
 	skaffold.Build().InDir("testdata/dev").RunOrFail(t)
 
-	ns, client := SetupNamespace(t)
+	ns, _ := SetupNamespace(t)
 
 	rpcAddr := randomPort()
 	skaffold.Dev("--auto-build=false", "--auto-sync=false", "--auto-deploy=false", "--rpc-port", rpcAddr, "--cache-artifacts=false").InDir("testdata/dev").InNs(ns.Name).RunBackground(t)
 
 	rpcClient, entries := apiEvents(t, rpcAddr)
-	dep := client.GetDeployment(testDev)
+
+	// Wait for the first devloop to register target files to the monitor before running command to change target files
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
 
 	// Make a change to foo
 	Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
@@ -169,6 +184,33 @@ func TestDevAPITriggers(t *testing.T) {
 		return e.GetEvent().GetBuildEvent().GetArtifact() == testDev
 	})
 	failNowIfError(t, err)
+}
+
+func TestDevApiDeployTrigger(t *testing.T) {
+	MarkIntegrationTest(t, CanRunWithoutGcp)
+
+	Run(t, "testdata/dev", "sh", "-c", "echo foo > foo")
+	defer Run(t, "testdata/dev", "rm", "foo")
+
+	// Run skaffold build first to fail quickly on a build failure
+	skaffold.Build().InDir("testdata/dev").RunOrFail(t)
+
+	ns, client := SetupNamespace(t)
+
+	rpcAddr := randomPort()
+	skaffold.Dev("--auto-deploy=false", "--rpc-port", rpcAddr, "--cache-artifacts=false").InDir("testdata/dev").InNs(ns.Name).RunBackground(t)
+
+	rpcClient, entries := apiEvents(t, rpcAddr)
+	dep := client.GetDeployment(testDev)
+
+	// Wait for the first devloop to register target files to the monitor before running command to change target files
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
+
+	// Make a change to foo
+	Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
 
 	// Issue a deploy trigger
 	rpcClient.Execute(context.Background(), &proto.UserIntentRequest{
@@ -197,6 +239,12 @@ func TestDevAPIAutoTriggers(t *testing.T) {
 	rpcClient, entries := apiEvents(t, rpcAddr)
 	dep := client.GetDeployment(testDev)
 
+	// Wait for the first devloop to register target files to the monitor before running command to change target files
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
+
 	// Make a change to foo
 	Run(t, "testdata/dev", "sh", "-c", "echo bar > foo")
 
diff --git a/integration/fix_test.go b/integration/fix_test.go
index e968d151e..7e45cf109 100644
--- a/integration/fix_test.go
+++ b/integration/fix_test.go
@@ -45,6 +45,17 @@ func TestFixStdout(t *testing.T) {
 	skaffold.Run().WithConfig("-").InDir("testdata/fix").InNs(ns.Name).WithStdin(out).RunOrFail(t)
 }
 
+func TestFixV2Beta29ToV3Alpha1PatchProfiles(t *testing.T) {
+	MarkIntegrationTest(t, CanRunWithoutGcp)
+
+	out, err := skaffold.Fix().InDir("testdata/fix-v2beta29").RunWithCombinedOutput(t)
+	testutil.CheckError(t, false, err)
+	testutil.CheckContains(t, "/manifests/helm/releases/0/setValueTemplates/image.repository", string(out))
+	testutil.CheckContains(t, "{{.IMAGE_REPO_skaffold_helm_v2}}", string(out))
+	testutil.CheckContains(t, "/manifests/helm/releases/0/setValueTemplates/image.tag", string(out))
+	testutil.CheckContains(t, "{{.IMAGE_TAG_skaffold_helm_v2}}@{{.IMAGE_DIGEST_skaffold_helm_v2}}", string(out))
+}
+
 func TestFixOutputFile(t *testing.T) {
 	MarkIntegrationTest(t, CanRunWithoutGcp)
 
diff --git a/integration/modules_test.go b/integration/modules_test.go
index b5cf7eb36..da6806ed9 100644
--- a/integration/modules_test.go
+++ b/integration/modules_test.go
@@ -23,20 +23,30 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
+	V2proto "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 )
 
 func TestModules_BuildDependency(t *testing.T) {
 	t.Run("build dependency between artifacts of different modules", func(t *testing.T) {
 		MarkIntegrationTest(t, CanRunWithoutGcp)
 		ns, client := SetupNamespace(t)
-
-		skaffold.Dev().InDir("testdata/modules").InNs(ns.Name).RunBackground(t)
+		rpcAddr := randomPort()
+		skaffold.Dev("--rpc-port", rpcAddr).InDir("testdata/modules").InNs(ns.Name).RunBackground(t)
 		client.waitForDeploymentsToStabilizeWithTimeout(3*time.Minute, "app1", "app2", "app3")
 
 		dep1 := client.GetDeployment("app1")
 		dep2 := client.GetDeployment("app2")
 		dep3 := client.GetDeployment("app3")
 
+		_, entries := v2apiEvents(t, rpcAddr)
+
+		failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+			taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+			return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+		}))
+
 		// Make a change to app3/foo so that dev is forced to delete the Deployment and redeploy app1, app2 and app3,
 		// since app2 depends on app3 and app1 depends on app2
 		Run(t, "testdata/modules/app3", "sh", "-c", "echo bar > foo")
diff --git a/integration/rpc_test.go b/integration/rpc_test.go
index 951466f03..db41cd4fc 100644
--- a/integration/rpc_test.go
+++ b/integration/rpc_test.go
@@ -32,10 +32,12 @@ import (
 	"github.com/golang/protobuf/jsonpb"
 	"github.com/golang/protobuf/ptypes/empty"
 	"google.golang.org/grpc"
+	"google.golang.org/protobuf/types/known/emptypb"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
 	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
+	protoV2 "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/testutil"
 )
 
@@ -454,3 +456,87 @@ func setupRPCClient(t *testing.T, port string) proto.SkaffoldServiceClient {
 
 	return client
 }
+
+func setupV2RPCClient(t *testing.T, port string) protoV2.SkaffoldV2ServiceClient {
+	// start a grpc client
+	var (
+		conn   *grpc.ClientConn
+		err    error
+		client protoV2.SkaffoldV2ServiceClient
+	)
+
+	// connect to the skaffold grpc server
+	for i := 0; i < connectionRetries; i++ {
+		conn, err = grpc.Dial(fmt.Sprintf(":%s", port), grpc.WithInsecure(), grpc.WithBackoffMaxDelay(10*time.Second))
+		if err != nil {
+			t.Logf("unable to establish skaffold grpc connection: retrying...")
+			time.Sleep(waitTime)
+			continue
+		}
+
+		client = protoV2.NewSkaffoldV2ServiceClient(conn)
+		break
+	}
+
+	if client == nil {
+		t.Fatalf("error establishing skaffold grpc connection")
+	}
+
+	t.Cleanup(func() { conn.Close() })
+
+	return client
+}
+
+func readV2EventAPIStream(client protoV2.SkaffoldV2ServiceClient, t *testing.T, retries int) (protoV2.SkaffoldV2Service_EventsClient, error) {
+	t.Helper()
+	// read the event log stream from the skaffold grpc server
+	var stream protoV2.SkaffoldV2Service_EventsClient
+	var err error
+	var protoReq emptypb.Empty
+	for i := 0; i < retries; i++ {
+		stream, err = client.Events(context.Background(), &protoReq, grpc.WaitForReady(true))
+		if err == nil {
+			break
+		}
+		t.Logf("waiting for connection...")
+		time.Sleep(waitTime)
+	}
+	return stream, err
+}
+
+func v2apiEvents(t *testing.T, rpcAddr string) (protoV2.SkaffoldV2ServiceClient, chan *protoV2.Event) { // nolint
+	client := setupV2RPCClient(t, rpcAddr)
+
+	stream, err := readV2EventAPIStream(client, t, readRetries)
+	if stream == nil {
+		t.Fatalf("error retrieving event log: %v\n", err)
+	}
+
+	// read entries from the log
+	entries := make(chan *protoV2.Event)
+	go func() {
+		for {
+			entry, _ := stream.Recv()
+			if entry != nil {
+				entries <- entry
+			}
+		}
+	}()
+
+	return client, entries
+}
+
+func waitForV2Event(timeout time.Duration, entries chan *protoV2.Event, condition func(event2 *protoV2.Event) bool) error {
+	ctx, cancelTimeout := context.WithTimeout(context.Background(), timeout)
+	defer cancelTimeout()
+	for {
+		select {
+		case <-ctx.Done():
+			return fmt.Errorf("timed out waiting for condition on log entry")
+		case ev := <-entries:
+			if condition(ev) {
+				return nil
+			}
+		}
+	}
+}
diff --git a/integration/sync_test.go b/integration/sync_test.go
index f4d7b9b08..c620dbd04 100644
--- a/integration/sync_test.go
+++ b/integration/sync_test.go
@@ -30,7 +30,10 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 
 	"github.com/GoogleContainerTools/skaffold/v2/integration/skaffold"
+	"github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/constants"
+	event "github.com/GoogleContainerTools/skaffold/v2/pkg/skaffold/event/v2"
 	"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
+	V2proto "github.com/GoogleContainerTools/skaffold/v2/proto/v2"
 )
 
 // TODO: remove nolint once we've reenabled integration tests
@@ -65,10 +68,19 @@ func TestDevSync(t *testing.T) {
 
 			ns, client := SetupNamespace(t)
 
-			skaffold.Dev("--trigger", test.trigger).InDir("testdata/file-sync").WithConfig(test.config).InNs(ns.Name).RunBackground(t)
+			rpcAddr := randomPort()
+
+			skaffold.Dev("--rpc-port", rpcAddr, "--trigger", test.trigger).InDir("testdata/file-sync").WithConfig(test.config).InNs(ns.Name).RunBackground(t)
 
 			client.WaitForPodsReady("test-file-sync")
 
+			_, entries := v2apiEvents(t, rpcAddr)
+
+			failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+				taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+				return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+			}))
+
 			os.WriteFile("testdata/file-sync/foo", []byte("foo"), 0644)
 			defer func() { os.Truncate("testdata/file-sync/foo", 0) }()
 
@@ -110,11 +122,20 @@ func TestDevSyncDefaultNamespace(t *testing.T) {
 
 			_, client := DefaultNamespace(t)
 
-			skaffold.Dev("--trigger", test.trigger).InDir("testdata/file-sync").WithConfig(test.config).RunBackground(t)
+			rpcAddr := randomPort()
+
+			skaffold.Dev("--rpc-port", rpcAddr, "--trigger", test.trigger).InDir("testdata/file-sync").WithConfig(test.config).RunBackground(t)
+
 			defer skaffold.Delete().InDir("testdata/file-sync").WithConfig(test.config).Run(t)
 
 			client.WaitForPodsReady(id)
 
+			_, entries := v2apiEvents(t, rpcAddr)
+			failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+				taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+				return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+			}))
+
 			os.WriteFile("testdata/file-sync/foo", []byte("foo"), 0644)
 			defer func() { os.Truncate("testdata/file-sync/foo", 0) }()
 
@@ -153,7 +174,8 @@ func TestDevAutoSync(t *testing.T) {
 
 			ns, client := SetupNamespace(t)
 
-			output := skaffold.Dev("--trigger", "notify").WithConfig(test.configFile).InDir(dir).InNs(ns.Name).RunLive(t)
+			rpcAddr := randomPort()
+			output := skaffold.Dev("--trigger", "notify", "--rpc-port", rpcAddr).WithConfig(test.configFile).InDir(dir).InNs(ns.Name).RunLive(t)
 
 			client.WaitForPodsReady("test-file-sync")
 
@@ -172,6 +194,13 @@ func TestDevAutoSync(t *testing.T) {
 				}
 			}
 
+			_, entries := v2apiEvents(t, rpcAddr)
+
+			failNowIfError(t, waitForV2Event(90*time.Second, entries, func(e *V2proto.Event) bool {
+				taskEvent, ok := e.EventType.(*V2proto.Event_TaskEvent)
+				return ok && taskEvent.TaskEvent.Task == string(constants.DevLoop) && taskEvent.TaskEvent.Status == event.Succeeded
+			}))
+
 			// direct file sync (this file is an existing file checked in for this testdata)
 			directFile := "direct-file"
 			directFilePath := dir + "src/main/jib/" + directFile
@@ -222,6 +251,10 @@ func TestDevSyncAPITrigger(t *testing.T) {
 
 	rpcClient, entries := apiEvents(t, rpcAddr)
 	client.WaitForPodsReady("test-file-sync")
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
 
 	os.WriteFile("testdata/file-sync/foo", []byte("foo"), 0644)
 	defer func() { os.Truncate("testdata/file-sync/foo", 0) }()
@@ -253,6 +286,11 @@ func TestDevAutoSyncAPITrigger(t *testing.T) {
 
 	client.WaitForPodsReady("test-file-sync")
 
+	failNowIfError(t, waitForEvent(90*time.Second, entries, func(e *proto.LogEntry) bool {
+		dle, ok := e.Event.EventType.(*proto.Event_DevLoopEvent)
+		return ok && dle.DevLoopEvent.Status == event.Succeeded
+	}))
+
 	os.WriteFile("testdata/file-sync/foo", []byte("foo"), 0644)
 	defer func() { os.Truncate("testdata/file-sync/foo", 0) }()
 
diff --git a/integration/testdata/fix-v2beta29/skaffold.yaml b/integration/testdata/fix-v2beta29/skaffold.yaml
new file mode 100644
index 000000000..4d54d946f
--- /dev/null
+++ b/integration/testdata/fix-v2beta29/skaffold.yaml
@@ -0,0 +1,24 @@
+apiVersion: skaffold/v2beta29
+kind: Config
+build:
+  artifacts:
+  - image: skaffold-helm
+deploy:
+ helm:
+   releases:
+     - name: skaffold-helm
+       chartPath: charts
+       namespace: repro-ns
+       recreatePods: false
+       useHelmSecrets: false
+       wait: false
+       artifactOverrides:
+         image: skaffold-helm
+       imageStrategy:
+         helm: {}
+profiles:
+ - name: skaffold-helm-v2
+   patches:
+      - op: replace
+        path: /deploy/helm/releases/0/artifactOverrides/image
+        value: skaffold-helm-v2
\ No newline at end of file
diff --git a/pkg/skaffold/build/gcb/kaniko_test.go b/pkg/skaffold/build/gcb/kaniko_test.go
index 57f30a3b4..d9c8d0e6d 100644
--- a/pkg/skaffold/build/gcb/kaniko_test.go
+++ b/pkg/skaffold/build/gcb/kaniko_test.go
@@ -397,6 +397,17 @@ func TestKanikoBuildSpec(t *testing.T) {
 				kaniko.LabelFlag, "label2",
 			},
 		},
+		{
+			description: "with IgnorePaths",
+			artifact: &latest.KanikoArtifact{
+				DockerfilePath: "Dockerfile",
+				IgnorePaths:    []string{"/var", "/tmp"},
+			},
+			expectedArgs: []string{
+				kaniko.IgnorePathFlag, "/var",
+				kaniko.IgnorePathFlag, "/tmp",
+			},
+		},
 	}
 	store := mockArtifactStore{
 		"img2": "img2:tag",
diff --git a/pkg/skaffold/build/kaniko/args_test.go b/pkg/skaffold/build/kaniko/args_test.go
index fd9874c09..f00c65de3 100644
--- a/pkg/skaffold/build/kaniko/args_test.go
+++ b/pkg/skaffold/build/kaniko/args_test.go
@@ -407,6 +407,18 @@ func TestArgs(t *testing.T) {
 			},
 			wantErr: false,
 		},
+		{
+			description: "with IgnorePaths",
+			artifact: &latest.KanikoArtifact{
+				DockerfilePath: "dir/Dockerfile",
+				IgnorePaths:    []string{"/tmp", "/proc"},
+			},
+			expectedArgs: []string{
+				IgnorePathFlag, "/tmp",
+				IgnorePathFlag, "/proc",
+			},
+			wantErr: false,
+		},
 	}
 
 	defaultExpectedArgs := []string{
diff --git a/pkg/skaffold/schema/latest/config.go b/pkg/skaffold/schema/latest/config.go
index 10266ccd6..b87e0d958 100644
--- a/pkg/skaffold/schema/latest/config.go
+++ b/pkg/skaffold/schema/latest/config.go
@@ -1387,6 +1387,9 @@ type KanikoArtifact struct {
 
 	// ContextSubPath is to specify a sub path within the context.
 	ContextSubPath string `yaml:"contextSubPath,omitempty" skaffold:"filepath"`
+
+	// IgnorePaths is a list of ignored paths when making an image snapshot.
+	IgnorePaths []string `yaml:"ignorePaths,omitempty"`
 }
 
 // DockerArtifact describes an artifact built from a Dockerfile,