Repository: GoogleContainerTools/skaffold
Pull request #7657 (2022-07-19 18:09:03)

fix: skaffold delete command and cleanup function
<!-- Thank you for your contribution! -->

<!-- Include if applicable: -->
Fixes: #7648 , #7644 <!-- tracking issues that this PR will close -->
**Related**: _Relevant tracking issues, for context_
**Merge before/after**: _Dependent or prerequisite PRs_

**Description**
 - fix ``skaffold delete`` command so users can use it to delete resources built from ``skaffold run``
 - fix ``deployer cleanup`` method, so resources can be cleaned up after dev command terminates
 - remove unnecessary code
 - add test util method for testing, deployments/pods deletions and deployer cleanup method.
 - add tests

**User facing changes (remove if N/A)**
 - fix ``skaffold delete`` command so users can use it to delete resources built from ``skaffold run``
 - fix ``deployer cleanup`` method, so resources can be cleaned up after dev command terminates


**Follow-up Work (remove if N/A)**
<!-- Mention any related follow up work to this PR. -->


<!--
Please be sure your PR includes unit tests - we don't merge code that brings down test coverage! 
Integration tests are sometimes an appropriate substitute. 
-->


codecov[bot]: # [Codecov](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) Report
> Merging [#7657](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (89f17de) into [main](https://codecov.io/gh/GoogleContainerTools/skaffold/commit/290280e8c6c19231e9ed5b5c027f2c01d22f75be?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (290280e) will **decrease** coverage by `3.85%`.
> The diff coverage is `53.95%`.

```diff
@@            Coverage Diff             @@
##             main    #7657      +/-   ##
==========================================
- Coverage   70.48%   66.62%   -3.86%     
==========================================
  Files         515      585      +70     
  Lines       23150    28046    +4896     
==========================================
+ Hits        16317    18686    +2369     
- Misses       5776     7993    +2217     
- Partials     1057     1367     +310     
```


| [Impacted Files](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | Coverage Δ | |
|---|---|---|
| [cmd/skaffold/app/cmd/credits/export.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY3JlZGl0cy9leHBvcnQuZ28=) | `0.00% <0.00%> (ø)` | |
| [cmd/skaffold/app/cmd/deploy.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZGVwbG95Lmdv) | `40.90% <0.00%> (-12.94%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/test.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvdGVzdC5nbw==) | `44.44% <0.00%> (ø)` | |
| [cmd/skaffold/skaffold.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL3NrYWZmb2xkLmdv) | `0.00% <0.00%> (ø)` | |
| [cmd/skaffold/app/cmd/inspect\_tests.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvaW5zcGVjdF90ZXN0cy5nbw==) | `62.50% <14.28%> (-1.14%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/render.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvcmVuZGVyLmdv) | `35.48% <18.18%> (-5.90%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/lsp.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvbHNwLmdv) | `28.12% <28.12%> (ø)` | |
| [cmd/skaffold/app/cmd/run.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvcnVuLmdv) | `64.28% <33.33%> (-9.63%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/fix.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZml4Lmdv) | `55.93% <38.88%> (-20.54%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/verify.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvdmVyaWZ5Lmdv) | `41.17% <41.17%> (ø)` | |
| ... and [327 more](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7657/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | |

:mega: Codecov can now indicate which changes are the most critical in Pull Requests. [Learn more](https://about.codecov.io/product/feature/runtime-insights/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools)


Patch:
diff --git a/cmd/skaffold/app/cmd/delete.go b/cmd/skaffold/app/cmd/delete.go
index 3a0285529..a2e350ab9 100644
--- a/cmd/skaffold/app/cmd/delete.go
+++ b/cmd/skaffold/app/cmd/delete.go
@@ -18,12 +18,11 @@ package cmd
 
 import (
 	"context"
-	"fmt"
 	"io"
 
 	"github.com/spf13/cobra"
 
-	deployutil "github.com/GoogleContainerTools/skaffold/pkg/skaffold/deploy/util"
+	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/graph"
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/runner"
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/util"
 )
@@ -46,10 +45,10 @@ func NewCmdDelete() *cobra.Command {
 
 func doDelete(ctx context.Context, out io.Writer) error {
 	return withRunner(ctx, out, func(r runner.Runner, configs []util.VersionedConfig) error {
-		manifests, err := deployutil.GetManifestsFromHydrationDir(ctx, opts)
+		manifestListByConfig, err := r.Render(ctx, io.Discard, []graph.Artifact{}, false)
 		if err != nil {
-			return fmt.Errorf("getting manifests from hydration dir: %w", err)
+			return err
 		}
-		return r.Cleanup(ctx, out, dryRun, manifests)
+		return r.Cleanup(ctx, out, dryRun, manifestListByConfig)
 	})
 }
diff --git a/cmd/skaffold/app/cmd/dev.go b/cmd/skaffold/app/cmd/dev.go
index 5b41a986a..8c00af02a 100644
--- a/cmd/skaffold/app/cmd/dev.go
+++ b/cmd/skaffold/app/cmd/dev.go
@@ -24,7 +24,7 @@ import (
 
 	"github.com/spf13/cobra"
 
-	deployutil "github.com/GoogleContainerTools/skaffold/pkg/skaffold/deploy/util"
+	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/graph"
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/output/log"
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/runner"
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/schema/latest"
@@ -74,11 +74,11 @@ func runDev(ctx context.Context, out io.Writer) error {
 
 				if r.HasDeployed() {
 					cleanup = func() {
-						manifests, err := deployutil.GetManifestsFromHydrationDir(ctx, opts)
+						manifestsByConfig, err := r.Render(ctx, io.Discard, []graph.Artifact{}, false)
 						if err != nil {
-							log.Entry(ctx).Warn(fmt.Errorf("getting manifests from hydration dir: %w", err))
+							log.Entry(ctx).Warn(fmt.Errorf("failed to render manifests: %w", err))
 						}
-						if err := r.Cleanup(context.Background(), out, false, manifests); err != nil {
+						if err := r.Cleanup(context.Background(), out, false, manifestsByConfig); err != nil {
 							log.Entry(ctx).Warn("deployer cleanup:", err)
 						}
 					}
diff --git a/pkg/skaffold/build/local/local.go b/pkg/skaffold/build/local/local.go
index d0a414695..5336d18f8 100644
--- a/pkg/skaffold/build/local/local.go
+++ b/pkg/skaffold/build/local/local.go
@@ -18,7 +18,6 @@ package local
 
 import (
 	"context"
-	"fmt"
 	"io"
 
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/build"
@@ -114,7 +113,8 @@ func (b *Builder) runBuildForArtifact(ctx context.Context, out io.Writer, a *lat
 		if p := platforms.Intersect(supported); p.IsNotEmpty() {
 			platforms = p
 		} else {
-			return "", fmt.Errorf("builder for artifact %q doesn't support building for target platforms: %q. Supported platforms are %q", a.ImageName, platforms, supported)
+			log.Entry(ctx).Warnf("builder for artifact %q doesn't support building for target platforms: %q. Building for supported platforms %q instead.", a.ImageName, platforms, supported)
+			platforms = supported
 		}
 	}
 	return builder.Build(ctx, out, a, tag, platforms)
diff --git a/pkg/skaffold/deploy/cloudrun/deploy.go b/pkg/skaffold/deploy/cloudrun/deploy.go
index 2ca2ed399..949e09e28 100644
--- a/pkg/skaffold/deploy/cloudrun/deploy.go
+++ b/pkg/skaffold/deploy/cloudrun/deploy.go
@@ -101,8 +101,8 @@ func (d *Deployer) Dependencies() ([]string, error) {
 }
 
 // Cleanup deletes the created Cloud Run services
-func (d *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, manifests manifest.ManifestList) error {
-	return d.deleteRunService(ctx, out, dryRun, manifests)
+func (d *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, byConfig *manifest.ManifestListByConfig) error {
+	return d.deleteRunService(ctx, out, dryRun, byConfig.GetForConfig(d.configName))
 }
 
 // GetDebugger Get the Debugger for Cloud Run. Not supported by this deployer.
diff --git a/pkg/skaffold/deploy/deploy.go b/pkg/skaffold/deploy/deploy.go
index bae499f47..3456796e1 100644
--- a/pkg/skaffold/deploy/deploy.go
+++ b/pkg/skaffold/deploy/deploy.go
@@ -41,7 +41,7 @@ type Deployer interface {
 	Dependencies() ([]string, error)
 
 	// Cleanup deletes what was deployed by calling Deploy.
-	Cleanup(context.Context, io.Writer, bool, manifest.ManifestList) error
+	Cleanup(context.Context, io.Writer, bool, *manifest.ManifestListByConfig) error
 
 	// GetDebugger returns a Deployer's implementation of a Debugger
 	GetDebugger() debug.Debugger
diff --git a/pkg/skaffold/deploy/deploy_mux.go b/pkg/skaffold/deploy/deploy_mux.go
index 13b19ea28..953314b07 100644
--- a/pkg/skaffold/deploy/deploy_mux.go
+++ b/pkg/skaffold/deploy/deploy_mux.go
@@ -160,13 +160,13 @@ func (m DeployerMux) Dependencies() ([]string, error) {
 	return deps.ToList(), nil
 }
 
-func (m DeployerMux) Cleanup(ctx context.Context, w io.Writer, dryRun bool, list manifest.ManifestList) error {
+func (m DeployerMux) Cleanup(ctx context.Context, w io.Writer, dryRun bool, manifestsByConfig *manifest.ManifestListByConfig) error {
 	for _, deployer := range m.deployers {
 		ctx, endTrace := instrumentation.StartTrace(ctx, "Cleanup")
 		if dryRun {
 			output.Yellow.Fprintln(w, "Following resources would be deleted:")
 		}
-		if err := deployer.Cleanup(ctx, w, dryRun, list); err != nil {
+		if err := deployer.Cleanup(ctx, w, dryRun, manifestsByConfig); err != nil {
 			return err
 		}
 		endTrace()
diff --git a/pkg/skaffold/deploy/docker/deploy.go b/pkg/skaffold/deploy/docker/deploy.go
index b351448d1..1226eb069 100644
--- a/pkg/skaffold/deploy/docker/deploy.go
+++ b/pkg/skaffold/deploy/docker/deploy.go
@@ -295,7 +295,7 @@ func (d *Deployer) Dependencies() ([]string, error) {
 	return nil, nil
 }
 
-func (d *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, list manifest.ManifestList) error {
+func (d *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, _ *manifest.ManifestListByConfig) error {
 	if dryRun {
 		for _, container := range d.tracker.DeployedContainers() {
 			output.Yellow.Fprintln(out, container.ID)
diff --git a/pkg/skaffold/deploy/helm/helm.go b/pkg/skaffold/deploy/helm/helm.go
index fa10f3600..9dbb6c0d4 100644
--- a/pkg/skaffold/deploy/helm/helm.go
+++ b/pkg/skaffold/deploy/helm/helm.go
@@ -355,7 +355,7 @@ func (h *Deployer) Dependencies() ([]string, error) {
 }
 
 // Cleanup deletes what was deployed by calling Deploy.
-func (h *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, list manifest.ManifestList) error {
+func (h *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, _ *manifest.ManifestListByConfig) error {
 	instrumentation.AddAttributesToCurrentSpanFromContext(ctx, map[string]string{
 		"DeployerType": "helm",
 	})
diff --git a/pkg/skaffold/deploy/kpt/kpt.go b/pkg/skaffold/deploy/kpt/kpt.go
index f2c9b0ed0..d0e50bc96 100644
--- a/pkg/skaffold/deploy/kpt/kpt.go
+++ b/pkg/skaffold/deploy/kpt/kpt.go
@@ -367,7 +367,7 @@ func (k *Deployer) Dependencies() ([]string, error) {
 }
 
 // Cleanup deletes what was deployed by calling `kpt live destroy`.
-func (k *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, list manifest.ManifestList) error {
+func (k *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, _ *manifest.ManifestListByConfig) error {
 	instrumentation.AddAttributesToCurrentSpanFromContext(ctx, map[string]string{
 		"DeployerType": deployerName,
 	})
diff --git a/pkg/skaffold/deploy/kubectl/errors.go b/pkg/skaffold/deploy/kubectl/errors.go
index 45ce01a7a..5b3f35bbe 100644
--- a/pkg/skaffold/deploy/kubectl/errors.go
+++ b/pkg/skaffold/deploy/kubectl/errors.go
@@ -43,20 +43,6 @@ func versionGetErr(err error) error {
 		})
 }
 
-func offlineModeErr() error {
-	return sErrors.NewErrorWithStatusCode(
-		&proto.ActionableErr{
-			Message: "Cannot use offline mode if URL manifests are configured",
-			ErrCode: proto.StatusCode_DEPLOY_KUBECTL_OFFLINE_MODE_ERR,
-			Suggestions: []*proto.Suggestion{
-				{
-					SuggestionCode: proto.SuggestionCode_SET_RENDER_FLAG_OFFLINE_FALSE,
-					Action:         "Please rerun with --offline=false",
-				},
-			},
-		})
-}
-
 func waitForDeletionErr(err error) error {
 	return sErrors.NewError(err,
 		&proto.ActionableErr{
diff --git a/pkg/skaffold/deploy/kubectl/kubectl.go b/pkg/skaffold/deploy/kubectl/kubectl.go
index fd4f44c25..96de62064 100644
--- a/pkg/skaffold/deploy/kubectl/kubectl.go
+++ b/pkg/skaffold/deploy/kubectl/kubectl.go
@@ -21,7 +21,6 @@ import (
 	"context"
 	"fmt"
 	"io"
-	"os"
 	"strings"
 
 	"github.com/segmentio/textio"
@@ -328,50 +327,6 @@ func (k *Deployer) manifestFiles(manifests []string) ([]string, error) {
 	return filteredManifests, nil
 }
 
-// readManifests reads the manifests to deploy/delete.
-func (k *Deployer) readManifests(ctx context.Context, offline bool) (manifest.ManifestList, error) {
-	var manifests []string
-
-	// Clean the temporary directory that holds the manifests downloaded from GCS
-	defer os.RemoveAll(k.gcsManifestDir)
-
-	// Append URL manifests. URL manifests are excluded from `Dependencies`.
-	hasURLManifest := false
-	for _, manifest := range k.KubectlDeploy.Manifests {
-		if util.IsURL(manifest) {
-			manifests = append(manifests, manifest)
-			hasURLManifest = true
-		}
-	}
-
-	if len(manifests) == 0 {
-		return manifest.ManifestList{}, nil
-	}
-
-	if !offline {
-		return k.kubectl.ReadManifests(ctx, manifests)
-	}
-
-	// In case no URLs are provided, we can stay offline - no need to run "kubectl create" which
-	// would try to connect to a cluster (https://github.com/kubernetes/kubernetes/issues/51475)
-	if hasURLManifest {
-		return nil, offlineModeErr()
-	}
-	return createManifestList(manifests)
-}
-
-func createManifestList(manifests []string) (manifest.ManifestList, error) {
-	var manifestList manifest.ManifestList
-	for _, manifestFilePath := range manifests {
-		manifestFileContent, err := os.ReadFile(manifestFilePath)
-		if err != nil {
-			return nil, readManifestErr(fmt.Errorf("reading manifest file %v: %w", manifestFilePath, err))
-		}
-		manifestList.Append(manifestFileContent)
-	}
-	return manifestList, nil
-}
-
 // readRemoteManifests will try to read manifests from the given kubernetes
 // context in the specified namespace and for the specified type
 func (k *Deployer) readRemoteManifest(ctx context.Context, name string) ([]byte, error) {
@@ -393,15 +348,14 @@ func (k *Deployer) readRemoteManifest(ctx context.Context, name string) ([]byte,
 }
 
 // Cleanup deletes what was deployed by calling Deploy.
-func (k *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, manifests manifest.ManifestList) error {
+func (k *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, manifestsByConfig *manifest.ManifestListByConfig) error {
+	var manifests manifest.ManifestList
+	if manifestsByConfig != nil {
+		manifests = append(manifests, manifestsByConfig.GetForConfig(k.ConfigName())...)
+	}
 	instrumentation.AddAttributesToCurrentSpanFromContext(ctx, map[string]string{
 		"DeployerType": "kubectl",
 	})
-	other, err := k.readManifests(ctx, false)
-	manifests = append(manifests, other...)
-	if err != nil {
-		return err
-	}
 	if dryRun {
 		for _, manifest := range manifests {
 			output.White.Fprintf(out, "---\n%s", manifest)
@@ -411,6 +365,7 @@ func (k *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, mani
 	// revert remote manifests
 	// TODO(dgageot): That seems super dangerous and I don't understand
 	// why we need to update resources just before we delete them.
+	// todo clean up this block as RemoteManifests is no longer used for deployer
 	if len(k.RemoteManifests) > 0 {
 		var rm manifest.ManifestList
 		for _, m := range k.RemoteManifests {
diff --git a/pkg/skaffold/deploy/kustomize/kustomize.go b/pkg/skaffold/deploy/kustomize/kustomize.go
index 2f147a04f..d002cde0b 100644
--- a/pkg/skaffold/deploy/kustomize/kustomize.go
+++ b/pkg/skaffold/deploy/kustomize/kustomize.go
@@ -369,7 +369,7 @@ func (k *Deployer) renderManifests(ctx context.Context, out io.Writer, builds []
 }
 
 // Cleanup deletes what was deployed by calling Deploy.
-func (k *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, list manifest.ManifestList) error {
+func (k *Deployer) Cleanup(ctx context.Context, out io.Writer, dryRun bool, _ *manifest.ManifestListByConfig) error {
 	instrumentation.AddAttributesToCurrentSpanFromContext(ctx, map[string]string{
 		"DeployerType": "kustomize",
 	})
diff --git a/pkg/skaffold/render/renderer/renderer.go b/pkg/skaffold/render/renderer/renderer.go
index 3985eac7c..aecb245ba 100644
--- a/pkg/skaffold/render/renderer/renderer.go
+++ b/pkg/skaffold/render/renderer/renderer.go
@@ -39,8 +39,8 @@ type Renderer interface {
 }
 
 // New creates a new Renderer object from the latestV2 API schema.
-func New(cfg render.Config, renderCfg latest.RenderConfig, hydrationDir string, labels map[string]string, usingLegacyHelmDeploy bool, configName string) (Renderer, error) {
-	if usingLegacyHelmDeploy {
+func New(cfg render.Config, renderCfg latest.RenderConfig, hydrationDir string, labels map[string]string, usingLegacyHelmDeploy bool, command string, configName string) (Renderer, error) {
+	if usingLegacyHelmDeploy && command != "render" {
 		return noop.New(renderCfg, cfg.GetWorkingDir(), hydrationDir, labels)
 	}
 	if renderCfg.Validate == nil && renderCfg.Transform == nil && renderCfg.Helm != nil {
diff --git a/pkg/skaffold/runner/renderer.go b/pkg/skaffold/runner/renderer.go
index 432d01fc9..20b23a269 100644
--- a/pkg/skaffold/runner/renderer.go
+++ b/pkg/skaffold/runner/renderer.go
@@ -29,7 +29,7 @@ func GetRenderer(ctx context.Context, runCtx *runcontext.RunContext, hydrationDi
 
 	var renderers []renderer.Renderer
 	for configName, p := range ps {
-		r, err := renderer.New(runCtx, p.Render, hydrationDir, labels, usingLegacyHelmDeploy, configName)
+		r, err := renderer.New(runCtx, p.Render, hydrationDir, labels, usingLegacyHelmDeploy, runCtx.Opts.Command, configName)
 		if err != nil {
 			return nil, err
 		}
diff --git a/pkg/skaffold/runner/runner.go b/pkg/skaffold/runner/runner.go
index e37ddba28..56053140f 100644
--- a/pkg/skaffold/runner/runner.go
+++ b/pkg/skaffold/runner/runner.go
@@ -35,7 +35,7 @@ type Runner interface {
 	Apply(context.Context, io.Writer) error
 	ApplyDefaultRepo(tag string) (string, error)
 	Build(context.Context, io.Writer, []*latest.Artifact) ([]graph.Artifact, error)
-	Cleanup(context.Context, io.Writer, bool, manifest.ManifestList) error
+	Cleanup(context.Context, io.Writer, bool, *manifest.ManifestListByConfig) error
 	Dev(context.Context, io.Writer, []*latest.Artifact) error
 	// Deploy and DeployAndLog: Do they need the `graph.Artifact` and could use render output.
 	Deploy(context.Context, io.Writer, []graph.Artifact, *manifest.ManifestListByConfig) error
diff --git a/pkg/skaffold/runner/timings.go b/pkg/skaffold/runner/timings.go
index f2a660f1b..6d2be1104 100644
--- a/pkg/skaffold/runner/timings.go
+++ b/pkg/skaffold/runner/timings.go
@@ -107,11 +107,11 @@ func (w withTimings) Deploy(ctx context.Context, out io.Writer, builds []graph.A
 	return err
 }
 
-func (w withTimings) Cleanup(ctx context.Context, out io.Writer, dryRun bool, list manifest.ManifestList) error {
+func (w withTimings) Cleanup(ctx context.Context, out io.Writer, dryRun bool, config *manifest.ManifestListByConfig) error {
 	start := time.Now()
 	output.Default.Fprintln(out, "Cleaning up...")
 
-	err := w.Deployer.Cleanup(ctx, out, dryRun, list)
+	err := w.Deployer.Cleanup(ctx, out, dryRun, config)
 	if err != nil {
 		return err
 	}
diff --git a/pkg/skaffold/runner/v2/cleanup.go b/pkg/skaffold/runner/v2/cleanup.go
index 729215738..1fb93e7ec 100644
--- a/pkg/skaffold/runner/v2/cleanup.go
+++ b/pkg/skaffold/runner/v2/cleanup.go
@@ -22,6 +22,6 @@ import (
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes/manifest"
 )
 
-func (r *SkaffoldRunner) Cleanup(ctx context.Context, out io.Writer, dryRun bool, list manifest.ManifestList) error {
-	return r.deployer.Cleanup(ctx, out, dryRun, list)
+func (r *SkaffoldRunner) Cleanup(ctx context.Context, out io.Writer, dryRun bool, manifestListByConfig *manifest.ManifestListByConfig) error {
+	return r.deployer.Cleanup(ctx, out, dryRun, manifestListByConfig)
 }


Test patch:
diff --git a/cmd/skaffold/app/cmd/dev_test.go b/cmd/skaffold/app/cmd/dev_test.go
index 6ab37b8ca..ff5d3347b 100644
--- a/cmd/skaffold/app/cmd/dev_test.go
+++ b/cmd/skaffold/app/cmd/dev_test.go
@@ -22,6 +22,7 @@ import (
 	"testing"
 
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/config"
+	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/graph"
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/kubernetes/manifest"
 	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/runner"
 	runcontext "github.com/GoogleContainerTools/skaffold/pkg/skaffold/runner/runcontext/v2"
@@ -32,10 +33,11 @@ import (
 
 type mockDevRunner struct {
 	runner.Runner
-	hasBuilt    bool
-	hasDeployed bool
-	errDev      error
-	calls       []string
+	hasBuilt          bool
+	hasDeployed       bool
+	errDev            error
+	calls             []string
+	manifestsByConfig *manifest.ManifestListByConfig
 }
 
 func (r *mockDevRunner) Dev(context.Context, io.Writer, []*latest.Artifact) error {
@@ -58,11 +60,16 @@ func (r *mockDevRunner) Prune(context.Context, io.Writer) error {
 	return nil
 }
 
-func (r *mockDevRunner) Cleanup(context.Context, io.Writer, bool, manifest.ManifestList) error {
+func (r *mockDevRunner) Cleanup(context.Context, io.Writer, bool, *manifest.ManifestListByConfig) error {
 	r.calls = append(r.calls, "Cleanup")
 	return nil
 }
 
+func (r *mockDevRunner) Render(ctx context.Context, out io.Writer, builds []graph.Artifact, offline bool) (*manifest.ManifestListByConfig, error) {
+	r.calls = append(r.calls, "Render")
+	return r.manifestsByConfig, nil
+}
+
 func TestDoDev(t *testing.T) {
 	tests := []struct {
 		description   string
@@ -74,7 +81,7 @@ func TestDoDev(t *testing.T) {
 			description:   "cleanup and then prune",
 			hasBuilt:      true,
 			hasDeployed:   true,
-			expectedCalls: []string{"Dev", "HasDeployed", "HasBuilt", "Cleanup", "Prune"},
+			expectedCalls: []string{"Dev", "HasDeployed", "HasBuilt", "Render", "Cleanup", "Prune"},
 		},
 		{
 			description:   "hasn't deployed",
@@ -139,7 +146,11 @@ func (m *mockConfigChangeRunner) Prune(context.Context, io.Writer) error {
 	return nil
 }
 
-func (m *mockConfigChangeRunner) Cleanup(context.Context, io.Writer, bool, manifest.ManifestList) error {
+func (m *mockConfigChangeRunner) Render(ctx context.Context, out io.Writer, builds []graph.Artifact, offline bool) (*manifest.ManifestListByConfig, error) {
+	return nil, nil
+}
+
+func (m *mockConfigChangeRunner) Cleanup(context.Context, io.Writer, bool, *manifest.ManifestListByConfig) error {
 	return nil
 }
 
diff --git a/integration/delete_test.go b/integration/delete_test.go
new file mode 100644
index 000000000..e37004b7b
--- /dev/null
+++ b/integration/delete_test.go
@@ -0,0 +1,73 @@
+/*
+Copyright 2022 The Skaffold Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package integration
+
+import (
+	"testing"
+	"time"
+
+	"github.com/GoogleContainerTools/skaffold/integration/skaffold"
+)
+
+func TestDelete(t *testing.T) {
+	var tests = []struct {
+		description string
+		dir         string
+		args        []string
+		pods        []string
+		deployments []string
+		env         []string
+	}{
+		{
+			description: "getting-started",
+			dir:         "testdata/getting-started",
+			pods:        []string{"getting-started"},
+		},
+		{
+			description: "microservices",
+			dir:         "examples/microservices",
+			args:        []string{"--status-check=false"},
+			deployments: []string{"leeroy-app", "leeroy-web"},
+		},
+		{
+			description: "multi-config-microservices",
+			dir:         "examples/multi-config-microservices",
+			deployments: []string{"leeroy-app", "leeroy-web"},
+		},
+		{
+			description: "multiple deployers",
+			dir:         "testdata/deploy-multiple",
+			pods:        []string{"deploy-kubectl", "deploy-kustomize"},
+		},
+	}
+
+	MarkIntegrationTest(t, CanRunWithoutGcp)
+
+	for _, test := range tests {
+		t.Run(test.description, func(t *testing.T) {
+			ns, client := SetupNamespace(t)
+
+			args := append(test.args, "--cache-artifacts=false")
+			skaffold.Run(args...).InDir(test.dir).InNs(ns.Name).WithEnv(test.env).RunOrFail(t)
+
+			client.WaitForPodsReady(test.pods...)
+			client.waitForDeploymentsToStabilizeWithTimeout(time.Minute*2, test.deployments...)
+
+			skaffold.Delete().InDir(test.dir).InNs(ns.Name).WithEnv(test.env).RunOrFail(t)
+		})
+	}
+}
diff --git a/pkg/skaffold/deploy/cloudrun/deploy_test.go b/pkg/skaffold/deploy/cloudrun/deploy_test.go
index c0b55254c..382ba5292 100644
--- a/pkg/skaffold/deploy/cloudrun/deploy_test.go
+++ b/pkg/skaffold/deploy/cloudrun/deploy_test.go
@@ -327,9 +327,11 @@ func TestCleanup(tOuter *testing.T) {
 			deployer, _ := NewDeployer(&runcontext.RunContext{}, &label.DefaultLabeller{}, &latest.CloudRunDeploy{ProjectID: test.defaultProject, Region: test.region}, configName)
 			deployer.clientOptions = append(deployer.clientOptions, option.WithEndpoint(ts.URL), option.WithoutAuthentication())
 			deployer.useGcpOptions = false
+			manifestListByConfig := manifest.NewManifestListByConfig()
 			manifest, _ := json.Marshal(test.toDelete)
 			manifests := [][]byte{manifest}
-			err := deployer.Cleanup(context.Background(), os.Stderr, false, manifests)
+			manifestListByConfig.Add(configName, manifests)
+			err := deployer.Cleanup(context.Background(), os.Stderr, false, &manifestListByConfig)
 			if test.httpErr == 0 && err != nil {
 				t.Fatalf("Expected success but got err: %v", err)
 			} else if test.httpErr != 0 && err == nil {
diff --git a/pkg/skaffold/deploy/deploy_mux_test.go b/pkg/skaffold/deploy/deploy_mux_test.go
index f6a4212f5..9f9fc6796 100644
--- a/pkg/skaffold/deploy/deploy_mux_test.go
+++ b/pkg/skaffold/deploy/deploy_mux_test.go
@@ -87,7 +87,7 @@ func (m *MockDeployer) Dependencies() ([]string, error) {
 	return m.dependencies, m.dependenciesErr
 }
 
-func (m *MockDeployer) Cleanup(context.Context, io.Writer, bool, manifest.ManifestList) error {
+func (m *MockDeployer) Cleanup(context.Context, io.Writer, bool, *manifest.ManifestListByConfig) error {
 	return m.cleanupErr
 }
 
diff --git a/pkg/skaffold/deploy/kubectl/kubectl_test.go b/pkg/skaffold/deploy/kubectl/kubectl_test.go
index b8a935441..a9edb4d47 100644
--- a/pkg/skaffold/deploy/kubectl/kubectl_test.go
+++ b/pkg/skaffold/deploy/kubectl/kubectl_test.go
@@ -337,7 +337,7 @@ func TestKubectlCleanup(t *testing.T) {
 				true)
 			t.CheckNoError(errR)
 
-			err = k.Cleanup(context.Background(), io.Discard, test.dryRun, m.GetForConfig(configName))
+			err = k.Cleanup(context.Background(), io.Discard, test.dryRun, m)
 
 			t.CheckError(test.shouldErr, err)
 		})
diff --git a/pkg/skaffold/render/generate/generate_test.go b/pkg/skaffold/render/generate/generate_test.go
index f15e39b2a..d638b6e5d 100644
--- a/pkg/skaffold/render/generate/generate_test.go
+++ b/pkg/skaffold/render/generate/generate_test.go
@@ -183,7 +183,7 @@ func TestGenerate(t *testing.T) {
 
 func TestGenerateFromURLManifest(t *testing.T) {
 	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		fmt.Fprintln(w, podYaml)
+		fmt.Fprint(w, podYaml)
 	}))
 
 	defer ts.Close()
diff --git a/pkg/skaffold/runner/timings_test.go b/pkg/skaffold/runner/timings_test.go
index 7553022e1..43ad2e134 100644
--- a/pkg/skaffold/runner/timings_test.go
+++ b/pkg/skaffold/runner/timings_test.go
@@ -100,7 +100,7 @@ func (m *mockDeployer) Deploy(context.Context, io.Writer, []graph.Artifact, *man
 	return nil
 }
 
-func (m *mockDeployer) Cleanup(context.Context, io.Writer, bool, manifest.ManifestList) error {
+func (m *mockDeployer) Cleanup(context.Context, io.Writer, bool, *manifest.ManifestListByConfig) error {
 	if m.err {
 		return errors.New("Unable to cleanup")
 	}
diff --git a/pkg/skaffold/runner/v2/runner_test.go b/pkg/skaffold/runner/v2/runner_test.go
index 5c16cc9a4..07feabdce 100644
--- a/pkg/skaffold/runner/v2/runner_test.go
+++ b/pkg/skaffold/runner/v2/runner_test.go
@@ -143,7 +143,7 @@ func (t *TestBench) TestDependencies(context.Context, *latest.Artifact) ([]strin
 	return nil, nil
 }
 func (t *TestBench) Dependencies() ([]string, error) { return nil, nil }
-func (t *TestBench) Cleanup(ctx context.Context, out io.Writer, dryRun bool, list manifest.ManifestList) error {
+func (t *TestBench) Cleanup(ctx context.Context, out io.Writer, dryRun bool, byConfig *manifest.ManifestListByConfig) error {
 	return nil
 }
 func (t *TestBench) Prune(ctx context.Context, out io.Writer) error { return nil }