Repository: GoogleContainerTools/skaffold
Pull request #7789 (2022-08-20 10:39:19)

feat: add toleration for GKE ARM nodes taint
<!-- Thank you for your contribution! -->

<!-- Include if applicable: -->
Fixes: #7779 <!-- tracking issues that this PR will close -->

**Description**
<!-- Describe your changes here. The more detail, the easier the review! -->
The GKE ARM nodes are tainted with `kubernetes.io/arch="arm64":NoSchedule` by default to prevent workloads from being scheduled. In this PR, Skaffold will automatically add the corresponding toleration to the manifest for ARM image Pods against GKE clusters. 
This can be turned off with the flag `--enable-gke-arm-node-toleration=false`.

**Testing Instruction**
* Use a GKE ARM node cluster (node pool version `1.23.7-gke.1400` or greater). Googlers can use [this](https://pantheon.corp.google.com/kubernetes/clusters/details/us-central1-a/integration-tests-hybird/details?project=k8s-skaffold) cluster
* In the `integration/examples/cross-platform-builds` project, with Skaffold built from `main` branch HEAD, run:
  ```
  skaffold dev --default-repo=gcr.io/k8s-skaffold --platform=linux/arm64
  ```
  This should fail due to the node Taint.
  ```
  Starting deploy...
   - pod/getting-started created
  Waiting for deployments to stabilize...
   - pods: Unschedulable: 0/1 nodes available: 
      - pod/getting-started: Unschedulable: 0/1 nodes available: 
   - pods: 0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 node(s) had taint {kubernetes.io/arch: arm64}, that the pod didn't tolerate.
      - pod/getting-started: 0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 node(s) had taint {kubernetes.io/arch: arm64}, that the pod didn't tolerate.
   - pods failed. Error: 0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 node(s) had taint {kubernetes.io/arch: arm64}, that the pod didn't tolerate..
  ```
* Now, build Skaffold from this PR branch and run:
  ```
  skaffold dev --default-repo=gcr.io/k8s-skaffold --platform=linux/arm64
  ```
  This should now succeed.
  ```
  Starting deploy...
   - pod/getting-started created
  Waiting for deployments to stabilize...
   - pods is ready.
  Deployments stabilized in 6.941 seconds
  Press Ctrl+C to exit
  Watching for changes...
  [getting-started] Hello world! Running on linux/arm64
  ```


codecov[bot]: # [Codecov](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) Report
> Merging [#7789](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (094d94e) into [main](https://codecov.io/gh/GoogleContainerTools/skaffold/commit/290280e8c6c19231e9ed5b5c027f2c01d22f75be?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) (290280e) will **decrease** coverage by `3.92%`.
> The diff coverage is `53.59%`.

```diff
@@            Coverage Diff             @@
##             main    #7789      +/-   ##
==========================================
- Coverage   70.48%   66.56%   -3.93%     
==========================================
  Files         515      591      +76     
  Lines       23150    28567    +5417     
==========================================
+ Hits        16317    19015    +2698     
- Misses       5776     8145    +2369     
- Partials     1057     1407     +350     
```


| [Impacted Files](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | Coverage Δ | |
|---|---|---|
| [cmd/skaffold/app/cmd/credits/export.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvY3JlZGl0cy9leHBvcnQuZ28=) | `0.00% <0.00%> (ø)` | |
| [cmd/skaffold/app/cmd/deploy.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZGVwbG95Lmdv) | `40.90% <0.00%> (-12.94%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/test.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvdGVzdC5nbw==) | `44.44% <0.00%> (ø)` | |
| [cmd/skaffold/skaffold.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL3NrYWZmb2xkLmdv) | `0.00% <0.00%> (ø)` | |
| [cmd/skaffold/app/cmd/inspect\_tests.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvaW5zcGVjdF90ZXN0cy5nbw==) | `62.50% <14.28%> (-1.14%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/render.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvcmVuZGVyLmdv) | `35.48% <18.18%> (-5.90%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/lsp.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvbHNwLmdv) | `28.12% <28.12%> (ø)` | |
| [cmd/skaffold/app/cmd/run.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvcnVuLmdv) | `64.28% <33.33%> (-9.63%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/fix.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvZml4Lmdv) | `56.41% <37.50%> (-20.07%)` | :arrow_down: |
| [cmd/skaffold/app/cmd/verify.go](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools#diff-Y21kL3NrYWZmb2xkL2FwcC9jbWQvdmVyaWZ5Lmdv) | `41.17% <41.17%> (ø)` | |
| ... and [360 more](https://codecov.io/gh/GoogleContainerTools/skaffold/pull/7789/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools) | |

:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=GoogleContainerTools)


Patch:
diff --git a/cmd/skaffold/app/cmd/flags.go b/cmd/skaffold/app/cmd/flags.go
index 9bb7584fb..b3b54ef4a 100644
--- a/cmd/skaffold/app/cmd/flags.go
+++ b/cmd/skaffold/app/cmd/flags.go
@@ -657,6 +657,15 @@ var flagRegistry = []Flag{
 		FlagAddMethod: "BoolVar",
 		DefinedOn:     []string{"dev", "render", "run", "debug", "deploy"},
 	},
+	{
+		Name:          "enable-gke-arm-node-toleration",
+		Usage:         "Setting this flag provides the appropriate toleration for Pods to be scheduled on GKE ARM nodes that are tainted to disallow workloads by default.",
+		Value:         &opts.EnableGKEARMNodeToleration,
+		DefValue:      true,
+		FlagAddMethod: "BoolVar",
+		DefinedOn:     []string{"dev", "render", "run", "debug", "deploy"},
+		Hidden:        true,
+	},
 	{
 		Name:     "disable-multi-platform-build",
 		Usage:    "When set to true, forces only single platform image builds even when multiple target platforms are specified. Enabled by default for `dev` and `debug` modes, to keep dev-loop fast",
diff --git a/docs-v2/content/en/schemas/v3alpha1.json b/docs-v2/content/en/schemas/v3alpha1.json
index 98dda3fc4..712fbd9b4 100755
--- a/docs-v2/content/en/schemas/v3alpha1.json
+++ b/docs-v2/content/en/schemas/v3alpha1.json
@@ -3420,15 +3420,6 @@
         "groupKind"
       ],
       "properties": {
-        "affinity": {
-          "items": {
-            "type": "string"
-          },
-          "type": "array",
-          "description": "an optional slice of JSON-path-like paths of where to add `affinity` definitions for scheduling Pods.",
-          "x-intellij-html-description": "an optional slice of JSON-path-like paths of where to add <code>affinity</code> definitions for scheduling Pods.",
-          "default": "[]"
-        },
         "groupKind": {
           "type": "string",
           "description": "compact format of a resource type.",
@@ -3451,13 +3442,22 @@
           "description": "an optional slice of JSON-path-like paths of where to add a labels block if missing.",
           "x-intellij-html-description": "an optional slice of JSON-path-like paths of where to add a labels block if missing.",
           "default": "[]"
+        },
+        "podSpec": {
+          "items": {
+            "type": "string"
+          },
+          "type": "array",
+          "description": "an optional slice of JSON-path-like paths of where pod spec properties can be overwritten.",
+          "x-intellij-html-description": "an optional slice of JSON-path-like paths of where pod spec properties can be overwritten.",
+          "default": "[]"
         }
       },
       "preferredOrder": [
         "groupKind",
         "image",
         "labels",
-        "affinity"
+        "podSpec"
       ],
       "additionalProperties": false,
       "type": "object",
diff --git a/pkg/skaffold/config/options.go b/pkg/skaffold/config/options.go
index 6a6036fc9..3d7bc05b4 100644
--- a/pkg/skaffold/config/options.go
+++ b/pkg/skaffold/config/options.go
@@ -59,6 +59,7 @@ type SkaffoldOptions struct {
 	Tail                       bool
 	WaitForConnection          bool
 	EnablePlatformNodeAffinity bool
+	EnableGKEARMNodeToleration bool
 	DisableMultiPlatformBuild  bool
 	CheckClusterNodePlatforms  bool
 	MakePathsAbsolute          *bool
diff --git a/pkg/skaffold/deploy/kubectl/cli.go b/pkg/skaffold/deploy/kubectl/cli.go
index bb5931833..15a8628e4 100644
--- a/pkg/skaffold/deploy/kubectl/cli.go
+++ b/pkg/skaffold/deploy/kubectl/cli.go
@@ -63,6 +63,7 @@ type Config interface {
 	PipelineForImage(imageName string) (latest.Pipeline, bool)
 	JSONParseConfig() latest.JSONParseConfig
 	EnablePlatformNodeAffinityInRenderedManifests() bool
+	EnableGKEARMNodeTolerationInRenderedManifests() bool
 }
 
 func NewCLI(cfg Config, flags latest.KubectlFlags, defaultNamespace string) CLI {
diff --git a/pkg/skaffold/kubernetes/manifest/affinity.go b/pkg/skaffold/kubernetes/manifest/affinity.go
index 1c12a0ab7..b5ab88307 100644
--- a/pkg/skaffold/kubernetes/manifest/affinity.go
+++ b/pkg/skaffold/kubernetes/manifest/affinity.go
@@ -45,21 +45,21 @@ const (
 	nodeArchitectureLabel    = "kubernetes.io/arch"
 )
 
-type ResourceSelectorAffinity struct {
+type ResourceSelectorPodSpec struct {
 	allowlist map[apimachinery.GroupKind]latest.ResourceFilter
 	denylist  map[apimachinery.GroupKind]latest.ResourceFilter
 }
 
-func NewResourceSelectorAffinity(allowlist map[apimachinery.GroupKind]latest.ResourceFilter, denylist map[apimachinery.GroupKind]latest.ResourceFilter) *ResourceSelectorAffinity {
-	return &ResourceSelectorAffinity{
+func NewResourceSelectorPodSpec(allowlist map[apimachinery.GroupKind]latest.ResourceFilter, denylist map[apimachinery.GroupKind]latest.ResourceFilter) *ResourceSelectorPodSpec {
+	return &ResourceSelectorPodSpec{
 		allowlist: allowlist,
 		denylist:  denylist,
 	}
 }
 
-func (rsa *ResourceSelectorAffinity) allowByGroupKind(gk apimachinery.GroupKind) bool {
-	if _, allowed := rsa.allowlist[gk]; allowed {
-		if rf, disallowed := rsa.denylist[gk]; disallowed {
+func (rs *ResourceSelectorPodSpec) allowByGroupKind(gk apimachinery.GroupKind) bool {
+	if _, allowed := rs.allowlist[gk]; allowed {
+		if rf, disallowed := rs.denylist[gk]; disallowed {
 			for _, s := range rf.Labels {
 				if s == ".*" {
 					return false
@@ -70,7 +70,7 @@ func (rsa *ResourceSelectorAffinity) allowByGroupKind(gk apimachinery.GroupKind)
 					return false
 				}
 			}
-			for _, s := range rf.Affinity {
+			for _, s := range rf.PodSpec {
 				if s == ".*" {
 					return false
 				}
@@ -82,15 +82,15 @@ func (rsa *ResourceSelectorAffinity) allowByGroupKind(gk apimachinery.GroupKind)
 }
 
 //nolint:unparam
-func (rsa *ResourceSelectorAffinity) allowByNavpath(gk apimachinery.GroupKind, navpath string, k string) (string, bool) {
+func (rs *ResourceSelectorPodSpec) allowByNavpath(gk apimachinery.GroupKind, navpath string, k string) (string, bool) {
 	for _, w := range ConfigConnectorResourceSelector {
 		if w.Matches(gk.Group, gk.Kind) {
 			return "", false
 		}
 	}
 
-	if rf, ok := rsa.denylist[gk]; ok {
-		for _, denypath := range rf.Affinity {
+	if rf, ok := rs.denylist[gk]; ok {
+		for _, denypath := range rf.PodSpec {
 			if denypath == ".*" {
 				return "", false
 			}
@@ -100,8 +100,8 @@ func (rsa *ResourceSelectorAffinity) allowByNavpath(gk apimachinery.GroupKind, n
 		}
 	}
 
-	if rf, ok := rsa.allowlist[gk]; ok {
-		for _, allowpath := range rf.Affinity {
+	if rf, ok := rs.allowlist[gk]; ok {
+		for _, allowpath := range rf.PodSpec {
 			if allowpath == ".*" {
 				if k != specField {
 					return "", false
diff --git a/pkg/skaffold/kubernetes/manifest/gke_toleration.go b/pkg/skaffold/kubernetes/manifest/gke_toleration.go
new file mode 100644
index 000000000..dec16f9ef
--- /dev/null
+++ b/pkg/skaffold/kubernetes/manifest/gke_toleration.go
@@ -0,0 +1,115 @@
+/*
+Copyright 2022 The Skaffold Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package manifest
+
+import (
+	"context"
+	"encoding/json"
+
+	"github.com/containerd/containerd/platforms"
+	spec "github.com/opencontainers/image-spec/specs-go/v1"
+	v1 "k8s.io/api/core/v1"
+	apimachinery "k8s.io/apimachinery/pkg/runtime/schema"
+
+	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/output/log"
+	"github.com/GoogleContainerTools/skaffold/pkg/skaffold/yaml"
+)
+
+const (
+	tolerationField = "tolerations"
+)
+
+var gkeARMToleration = v1.Toleration{
+	Key:      "kubernetes.io/arch",
+	Operator: v1.TolerationOpEqual,
+	Value:    "arm64",
+	Effect:   v1.TaintEffectNoSchedule,
+}
+
+// SetGKEARMToleration adds a toleration for GKE ARM node taint `kubernetes.io/arch="arm64":NoSchedule`.
+func (l *ManifestList) SetGKEARMToleration(ctx context.Context, rs ResourceSelector, platforms PodPlatforms) (ManifestList, error) {
+	r := &gkeTolerationSetter{ctx: ctx, platforms: platforms}
+	return l.Visit(r, rs)
+}
+
+type gkeTolerationSetter struct {
+	ctx       context.Context
+	platforms PodPlatforms
+}
+
+func (s *gkeTolerationSetter) Visit(gk apimachinery.GroupKind, navpath string, o map[string]interface{}, k string, v interface{}, rs ResourceSelector) bool {
+	if _, ok := rs.allowByNavpath(gk, navpath, k); !ok {
+		return true
+	}
+	if len(s.platforms) == 0 {
+		return false
+	}
+	pls, ok := s.platforms[navpath]
+	if !ok || len(pls) == 0 {
+		return true
+	}
+	matcher := platforms.Any(pls...)
+	if !matcher.Match(spec.Platform{OS: "linux", Architecture: "arm64"}) {
+		return false
+	}
+
+	spec, ok := v.(map[string]interface{})
+	if !ok {
+		return true
+	}
+
+	tolerations, err := addGKEARMToleration(spec[tolerationField])
+	if err != nil {
+		log.Entry(s.ctx).Debugf("failed to update spec tolerations: %s", err.Error())
+		return true
+	}
+	spec[tolerationField] = tolerations
+	return false
+}
+
+func addGKEARMToleration(data interface{}) ([]interface{}, error) {
+	var tolerations []v1.Toleration
+	if data != nil {
+		b, err := json.Marshal(data)
+		if err != nil {
+			return nil, err
+		}
+		if err = json.Unmarshal(b, &tolerations); err != nil {
+			return nil, err
+		}
+	}
+	exists := false
+	for _, t := range tolerations {
+		if t == gkeARMToleration {
+			exists = true
+			break
+		}
+	}
+	if !exists {
+		tolerations = append(tolerations, gkeARMToleration)
+	}
+	bytes, err := json.Marshal(tolerations)
+	if err != nil {
+		return nil, err
+	}
+
+	var sl []interface{}
+	if err := yaml.Unmarshal(bytes, &sl); err != nil {
+		return nil, err
+	}
+	return sl, err
+}
diff --git a/pkg/skaffold/kubernetes/manifest/visitor.go b/pkg/skaffold/kubernetes/manifest/visitor.go
index 7dca96d89..3ba7229c1 100644
--- a/pkg/skaffold/kubernetes/manifest/visitor.go
+++ b/pkg/skaffold/kubernetes/manifest/visitor.go
@@ -38,67 +38,67 @@ var TransformAllowlist = map[apimachinery.GroupKind]latest.ResourceFilter{
 		GroupKind: "Pod",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "", Kind: "Service"}: {
 		GroupKind: "Service",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "apps", Kind: "DaemonSet"}: {
 		GroupKind: "DaemonSet.apps",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "apps", Kind: "Deployment"}: {
 		GroupKind: "Deployment.apps",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "apps", Kind: "ReplicaSet"}: {
 		GroupKind: "ReplicaSet.apps",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "apps", Kind: "StatefulSet"}: {
 		GroupKind: "StatefulSet.apps",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "batch", Kind: "CronJob"}: {
 		GroupKind: "CronJob.batch",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "batch", Kind: "Job"}: {
 		GroupKind: "Job.batch",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "extensions", Kind: "DaemonSet"}: {
 		GroupKind: "DaemonSet.extensions",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "extensions", Kind: "Deployment"}: {
 		GroupKind: "Deployment.extensions",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	{Group: "extensions", Kind: "ReplicaSet"}: {
 		GroupKind: "ReplicaSet.extensions",
 		Image:     []string{".*"},
 		Labels:    []string{".*"},
-		Affinity:  []string{".*"},
+		PodSpec:   []string{".*"},
 	},
 	// TODO: Investigate exact requirements for adding `affinity` definitions for the following custom resource kinds
 	{Group: "serving.knative.dev", Kind: "Service"}: {
diff --git a/pkg/skaffold/render/config.go b/pkg/skaffold/render/config.go
index 4eeeb6f36..f4833ec4a 100644
--- a/pkg/skaffold/render/config.go
+++ b/pkg/skaffold/render/config.go
@@ -32,6 +32,7 @@ type Config interface {
 	GetNamespace() string
 	Mode() config.RunMode
 	EnablePlatformNodeAffinityInRenderedManifests() bool
+	EnableGKEARMNodeTolerationInRenderedManifests() bool
 }
 
 type MockConfig struct {
@@ -48,4 +49,5 @@ func (mc MockConfig) GetKubeConfig() string                               { retu
 func (mc MockConfig) GetKubeContext() string                              { return "" }
 func (mc MockConfig) Mode() config.RunMode                                { return "" }
 func (mc MockConfig) EnablePlatformNodeAffinityInRenderedManifests() bool { return true }
+func (mc MockConfig) EnableGKEARMNodeTolerationInRenderedManifests() bool { return true }
 func (mc MockConfig) GetNamespace() string                                { return mc.Namespace }
diff --git a/pkg/skaffold/render/renderer/kpt/kpt.go b/pkg/skaffold/render/renderer/kpt/kpt.go
index 68ec3866a..c5b8b0819 100644
--- a/pkg/skaffold/render/renderer/kpt/kpt.go
+++ b/pkg/skaffold/render/renderer/kpt/kpt.go
@@ -154,7 +154,9 @@ func (r *Kpt) Render(ctx context.Context, out io.Writer, builds []graph.Artifact
 		TransformAllowList:         r.transformAllowlist,
 		TransformDenylist:          r.transformDenylist,
 		EnablePlatformNodeAffinity: r.cfg.EnablePlatformNodeAffinityInRenderedManifests(),
+		EnableGKEARMNodeToleration: r.cfg.EnableGKEARMNodeTolerationInRenderedManifests(),
 		Offline:                    offline,
+		KubeContext:                r.cfg.GetKubeContext(),
 	}
 	manifests, errH := rUtil.GenerateHydratedManifests(ctx, out, builds, r.Generator, r.labels, r.namespace, opts)
 	if errH != nil {
diff --git a/pkg/skaffold/render/renderer/kubectl/kubectl.go b/pkg/skaffold/render/renderer/kubectl/kubectl.go
index 99598dec0..7aa48e595 100644
--- a/pkg/skaffold/render/renderer/kubectl/kubectl.go
+++ b/pkg/skaffold/render/renderer/kubectl/kubectl.go
@@ -73,7 +73,9 @@ func (r Kubectl) Render(ctx context.Context, out io.Writer, builds []graph.Artif
 		TransformAllowList:         r.transformAllowlist,
 		TransformDenylist:          r.transformDenylist,
 		EnablePlatformNodeAffinity: r.cfg.EnablePlatformNodeAffinityInRenderedManifests(),
+		EnableGKEARMNodeToleration: r.cfg.EnableGKEARMNodeTolerationInRenderedManifests(),
 		Offline:                    offline,
+		KubeContext:                r.cfg.GetKubeContext(),
 	}
 	manifests, err := util.GenerateHydratedManifests(ctx, out, builds, r.Generator, r.labels, r.namespace, opts)
 	endTrace()
diff --git a/pkg/skaffold/render/renderer/util/util.go b/pkg/skaffold/render/renderer/util/util.go
index 8d613b274..074f63706 100644
--- a/pkg/skaffold/render/renderer/util/util.go
+++ b/pkg/skaffold/render/renderer/util/util.go
@@ -37,7 +37,9 @@ type GenerateHydratedManifestsOptions struct {
 	TransformAllowList         map[apim.GroupKind]latest.ResourceFilter
 	TransformDenylist          map[apim.GroupKind]latest.ResourceFilter
 	EnablePlatformNodeAffinity bool
+	EnableGKEARMNodeToleration bool
 	Offline                    bool
+	KubeContext                string
 }
 
 func GenerateHydratedManifests(ctx context.Context, out io.Writer, builds []graph.Artifact, g generate.Generator, labels map[string]string, ns string, opts GenerateHydratedManifestsOptions) (manifest.ManifestList, error) {
@@ -69,6 +71,19 @@ func GenerateHydratedManifests(ctx context.Context, out io.Writer, builds []grap
 		}
 		endTrace()
 	}
+	var platforms manifest.PodPlatforms
+
+	if opts.EnableGKEARMNodeToleration && isGKECluster(opts.KubeContext) {
+		rCtx, endTrace = instrumentation.StartTrace(ctx, "Render_setGKEARMToleration")
+		platforms, err = manifests.GetImagePlatforms(rCtx, manifest.NewResourceSelectorImages(opts.TransformAllowList, opts.TransformDenylist))
+		if err != nil {
+			return nil, err
+		}
+		if manifests, err = manifests.SetGKEARMToleration(rCtx, manifest.NewResourceSelectorPodSpec(opts.TransformAllowList, opts.TransformDenylist), platforms); err != nil {
+			return nil, err
+		}
+		endTrace()
+	}
 
 	if !opts.EnablePlatformNodeAffinity {
 		// TODO (gaghosh): To support platform node affinity in offline mode, we'll need to save the image platform
@@ -76,11 +91,13 @@ func GenerateHydratedManifests(ctx context.Context, out io.Writer, builds []grap
 		return manifests, nil
 	}
 	rCtx, endTrace = instrumentation.StartTrace(ctx, "Render_setPlatformNodeAffinity")
-	platforms, err := manifests.GetImagePlatforms(rCtx, manifest.NewResourceSelectorImages(opts.TransformAllowList, opts.TransformDenylist))
-	if err != nil {
-		return nil, err
+	if platforms == nil {
+		platforms, err = manifests.GetImagePlatforms(rCtx, manifest.NewResourceSelectorImages(opts.TransformAllowList, opts.TransformDenylist))
+		if err != nil {
+			return nil, err
+		}
 	}
-	if manifests, err = manifests.SetPlatformNodeAffinity(rCtx, manifest.NewResourceSelectorAffinity(opts.TransformAllowList, opts.TransformDenylist), platforms); err != nil {
+	if manifests, err = manifests.SetPlatformNodeAffinity(rCtx, manifest.NewResourceSelectorPodSpec(opts.TransformAllowList, opts.TransformDenylist), platforms); err != nil {
 		return nil, err
 	}
 	endTrace()
@@ -150,17 +167,17 @@ func ConsolidateTransformConfiguration(cfg render.Config) (map[apim.GroupKind]la
 func ConvertJSONPathIndex(rf latest.ResourceFilter) latest.ResourceFilter {
 	nrf := latest.ResourceFilter{}
 	nrf.GroupKind = rf.GroupKind
-	if len(rf.Affinity) > 0 {
-		naffinity := []string{}
-		for _, str := range rf.Affinity {
+	if len(rf.PodSpec) > 0 {
+		nspec := []string{}
+		for _, str := range rf.PodSpec {
 			if str == ".*" {
-				naffinity = append(naffinity, str)
+				nspec = append(nspec, str)
 				continue
 			}
 			nstr := strings.ReplaceAll(str, ".*", "")
-			naffinity = append(naffinity, nstr)
+			nspec = append(nspec, nstr)
 		}
-		nrf.Affinity = naffinity
+		nrf.PodSpec = nspec
 	}
 
 	if len(rf.Labels) > 0 {
@@ -191,3 +208,5 @@ func ConvertJSONPathIndex(rf latest.ResourceFilter) latest.ResourceFilter {
 
 	return nrf
 }
+
+func isGKECluster(kubeContext string) bool { return strings.HasPrefix(kubeContext, "gke") }
diff --git a/pkg/skaffold/runner/runcontext/context.go b/pkg/skaffold/runner/runcontext/context.go
index 3c876742c..1fbaa7421 100644
--- a/pkg/skaffold/runner/runcontext/context.go
+++ b/pkg/skaffold/runner/runcontext/context.go
@@ -284,6 +284,9 @@ func (rc *RunContext) JSONParseConfig() latest.JSONParseConfig {
 func (rc *RunContext) EnablePlatformNodeAffinityInRenderedManifests() bool {
 	return rc.Opts.EnablePlatformNodeAffinity && rc.Cluster.IsMixedPlatform
 }
+func (rc *RunContext) EnableGKEARMNodeTolerationInRenderedManifests() bool {
+	return rc.Opts.EnableGKEARMNodeToleration
+}
 
 // GetRenderConfig returns the top tier RenderConfig.
 // TODO: design how to support multi-module.


Test patch:
diff --git a/pkg/skaffold/kubernetes/manifest/affinity_test.go b/pkg/skaffold/kubernetes/manifest/affinity_test.go
index d0634953d..566223e7f 100644
--- a/pkg/skaffold/kubernetes/manifest/affinity_test.go
+++ b/pkg/skaffold/kubernetes/manifest/affinity_test.go
@@ -428,7 +428,7 @@ spec:
 				return test.platforms[image], nil
 			})
 			ml := ManifestList{[]byte(test.manifest)}
-			m, err := ml.SetPlatformNodeAffinity(context.Background(), NewResourceSelectorAffinity(TransformAllowlist, TransformDenylist), test.platforms)
+			m, err := ml.SetPlatformNodeAffinity(context.Background(), NewResourceSelectorPodSpec(TransformAllowlist, TransformDenylist), test.platforms)
 			t.CheckNoError(err)
 			expected := ManifestList{[]byte(test.expected)}
 			t.CheckDeepEqual(expected.String(), m.String())
diff --git a/pkg/skaffold/kubernetes/manifest/gke_toleration_test.go b/pkg/skaffold/kubernetes/manifest/gke_toleration_test.go
new file mode 100644
index 000000000..6ea4176ef
--- /dev/null
+++ b/pkg/skaffold/kubernetes/manifest/gke_toleration_test.go
@@ -0,0 +1,213 @@
+/*
+Copyright 2019 The Skaffold Authors
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package manifest
+
+import (
+	"context"
+	"testing"
+
+	spec "github.com/opencontainers/image-spec/specs-go/v1"
+
+	"github.com/GoogleContainerTools/skaffold/testutil"
+)
+
+func TestSetGKEARMToleration(t *testing.T) {
+	tests := []struct {
+		description string
+		manifest    string
+		platforms   PodPlatforms
+		expected    string
+	}{
+		{
+			description: "ARM architecture image; no existing tolerations",
+			manifest: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example`,
+			platforms: map[string][]spec.Platform{
+				".spec": {{Architecture: "amd64", OS: "linux"}, {Architecture: "arm64", OS: "linux"}},
+			},
+			expected: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example
+  tolerations:
+  - effect: NoSchedule
+    key: kubernetes.io/arch
+    operator: Equal
+    value: arm64`,
+		},
+		{
+			description: "ARM architecture image; existing tolerations",
+			manifest: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example
+  tolerations:
+  - key: foo
+    operator: Equal
+    value: bar
+    effect: NoSchedule`,
+			platforms: map[string][]spec.Platform{
+				".spec": {{Architecture: "amd64", OS: "linux"}, {Architecture: "arm64", OS: "linux"}},
+			},
+			expected: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example
+  tolerations:
+  - effect: NoSchedule
+    key: foo
+    operator: Equal
+    value: bar
+  - effect: NoSchedule
+    key: kubernetes.io/arch
+    operator: Equal
+    value: arm64`,
+		},
+		{
+			description: "ARM architecture image; existing ARM toleration",
+			manifest: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example
+  tolerations:
+  - key: foo
+    operator: Equal
+    value: bar
+    effect: NoSchedule
+  - key: kubernetes.io/arch
+    operator: Equal
+    value: arm64
+    effect: NoSchedule`,
+			platforms: map[string][]spec.Platform{
+				".spec": {{Architecture: "amd64", OS: "linux"}, {Architecture: "arm64", OS: "linux"}},
+			},
+			expected: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example
+  tolerations:
+  - effect: NoSchedule
+    key: foo
+    operator: Equal
+    value: bar
+  - effect: NoSchedule
+    key: kubernetes.io/arch
+    operator: Equal
+    value: arm64`,
+		},
+		{
+			description: "non-ARM architecture image; no existing tolerations",
+			manifest: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example`,
+			platforms: map[string][]spec.Platform{
+				".spec": {{Architecture: "amd64", OS: "linux"}},
+			},
+			expected: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example`,
+		},
+		{
+			description: "non-ARM architecture image; existing tolerations",
+			manifest: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example
+  tolerations:
+  - key: foo
+    operator: Equal
+    value: bar
+    effect: NoSchedule`,
+			platforms: map[string][]spec.Platform{
+				".spec": {{Architecture: "amd64", OS: "linux"}},
+			},
+			expected: `
+apiVersion: v1
+kind: Pod
+metadata:
+  name: getting-started
+spec:
+  containers:
+  - image: gcr.io/k8s-skaffold/example:latest
+    name: example
+  tolerations:
+  - effect: NoSchedule
+    key: foo
+    operator: Equal
+    value: bar`,
+		},
+	}
+
+	for _, test := range tests {
+		testutil.Run(t, test.description, func(t *testutil.T) {
+			ml := ManifestList{[]byte(test.manifest)}
+			m, err := ml.SetGKEARMToleration(context.Background(), NewResourceSelectorPodSpec(TransformAllowlist, TransformDenylist), test.platforms)
+			t.CheckNoError(err)
+			expected := ManifestList{[]byte(test.expected)}
+			t.CheckDeepEqual(expected.String(), m.String())
+		})
+	}
+}
diff --git a/pkg/skaffold/schema/latest/config.go b/pkg/skaffold/schema/latest/config.go
index 6b263b257..2ca997d5f 100644
--- a/pkg/skaffold/schema/latest/config.go
+++ b/pkg/skaffold/schema/latest/config.go
@@ -1573,8 +1573,8 @@ type ResourceFilter struct {
 	Image []string `yaml:"image,omitempty"`
 	// Labels is an optional slice of JSON-path-like paths of where to add a labels block if missing.
 	Labels []string `yaml:"labels,omitempty"`
-	// Affinity is an optional slice of JSON-path-like paths of where to add `affinity` definitions for scheduling Pods.
-	Affinity []string `yaml:"affinity,omitempty"`
+	// PodSpec is an optional slice of JSON-path-like paths of where pod spec properties can be overwritten.
+	PodSpec []string `yaml:"podSpec,omitempty"`
 }
 
 // UnmarshalYAML provides a custom unmarshaller to deal with