Repository: kubernetes-sigs/node-feature-discovery
Pull request #336 (2020-08-20 11:07:45)

nfd-master: patch node object instead of rewriting it
When updating node labels and annotations use JSON patches instead of
doing a read-modify-write on the whole node object. Patching is already
being used in managing extended resources so some of the existing code
was re-usable. This should mitigate the problem of node update failures caused by
race conditions (a change in the node object between our read and write)
resulting e.g. in errors/restarts in nfd worker pods.

This patchset also slightly improves the unit tests.

Addresses #122

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/node-feature-discovery/pull/336#" title="Author self-approved">marquiz</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fnode-feature-discovery).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/node-feature-discovery/blob/master/OWNERS)~~ [marquiz]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
marquiz: Rebased
marquiz: /assign @Ethyling 
k8s-ci-robot: @marquiz: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-node-feature-discovery-build-gh-pages | db3c82973f3fadf7c0c86ee12387f53363f36cc1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_node-feature-discovery/336/pull-node-feature-discovery-build-gh-pages/1319246810086117376) | `/test pull-node-feature-discovery-build-gh-pages`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=node-feature-discovery&pr=336). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Amarquiz). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/node-feature-discovery/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
ArangoGutierrez: so much water has gone below the bridge after this PR....
We need to rebase it on top of master 
k8s-ci-robot: @marquiz: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
Ethyling: Seems to be working as expected
Side note: The ClusterRole in `nfd-daemonset-combined.yaml.template` is not complete compared to the one in `nfd-master.yaml.template` and miss the `nodes/status` comment and the `list` verb to prune the node. Should we fix this in this PR or do another PR?
marquiz: > Side note: The ClusterRole in `nfd-daemonset-combined.yaml.template` is not complete compared to the one in `nfd-master.yaml.template` and miss the `nodes/status` comment and the `list` verb to prune the node. Should we fix this in this PR or do another PR?

Thanks for pointing this out. I submitted #400 to address that

ArangoGutierrez: What's the status of this discussion? do we need more changes or is this good to roll in?
marquiz: > What's the status of this discussion? do we need more changes or is this good to roll in?

From testing point of view it should be good to go. It's been tested by me, @uniemimu and @Ethyling 

Code review is missing, I suppose
Ethyling: I will try to do a final review tomorrow and then we can merge it I think
ArangoGutierrez: I am getting , on kubernetes v1.18.9 , KOPS deployed 
```
Nov 23 15:43:39.578: INFO: Waiting up to 3m0s for all (but 0) nodes to be ready
STEP: Destroying namespace "node-feature-discovery-3939" for this suite.

• Failure [10.500 seconds]
[k8s.io] [NFD] Node Feature Discovery
/home/eduardo/go/pkg/mod/k8s.io/kubernetes@v1.19.4/test/e2e/framework/framework.go:592
  when deploying a single nfd-master pod
  /home/eduardo/sdk/github/kubernetes-sigs/node-feature-discovery/test/e2e/node_feature_discovery.go:414
    and a single worker pod with fake source enabled
    /home/eduardo/sdk/github/kubernetes-sigs/node-feature-discovery/test/e2e/node_feature_discovery.go:448
      it should decorate the node with the fake feature labels [It]
      /home/eduardo/sdk/github/kubernetes-sigs/node-feature-discovery/test/e2e/node_feature_discovery.go:449

      Nov 23 15:43:35.635: Expected
          <string>: 
      to equal
          <string>: true

      /home/eduardo/sdk/github/kubernetes-sigs/node-feature-discovery/test/e2e/node_feature_discovery.go:476
------------------------------
{"msg":"FAILED [k8s.io] [NFD] Node Feature Discovery when deploying a single nfd-master pod and a single worker pod with fake source enabled it should decorate the node with the fake feature labels","total":2,"completed":0,"skipped":73,"failed":1,"failures":["[k8s.io] [NFD] Node Feature Discovery when deploying a single nfd-master pod and a single worker pod with fake source enabled it should decorate the node with the fake feature labels"]}
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSNov 23 15:43:39.759: INFO: Running AfterSuite actions on all nodes
Nov 23 15:43:39.759: INFO: Running AfterSuite actions on node 1
{"msg":"Test Suite completed","total":2,"completed":0,"skipped":218,"failed":1,"failures":["[k8s.io] [NFD] Node Feature Discovery when deploying a single nfd-master pod and a single worker pod with fake source enabled it should decorate the node with the fake feature labels"]}


Summarizing 1 Failure:

[Fail] [k8s.io] [NFD] Node Feature Discovery when deploying a single nfd-master pod and a single worker pod with fake source enabled [It] it should decorate the node with the fake feature labels 
/home/eduardo/sdk/github/kubernetes-sigs/node-feature-discovery/test/e2e/node_feature_discovery.go:476

Ran 1 of 219 Specs in 15.568 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 218 Skipped
--- FAIL: TestE2E (15.57s)
FAIL
FAIL    sigs.k8s.io/node-feature-discovery/test/e2e     15.645s
FAIL
make: *** [Makefile:107: e2e-test] Error 1
```
ArangoGutierrez: Once we rebase on #403 I think this PR is good to go
marquiz: > Once we rebase on #403 I think this PR is good to go

Hmm, this is a different one 🤔 

/hold

Patch:
diff --git a/cmd/nfd-master/main.go b/cmd/nfd-master/main.go
index caf1e03f0..5247477d0 100644
--- a/cmd/nfd-master/main.go
+++ b/cmd/nfd-master/main.go
@@ -122,7 +122,10 @@ func argsParse(argv []string) (master.Args, error) {
 		return args, fmt.Errorf("error parsing whitelist regex (%s): %s", arguments["--label-whitelist"], err)
 	}
 	args.VerifyNodeName = arguments["--verify-node-name"].(bool)
-	args.ExtraLabelNs = strings.Split(arguments["--extra-label-ns"].(string), ",")
+	args.ExtraLabelNs = map[string]struct{}{}
+	for _, n := range strings.Split(arguments["--extra-label-ns"].(string), ",") {
+		args.ExtraLabelNs[n] = struct{}{}
+	}
 	args.ResourceLabels = strings.Split(arguments["--resource-labels"].(string), ",")
 	args.Prune = arguments["--prune"].(bool)
 	args.Kubeconfig = arguments["--kubeconfig"].(string)
diff --git a/go.mod b/go.mod
index bf92484f7..b8e803e7d 100644
--- a/go.mod
+++ b/go.mod
@@ -8,6 +8,7 @@ require (
 	github.com/klauspost/cpuid v1.2.3
 	github.com/onsi/ginkgo v1.11.0
 	github.com/onsi/gomega v1.7.0
+	github.com/smartystreets/assertions v1.2.0
 	github.com/smartystreets/goconvey v1.6.4
 	github.com/stretchr/testify v1.4.0
 	github.com/vektra/errors v0.0.0-20140903201135-c64d83aba85a
diff --git a/go.sum b/go.sum
index 913d5391c..1efa099fe 100644
--- a/go.sum
+++ b/go.sum
@@ -572,6 +572,8 @@ github.com/sirupsen/logrus v1.6.0 h1:UBcNElsrwanuuMsnGSlYmtmgbb23qDR5dG+6X6Oo89I
 github.com/sirupsen/logrus v1.6.0/go.mod h1:7uNnSEd1DgxDLC74fIahvMZmmYsHGZGEOFrfsX/uA88=
 github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
 github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
+github.com/smartystreets/assertions v1.2.0 h1:42S6lae5dvLc7BrLu/0ugRtcFVjoJNMC/N3yZFZkDFs=
+github.com/smartystreets/assertions v1.2.0/go.mod h1:tcbTF8ujkAEcZ8TElKY+i30BzYlVhC/LOxJk7iOWnoo=
 github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
 github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
 github.com/soheilhy/cmux v0.1.4 h1:0HKaf1o97UwFjHH9o5XsHUOF+tqmdA7KEzXLpiyaw0E=
diff --git a/pkg/apihelper/apihelpers.go b/pkg/apihelper/apihelpers.go
index 59e1b285e..8a7edc98e 100644
--- a/pkg/apihelper/apihelpers.go
+++ b/pkg/apihelper/apihelpers.go
@@ -35,6 +35,9 @@ type APIHelpers interface {
 	// UpdateNode updates the node via the API server using a client.
 	UpdateNode(*k8sclient.Clientset, *api.Node) error
 
-	// PatchStatus updates the node status via the API server using a client.
-	PatchStatus(*k8sclient.Clientset, string, interface{}) error
+	// PatchNode updates the node object via the API server using a client.
+	PatchNode(*k8sclient.Clientset, string, []JsonPatch) error
+
+	// PatchNodeStatus updates the node status via the API server using a client.
+	PatchNodeStatus(*k8sclient.Clientset, string, []JsonPatch) error
 }
diff --git a/pkg/apihelper/jsonpatch.go b/pkg/apihelper/jsonpatch.go
new file mode 100644
index 000000000..0939cc8b6
--- /dev/null
+++ b/pkg/apihelper/jsonpatch.go
@@ -0,0 +1,34 @@
+/*
+Copyright 2020 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package apihelper
+
+import (
+	"path/filepath"
+	"strings"
+)
+
+// JsonPatch is a json marshaling helper used for patching API objects
+type JsonPatch struct {
+	Op    string `json:"op"`
+	Path  string `json:"path"`
+	Value string `json:"value,omitempty"`
+}
+
+// NewJsonPatch returns a new JsonPatch object
+func NewJsonPatch(verb string, path string, key string, value string) JsonPatch {
+	return JsonPatch{verb, filepath.Join(path, strings.ReplaceAll(key, "/", "~1")), value}
+}
diff --git a/pkg/apihelper/k8shelpers.go b/pkg/apihelper/k8shelpers.go
index 8a4186065..d029b02c0 100644
--- a/pkg/apihelper/k8shelpers.go
+++ b/pkg/apihelper/k8shelpers.go
@@ -78,12 +78,25 @@ func (h K8sHelpers) UpdateNode(c *k8sclient.Clientset, n *api.Node) error {
 	return nil
 }
 
-func (h K8sHelpers) PatchStatus(c *k8sclient.Clientset, nodeName string, marshalable interface{}) error {
-	// Send the updated node to the apiserver.
-	patch, err := json.Marshal(marshalable)
-	if err == nil {
-		_, err = c.CoreV1().Nodes().Patch(context.TODO(), nodeName, types.JSONPatchType, patch, meta_v1.PatchOptions{}, "status")
+func (h K8sHelpers) PatchNode(c *k8sclient.Clientset, nodeName string, patches []JsonPatch) error {
+	if len(patches) > 0 {
+		data, err := json.Marshal(patches)
+		if err == nil {
+			_, err = c.CoreV1().Nodes().Patch(context.TODO(), nodeName, types.JSONPatchType, data, meta_v1.PatchOptions{})
+		}
+		return err
+	}
+	return nil
+}
+
+func (h K8sHelpers) PatchNodeStatus(c *k8sclient.Clientset, nodeName string, patches []JsonPatch) error {
+	if len(patches) > 0 {
+		data, err := json.Marshal(patches)
+		if err == nil {
+			_, err = c.CoreV1().Nodes().Patch(context.TODO(), nodeName, types.JSONPatchType, data, meta_v1.PatchOptions{}, "status")
+		}
+		return err
 	}
+	return nil
 
-	return err
 }
diff --git a/pkg/apihelper/mock_APIHelpers.go b/pkg/apihelper/mock_APIHelpers.go
index d79f56326..136a618fc 100644
--- a/pkg/apihelper/mock_APIHelpers.go
+++ b/pkg/apihelper/mock_APIHelpers.go
@@ -85,12 +85,26 @@ func (_m *MockAPIHelpers) GetNodes(_a0 *kubernetes.Clientset) (*v1.NodeList, err
 	return r0, r1
 }
 
-// PatchStatus provides a mock function with given fields: _a0, _a1, _a2
-func (_m *MockAPIHelpers) PatchStatus(_a0 *kubernetes.Clientset, _a1 string, _a2 interface{}) error {
+// PatchNode provides a mock function with given fields: _a0, _a1, _a2
+func (_m *MockAPIHelpers) PatchNode(_a0 *kubernetes.Clientset, _a1 string, _a2 []JsonPatch) error {
 	ret := _m.Called(_a0, _a1, _a2)
 
 	var r0 error
-	if rf, ok := ret.Get(0).(func(*kubernetes.Clientset, string, interface{}) error); ok {
+	if rf, ok := ret.Get(0).(func(*kubernetes.Clientset, string, []JsonPatch) error); ok {
+		r0 = rf(_a0, _a1, _a2)
+	} else {
+		r0 = ret.Error(0)
+	}
+
+	return r0
+}
+
+// PatchNodeStatus provides a mock function with given fields: _a0, _a1, _a2
+func (_m *MockAPIHelpers) PatchNodeStatus(_a0 *kubernetes.Clientset, _a1 string, _a2 []JsonPatch) error {
+	ret := _m.Called(_a0, _a1, _a2)
+
+	var r0 error
+	if rf, ok := ret.Get(0).(func(*kubernetes.Clientset, string, []JsonPatch) error); ok {
 		r0 = rf(_a0, _a1, _a2)
 	} else {
 		r0 = ret.Error(0)
diff --git a/pkg/nfd-master/nfd-master.go b/pkg/nfd-master/nfd-master.go
index 66ae9799b..c58d78d18 100644
--- a/pkg/nfd-master/nfd-master.go
+++ b/pkg/nfd-master/nfd-master.go
@@ -24,6 +24,7 @@ import (
 	"log"
 	"net"
 	"os"
+	"path/filepath"
 	"regexp"
 	"sort"
 	"strconv"
@@ -42,10 +43,16 @@ import (
 
 const (
 	// Namespace for feature labels
-	LabelNs = "feature.node.kubernetes.io/"
+	LabelNs = "feature.node.kubernetes.io"
 
 	// Namespace for all NFD-related annotations
-	AnnotationNs = "nfd.node.kubernetes.io/"
+	AnnotationNs = "nfd.node.kubernetes.io"
+
+	// NFD Annotations
+	extendedResourceAnnotation = AnnotationNs + "/extended-resources"
+	featureLabelAnnotation     = AnnotationNs + "/feature-labels"
+	masterVersionAnnotation    = AnnotationNs + "/master.version"
+	workerVersionAnnotation    = AnnotationNs + "/worker.version"
 )
 
 // package loggers
@@ -68,7 +75,7 @@ type Annotations map[string]string
 type Args struct {
 	CaFile         string
 	CertFile       string
-	ExtraLabelNs   []string
+	ExtraLabelNs   map[string]struct{}
 	KeyFile        string
 	Kubeconfig     string
 	LabelWhiteList *regexp.Regexp
@@ -92,21 +99,6 @@ type nfdMaster struct {
 	apihelper apihelper.APIHelpers
 }
 
-// statusOp is a json marshaling helper used for patching node status
-type statusOp struct {
-	Op    string `json:"op"`
-	Path  string `json:"path"`
-	Value string `json:"value,omitempty"`
-}
-
-func createStatusOp(verb string, resource string, path string, value string) statusOp {
-	if !strings.Contains(resource, "/") {
-		resource = LabelNs + resource
-	}
-	res := strings.ReplaceAll(resource, "/", "~1")
-	return statusOp{verb, "/status/" + path + "/" + res, value}
-}
-
 // Create new NfdMaster server instance.
 func NewNfdMaster(args Args) (NfdMaster, error) {
 	nfd := &nfdMaster{args: args, ready: make(chan bool, 1)}
@@ -260,60 +252,62 @@ func updateMasterNode(helper apihelper.APIHelpers) error {
 	}
 
 	// Advertise NFD version as an annotation
-	addAnnotations(node, Annotations{"master.version": version.Get()})
-	err = helper.UpdateNode(cli, node)
+	p := createPatches(nil, node.Annotations, Annotations{masterVersionAnnotation: version.Get()}, "/metadata/annotations")
+	err = helper.PatchNode(cli, node.Name, p)
 	if err != nil {
-		stderrLogger.Printf("can't update node: %s", err.Error())
+		stderrLogger.Printf("failed to patch node annotations: %v", err)
 		return err
 	}
 
 	return nil
 }
 
-// Filter labels by namespace and name whitelist
-func filterFeatureLabels(labels Labels, extraLabelNs []string, labelWhiteList *regexp.Regexp, extendedResourceNames []string) (Labels, ExtendedResources) {
-	for label := range labels {
-		split := strings.SplitN(label, "/", 2)
-		name := split[0]
-
-		// Check namespaced labels, filter out if ns is not whitelisted
-		if len(split) == 2 {
-			ns := split[0]
-			name = split[1]
-			for i, extraNs := range extraLabelNs {
-				if ns == extraNs {
-					break
-				} else if i == len(extraLabelNs)-1 {
-					stderrLogger.Printf("Namespace '%s' is not allowed. Ignoring label '%s'\n", ns, label)
-					delete(labels, label)
-				}
+// Filter labels by namespace and name whitelist, and, turn selected labels
+// into extended resources. This function also handles proper namespacing of
+// labels and ERs, i.e. adds the possibly missing default namespace for labels
+// arriving through the gRPC API.
+func filterFeatureLabels(labels Labels, extraLabelNs map[string]struct{}, labelWhiteList *regexp.Regexp, extendedResourceNames []string) (Labels, ExtendedResources) {
+	outLabels := Labels{}
+
+	for label, value := range labels {
+		// Add possibly missing default ns
+		label := addNs(label, LabelNs)
+
+		ns, name := splitNs(label)
+
+		// Check label namespace, filter out if ns is not whitelisted
+		if ns != LabelNs {
+			if _, ok := extraLabelNs[ns]; !ok {
+				stderrLogger.Printf("Namespace '%s' is not allowed. Ignoring label '%s'\n", ns, label)
+				continue
 			}
 		}
 
 		// Skip if label doesn't match labelWhiteList
 		if !labelWhiteList.MatchString(name) {
-			stderrLogger.Printf("%s does not match the whitelist (%s) and will not be published.", name, labelWhiteList.String())
-			delete(labels, label)
+			stderrLogger.Printf("%s (%s) does not match the whitelist (%s) and will not be published.", name, label, labelWhiteList.String())
+			continue
 		}
+		outLabels[label] = value
 	}
 
 	// Remove labels which are intended to be extended resources
 	extendedResources := ExtendedResources{}
 	for _, extendedResourceName := range extendedResourceNames {
-		// remove possibly given default LabelNs to keep annotations shorter
-		extendedResourceName = strings.TrimPrefix(extendedResourceName, LabelNs)
-		if _, ok := labels[extendedResourceName]; ok {
-			if _, err := strconv.Atoi(labels[extendedResourceName]); err != nil {
-				stderrLogger.Printf("bad label value encountered for extended resource: %s", err.Error())
+		// Add possibly missing default ns
+		extendedResourceName = addNs(extendedResourceName, LabelNs)
+		if value, ok := outLabels[extendedResourceName]; ok {
+			if _, err := strconv.Atoi(value); err != nil {
+				stderrLogger.Printf("bad label value (%s: %s) encountered for extended resource: %s", extendedResourceName, value, err.Error())
 				continue // non-numeric label can't be used
 			}
 
-			extendedResources[extendedResourceName] = labels[extendedResourceName]
-			delete(labels, extendedResourceName)
+			extendedResources[extendedResourceName] = value
+			delete(outLabels, extendedResourceName)
 		}
 	}
 
-	return labels, extendedResources
+	return outLabels, extendedResources
 }
 
 // Implement LabelerServer
@@ -352,23 +346,8 @@ func (s *labelerServer) SetLabels(c context.Context, r *pb.SetLabelsRequest) (*p
 	labels, extendedResources := filterFeatureLabels(r.Labels, s.args.ExtraLabelNs, s.args.LabelWhiteList, s.args.ResourceLabels)
 
 	if !s.args.NoPublish {
-		// Advertise NFD worker version, label names and extended resources as annotations
-		labelKeys := make([]string, 0, len(labels))
-		for k := range labels {
-			labelKeys = append(labelKeys, k)
-		}
-		sort.Strings(labelKeys)
-
-		extendedResourceKeys := make([]string, 0, len(extendedResources))
-		for key := range extendedResources {
-			extendedResourceKeys = append(extendedResourceKeys, key)
-		}
-		sort.Strings(extendedResourceKeys)
-
-		annotations := Annotations{"worker.version": r.NfdVersion,
-			"feature-labels":     strings.Join(labelKeys, ","),
-			"extended-resources": strings.Join(extendedResourceKeys, ","),
-		}
+		// Advertise NFD worker version as an annotation
+		annotations := Annotations{workerVersionAnnotation: r.NfdVersion}
 
 		err := updateNodeFeatures(s.apiHelper, r.NodeName, labels, annotations, extendedResources)
 		if err != nil {
@@ -394,78 +373,106 @@ func updateNodeFeatures(helper apihelper.APIHelpers, nodeName string, labels Lab
 		return err
 	}
 
-	// Resolve publishable extended resources before node is modified
-	statusOps := getExtendedResourceOps(node, extendedResources)
-
-	// Remove old labels
-	if l, ok := node.Annotations[AnnotationNs+"feature-labels"]; ok {
-		oldLabels := strings.Split(l, ",")
-		removeLabels(node, oldLabels)
+	// Store names of labels in an annotation
+	labelKeys := make([]string, 0, len(labels))
+	for key := range labels {
+		// Drop the ns part for labels in the default ns
+		labelKeys = append(labelKeys, strings.TrimPrefix(key, LabelNs+"/"))
 	}
+	sort.Strings(labelKeys)
+	annotations[featureLabelAnnotation] = strings.Join(labelKeys, ",")
+
+	// Store names of extended resources in an annotation
+	extendedResourceKeys := make([]string, 0, len(extendedResources))
+	for key := range extendedResources {
+		// Drop the ns part if in the default ns
+		extendedResourceKeys = append(extendedResourceKeys, strings.TrimPrefix(key, LabelNs+"/"))
+	}
+	sort.Strings(extendedResourceKeys)
+	annotations[extendedResourceAnnotation] = strings.Join(extendedResourceKeys, ",")
 
-	// Also, remove all labels with the old prefix, and the old version label
-	removeLabelsWithPrefix(node, "node.alpha.kubernetes-incubator.io/nfd")
-	removeLabelsWithPrefix(node, "node.alpha.kubernetes-incubator.io/node-feature-discovery")
-
-	// Add labels to the node object.
-	addLabels(node, labels)
+	// Create JSON patches for changes in labels and annotations
+	oldLabels := stringToNsNames(node.Annotations[featureLabelAnnotation], LabelNs)
+	patches := createPatches(oldLabels, node.Labels, labels, "/metadata/labels")
+	patches = append(patches, createPatches(nil, node.Annotations, annotations, "/metadata/annotations")...)
 
-	// Add annotations
-	addAnnotations(node, annotations)
+	// Also, remove all labels with the old prefix, and the old version label
+	patches = append(patches, removeLabelsWithPrefix(node, "node.alpha.kubernetes-incubator.io/nfd")...)
+	patches = append(patches, removeLabelsWithPrefix(node, "node.alpha.kubernetes-incubator.io/node-feature-discovery")...)
 
-	// Send the updated node to the apiserver.
-	err = helper.UpdateNode(cli, node)
+	// Patch the node object in the apiserver
+	err = helper.PatchNode(cli, node.Name, patches)
 	if err != nil {
-		stderrLogger.Printf("can't update node: %s", err.Error())
+		stderrLogger.Printf("error while patching node object: %s", err.Error())
 		return err
 	}
 
 	// patch node status with extended resource changes
-	if len(statusOps) > 0 {
-		err = helper.PatchStatus(cli, node.Name, statusOps)
-		if err != nil {
-			stderrLogger.Printf("error while patching extended resources: %s", err.Error())
-			return err
-		}
+	patches = createExtendedResourcePatches(node, extendedResources)
+	err = helper.PatchNodeStatus(cli, node.Name, patches)
+	if err != nil {
+		stderrLogger.Printf("error while patching extended resources: %s", err.Error())
+		return err
 	}
 
 	return err
 }
 
 // Remove any labels having the given prefix
-func removeLabelsWithPrefix(n *api.Node, search string) {
+func removeLabelsWithPrefix(n *api.Node, search string) []apihelper.JsonPatch {
+	var p []apihelper.JsonPatch
+
 	for k := range n.Labels {
 		if strings.HasPrefix(k, search) {
-			delete(n.Labels, k)
+			p = append(p, apihelper.NewJsonPatch("remove", "/metadata/labels", k, ""))
 		}
 	}
+
+	return p
 }
 
-// Removes NFD labels from a Node object
-func removeLabels(n *api.Node, labelNames []string) {
-	for _, l := range labelNames {
-		if strings.Contains(l, "/") {
-			delete(n.Labels, l)
+// createPatches is a generic helper that returns json patch operations to perform
+func createPatches(removeKeys []string, oldItems map[string]string, newItems map[string]string, jsonPath string) []apihelper.JsonPatch {
+	patches := []apihelper.JsonPatch{}
+
+	// Determine items to remove
+	for _, key := range removeKeys {
+		if _, ok := oldItems[key]; ok {
+			if _, ok := newItems[key]; !ok {
+				patches = append(patches, apihelper.NewJsonPatch("remove", jsonPath, key, ""))
+			}
+		}
+	}
+
+	// Determine items to add or replace
+	for key, newVal := range newItems {
+		if oldVal, ok := oldItems[key]; ok {
+			if newVal != oldVal {
+				patches = append(patches, apihelper.NewJsonPatch("replace", jsonPath, key, newVal))
+			}
 		} else {
-			delete(n.Labels, LabelNs+l)
+			patches = append(patches, apihelper.NewJsonPatch("add", jsonPath, key, newVal))
 		}
 	}
+
+	return patches
 }
 
-// getExtendedResourceOps returns a slice of operations to perform on the node status
-func getExtendedResourceOps(n *api.Node, extendedResources ExtendedResources) []statusOp {
-	var statusOps []statusOp
+// createExtendedResourcePatches returns a slice of operations to perform on
+// the node status
+func createExtendedResourcePatches(n *api.Node, extendedResources ExtendedResources) []apihelper.JsonPatch {
+	patches := []apihelper.JsonPatch{}
 
-	oldResources := strings.Split(n.Annotations[AnnotationNs+"extended-resources"], ",")
+	// Form a list of namespaced resource names managed by us
+	oldResources := stringToNsNames(n.Annotations[extendedResourceAnnotation], LabelNs)
 
 	// figure out which resources to remove
 	for _, resource := range oldResources {
-		if _, ok := n.Status.Capacity[api.ResourceName(addNs(resource, LabelNs))]; ok {
+		if _, ok := n.Status.Capacity[api.ResourceName(resource)]; ok {
 			// check if the ext resource is still needed
-			_, extResNeeded := extendedResources[resource]
-			if !extResNeeded {
-				statusOps = append(statusOps, createStatusOp("remove", resource, "capacity", ""))
-				statusOps = append(statusOps, createStatusOp("remove", resource, "allocatable", ""))
+			if _, extResNeeded := extendedResources[resource]; !extResNeeded {
+				patches = append(patches, apihelper.NewJsonPatch("remove", "/status/capacity", resource, ""))
+				patches = append(patches, apihelper.NewJsonPatch("remove", "/status/allocatable", resource, ""))
 			}
 		}
 	}
@@ -473,43 +480,48 @@ func getExtendedResourceOps(n *api.Node, extendedResources ExtendedResources) []
 	// figure out which resources to replace and which to add
 	for resource, value := range extendedResources {
 		// check if the extended resource already exists with the same capacity in the node
-		if quantity, ok := n.Status.Capacity[api.ResourceName(addNs(resource, LabelNs))]; ok {
+		if quantity, ok := n.Status.Capacity[api.ResourceName(resource)]; ok {
 			val, _ := quantity.AsInt64()
 			if strconv.FormatInt(val, 10) != value {
-				statusOps = append(statusOps, createStatusOp("replace", resource, "capacity", value))
-				statusOps = append(statusOps, createStatusOp("replace", resource, "allocatable", value))
+				patches = append(patches, apihelper.NewJsonPatch("replace", "/status/capacity", resource, value))
+				patches = append(patches, apihelper.NewJsonPatch("replace", "/status/allocatable", resource, value))
 			}
 		} else {
-			statusOps = append(statusOps, createStatusOp("add", resource, "capacity", value))
+			patches = append(patches, apihelper.NewJsonPatch("add", "/status/capacity", resource, value))
 			// "allocatable" gets added implicitly after adding to capacity
 		}
 	}
 
-	return statusOps
+	return patches
 }
 
-// Add NFD labels to a Node object.
-func addLabels(n *api.Node, labels map[string]string) {
-	for k, v := range labels {
-		if strings.Contains(k, "/") {
-			n.Labels[k] = v
-		} else {
-			n.Labels[LabelNs+k] = v
-		}
+// addNs adds a namespace if one isn't already found from src string
+func addNs(src string, nsToAdd string) string {
+	if strings.Contains(src, "/") {
+		return src
 	}
+	return filepath.Join(nsToAdd, src)
 }
 
-// Add Annotations to a Node object
-func addAnnotations(n *api.Node, annotations map[string]string) {
-	for k, v := range annotations {
-		n.Annotations[AnnotationNs+k] = v
+// splitNs splits a name into its namespace and name parts
+func splitNs(fullname string) (string, string) {
+	split := strings.SplitN(fullname, "/", 2)
+	if len(split) == 2 {
+		return split[0], split[1]
 	}
+	return "", fullname
 }
 
-// addNs adds a namespace if one isn't already found from src string
-func addNs(src string, nsToAdd string) string {
-	if strings.Contains(src, "/") {
-		return src
+// stringToNsNames is a helper for converting a string of comma-separated names
+// into a slice of fully namespaced names
+func stringToNsNames(cslist, ns string) []string {
+	var names []string
+	if cslist != "" {
+		names = strings.Split(cslist, ",")
+		for i, name := range names {
+			// Expect that names may omit the ns part
+			names[i] = addNs(name, ns)
+		}
 	}
-	return nsToAdd + src
+	return names
 }


Test patch:
diff --git a/pkg/nfd-master/nfd-master-internal_test.go b/pkg/nfd-master/nfd-master-internal_test.go
index 7810c08bc..fbc7e3a70 100644
--- a/pkg/nfd-master/nfd-master-internal_test.go
+++ b/pkg/nfd-master/nfd-master-internal_test.go
@@ -22,6 +22,7 @@ import (
 	"strings"
 	"testing"
 
+	"github.com/smartystreets/assertions"
 	. "github.com/smartystreets/goconvey/convey"
 	"github.com/stretchr/testify/mock"
 	"github.com/vektra/errors"
@@ -33,6 +34,7 @@ import (
 	"sigs.k8s.io/node-feature-discovery/pkg/apihelper"
 	"sigs.k8s.io/node-feature-discovery/pkg/labeler"
 	"sigs.k8s.io/node-feature-discovery/pkg/version"
+	"sigs.k8s.io/yaml"
 )
 
 const (
@@ -54,43 +56,58 @@ func newMockNode() *api.Node {
 
 func TestUpdateNodeFeatures(t *testing.T) {
 	Convey("When I update the node using fake client", t, func() {
-		fakeFeatureLabels := map[string]string{"source-feature.1": "1", "source-feature.2": "2", "source-feature.3": "val3"}
-		fakeAnnotations := map[string]string{"version": version.Get()}
-		fakeExtResources := ExtendedResources{"source-feature.1": "", "source-feature.2": ""}
+		fakeFeatureLabels := map[string]string{LabelNs + "/source-feature.1": "1", LabelNs + "/source-feature.2": "2", LabelNs + "/source-feature.3": "val3"}
+		fakeAnnotations := map[string]string{"my-annotation": "my-val"}
+		fakeExtResources := ExtendedResources{LabelNs + "/source-feature.1": "1", LabelNs + "/source-feature.2": "2"}
+
 		fakeFeatureLabelNames := make([]string, 0, len(fakeFeatureLabels))
 		for k := range fakeFeatureLabels {
-			fakeFeatureLabelNames = append(fakeFeatureLabelNames, k)
+			fakeFeatureLabelNames = append(fakeFeatureLabelNames, strings.TrimPrefix(k, LabelNs+"/"))
 		}
 		sort.Strings(fakeFeatureLabelNames)
-		fakeAnnotations["feature-labels"] = strings.Join(fakeFeatureLabelNames, ",")
+
+		fakeExtResourceNames := make([]string, 0, len(fakeExtResources))
+		for k := range fakeExtResources {
+			fakeExtResourceNames = append(fakeExtResourceNames, strings.TrimPrefix(k, LabelNs+"/"))
+		}
+		sort.Strings(fakeExtResourceNames)
 
 		mockAPIHelper := new(apihelper.MockAPIHelpers)
 		mockClient := &k8sclient.Clientset{}
 		// Mock node with old features
 		mockNode := newMockNode()
-		mockNode.Labels[LabelNs+"old-feature"] = "old-value"
-		mockNode.Annotations[AnnotationNs+"feature-labels"] = "old-feature"
+		mockNode.Labels[LabelNs+"/old-feature"] = "old-value"
+		mockNode.Annotations[AnnotationNs+"/feature-labels"] = "old-feature"
 
 		Convey("When I successfully update the node with feature labels", func() {
+			// Create a list of expected node metadata patches
+			metadataPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("replace", "/metadata/annotations", AnnotationNs+"/feature-labels", strings.Join(fakeFeatureLabelNames, ",")),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", AnnotationNs+"/extended-resources", strings.Join(fakeExtResourceNames, ",")),
+				apihelper.NewJsonPatch("remove", "/metadata/labels", LabelNs+"/old-feature", ""),
+			}
+			for k, v := range fakeFeatureLabels {
+				metadataPatches = append(metadataPatches, apihelper.NewJsonPatch("add", "/metadata/labels", k, v))
+			}
+			for k, v := range fakeAnnotations {
+				metadataPatches = append(metadataPatches, apihelper.NewJsonPatch("add", "/metadata/annotations", k, v))
+			}
+
+			// Create a list of expected node status patches
+			statusPatches := []apihelper.JsonPatch{}
+			for k, v := range fakeExtResources {
+				statusPatches = append(statusPatches, apihelper.NewJsonPatch("add", "/status/capacity", k, v))
+			}
+
 			mockAPIHelper.On("GetClient").Return(mockClient, nil)
 			mockAPIHelper.On("GetNode", mockClient, mockNodeName).Return(mockNode, nil).Once()
-			mockAPIHelper.On("UpdateNode", mockClient, mockNode).Return(nil).Once()
-			mockAPIHelper.On("PatchStatus", mockClient, mockNodeName, mock.Anything).Return(nil).Twice()
+			mockAPIHelper.On("PatchNode", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(metadataPatches))).Return(nil)
+			mockAPIHelper.On("PatchNodeStatus", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(statusPatches))).Return(nil)
 			err := updateNodeFeatures(mockAPIHelper, mockNodeName, fakeFeatureLabels, fakeAnnotations, fakeExtResources)
 
 			Convey("Error is nil", func() {
 				So(err, ShouldBeNil)
 			})
-			Convey("Node object should have updated with labels and annotations", func() {
-				So(len(mockNode.Labels), ShouldEqual, len(fakeFeatureLabels))
-				for k, v := range fakeFeatureLabels {
-					So(mockNode.Labels[LabelNs+k], ShouldEqual, v)
-				}
-				So(len(mockNode.Annotations), ShouldEqual, len(fakeAnnotations))
-				for k, v := range fakeAnnotations {
-					So(mockNode.Annotations[AnnotationNs+k], ShouldEqual, v)
-				}
-			})
 		})
 
 		Convey("When I fail to update the node with feature labels", func() {
@@ -128,7 +145,7 @@ func TestUpdateNodeFeatures(t *testing.T) {
 			expectedError := errors.New("fake error")
 			mockAPIHelper.On("GetClient").Return(mockClient, nil)
 			mockAPIHelper.On("GetNode", mockClient, mockNodeName).Return(mockNode, nil).Once()
-			mockAPIHelper.On("UpdateNode", mockClient, mockNode).Return(expectedError).Once()
+			mockAPIHelper.On("PatchNode", mockClient, mockNodeName, mock.Anything).Return(expectedError).Once()
 			err := updateNodeFeatures(mockAPIHelper, mockNodeName, fakeFeatureLabels, fakeAnnotations, fakeExtResources)
 
 			Convey("Error is produced", func() {
@@ -145,9 +162,11 @@ func TestUpdateMasterNode(t *testing.T) {
 		mockClient := &k8sclient.Clientset{}
 		mockNode := newMockNode()
 		Convey("When update operation succeeds", func() {
+			expectedPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", "/metadata/annotations", AnnotationNs+"/master.version", version.Get())}
 			mockHelper.On("GetClient").Return(mockClient, nil)
 			mockHelper.On("GetNode", mockClient, mockNodeName).Return(mockNode, nil)
-			mockHelper.On("UpdateNode", mockClient, mockNode).Return(nil)
+			mockHelper.On("PatchNode", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedPatches))).Return(nil)
 			err := updateMasterNode(mockHelper)
 			Convey("No error should be returned", func() {
 				So(err, ShouldBeNil)
@@ -175,7 +194,7 @@ func TestUpdateMasterNode(t *testing.T) {
 		Convey("When updating node object fails", func() {
 			mockHelper.On("GetClient").Return(mockClient, nil)
 			mockHelper.On("GetNode", mockClient, mockNodeName).Return(mockNode, nil)
-			mockHelper.On("UpdateNode", mockClient, mockNode).Return(mockErr)
+			mockHelper.On("PatchNode", mockClient, mockNodeName, mock.Anything).Return(mockErr)
 			err := updateMasterNode(mockHelper)
 			Convey("An error should be returned", func() {
 				So(err, ShouldEqual, mockErr)
@@ -189,31 +208,39 @@ func TestAddingExtResources(t *testing.T) {
 		Convey("When there are no matching labels", func() {
 			mockNode := newMockNode()
 			mockResourceLabels := ExtendedResources{}
-			resourceOps := getExtendedResourceOps(mockNode, mockResourceLabels)
-			So(len(resourceOps), ShouldEqual, 0)
+			patches := createExtendedResourcePatches(mockNode, mockResourceLabels)
+			So(len(patches), ShouldEqual, 0)
 		})
 
 		Convey("When there are matching labels", func() {
 			mockNode := newMockNode()
 			mockResourceLabels := ExtendedResources{"feature-1": "1", "feature-2": "2"}
-			resourceOps := getExtendedResourceOps(mockNode, mockResourceLabels)
-			So(len(resourceOps), ShouldBeGreaterThan, 0)
+			expectedPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", "/status/capacity", "feature-1", "1"),
+				apihelper.NewJsonPatch("add", "/status/capacity", "feature-2", "2"),
+			}
+			patches := createExtendedResourcePatches(mockNode, mockResourceLabels)
+			So(sortJsonPatches(patches), ShouldResemble, sortJsonPatches(expectedPatches))
 		})
 
 		Convey("When the resource already exists", func() {
 			mockNode := newMockNode()
-			mockNode.Status.Capacity[api.ResourceName(LabelNs+"feature-1")] = *resource.NewQuantity(1, resource.BinarySI)
-			mockResourceLabels := ExtendedResources{"feature-1": "1"}
-			resourceOps := getExtendedResourceOps(mockNode, mockResourceLabels)
-			So(len(resourceOps), ShouldEqual, 0)
+			mockNode.Status.Capacity[api.ResourceName(LabelNs+"/feature-1")] = *resource.NewQuantity(1, resource.BinarySI)
+			mockResourceLabels := ExtendedResources{LabelNs + "/feature-1": "1"}
+			patches := createExtendedResourcePatches(mockNode, mockResourceLabels)
+			So(len(patches), ShouldEqual, 0)
 		})
 
 		Convey("When the resource already exists but its capacity has changed", func() {
 			mockNode := newMockNode()
-			mockNode.Status.Capacity[api.ResourceName(LabelNs+"feature-1")] = *resource.NewQuantity(2, resource.BinarySI)
+			mockNode.Status.Capacity[api.ResourceName("feature-1")] = *resource.NewQuantity(2, resource.BinarySI)
 			mockResourceLabels := ExtendedResources{"feature-1": "1"}
-			resourceOps := getExtendedResourceOps(mockNode, mockResourceLabels)
-			So(len(resourceOps), ShouldBeGreaterThan, 0)
+			expectedPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("replace", "/status/capacity", "feature-1", "1"),
+				apihelper.NewJsonPatch("replace", "/status/allocatable", "feature-1", "1"),
+			}
+			patches := createExtendedResourcePatches(mockNode, mockResourceLabels)
+			So(sortJsonPatches(patches), ShouldResemble, sortJsonPatches(expectedPatches))
 		})
 	})
 }
@@ -222,30 +249,30 @@ func TestRemovingExtResources(t *testing.T) {
 	Convey("When removing extended resources", t, func() {
 		Convey("When none are removed", func() {
 			mockNode := newMockNode()
-			mockResourceLabels := ExtendedResources{"feature-1": "1", "feature-2": "2"}
-			mockNode.Annotations[AnnotationNs+"extended-resources"] = "feature-1,feature-2"
-			mockNode.Status.Capacity[api.ResourceName(LabelNs+"feature-1")] = *resource.NewQuantity(1, resource.BinarySI)
-			mockNode.Status.Capacity[api.ResourceName(LabelNs+"feature-2")] = *resource.NewQuantity(2, resource.BinarySI)
-			resourceOps := getExtendedResourceOps(mockNode, mockResourceLabels)
-			So(len(resourceOps), ShouldEqual, 0)
+			mockResourceLabels := ExtendedResources{LabelNs + "/feature-1": "1", LabelNs + "/feature-2": "2"}
+			mockNode.Annotations[AnnotationNs+"/extended-resources"] = "feature-1,feature-2"
+			mockNode.Status.Capacity[api.ResourceName(LabelNs+"/feature-1")] = *resource.NewQuantity(1, resource.BinarySI)
+			mockNode.Status.Capacity[api.ResourceName(LabelNs+"/feature-2")] = *resource.NewQuantity(2, resource.BinarySI)
+			patches := createExtendedResourcePatches(mockNode, mockResourceLabels)
+			So(len(patches), ShouldEqual, 0)
 		})
 		Convey("When the related label is gone", func() {
 			mockNode := newMockNode()
-			mockResourceLabels := ExtendedResources{"feature-4": "", "feature-2": "2"}
-			mockNode.Annotations[AnnotationNs+"extended-resources"] = "feature-4,feature-2"
-			mockNode.Status.Capacity[api.ResourceName(LabelNs+"feature-4")] = *resource.NewQuantity(4, resource.BinarySI)
-			mockNode.Status.Capacity[api.ResourceName(LabelNs+"feature-2")] = *resource.NewQuantity(2, resource.BinarySI)
-			resourceOps := getExtendedResourceOps(mockNode, mockResourceLabels)
-			So(len(resourceOps), ShouldBeGreaterThan, 0)
+			mockResourceLabels := ExtendedResources{LabelNs + "/feature-4": "", LabelNs + "/feature-2": "2"}
+			mockNode.Annotations[AnnotationNs+"/extended-resources"] = "feature-4,feature-2"
+			mockNode.Status.Capacity[api.ResourceName(LabelNs+"/feature-4")] = *resource.NewQuantity(4, resource.BinarySI)
+			mockNode.Status.Capacity[api.ResourceName(LabelNs+"/feature-2")] = *resource.NewQuantity(2, resource.BinarySI)
+			patches := createExtendedResourcePatches(mockNode, mockResourceLabels)
+			So(len(patches), ShouldBeGreaterThan, 0)
 		})
 		Convey("When the extended resource is no longer wanted", func() {
 			mockNode := newMockNode()
-			mockNode.Status.Capacity[api.ResourceName(LabelNs+"feature-1")] = *resource.NewQuantity(1, resource.BinarySI)
-			mockNode.Status.Capacity[api.ResourceName(LabelNs+"feature-2")] = *resource.NewQuantity(2, resource.BinarySI)
-			mockResourceLabels := ExtendedResources{"feature-2": "2"}
-			mockNode.Annotations[AnnotationNs+"extended-resources"] = "feature-1,feature-2"
-			resourceOps := getExtendedResourceOps(mockNode, mockResourceLabels)
-			So(len(resourceOps), ShouldBeGreaterThan, 0)
+			mockNode.Status.Capacity[api.ResourceName(LabelNs+"/feature-1")] = *resource.NewQuantity(1, resource.BinarySI)
+			mockNode.Status.Capacity[api.ResourceName(LabelNs+"/feature-2")] = *resource.NewQuantity(2, resource.BinarySI)
+			mockResourceLabels := ExtendedResources{LabelNs + "/feature-2": "2"}
+			mockNode.Annotations[AnnotationNs+"/extended-resources"] = "feature-1,feature-2"
+			patches := createExtendedResourcePatches(mockNode, mockResourceLabels)
+			So(len(patches), ShouldBeGreaterThan, 0)
 		})
 	})
 }
@@ -259,7 +286,8 @@ func TestSetLabels(t *testing.T) {
 		mockNode := newMockNode()
 		mockServer := labelerServer{args: Args{LabelWhiteList: regexp.MustCompile("")}, apiHelper: mockHelper}
 		mockCtx := context.Background()
-		mockLabels := map[string]string{"feature-1": "val-1", "feature-2": "val-2", "feature-3": "val-3"}
+		// In the gRPC request the label names may omit the default ns
+		mockLabels := map[string]string{"feature-1": "1", "feature-2": "val-2", "feature-3": "3"}
 		mockReq := &labeler.SetLabelsRequest{NodeName: workerName, NfdVersion: workerVer, Labels: mockLabels}
 
 		mockLabelNames := make([]string, 0, len(mockLabels))
@@ -267,69 +295,94 @@ func TestSetLabels(t *testing.T) {
 			mockLabelNames = append(mockLabelNames, k)
 		}
 		sort.Strings(mockLabelNames)
-		expectedAnnotations := map[string]string{"worker.version": workerVer}
-		expectedAnnotations["feature-labels"] = strings.Join(mockLabelNames, ",")
-		expectedAnnotations["extended-resources"] = ""
+
+		expectedStatusPatches := []apihelper.JsonPatch{}
 
 		Convey("When node update succeeds", func() {
+			expectedPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", "/metadata/annotations", workerVersionAnnotation, workerVer),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", featureLabelAnnotation, strings.Join(mockLabelNames, ",")),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", extendedResourceAnnotation, ""),
+			}
+			for k, v := range mockLabels {
+				expectedPatches = append(expectedPatches, apihelper.NewJsonPatch("add", "/metadata/labels", LabelNs+"/"+k, v))
+			}
+
 			mockHelper.On("GetClient").Return(mockClient, nil)
 			mockHelper.On("GetNode", mockClient, workerName).Return(mockNode, nil)
-			mockHelper.On("UpdateNode", mockClient, mockNode).Return(nil)
+			mockHelper.On("PatchNode", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedPatches))).Return(nil)
+			mockHelper.On("PatchNodeStatus", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedStatusPatches))).Return(nil)
 			_, err := mockServer.SetLabels(mockCtx, mockReq)
 			Convey("No error should be returned", func() {
 				So(err, ShouldBeNil)
 			})
-			Convey("Node object should have updated with labels and annotations", func() {
-				So(len(mockNode.Labels), ShouldEqual, len(mockLabels))
-				for k, v := range mockLabels {
-					So(mockNode.Labels[LabelNs+k], ShouldEqual, v)
-				}
-				So(len(mockNode.Annotations), ShouldEqual, len(expectedAnnotations))
-				for k, v := range expectedAnnotations {
-					So(mockNode.Annotations[AnnotationNs+k], ShouldEqual, v)
-				}
-			})
 		})
 
 		Convey("When --label-whitelist is specified", func() {
+			expectedPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", "/metadata/annotations", workerVersionAnnotation, workerVer),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", featureLabelAnnotation, "feature-2"),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", extendedResourceAnnotation, ""),
+				apihelper.NewJsonPatch("add", "/metadata/labels", LabelNs+"/feature-2", mockLabels["feature-2"]),
+			}
+
 			mockServer.args.LabelWhiteList = regexp.MustCompile("^f.*2$")
 			mockHelper.On("GetClient").Return(mockClient, nil)
 			mockHelper.On("GetNode", mockClient, workerName).Return(mockNode, nil)
-			mockHelper.On("UpdateNode", mockClient, mockNode).Return(nil)
+			mockHelper.On("PatchNode", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedPatches))).Return(nil)
+			mockHelper.On("PatchNodeStatus", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedStatusPatches))).Return(nil)
 			_, err := mockServer.SetLabels(mockCtx, mockReq)
 			Convey("Error is nil", func() {
 				So(err, ShouldBeNil)
 			})
-			Convey("Node object should only have whitelisted labels", func() {
-				So(len(mockNode.Labels), ShouldEqual, 1)
-				So(mockNode.Labels, ShouldResemble, map[string]string{LabelNs + "feature-2": "val-2"})
-
-				a := map[string]string{AnnotationNs + "worker.version": workerVer, AnnotationNs + "feature-labels": "feature-2", AnnotationNs + "extended-resources": ""}
-				So(len(mockNode.Annotations), ShouldEqual, len(a))
-				So(mockNode.Annotations, ShouldResemble, a)
-			})
 		})
 
 		Convey("When --extra-label-ns is specified", func() {
-			mockServer.args.ExtraLabelNs = []string{"valid.ns"}
-			mockHelper.On("GetClient").Return(mockClient, nil)
-			mockHelper.On("GetNode", mockClient, workerName).Return(mockNode, nil)
-			mockHelper.On("UpdateNode", mockClient, mockNode).Return(nil)
+			// In the gRPC request the label names may omit the default ns
 			mockLabels := map[string]string{"feature-1": "val-1",
 				"valid.ns/feature-2":   "val-2",
 				"invalid.ns/feature-3": "val-3"}
+			expectedPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", "/metadata/annotations", workerVersionAnnotation, workerVer),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", featureLabelAnnotation, "feature-1,valid.ns/feature-2"),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", extendedResourceAnnotation, ""),
+				apihelper.NewJsonPatch("add", "/metadata/labels", LabelNs+"/feature-1", mockLabels["feature-1"]),
+				apihelper.NewJsonPatch("add", "/metadata/labels", "valid.ns/feature-2", mockLabels["valid.ns/feature-2"]),
+			}
+
+			mockServer.args.ExtraLabelNs = map[string]struct{}{"valid.ns": struct{}{}}
+			mockHelper.On("GetClient").Return(mockClient, nil)
+			mockHelper.On("GetNode", mockClient, workerName).Return(mockNode, nil)
+			mockHelper.On("PatchNode", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedPatches))).Return(nil)
+			mockHelper.On("PatchNodeStatus", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedStatusPatches))).Return(nil)
 			mockReq := &labeler.SetLabelsRequest{NodeName: workerName, NfdVersion: workerVer, Labels: mockLabels}
 			_, err := mockServer.SetLabels(mockCtx, mockReq)
 			Convey("Error is nil", func() {
 				So(err, ShouldBeNil)
 			})
-			Convey("Node object should only have allowed label namespaces", func() {
-				So(len(mockNode.Labels), ShouldEqual, 2)
-				So(mockNode.Labels, ShouldResemble, map[string]string{LabelNs + "feature-1": "val-1", "valid.ns/feature-2": "val-2"})
 
-				a := map[string]string{AnnotationNs + "worker.version": workerVer, AnnotationNs + "feature-labels": "feature-1,valid.ns/feature-2", AnnotationNs + "extended-resources": ""}
-				So(len(mockNode.Annotations), ShouldEqual, len(a))
-				So(mockNode.Annotations, ShouldResemble, a)
+		})
+
+		Convey("When --resource-labels is specified", func() {
+			expectedPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", "/metadata/annotations", workerVersionAnnotation, workerVer),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", featureLabelAnnotation, "feature-2"),
+				apihelper.NewJsonPatch("add", "/metadata/annotations", extendedResourceAnnotation, "feature-1,feature-3"),
+				apihelper.NewJsonPatch("add", "/metadata/labels", LabelNs+"/feature-2", mockLabels["feature-2"]),
+			}
+			expectedStatusPatches := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", "/status/capacity", LabelNs+"/feature-1", mockLabels["feature-1"]),
+				apihelper.NewJsonPatch("add", "/status/capacity", LabelNs+"/feature-3", mockLabels["feature-3"]),
+			}
+
+			mockServer.args.ResourceLabels = []string{"feature-3", "feature-1"}
+			mockHelper.On("GetClient").Return(mockClient, nil)
+			mockHelper.On("GetNode", mockClient, workerName).Return(mockNode, nil)
+			mockHelper.On("PatchNode", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedPatches))).Return(nil)
+			mockHelper.On("PatchNodeStatus", mockClient, mockNodeName, mock.MatchedBy(jsonPatchMatcher(expectedStatusPatches))).Return(nil)
+			_, err := mockServer.SetLabels(mockCtx, mockReq)
+			Convey("Error is nil", func() {
+				So(err, ShouldBeNil)
 			})
 		})
 
@@ -352,28 +405,46 @@ func TestSetLabels(t *testing.T) {
 	})
 }
 
-func TestAddLabels(t *testing.T) {
-	Convey("When adding labels", t, func() {
-		labels := map[string]string{}
-		n := &api.Node{
-			ObjectMeta: meta_v1.ObjectMeta{
-				Labels: map[string]string{},
-			},
-		}
+func TestCreatePatches(t *testing.T) {
+	Convey("When creating JSON patches", t, func() {
+		existingItems := map[string]string{"key-1": "val-1", "key-2": "val-2", "key-3": "val-3"}
+		jsonPath := "/root"
 
-		Convey("If no labels are passed", func() {
-			addLabels(n, labels)
+		Convey("When when there are neither itmes to remoe nor to add or update", func() {
+			p := createPatches([]string{"foo", "bar"}, existingItems, map[string]string{}, jsonPath)
+			So(len(p), ShouldEqual, 0)
+		})
 
-			Convey("None should be added", func() {
-				So(len(n.Labels), ShouldEqual, 0)
-			})
+		Convey("When when there are itmes to remoe but none to add or update", func() {
+			p := createPatches([]string{"key-2", "key-3", "foo"}, existingItems, map[string]string{}, jsonPath)
+			expected := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("remove", jsonPath, "key-2", ""),
+				apihelper.NewJsonPatch("remove", jsonPath, "key-3", ""),
+			}
+			So(sortJsonPatches(p), ShouldResemble, sortJsonPatches(expected))
 		})
 
-		Convey("They should be added to the node.Labels", func() {
-			test1 := "test1"
-			labels[test1] = "true"
-			addLabels(n, labels)
-			So(n.Labels, ShouldContainKey, LabelNs+test1)
+		Convey("When when there are no itmes to remove but new items to add", func() {
+			newItems := map[string]string{"new-key": "new-val", "key-1": "new-1"}
+			p := createPatches([]string{"key-1"}, existingItems, newItems, jsonPath)
+			expected := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", jsonPath, "new-key", newItems["new-key"]),
+				apihelper.NewJsonPatch("replace", jsonPath, "key-1", newItems["key-1"]),
+			}
+			So(sortJsonPatches(p), ShouldResemble, sortJsonPatches(expected))
+		})
+
+		Convey("When when there are items to remove add and update", func() {
+			newItems := map[string]string{"new-key": "new-val", "key-2": "new-2", "key-4": "val-4"}
+			p := createPatches([]string{"key-1", "key-2", "key-3", "foo"}, existingItems, newItems, jsonPath)
+			expected := []apihelper.JsonPatch{
+				apihelper.NewJsonPatch("add", jsonPath, "new-key", newItems["new-key"]),
+				apihelper.NewJsonPatch("add", jsonPath, "key-4", newItems["key-4"]),
+				apihelper.NewJsonPatch("replace", jsonPath, "key-2", newItems["key-2"]),
+				apihelper.NewJsonPatch("remove", jsonPath, "key-1", ""),
+				apihelper.NewJsonPatch("remove", jsonPath, "key-3", ""),
+			}
+			So(sortJsonPatches(p), ShouldResemble, sortJsonPatches(expected))
 		})
 	})
 }
@@ -391,16 +462,16 @@ func TestRemoveLabelsWithPrefix(t *testing.T) {
 		}
 
 		Convey("a unique label should be removed", func() {
-			removeLabelsWithPrefix(n, "single")
-			So(len(n.Labels), ShouldEqual, 2)
-			So(n.Labels, ShouldNotContainKey, "single")
+			p := removeLabelsWithPrefix(n, "single")
+			So(p, ShouldResemble, []apihelper.JsonPatch{apihelper.NewJsonPatch("remove", "/metadata/labels", "single-label", "")})
 		})
 
 		Convey("a non-unique search string should remove all matching keys", func() {
-			removeLabelsWithPrefix(n, "multiple")
-			So(len(n.Labels), ShouldEqual, 1)
-			So(n.Labels, ShouldNotContainKey, "multiple_A")
-			So(n.Labels, ShouldNotContainKey, "multiple_B")
+			p := removeLabelsWithPrefix(n, "multiple")
+			So(sortJsonPatches(p), ShouldResemble, sortJsonPatches([]apihelper.JsonPatch{
+				apihelper.NewJsonPatch("remove", "/metadata/labels", "multiple_A", ""),
+				apihelper.NewJsonPatch("remove", "/metadata/labels", "multiple_B", ""),
+			}))
 		})
 
 		Convey("a search string with no matches should not alter labels", func() {
@@ -412,3 +483,25 @@ func TestRemoveLabelsWithPrefix(t *testing.T) {
 		})
 	})
 }
+
+func jsonPatchMatcher(expected []apihelper.JsonPatch) func([]apihelper.JsonPatch) bool {
+	return func(actual []apihelper.JsonPatch) bool {
+		// We don't care about modifying the original slices
+		ok, msg := assertions.So(sortJsonPatches(actual), ShouldResemble, sortJsonPatches(expected))
+		if !ok {
+			// We parse the cryptic string message for better readability
+			var f assertions.FailureView
+			if err := yaml.Unmarshal([]byte(msg), &f); err == nil {
+				Printf("%s\n", f.Message)
+			} else {
+				Printf("%s\n", msg)
+			}
+		}
+		return ok
+	}
+}
+
+func sortJsonPatches(p []apihelper.JsonPatch) []apihelper.JsonPatch {
+	sort.Slice(p, func(i, j int) bool { return p[i].Path < p[j].Path })
+	return p
+}
diff --git a/test/e2e/node_feature_discovery.go b/test/e2e/node_feature_discovery.go
index e2cf763f2..e80b1f76c 100644
--- a/test/e2e/node_feature_discovery.go
+++ b/test/e2e/node_feature_discovery.go
@@ -458,9 +458,9 @@ var _ = framework.KubeDescribe("[NFD] Node Feature Discovery", func() {
 			ginkgo.It("it should decorate the node with the fake feature labels", func() {
 
 				fakeFeatureLabels := map[string]string{
-					master.LabelNs + "fake-fakefeature1": "true",
-					master.LabelNs + "fake-fakefeature2": "true",
-					master.LabelNs + "fake-fakefeature3": "true",
+					master.LabelNs + "/fake-fakefeature1": "true",
+					master.LabelNs + "/fake-fakefeature2": "true",
+					master.LabelNs + "/fake-fakefeature3": "true",
 				}
 
 				// Remove pre-existing stale annotations and labels