Repository: VictoriaMetrics/operator

Issue:
Support using DaemonSets
It would be very useful to be able to run at least some of the components as DaemonSets instead of scaling them manually. We have dedicated nodes for vmstorage for example and a DaemonSet would allow automatically scaling when adding a new node.

If this is already possible I couldn't find it from the docs.

Thanks!

Discussion:
@f41gh7:  Hello, currently it's not possible and we don't have such plans.

 I'd recommend to take look at [affinity](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/) rules for pod scheduling. It's possible to schedule pod per node with it.  @zekker6: Support of using DaemonSets might be helpful for vmagent and vmauth for cases of reducing inter-AZ traffic.
For example, vmagent can scrape metrics only from targets within the same AZ (by using relabeling + propagation of properties via downward API) and than route metrics to the destination via vmauth running on the same node.
@f41gh7: > Support of using DaemonSets might be helpful for vmagent and vmauth for cases of reducing inter-AZ traffic. For example, vmagent can scrape metrics only from targets within the same AZ (by using relabeling + propagation of properties via downward API) and than route metrics to the destination via vmauth running on the same node.

 It's already possible to get rid of cross-AZ traffic with labels. See https://tanmay-bhat.medium.com/reducing-inter-az-traffic-in-victoriametrics-with-zonekeeper-3bd7e1526796

 Also I'd not recommend to run vmagent as daemonset if it uses non-local service-discovery. Like `kubernetes_sd_configs`. It may have significant impact on Kubernetes api-servers and probably lead to whole cluster downtime ( like chatGPT had). @zekker6: @f41gh7 There is another use-case where applications are pushing metrics. In this case apps could push to a local address and vmagent would push the data to the receiver after applying compression.@f41gh7:  Okay, we agreed to implement this feature. However, it'll have some limitations.

 Goals:
* provide seamless migration from deployment/statefulset to daemonset
* daemonset mode must be compatible with the most common scrape objects - `VMPodScrape`, `VMServiceScrape`
* daemonset mode must efficiently utilize Kubernetes api-server. 

Limitations:
* HPA is not supported
* PDB is not supported
* Sharding is not supported
* storage could be hostPath or emptyDir
* VMNodeScrape, VMStaticScrape, VMScrapeConfig and VMProbeScrape are not supported


 Viable Kubernetes api-server interaction optimizations:
* role: pod must have field selector: 
 ```yaml
  selectors:
  - role: pod
    field: spec.nodeName=%{NODE_NAME}
```
* VMServiceScrape must use endpointslices instead of endpoints
* VMServiceScrape with discoveryRole = `service` must be rejected.
 @f41gh7: Feature was included into [v0.55.0 release](https://docs.victoriametrics.com/operator/changelog/#v0550)

Solution patch:
diff --git a/api/operator/v1beta1/vmagent_types.go b/api/operator/v1beta1/vmagent_types.go
index a8e650c2..84ca8991 100644
--- a/api/operator/v1beta1/vmagent_types.go
+++ b/api/operator/v1beta1/vmagent_types.go
@@ -281,6 +281,12 @@ type VMAgentSpec struct {
 	// MaxScrapeInterval allows limiting maximum scrape interval for VMServiceScrape, VMPodScrape and other scrapes
 	// If interval is higher than defined limit, `maxScrapeInterval` will be used.
 	MaxScrapeInterval *string `json:"maxScrapeInterval,omitempty"`
+	// DaemonSetMode enables DaemonSet deployment mode instead of Deployment.
+	// Supports only VMPodScrape
+	// (available from v0.55.0).
+	// Cannot be used with statefulMode
+	// +optional
+	DaemonSetMode bool `json:"daemonSetMode,omitempty"`
 	// StatefulMode enables StatefulSet for `VMAgent` instead of Deployment
 	// it allows using persistent storage for vmagent's persistentQueue
 	// +optional
@@ -359,7 +365,17 @@ func (cr *VMAgent) Validate() error {
 			}
 		}
 	}
-
+	if cr.Spec.DaemonSetMode && cr.Spec.StatefulMode {
+		return fmt.Errorf("daemonSetMode and statefulMode cannot be used in the same time")
+	}
+	if cr.Spec.DaemonSetMode {
+		if cr.Spec.PodDisruptionBudget != nil {
+			return fmt.Errorf("podDisruptionBudget cannot be used with daemonSetMode")
+		}
+		if cr.Spec.EnableKubernetesAPISelectors {
+			return fmt.Errorf("enableKubernetesAPISelectors cannot be used with daemonSetMode")
+		}
+	}
 	return nil
 }
 
diff --git a/config/crd/overlay/crd.yaml b/config/crd/overlay/crd.yaml
index aa3a126e..038cc276 100644
--- a/config/crd/overlay/crd.yaml
+++ b/config/crd/overlay/crd.yaml
@@ -1977,6 +1977,13 @@ spec:
                   type: object
                   x-kubernetes-preserve-unknown-fields: true
                 type: array
+              daemonSetMode:
+                description: |-
+                  DaemonSetMode enables DaemonSet deployment mode instead of Deployment.
+                  Supports only VMPodScrape
+                  (available from v0.55.0).
+                  Cannot be used with statefulMode
+                type: boolean
               disableAutomountServiceAccountToken:
                 description: |-
                   DisableAutomountServiceAccountToken whether to disable serviceAccount auto mount by Kubernetes (available from v0.54.0).
diff --git a/config/examples/vmagent_daemonset.yaml b/config/examples/vmagent_daemonset.yaml
new file mode 100644
index 00000000..635a6841
--- /dev/null
+++ b/config/examples/vmagent_daemonset.yaml
@@ -0,0 +1,27 @@
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMAgent
+metadata:
+  name: per-node
+spec:
+  selectAllByDefault: true
+  resources:
+    requests:
+      cpu: "50m"
+      memory: "350Mi"
+    limits:
+      cpu: "500m"
+      memory: "850Mi"
+  daemonSetMode: true
+  useVMConfigReloader: true
+  remoteWrite:
+    - url: "http://vmsingle-dc-1.default.svc:8429/api/v1/write"
+      maxDiskUsage: 5GB
+  remoteWriteSettings:
+    tmpDataPath: "/opt/queue"
+  volumes:
+    - name: persistent-queue
+      hostPath:
+        path: /opt/mnt/vmagent-queue
+  volumeMounts:
+    - mountPath: /opt/queue
+      name: persistent-queue
diff --git a/config/manifests/bases/victoriametrics-operator.clusterserviceversion.yaml b/config/manifests/bases/victoriametrics-operator.clusterserviceversion.yaml
index c7da0b07..6e83af3d 100644
--- a/config/manifests/bases/victoriametrics-operator.clusterserviceversion.yaml
+++ b/config/manifests/bases/victoriametrics-operator.clusterserviceversion.yaml
@@ -442,6 +442,14 @@ spec:
           - apps
           resources:
           - deployments
+          - deployments/finalizers
+          verbs:
+          - '*'
+        - apiGroups:
+          - apps
+          resources:
+          - daemonsets
+          - daemonsets/finalizers
           verbs:
           - '*'
         - apiGroups:
diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml
index 23c59a9f..7e262d26 100644
--- a/config/rbac/role.yaml
+++ b/config/rbac/role.yaml
@@ -49,6 +49,10 @@ rules:
   resources:
   - deployments
   - deployments/finalizers
+  - statefulsets
+  - statefulsets/finalizers
+  - daemonsets
+  - daemonsets/finalizers
   - replicasets
   - statefulsets
   - statefulsets/finalizers
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index 3a80754e..6d69d720 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -23,6 +23,7 @@ aliases:
 * FEATURE: [operator](https://docs.victoriametrics.com/operator: log fields changes diff for `Deployment`, `StatefulSet`, `Service`, `PDB`, `HPA` and `VMServiceScrape` during reconcile process. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1271) for details.
 * FEATURE: [operator](https://docs.victoriametrics.com/operator: add global env variables `VM_CONFIG_RELOADER_LIMIT_CPU` and `VM_CONFIG_RELOADER_LIMIT_MEMORY` with `unlimited` as default value. It controls global limits for config-reloader containers. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1283) for details.
 * FEATURE: [operator](https://docs.victoriametrics.com/operator: add global env variables `VM_CONFIG_RELOADER_REQUEST_CPU` and `VM_CONFIG_RELOADER_REQUEST_MEMORY` with empty as default value. It controls global requests for config-reloader containers. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1283) for details. All per resource config-reloader requests env variables are now deprecated.
+* FEATURE: [vmagent](https://docs.victoriametrics.com/operator/resources/vmagent/): introduce `daemonSetMode` as a beta feature. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1103) and this [docs](https://docs.victoriametrics.com/operator/resources/vmagent/#daemonSet-mode) for details.
 * FEATURE: [vmagent](https://docs.victoriametrics.com/operator/resources/vmagent/): reduce Kubernetes API server load on large scale by removing `selectors` from `VMPodscrape` `kubernetes_sd_configs`. Add new field `VMAgent.spec.enableKubernetesAPISelectors`, which restores original behaviour. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1283) for details.
 * FEATURE: [vmagent](https://docs.victoriametrics.com/operator/resources/vmagent/): allow to define `remoteWrite.MaxDiskUsage` as integer and adds validation to it. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1256) for details.
 * FEATURE: [vmagent](https://docs.victoriametrics.com/operator/resources/vmagent/): allow to define `remoteWriteSettings.maxDiskUsagePerURL` as string with bytes suffix and adds validation to it. See [this issue](https://github.com/VictoriaMetrics/operator/issues/1256) for details.
diff --git a/docs/api.md b/docs/api.md
index 3e4679ff..102a39da 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -2839,6 +2839,7 @@ _Appears in:_
 | <a href="#vmagentspec-configreloaderimagetag"><code id="vmagentspec-configreloaderimagetag">configReloaderImageTag</code></a><br/>_string_ | _(Optional)_<br/>ConfigReloaderImageTag defines image:tag for config-reloader container |
 | <a href="#vmagentspec-configreloaderresources"><code id="vmagentspec-configreloaderresources">configReloaderResources</code></a><br/>_[ResourceRequirements](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#resourcerequirements-v1-core)_ | _(Optional)_<br/>ConfigReloaderResources config-reloader container resource request and limits, https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/<br />if not defined default resources from operator config will be used |
 | <a href="#vmagentspec-containers"><code id="vmagentspec-containers">containers</code></a><br/>_[Container](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#container-v1-core) array_ | _(Optional)_<br/>Containers property allows to inject additions sidecars or to patch existing containers.<br />It can be useful for proxies, backup, etc. |
+| <a href="#vmagentspec-daemonsetmode"><code id="vmagentspec-daemonsetmode">daemonSetMode</code></a><br/>_boolean_ | _(Optional)_<br/>DaemonSetMode enables DaemonSet deployment mode instead of Deployment.<br />Supports only VMPodScrape<br />(available from v0.55.0).<br />Cannot be used with statefulMode |
 | <a href="#vmagentspec-disableautomountserviceaccounttoken"><code id="vmagentspec-disableautomountserviceaccounttoken">disableAutomountServiceAccountToken</code></a><br/>_boolean_ | _(Optional)_<br/>DisableAutomountServiceAccountToken whether to disable serviceAccount auto mount by Kubernetes (available from v0.54.0).<br />Operator will conditionally create volumes and volumeMounts for containers if it requires k8s API access.<br />For example, vmagent and vm-config-reloader requires k8s API access.<br />Operator creates volumes with name: "kube-api-access", which can be used as volumeMount for extraContainers if needed.<br />And also adds VolumeMounts at /var/run/secrets/kubernetes.io/serviceaccount. |
 | <a href="#vmagentspec-disableselfservicescrape"><code id="vmagentspec-disableselfservicescrape">disableSelfServiceScrape</code></a><br/>_boolean_ | _(Optional)_<br/>DisableSelfServiceScrape controls creation of VMServiceScrape by operator<br />for the application.<br />Has priority over `VM_DISABLESELFSERVICESCRAPECREATION` operator env variable |
 | <a href="#vmagentspec-dnsconfig"><code id="vmagentspec-dnsconfig">dnsConfig</code></a><br/>_[PodDNSConfig](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.30/#poddnsconfig-v1-core)_ | _(Optional)_<br/>Specifies the DNS parameters of a pod.<br />Parameters specified here will be merged to the generated DNS<br />configuration based on DNSPolicy. |
diff --git a/docs/resources/vmagent.md b/docs/resources/vmagent.md
index 9e4bc758..f4d64807 100644
--- a/docs/resources/vmagent.md
+++ b/docs/resources/vmagent.md
@@ -775,3 +775,48 @@ spec:
     - action: labeldrop
       regex: "temp.*"
 ```
+
+
+### DaemonSet mode
+
+ It's possible to configure vmagent to use [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) instead of Deployment and StatefulSet. Operator provides seamless transition between launch modes - daemonSetMode, statefulMode or defaultMode.
+
+Key features:
+* reduce network traffic for metric scrapping.
+* spread load for metrics collection.
+* provide resilience for single pod failure.
+
+ In this scenario, VMAgent's pods will be launched on each Kubernetes Node. Operator configures VMAgent to apply `spec.nodeName` pod [field selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/#list-of-supported-fields) for Kubernetes API requests.
+This field selector is only supported by `role: pod`, which could be used only with `VMPodScrape`. It limits scope of objects selectable by VMAgent.
+An example of configuration:
+```yaml
+kubernetes_sd_configs:
+- role: pod
+  namespaces:
+    names:
+    - default
+  selectors:
+  - role: pod
+    field: spec.nodeName=%{KUBE_NODE_NAME}
+```
+
+ An example of VMAgent object:
+```yaml
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMAgent
+metadata:
+  name: per-node
+spec:
+  selectAllByDefault: true
+  daemonSetMode: true
+  remoteWrite:
+    - url: "http://vmsingle-example.default.svc:8428/api/v1/write"
+```
+
+ daemonSetMode has the following restrictions and limitations:
+* sharding not supported.
+* podDisruptionPudget not supported.
+* horizontalPodAutoScraler not supported.
+* Volume for the persistent-queue could be mounted with `volumes` and must have either hostPath or emptyDir.
+* Only VMPodScrape supported.
+* vmagent restarts will lead to the small metric collection gaps. Only a single pod from DaemonSet deployed per node.
diff --git a/internal/controller/operator/factory/build/daemonset.go b/internal/controller/operator/factory/build/daemonset.go
new file mode 100644
index 00000000..8e8939f7
--- /dev/null
+++ b/internal/controller/operator/factory/build/daemonset.go
@@ -0,0 +1,36 @@
+package build
+
+import (
+	vmv1beta1 "github.com/VictoriaMetrics/operator/api/operator/v1beta1"
+
+	appsv1 "k8s.io/api/apps/v1"
+	"k8s.io/utils/ptr"
+)
+
+// DeploymentAddCommonParams adds common params for all deployments
+func DaemonSetAddCommonParams(dst *appsv1.DaemonSet, useStrictSecurity bool, params *vmv1beta1.CommonApplicationDeploymentParams) {
+	dst.Spec.Template.Spec.Affinity = params.Affinity
+	dst.Spec.Template.Spec.Tolerations = params.Tolerations
+	dst.Spec.Template.Spec.SchedulerName = params.SchedulerName
+	dst.Spec.Template.Spec.RuntimeClassName = params.RuntimeClassName
+	dst.Spec.Template.Spec.HostAliases = params.HostAliases
+	if len(params.HostAliasesUnderScore) > 0 {
+		dst.Spec.Template.Spec.HostAliases = params.HostAliasesUnderScore
+	}
+	dst.Spec.Template.Spec.PriorityClassName = params.PriorityClassName
+	dst.Spec.Template.Spec.HostNetwork = params.HostNetwork
+	dst.Spec.Template.Spec.DNSPolicy = params.DNSPolicy
+	dst.Spec.Template.Spec.DNSConfig = params.DNSConfig
+	dst.Spec.Template.Spec.NodeSelector = params.NodeSelector
+	dst.Spec.Template.Spec.SecurityContext = AddStrictSecuritySettingsToPod(params.SecurityContext, useStrictSecurity)
+	dst.Spec.Template.Spec.TerminationGracePeriodSeconds = params.TerminationGracePeriodSeconds
+	dst.Spec.Template.Spec.TopologySpreadConstraints = params.TopologySpreadConstraints
+	dst.Spec.Template.Spec.ImagePullSecrets = params.ImagePullSecrets
+	dst.Spec.Template.Spec.TerminationGracePeriodSeconds = params.TerminationGracePeriodSeconds
+	dst.Spec.Template.Spec.ReadinessGates = params.ReadinessGates
+	dst.Spec.MinReadySeconds = params.MinReadySeconds
+	dst.Spec.RevisionHistoryLimit = params.RevisionHistoryLimitCount
+	if params.DisableAutomountServiceAccountToken {
+		dst.Spec.Template.Spec.AutomountServiceAccountToken = ptr.To(false)
+	}
+}
diff --git a/internal/controller/operator/factory/build/vmservicescrape.go b/internal/controller/operator/factory/build/vmservicescrape.go
index 1cdb26f7..1b79a425 100644
--- a/internal/controller/operator/factory/build/vmservicescrape.go
+++ b/internal/controller/operator/factory/build/vmservicescrape.go
@@ -150,3 +150,84 @@ func vmServiceScrapeForServiceWithSpec(service *v1.Service, serviceScrapeSpec *v
 
 	return scrapeSvc
 }
+
+type podScrapeBuilder interface {
+	GetNamespace() string
+	PrefixedName() string
+	ProbePort() string
+	SelectorLabels() map[string]string
+	GetMetricPath() string
+}
+
+// VMPodScrapeForObjectWithSpec build VMPodScrape for given podScrapeBuilder with provided args
+// optionally could filter out ports from service
+func VMPodScrapeForObjectWithSpec(psb podScrapeBuilder, serviceScrapeSpec *vmv1beta1.VMServiceScrapeSpec, extraArgs map[string]string) *vmv1beta1.VMPodScrape {
+	var isTLS bool
+	v, ok := extraArgs["tls"]
+	if ok {
+		// tls is array flag type at VictoriaMetrics components
+		// use first value
+		firstIdx := strings.IndexByte(v, ',')
+		if firstIdx > 0 {
+			v = v[:firstIdx]
+		}
+		isTLS = strings.ToLower(v) == "true"
+	}
+	authKey := extraArgs["metricsAuthKey"]
+
+	defaultEP := vmv1beta1.PodMetricsEndpoint{
+		Port: ptr.To("http"),
+		EndpointScrapeParams: vmv1beta1.EndpointScrapeParams{
+			Path: psb.GetMetricPath(),
+		},
+	}
+	if isTLS {
+		defaultEP.Scheme = "https"
+		// add insecure by default
+		// if needed user will override it with direct config
+		defaultEP.TLSConfig = &vmv1beta1.TLSConfig{
+			InsecureSkipVerify: true,
+		}
+	}
+	if len(authKey) > 0 {
+		defaultEP.Params = map[string][]string{
+			"authKey": {authKey},
+		}
+	}
+
+	selectorLabels := psb.SelectorLabels()
+	podScrape := &vmv1beta1.VMPodScrape{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      psb.PrefixedName(),
+			Namespace: psb.GetNamespace(),
+			Labels:    selectorLabels,
+		},
+		Spec: vmv1beta1.VMPodScrapeSpec{
+			Selector: *metav1.SetAsLabelSelector(selectorLabels),
+			PodMetricsEndpoints: []vmv1beta1.PodMetricsEndpoint{
+				defaultEP,
+			},
+		},
+	}
+	if serviceScrapeSpec != nil {
+		for _, ssEP := range serviceScrapeSpec.Endpoints {
+			if ssEP.Port == *defaultEP.Port {
+				defaultEP.EndpointAuth = ssEP.EndpointAuth
+				defaultEP.EndpointScrapeParams = ssEP.EndpointScrapeParams
+				defaultEP.EndpointRelabelings = ssEP.EndpointRelabelings
+				continue
+			}
+			podScrape.Spec.PodMetricsEndpoints = append(podScrape.Spec.PodMetricsEndpoints, vmv1beta1.PodMetricsEndpoint{
+				Port:                 ptr.To(ssEP.Port),
+				EndpointAuth:         ssEP.EndpointAuth,
+				EndpointRelabelings:  ssEP.EndpointRelabelings,
+				EndpointScrapeParams: ssEP.EndpointScrapeParams,
+			})
+		}
+		podScrape.Spec.PodTargetLabels = serviceScrapeSpec.PodTargetLabels
+		podScrape.Spec.SampleLimit = serviceScrapeSpec.SampleLimit
+		podScrape.Spec.SeriesLimit = serviceScrapeSpec.SeriesLimit
+		podScrape.Spec.AttachMetadata = serviceScrapeSpec.AttachMetadata
+	}
+	return podScrape
+}
diff --git a/internal/controller/operator/factory/finalize/vmagent.go b/internal/controller/operator/factory/finalize/vmagent.go
index 9daa57f7..8567a9a3 100644
--- a/internal/controller/operator/factory/finalize/vmagent.go
+++ b/internal/controller/operator/factory/finalize/vmagent.go
@@ -21,6 +21,9 @@ func OnVMAgentDelete(ctx context.Context, rclient client.Client, crd *vmv1beta1.
 	if err := removeFinalizeObjByName(ctx, rclient, &appsv1.StatefulSet{}, crd.PrefixedName(), crd.Namespace); err != nil {
 		return err
 	}
+	if err := removeFinalizeObjByName(ctx, rclient, &appsv1.DaemonSet{}, crd.PrefixedName(), crd.Namespace); err != nil {
+		return err
+	}
 
 	if err := RemoveOrphanedDeployments(ctx, rclient, crd, nil); err != nil {
 		return err
diff --git a/internal/controller/operator/factory/reconcile/daemonset.go b/internal/controller/operator/factory/reconcile/daemonset.go
new file mode 100644
index 00000000..c0da360b
--- /dev/null
+++ b/internal/controller/operator/factory/reconcile/daemonset.go
@@ -0,0 +1,122 @@
+package reconcile
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	appsv1 "k8s.io/api/apps/v1"
+	"k8s.io/apimachinery/pkg/api/equality"
+	"k8s.io/apimachinery/pkg/api/errors"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/apimachinery/pkg/util/wait"
+	"k8s.io/client-go/util/retry"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	vmv1beta1 "github.com/VictoriaMetrics/operator/api/operator/v1beta1"
+	"github.com/VictoriaMetrics/operator/internal/controller/operator/factory/finalize"
+	"github.com/VictoriaMetrics/operator/internal/controller/operator/factory/logger"
+)
+
+// DaemonSet performs an update or create operator for daemonset and waits until it finishes update rollout
+func DaemonSet(ctx context.Context, rclient client.Client, newDs, prevDs *appsv1.DaemonSet) error {
+	var isPrevEqual bool
+	var prevSpecDiff string
+	if prevDs != nil {
+		isPrevEqual = equality.Semantic.DeepDerivative(prevDs.Spec, newDs.Spec)
+		if !isPrevEqual {
+			prevSpecDiff = diffDeepDerivative(prevDs.Spec, newDs.Spec)
+		}
+	}
+	rclient.Scheme().Default(newDs)
+
+	return retry.RetryOnConflict(retry.DefaultRetry, func() error {
+		var currentDs appsv1.DaemonSet
+		err := rclient.Get(ctx, types.NamespacedName{Name: newDs.Name, Namespace: newDs.Namespace}, &currentDs)
+		if err != nil {
+			if errors.IsNotFound(err) {
+				logger.WithContext(ctx).Info(fmt.Sprintf("creating new DaemonSet %s", newDs.Name))
+				if err := rclient.Create(ctx, newDs); err != nil {
+					return fmt.Errorf("cannot create new DaemonSet for app: %s, err: %w", newDs.Name, err)
+				}
+				return waitDaemonSetReady(ctx, rclient, newDs, appWaitReadyDeadline)
+			}
+			return fmt.Errorf("cannot get DaemonSet for app: %s err: %w", newDs.Name, err)
+		}
+		if err := finalize.FreeIfNeeded(ctx, rclient, &currentDs); err != nil {
+			return err
+		}
+		newDs.Status = currentDs.Status
+		var prevAnnotations, prevTemplateAnnotations map[string]string
+		if prevDs != nil {
+			prevAnnotations = prevDs.Annotations
+			prevTemplateAnnotations = prevDs.Spec.Template.Annotations
+		}
+		isEqual := equality.Semantic.DeepDerivative(newDs.Spec, currentDs.Spec)
+		if isEqual &&
+			isPrevEqual &&
+			equality.Semantic.DeepEqual(newDs.Labels, currentDs.Labels) &&
+			isAnnotationsEqual(currentDs.Annotations, newDs.Annotations, prevAnnotations) {
+			return waitDaemonSetReady(ctx, rclient, newDs, appWaitReadyDeadline)
+		}
+
+		vmv1beta1.AddFinalizer(newDs, &currentDs)
+		newDs.Annotations = mergeAnnotations(currentDs.Annotations, newDs.Annotations, prevAnnotations)
+		newDs.Spec.Template.Annotations = mergeAnnotations(currentDs.Spec.Template.Annotations, newDs.Spec.Template.Annotations, prevTemplateAnnotations)
+		cloneSignificantMetadata(newDs, &currentDs)
+
+		logMsg := fmt.Sprintf("updating DaemonSet %s configuration"+
+			"is_prev_equal=%v,is_current_equal=%v,is_prev_nil=%v",
+			newDs.Name, isPrevEqual, isEqual, prevDs == nil)
+
+		if len(prevSpecDiff) > 0 {
+			logMsg += fmt.Sprintf(", prev_spec_diff=%s", prevSpecDiff)
+		}
+		if !isEqual {
+			logMsg += fmt.Sprintf(", curr_spec_diff=%s", diffDeepDerivative(newDs.Spec, currentDs.Spec))
+		}
+
+		logger.WithContext(ctx).Info(logMsg)
+
+		if err := rclient.Update(ctx, newDs); err != nil {
+			return fmt.Errorf("cannot update DaemonSet for app: %s, err: %w", newDs.Name, err)
+		}
+
+		return waitDaemonSetReady(ctx, rclient, newDs, appWaitReadyDeadline)
+	})
+}
+
+// waitDeploymentReady waits until deployment's replicaSet rollouts and all new pods is ready
+func waitDaemonSetReady(ctx context.Context, rclient client.Client, ds *appsv1.DaemonSet, deadline time.Duration) error {
+	var isErrDealine bool
+	err := wait.PollUntilContextTimeout(ctx, time.Second, deadline, false, func(ctx context.Context) (done bool, err error) {
+		var daemon appsv1.DaemonSet
+		if err := rclient.Get(ctx, types.NamespacedName{Namespace: ds.Namespace, Name: ds.Name}, &daemon); err != nil {
+			return false, fmt.Errorf("cannot fetch actual daemonset state: %w", err)
+		}
+
+		// Based on recommendations from the kubernetes documentation
+		// this function uses the deployment readiness detection algorithm from `kubectl rollout status` command
+		// https://github.com/kubernetes/kubectl/blob/6e4fe32a45fdcbf61e5c30ebdc511d75e7242432/pkg/polymorphichelpers/rollout_status.go#L95
+		if daemon.Generation > daemon.Status.ObservedGeneration {
+			// Waiting for deployment spec update to be observed by controller...
+			return false, nil
+		}
+		if daemon.Status.UpdatedNumberScheduled < daemon.Status.DesiredNumberScheduled {
+			return false, nil
+		}
+		if daemon.Status.NumberAvailable < daemon.Status.DesiredNumberScheduled {
+			return false, nil
+		}
+		return true, nil
+	})
+	if err != nil {
+		podErr := reportFirstNotReadyPodOnError(ctx, rclient, fmt.Errorf("cannot wait for DaemonSet to become ready: %w", err), ds.Namespace, labels.SelectorFromSet(ds.Spec.Selector.MatchLabels), ds.Spec.MinReadySeconds)
+		if isErrDealine {
+			return err
+		}
+		return podErr
+	}
+	return nil
+}
diff --git a/internal/controller/operator/factory/reconcile/vmservicescrape.go b/internal/controller/operator/factory/reconcile/vmservicescrape.go
index 221fc2ba..63ae7f33 100644
--- a/internal/controller/operator/factory/reconcile/vmservicescrape.go
+++ b/internal/controller/operator/factory/reconcile/vmservicescrape.go
@@ -46,3 +46,35 @@ func VMServiceScrapeForCRD(ctx context.Context, rclient client.Client, vss *vmv1
 		return rclient.Update(ctx, &existVSS)
 	})
 }
+
+// VMPodScrapeForCRD creates or updates given object
+func VMPodScrapeForCRD(ctx context.Context, rclient client.Client, vps *vmv1beta1.VMPodScrape) error {
+	return retry.RetryOnConflict(retry.DefaultRetry, func() error {
+		var existVPS vmv1beta1.VMPodScrape
+		err := rclient.Get(ctx, types.NamespacedName{Namespace: vps.Namespace, Name: vps.Name}, &existVPS)
+		if err != nil {
+			if errors.IsNotFound(err) {
+				logger.WithContext(ctx).Info(fmt.Sprintf("creating VMPodScrape %s", vps.Name))
+				return rclient.Create(ctx, vps)
+			}
+			return err
+		}
+		if err := finalize.FreeIfNeeded(ctx, rclient, &existVPS); err != nil {
+			return err
+		}
+
+		if equality.Semantic.DeepEqual(vps.Spec, existVPS.Spec) &&
+			equality.Semantic.DeepEqual(vps.Labels, existVPS.Labels) &&
+			equality.Semantic.DeepEqual(vps.Annotations, existVPS.Annotations) {
+			return nil
+		}
+		// TODO: @f41gh7 allow 3rd party applications to add annotations for generated VMPodScrape
+		existVPS.Annotations = vps.Annotations
+		existVPS.Spec = vps.Spec
+		existVPS.Labels = vps.Labels
+		logMsg := fmt.Sprintf("updating VMPodScrape %s for CRD object spec_diff: %s", vps.Name, diffDeep(vps.Spec, existVPS.Spec))
+		logger.WithContext(ctx).Info(logMsg)
+
+		return rclient.Update(ctx, &existVPS)
+	})
+}
diff --git a/internal/controller/operator/factory/vmagent/collect_scrapes.go b/internal/controller/operator/factory/vmagent/collect_scrapes.go
index 505c342c..2e4622c3 100644
--- a/internal/controller/operator/factory/vmagent/collect_scrapes.go
+++ b/internal/controller/operator/factory/vmagent/collect_scrapes.go
@@ -13,6 +13,10 @@ import (
 )
 
 func selectScrapeConfig(ctx context.Context, cr *vmv1beta1.VMAgent, rclient client.Client) ([]*vmv1beta1.VMScrapeConfig, error) {
+	if cr.Spec.DaemonSetMode {
+		return nil, nil
+	}
+
 	var scrapeConfigsCombined []*vmv1beta1.VMScrapeConfig
 	var namespacedNames []string
 
@@ -60,6 +64,9 @@ func selectPodScrapes(ctx context.Context, cr *vmv1beta1.VMAgent, rclient client
 }
 
 func selectVMProbes(ctx context.Context, cr *vmv1beta1.VMAgent, rclient client.Client) ([]*vmv1beta1.VMProbe, error) {
+	if cr.Spec.DaemonSetMode {
+		return nil, nil
+	}
 	var probesCombined []*vmv1beta1.VMProbe
 	var namespacedNames []string
 	if err := k8stools.VisitObjectsForSelectorsAtNs(ctx, rclient, cr.Spec.ProbeNamespaceSelector, cr.Spec.ProbeSelector, cr.Namespace, cr.Spec.SelectAllByDefault,
@@ -82,6 +89,9 @@ func selectVMProbes(ctx context.Context, cr *vmv1beta1.VMAgent, rclient client.C
 }
 
 func selectVMNodeScrapes(ctx context.Context, cr *vmv1beta1.VMAgent, rclient client.Client) ([]*vmv1beta1.VMNodeScrape, error) {
+	if cr.Spec.DaemonSetMode {
+		return nil, nil
+	}
 	if !config.IsClusterWideAccessAllowed() && cr.IsOwnsServiceAccount() {
 		logger.WithContext(ctx).Info("cannot use VMNodeScrape at operator in single namespace mode with default permissions." +
 			" Create ServiceAccount for VMAgent manually if needed. Skipping config generation for it")
@@ -113,6 +123,9 @@ func selectVMNodeScrapes(ctx context.Context, cr *vmv1beta1.VMAgent, rclient cli
 }
 
 func selectStaticScrapes(ctx context.Context, cr *vmv1beta1.VMAgent, rclient client.Client) ([]*vmv1beta1.VMStaticScrape, error) {
+	if cr.Spec.DaemonSetMode {
+		return nil, nil
+	}
 	var staticScrapesCombined []*vmv1beta1.VMStaticScrape
 	var namespacedNames []string
 	if err := k8stools.VisitObjectsForSelectorsAtNs(ctx, rclient, cr.Spec.StaticScrapeNamespaceSelector, cr.Spec.StaticScrapeSelector, cr.Namespace, cr.Spec.SelectAllByDefault,
@@ -135,6 +148,10 @@ func selectStaticScrapes(ctx context.Context, cr *vmv1beta1.VMAgent, rclient cli
 }
 
 func selectServiceScrapes(ctx context.Context, cr *vmv1beta1.VMAgent, rclient client.Client) ([]*vmv1beta1.VMServiceScrape, error) {
+	if cr.Spec.DaemonSetMode {
+		return nil, nil
+	}
+
 	var servScrapesCombined []*vmv1beta1.VMServiceScrape
 	var namespacedNames []string
 	if err := k8stools.VisitObjectsForSelectorsAtNs(ctx, rclient, cr.Spec.ServiceScrapeNamespaceSelector, cr.Spec.ServiceScrapeSelector, cr.Namespace, cr.Spec.SelectAllByDefault,
diff --git a/internal/controller/operator/factory/vmagent/podscrape.go b/internal/controller/operator/factory/vmagent/podscrape.go
index 840d9f70..cdcb4e56 100644
--- a/internal/controller/operator/factory/vmagent/podscrape.go
+++ b/internal/controller/operator/factory/vmagent/podscrape.go
@@ -37,6 +37,9 @@ func generatePodScrapeConfig(
 		role:               kubernetesSDRolePod,
 		attachMetadata:     &ep.AttachMetadata,
 	}
+	if vmagentCR.Spec.DaemonSetMode {
+		k8sSDOpts.mustUseNodeSelector = true
+	}
 	cfg = append(cfg, generateK8SSDConfig(ssCache, k8sSDOpts))
 
 	// set defaults
diff --git a/internal/controller/operator/factory/vmagent/servicescrape.go b/internal/controller/operator/factory/vmagent/servicescrape.go
index 51b67e7a..84207f18 100644
--- a/internal/controller/operator/factory/vmagent/servicescrape.go
+++ b/internal/controller/operator/factory/vmagent/servicescrape.go
@@ -56,8 +56,6 @@ func generateServiceScrapeConfig(
 
 	var relabelings []yaml.MapSlice
 
-	// Filter targets by services selected by the scrape.
-
 	// Exact label matches.
 	skipRelabelSelectors := vmagentCR.Spec.EnableKubernetesAPISelectors
 	relabelings = addSelectorToRelabelingFor(relabelings, "service", m.Spec.Selector, skipRelabelSelectors)
diff --git a/internal/controller/operator/factory/vmagent/vmagent.go b/internal/controller/operator/factory/vmagent/vmagent.go
index 1a2391de..527c9735 100644
--- a/internal/controller/operator/factory/vmagent/vmagent.go
+++ b/internal/controller/operator/factory/vmagent/vmagent.go
@@ -43,6 +43,9 @@ const (
 	vmagentGzippedFilename = "vmagent.yaml.gz"
 	configEnvsubstFilename = "vmagent.env.yaml"
 	defaultMaxDiskUsage    = "1073741824"
+
+	kubeNodeEnvName     = "KUBE_NODE_NAME"
+	kubeNodeEnvTemplate = "%{" + kubeNodeEnvName + "}"
 )
 
 // To save compatibility in the single-shard version still need to fill in %SHARD_NUM% placeholder
@@ -94,7 +97,7 @@ func CreateOrUpdateVMAgent(ctx context.Context, cr *vmv1beta1.VMAgent, rclient c
 		prevCR = cr.DeepCopy()
 		prevCR.Spec = *cr.ParsedLastAppliedSpec
 	}
-	if err := deletePrevStateResources(ctx, cr, rclient); err != nil {
+	if err := deletePrevStateResources(ctx, rclient, cr, prevCR); err != nil {
 		return fmt.Errorf("cannot delete objects from prev state: %w", err)
 	}
 	if cr.IsOwnsServiceAccount() {
@@ -110,7 +113,6 @@ func CreateOrUpdateVMAgent(ctx context.Context, cr *vmv1beta1.VMAgent, rclient c
 				return fmt.Errorf("cannot create vmagent role and binding for it, err: %w", err)
 			}
 		}
-
 	}
 
 	svc, err := createOrUpdateVMAgentService(ctx, rclient, cr, prevCR)
@@ -119,9 +121,14 @@ func CreateOrUpdateVMAgent(ctx context.Context, cr *vmv1beta1.VMAgent, rclient c
 	}
 
 	if !ptr.Deref(cr.Spec.DisableSelfServiceScrape, false) {
-		err = reconcile.VMServiceScrapeForCRD(ctx, rclient, build.VMServiceScrapeForServiceWithSpec(svc, cr))
+		if cr.Spec.DaemonSetMode {
+			ps := build.VMPodScrapeForObjectWithSpec(cr, cr.Spec.ServiceScrapeSpec, cr.Spec.ExtraArgs)
+			err = reconcile.VMPodScrapeForCRD(ctx, rclient, ps)
+		} else {
+			err = reconcile.VMServiceScrapeForCRD(ctx, rclient, build.VMServiceScrapeForServiceWithSpec(svc, cr))
+		}
 		if err != nil {
-			return fmt.Errorf("cannot create serviceScrape: %w", err)
+			return fmt.Errorf("cannot create or update scrape object: %w", err)
 		}
 	}
 
@@ -138,7 +145,7 @@ func CreateOrUpdateVMAgent(ctx context.Context, cr *vmv1beta1.VMAgent, rclient c
 		return fmt.Errorf("cannot update stream aggregation config for vmagent: %w", err)
 	}
 
-	if cr.Spec.PodDisruptionBudget != nil {
+	if cr.Spec.PodDisruptionBudget != nil && !cr.Spec.DaemonSetMode {
 		var prevPDB *policyv1.PodDisruptionBudget
 		if prevCR != nil && prevCR.Spec.PodDisruptionBudget != nil {
 			prevPDB = build.PodDisruptionBudget(prevCR, prevCR.Spec.PodDisruptionBudget)
@@ -162,7 +169,7 @@ func CreateOrUpdateVMAgent(ctx context.Context, cr *vmv1beta1.VMAgent, rclient c
 		return fmt.Errorf("cannot build new deploy for vmagent: %w", err)
 	}
 
-	if cr.Spec.ShardCount != nil && *cr.Spec.ShardCount > 1 {
+	if !cr.Spec.DaemonSetMode && cr.Spec.ShardCount != nil && *cr.Spec.ShardCount > 1 {
 		return createOrUpdateShardedDeploy(ctx, rclient, cr, prevCR, newDeploy, prevDeploy)
 	}
 	return createOrUpdateDeploy(ctx, rclient, cr, prevCR, newDeploy, prevDeploy)
@@ -219,6 +226,19 @@ func createOrUpdateDeploy(ctx context.Context, rclient client.Client, cr, _ *vmv
 			return err
 		}
 		stsNames[newDeploy.Name] = struct{}{}
+	case *appsv1.DaemonSet:
+		var prevDeploy *appsv1.DaemonSet
+		if prevObjectSpec != nil {
+			prevAppObject, ok := prevObjectSpec.(*appsv1.DaemonSet)
+			if ok {
+				prevDeploy = prevAppObject
+			}
+		}
+		if err := reconcile.DaemonSet(ctx, rclient, newDeploy, prevDeploy); err != nil {
+			return err
+		}
+	default:
+		panic(fmt.Sprintf("BUG: unexpected deploy object type: %T", newDeploy))
 	}
 	if err := finalize.RemoveOrphanedDeployments(ctx, rclient, cr, deploymentNames); err != nil {
 		return err
@@ -226,6 +246,9 @@ func createOrUpdateDeploy(ctx context.Context, rclient client.Client, cr, _ *vmv
 	if err := finalize.RemoveOrphanedSTSs(ctx, rclient, cr, stsNames); err != nil {
 		return err
 	}
+	if err := removeStaleDaemonSet(ctx, rclient, cr); err != nil {
+		return fmt.Errorf("cannot remove vmagent daemonSet: %w", err)
+	}
 	return nil
 }
 
@@ -311,6 +334,9 @@ func createOrUpdateShardedDeploy(ctx context.Context, rclient client.Client, cr,
 	if err := finalize.RemoveOrphanedSTSs(ctx, rclient, cr, stsNames); err != nil {
 		return err
 	}
+	if err := removeStaleDaemonSet(ctx, rclient, cr); err != nil {
+		return fmt.Errorf("cannot remove vmagent daemonSet: %w", err)
+	}
 	return nil
 }
 
@@ -343,6 +369,35 @@ func newDeployForVMAgent(cr *vmv1beta1.VMAgent, ssCache *scrapesSecretsCache) (r
 	}
 	useStrictSecurity := ptr.Deref(cr.Spec.UseStrictSecurity, false)
 
+	if cr.Spec.DaemonSetMode {
+		dsSpec := &appsv1.DaemonSet{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:            cr.PrefixedName(),
+				Namespace:       cr.Namespace,
+				Labels:          cr.AllLabels(),
+				Annotations:     cr.AnnotationsFiltered(),
+				OwnerReferences: cr.AsOwner(),
+				Finalizers:      []string{vmv1beta1.FinalizerName},
+			},
+			Spec: appsv1.DaemonSetSpec{
+				Selector: &metav1.LabelSelector{
+					MatchLabels: cr.SelectorLabels(),
+				},
+				MinReadySeconds: cr.Spec.MinReadySeconds,
+				Template: corev1.PodTemplateSpec{
+					ObjectMeta: metav1.ObjectMeta{
+						Labels:      cr.PodLabels(),
+						Annotations: cr.PodAnnotations(),
+					},
+					Spec: *podSpec,
+				},
+			},
+		}
+		build.DaemonSetAddCommonParams(dsSpec, useStrictSecurity, &cr.Spec.CommonApplicationDeploymentParams)
+		dsSpec.Spec.Template.Spec.Volumes = build.AddServiceAccountTokenVolume(dsSpec.Spec.Template.Spec.Volumes, &cr.Spec.CommonApplicationDeploymentParams)
+
+		return dsSpec, nil
+	}
 	// fast path, use sts
 	if cr.Spec.StatefulMode {
 		stsSpec := &appsv1.StatefulSet{
@@ -452,6 +507,17 @@ func makeSpecForVMAgent(cr *vmv1beta1.VMAgent, ssCache *scrapesSecretsCache) (*c
 	var envs []corev1.EnvVar
 	envs = append(envs, cr.Spec.ExtraEnvs...)
 
+	if cr.Spec.DaemonSetMode {
+		envs = append(envs, corev1.EnvVar{
+			Name: kubeNodeEnvName,
+			ValueFrom: &corev1.EnvVarSource{
+				FieldRef: &corev1.ObjectFieldSelector{
+					FieldPath: "spec.nodeName",
+				},
+			},
+		})
+	}
+
 	var ports []corev1.ContainerPort
 	ports = append(ports, corev1.ContainerPort{Name: "http", Protocol: "TCP", ContainerPort: intstr.Parse(cr.Spec.Port).IntVal})
 	ports = build.AppendInsertPorts(ports, cr.Spec.InsertPorts)
@@ -1526,23 +1592,46 @@ func buildInitConfigContainer(useVMConfigReloader bool, cr *vmv1beta1.VMAgent, c
 	return []corev1.Container{initReloader}
 }
 
-func deletePrevStateResources(ctx context.Context, cr *vmv1beta1.VMAgent, rclient client.Client) error {
-	if cr.ParsedLastAppliedSpec == nil {
+func deletePrevStateResources(ctx context.Context, rclient client.Client, cr, prevCR *vmv1beta1.VMAgent) error {
+	if prevCR == nil {
 		return nil
 	}
 	// TODO check for stream aggr removed
 
-	prevSvc, currSvc := cr.ParsedLastAppliedSpec.ServiceSpec, cr.Spec.ServiceSpec
+	prevSvc, currSvc := prevCR.Spec.ServiceSpec, cr.Spec.ServiceSpec
 	if err := reconcile.AdditionalServices(ctx, rclient, cr.PrefixedName(), cr.Namespace, prevSvc, currSvc); err != nil {
 		return fmt.Errorf("cannot remove additional service: %w", err)
 	}
 	objMeta := metav1.ObjectMeta{Name: cr.PrefixedName(), Namespace: cr.Namespace}
-	if cr.Spec.PodDisruptionBudget == nil && cr.ParsedLastAppliedSpec.PodDisruptionBudget != nil {
+	if cr.Spec.PodDisruptionBudget == nil && prevCR.Spec.PodDisruptionBudget != nil {
 		if err := finalize.SafeDeleteWithFinalizer(ctx, rclient, &policyv1.PodDisruptionBudget{ObjectMeta: objMeta}); err != nil {
 			return fmt.Errorf("cannot delete PDB from prev state: %w", err)
 		}
 	}
 
+	if !prevCR.Spec.DaemonSetMode && cr.Spec.DaemonSetMode {
+		// transit into DaemonSetMode
+		if cr.Spec.PodDisruptionBudget != nil {
+			if err := finalize.SafeDeleteWithFinalizer(ctx, rclient, &policyv1.PodDisruptionBudget{ObjectMeta: objMeta}); err != nil {
+				return fmt.Errorf("cannot delete PDB from prev state: %w", err)
+			}
+		}
+		if !ptr.Deref(cr.Spec.DisableSelfServiceScrape, false) {
+			if err := finalize.SafeDeleteWithFinalizer(ctx, rclient, &vmv1beta1.VMServiceScrape{ObjectMeta: objMeta}); err != nil {
+				return fmt.Errorf("cannot delete VMServiceScrape during daemonset transition: %w", err)
+			}
+		}
+	}
+
+	if prevCR.Spec.DaemonSetMode && !cr.Spec.DaemonSetMode {
+		// transit into non DaemonSetMode
+		if !ptr.Deref(cr.Spec.DisableSelfServiceScrape, false) {
+			if err := finalize.SafeDeleteWithFinalizer(ctx, rclient, &vmv1beta1.VMPodScrape{ObjectMeta: objMeta}); err != nil {
+				return fmt.Errorf("cannot delete VMPodScrape during transition for non-daemonsetMode: %w", err)
+			}
+		}
+	}
+
 	if ptr.Deref(cr.Spec.DisableSelfServiceScrape, false) && !ptr.Deref(cr.ParsedLastAppliedSpec.DisableSelfServiceScrape, false) {
 		if err := finalize.SafeDeleteWithFinalizer(ctx, rclient, &vmv1beta1.VMServiceScrape{ObjectMeta: objMeta}); err != nil {
 			return fmt.Errorf("cannot remove serviceScrape: %w", err)
@@ -1551,3 +1640,16 @@ func deletePrevStateResources(ctx context.Context, cr *vmv1beta1.VMAgent, rclien
 
 	return nil
 }
+
+func removeStaleDaemonSet(ctx context.Context, rclient client.Client, cr *vmv1beta1.VMAgent) error {
+	if cr.Spec.DaemonSetMode {
+		return nil
+	}
+	ds := appsv1.DaemonSet{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      cr.PrefixedName(),
+			Namespace: cr.Namespace,
+		},
+	}
+	return finalize.SafeDeleteWithFinalizer(ctx, rclient, &ds)
+}
diff --git a/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig.go b/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig.go
index 23417f2a..3f2c9c0e 100644
--- a/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig.go
+++ b/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig.go
@@ -1373,12 +1373,13 @@ func getNamespacesFromNamespaceSelector(nsSelector *vmv1beta1.NamespaceSelector,
 }
 
 type generateK8SSDConfigOptions struct {
-	namespaces         []string
-	apiServerConfig    *vmv1beta1.APIServerConfig
-	role               string
-	attachMetadata     *vmv1beta1.AttachMetadata
-	shouldAddSelectors bool
-	selectors          metav1.LabelSelector
+	namespaces          []string
+	apiServerConfig     *vmv1beta1.APIServerConfig
+	role                string
+	attachMetadata      *vmv1beta1.AttachMetadata
+	shouldAddSelectors  bool
+	selectors           metav1.LabelSelector
+	mustUseNodeSelector bool
 }
 
 func generateK8SSDConfig(ssCache *scrapesSecretsCache, opts generateK8SSDConfigOptions) yaml.MapItem {
@@ -1436,9 +1437,23 @@ func generateK8SSDConfig(ssCache *scrapesSecretsCache, opts generateK8SSDConfigO
 		k8sSDConfig = addTLStoYaml(k8sSDConfig, "", apiserverConfig.TLSConfig, false)
 	}
 
+	var selectors []yaml.MapSlice
+
 	isEmptySelectors := len(opts.selectors.MatchLabels)+len(opts.selectors.MatchExpressions) == 0
-	if opts.shouldAddSelectors && !isEmptySelectors {
-		var selectors []yaml.MapSlice
+	switch {
+	case opts.mustUseNodeSelector:
+		var selector yaml.MapSlice
+		selector = append(selector, yaml.MapItem{
+			Key:   "role",
+			Value: kubernetesSDRolePod,
+		})
+		selector = append(selector, yaml.MapItem{
+			Key:   "field",
+			Value: "spec.nodeName=" + kubeNodeEnvTemplate,
+		})
+		selectors = append(selectors, selector)
+
+	case opts.shouldAddSelectors && !isEmptySelectors:
 		var selector yaml.MapSlice
 		selector = append(selector, yaml.MapItem{
 			Key:   "role",
@@ -1470,12 +1485,12 @@ func generateK8SSDConfig(ssCache *scrapesSecretsCache, opts generateK8SSDConfigO
 				})
 			}
 		}
-
+	}
+	if len(selectors) > 0 {
 		k8sSDConfig = append(k8sSDConfig, yaml.MapItem{
 			Key:   "selectors",
 			Value: selectors,
 		})
-
 	}
 
 	return yaml.MapItem{
diff --git a/internal/controller/operator/vmagent_controller.go b/internal/controller/operator/vmagent_controller.go
index 3eca633f..2e5d1929 100644
--- a/internal/controller/operator/vmagent_controller.go
+++ b/internal/controller/operator/vmagent_controller.go
@@ -74,6 +74,9 @@ func (r *VMAgentReconciler) Init(rclient client.Client, l logr.Logger, sc *runti
 // +kubebuilder:rbac:groups="rbac.authorization.k8s.io",resources=clusterrolebindings,verbs=get;create,update;list
 // +kubebuilder:rbac:groups="rbac.authorization.k8s.io",resources=clusterroles,verbs=get;create,update;list
 // +kubebuilder:rbac:groups="",resources=serviceaccounts,verbs=get;create,update;list
+// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=*
+// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=*
+// +kubebuilder:rbac:groups=apps,resources=daemonsets,verbs=*
 func (r *VMAgentReconciler) Reconcile(ctx context.Context, req ctrl.Request) (result ctrl.Result, err error) {
 	reqLogger := r.Log.WithValues("vmagent", req.Name, "namespace", req.Namespace)
 	ctx = logger.AddToContext(ctx, reqLogger)
diff --git a/internal/controller/operator/vmnodescrape_controller.go b/internal/controller/operator/vmnodescrape_controller.go
index 08dc9dc1..2df65baf 100644
--- a/internal/controller/operator/vmnodescrape_controller.go
+++ b/internal/controller/operator/vmnodescrape_controller.go
@@ -93,7 +93,9 @@ func (r *VMNodeScrapeReconciler) Reconcile(ctx context.Context, req ctrl.Request
 		currentVMagent := &vmagentItem
 		reqLogger := reqLogger.WithValues("vmagent", currentVMagent.Name, "parent_namespace", currentVMagent.Namespace)
 		ctx := logger.AddToContext(ctx, reqLogger)
-
+		if currentVMagent.Spec.DaemonSetMode {
+			continue
+		}
 		// only check selector when deleting object,
 		// since labels can be changed when updating and we can't tell if it was selected before, and we can't tell if it's creating or updating.
 		if !instance.DeletionTimestamp.IsZero() {
diff --git a/internal/controller/operator/vmprobe_controller.go b/internal/controller/operator/vmprobe_controller.go
index d5196955..74e6c821 100644
--- a/internal/controller/operator/vmprobe_controller.go
+++ b/internal/controller/operator/vmprobe_controller.go
@@ -88,6 +88,9 @@ func (r *VMProbeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (re
 			continue
 		}
 		currentVMagent := &vmagentItem
+		if currentVMagent.Spec.DaemonSetMode {
+			continue
+		}
 		reqLogger := reqLogger.WithValues("vmagent", currentVMagent.Name, "parent_namespace", currentVMagent.Namespace)
 		ctx := logger.AddToContext(ctx, reqLogger)
 
diff --git a/internal/controller/operator/vmscrapeconfig_controller.go b/internal/controller/operator/vmscrapeconfig_controller.go
index 03e46ce9..912357fc 100644
--- a/internal/controller/operator/vmscrapeconfig_controller.go
+++ b/internal/controller/operator/vmscrapeconfig_controller.go
@@ -89,7 +89,9 @@ func (r *VMScrapeConfigReconciler) Reconcile(ctx context.Context, req ctrl.Reque
 		currentVMagent := &vmagentItem
 		reqLogger := reqLogger.WithValues("vmagent", currentVMagent.Name, "parent_namespace", currentVMagent.Namespace)
 		ctx := logger.AddToContext(ctx, reqLogger)
-
+		if currentVMagent.Spec.DaemonSetMode {
+			continue
+		}
 		// only check selector when deleting object,
 		// since labels can be changed when updating and we can't tell if it was selected before, and we can't tell if it's creating or updating.
 		if !instance.DeletionTimestamp.IsZero() {
diff --git a/internal/controller/operator/vmservicescrape_controller.go b/internal/controller/operator/vmservicescrape_controller.go
index 27e04f1a..d20929b1 100644
--- a/internal/controller/operator/vmservicescrape_controller.go
+++ b/internal/controller/operator/vmservicescrape_controller.go
@@ -90,6 +90,9 @@ func (r *VMServiceScrapeReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 		currentVMagent := &vmagentItem
 		reqLogger := reqLogger.WithValues("vmagent", currentVMagent.Name, "parent_namespace", currentVMagent.Namespace)
 		ctx := logger.AddToContext(ctx, reqLogger)
+		if currentVMagent.Spec.DaemonSetMode {
+			continue
+		}
 		// only check selector when deleting object,
 		// since labels can be changed when updating and we can't tell if it was selected before, and we can't tell if it's creating or updating.
 		if !instance.DeletionTimestamp.IsZero() {
diff --git a/internal/controller/operator/vmstaticscrape_controller.go b/internal/controller/operator/vmstaticscrape_controller.go
index 00c4e982..2b48e08b 100644
--- a/internal/controller/operator/vmstaticscrape_controller.go
+++ b/internal/controller/operator/vmstaticscrape_controller.go
@@ -70,7 +70,9 @@ func (r *VMStaticScrapeReconciler) Reconcile(ctx context.Context, req ctrl.Reque
 		currentVMagent := &vmagentItem
 		reqLogger := reqLogger.WithValues("vmagent", currentVMagent.Name, "parent_namespace", currentVMagent.Namespace)
 		ctx := logger.AddToContext(ctx, reqLogger)
-
+		if currentVMagent.Spec.DaemonSetMode {
+			continue
+		}
 		// only check selector when deleting object,
 		// since labels can be changed when updating and we can't tell if it was selected before, and we can't tell if it's creating or updating.
 		if !instance.DeletionTimestamp.IsZero() {



Test patch:
diff --git a/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig_test.go b/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig_test.go
index c024ff05..1ac73dfa 100644
--- a/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig_test.go
+++ b/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig_test.go
@@ -1572,6 +1572,130 @@ scrape_configs:
       ca_file: /etc/vmagent-tls/certs/default_configmap_tls-default_CA
       cert_file: /etc/vmagent-tls/certs/default_tls-auth_CERT
       key_file: /etc/vmagent-tls/certs/default_tls-auth_SECRET_KEY
+`,
+		},
+		{
+			name: "daemonset mode",
+			args: args{
+				cr: &vmv1beta1.VMAgent{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "per-node",
+						Namespace: "default",
+					},
+					Spec: vmv1beta1.VMAgentSpec{
+						DaemonSetMode:      true,
+						SelectAllByDefault: true,
+					},
+				},
+			},
+			predefinedObjects: []runtime.Object{
+				&corev1.Namespace{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: "default-2",
+					},
+				},
+				&corev1.Namespace{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: "system",
+					},
+				},
+				&vmv1beta1.VMServiceScrape{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "svc-1",
+						Namespace: "default-1",
+					},
+					Spec: vmv1beta1.VMServiceScrapeSpec{
+						Endpoints: []vmv1beta1.Endpoint{
+							{
+								Port: "http",
+							},
+						},
+					},
+				},
+				&vmv1beta1.VMPodScrape{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "pod-1",
+						Namespace: "default",
+					},
+					Spec: vmv1beta1.VMPodScrapeSpec{
+						PodMetricsEndpoints: []vmv1beta1.PodMetricsEndpoint{
+							{
+								Port: ptr.To("web"),
+							},
+							{
+								PortNumber: ptr.To(int32(8085)),
+							},
+						},
+					},
+				},
+			},
+			wantConfig: `global:
+  scrape_interval: 30s
+  external_labels:
+    prometheus: default/per-node
+scrape_configs:
+- job_name: podScrape/default/pod-1/0
+  kubernetes_sd_configs:
+  - role: pod
+    namespaces:
+      names:
+      - default
+    selectors:
+    - role: pod
+      field: spec.nodeName=%{KUBE_NODE_NAME}
+  honor_labels: false
+  relabel_configs:
+  - action: drop
+    source_labels:
+    - __meta_kubernetes_pod_phase
+    regex: (Failed|Succeeded)
+  - action: keep
+    source_labels:
+    - __meta_kubernetes_pod_container_port_name
+    regex: web
+  - source_labels:
+    - __meta_kubernetes_namespace
+    target_label: namespace
+  - source_labels:
+    - __meta_kubernetes_pod_container_name
+    target_label: container
+  - source_labels:
+    - __meta_kubernetes_pod_name
+    target_label: pod
+  - target_label: job
+    replacement: default/pod-1
+  - target_label: endpoint
+    replacement: web
+- job_name: podScrape/default/pod-1/1
+  kubernetes_sd_configs:
+  - role: pod
+    namespaces:
+      names:
+      - default
+    selectors:
+    - role: pod
+      field: spec.nodeName=%{KUBE_NODE_NAME}
+  honor_labels: false
+  relabel_configs:
+  - action: drop
+    source_labels:
+    - __meta_kubernetes_pod_phase
+    regex: (Failed|Succeeded)
+  - action: keep
+    source_labels:
+    - __meta_kubernetes_pod_container_port_number
+    regex: 8085
+  - source_labels:
+    - __meta_kubernetes_namespace
+    target_label: namespace
+  - source_labels:
+    - __meta_kubernetes_pod_container_name
+    target_label: container
+  - source_labels:
+    - __meta_kubernetes_pod_name
+    target_label: pod
+  - target_label: job
+    replacement: default/pod-1
 `,
 		},
 	}
diff --git a/test/e2e/vmagent_test.go b/test/e2e/vmagent_test.go
index 476ad319..8879ca36 100644
--- a/test/e2e/vmagent_test.go
+++ b/test/e2e/vmagent_test.go
@@ -560,6 +560,42 @@ var _ = Describe("test  vmagent Controller", func() {
 					},
 				},
 			),
+			Entry("by transition into daemonSet and back", "daemonset-transition",
+				&v1beta1vm.VMAgent{Spec: v1beta1vm.VMAgentSpec{
+					CommonApplicationDeploymentParams: v1beta1vm.CommonApplicationDeploymentParams{
+						ReplicaCount: ptr.To[int32](1),
+					},
+					RemoteWrite: []v1beta1vm.VMAgentRemoteWriteSpec{
+						{URL: "http://some-vm-single:8428"},
+					},
+				},
+				},
+				testStep{
+					modify: func(cr *v1beta1vm.VMAgent) { cr.Spec.DaemonSetMode = true },
+					verify: func(cr *v1beta1vm.VMAgent) {
+						nsn := types.NamespacedName{Namespace: namespace, Name: cr.PrefixedName()}
+						Expect(k8sClient.Get(ctx, nsn, &appsv1.DaemonSet{})).To(Succeed())
+						Expect(k8sClient.Get(ctx, nsn, &v1beta1vm.VMPodScrape{})).To(Succeed())
+						Expect(k8sClient.Get(ctx, nsn, &appsv1.Deployment{})).To(MatchError(errors.IsNotFound, "IsNotFound"))
+						Expect(k8sClient.Get(ctx, nsn, &v1beta1vm.VMServiceScrape{})).To(MatchError(errors.IsNotFound, "IsNotFound"))
+					},
+				},
+				testStep{
+					modify: func(cr *v1beta1vm.VMAgent) {
+						cr.Spec.StatefulMode = true
+						cr.Spec.DaemonSetMode = false
+					},
+					verify: func(cr *v1beta1vm.VMAgent) {
+						nsn := types.NamespacedName{Namespace: namespace, Name: cr.PrefixedName()}
+						Expect(k8sClient.Get(ctx, nsn, &appsv1.StatefulSet{})).To(Succeed())
+						Expect(k8sClient.Get(ctx, nsn, &v1beta1vm.VMServiceScrape{})).To(Succeed())
+						Expect(k8sClient.Get(ctx, nsn, &appsv1.DaemonSet{})).To(MatchError(errors.IsNotFound, "IsNotFound"))
+						Expect(k8sClient.Get(ctx, nsn, &appsv1.Deployment{})).To(MatchError(errors.IsNotFound, "IsNotFound"))
+						Expect(k8sClient.Get(ctx, nsn, &v1beta1vm.VMPodScrape{})).To(MatchError(errors.IsNotFound, "IsNotFound"))
+
+					},
+				},
+			),
 		)
 	})
 })