Repository: VictoriaMetrics/VictoriaMetrics
Pull request #2838 (2022-07-06 14:21:18)

docs: mention deduplication issue for HA vmalert topology
Signed-off-by: hagen1778 <roman@victoriametrics.com>


Patch:
diff --git a/app/vmalert/README.md b/app/vmalert/README.md
index 52e2a6d0f..3b65798e7 100644
--- a/app/vmalert/README.md
+++ b/app/vmalert/README.md
@@ -390,6 +390,10 @@ Alertmanagers.
 To avoid recording rules results and alerts state duplication in VictoriaMetrics server
 don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication).
 The recommended value for `-dedup.minScrapeInterval` must be greater or equal to vmalert's `evaluation_interval`.
+If you observe inconsistent or "jumping" values in series produced by vmalert, try disabling `-datasource.queryTimeAlignment`
+command line flag. Because of alignment, two or more vmalert HA pairs will produce results with the same timestamps. 
+But due of backfilling (data delivered to the datasource with some delay) values of such results may differ, 
+which would affect deduplication logic and result into "jumping" datapoints.
 
 Alertmanager will automatically deduplicate alerts with identical labels, so ensure that
 all `vmalert`s are having the same config.
diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md
index ab488bb0a..3c9dacbf9 100644
--- a/docs/CHANGELOG.md
+++ b/docs/CHANGELOG.md
@@ -23,6 +23,7 @@ If you use alerting rules or Grafana dashboards, which rely on this metric, then
 * FEATURE: [vmui](https://docs.victoriametrics.com/#vmui): add an UI for [query tracing](https://docs.victoriametrics.com/#query-tracing). It can be enabled by clicking `enable query tracing` checkbox and re-running the query. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2703).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add `-remoteWrite.headers` command-line option for specifying optional HTTP headers to send to the configured `-remoteWrite.url`. For example, `-remoteWrite.headers='Foo:Bar^^Baz:x'` would send `Foo: Bar` and `Baz: x` HTTP headers with every request to `-remoteWrite.url`. See [this feature request](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2805).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): push per-target `scrape_samples_limit` metric to the cofigured `-remoteWrite.url` if `sample_limit` option is set for this target in [scrape_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config). See [this feature request](https://github.com/VictoriaMetrics/operator/issues/497).
+* FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): attach node-level labels to [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config) targets if `attach_metadata: {"node": true}` is set for `role: endpoints` and `role: endpointslice`. This is a feature backport from Prometheus 2.37 - see [this pull request](https://github.com/prometheus/prometheus/pull/10759).
 * FEATURE: [vmagent](https://docs.victoriametrics.com/vmagent.html): add ability to specify additional HTTP headers to send to scrape targets via `headers` section in `scrape_configs`. This can be used when the scrape target requires custom authorization and authentication like in [this stackoverflow question](https://stackoverflow.com/questions/66032498/prometheus-scrape-metric-with-custom-header). For example, the following config instructs sending `My-Auth: top-secret` and `TenantID: FooBar` headers with each request to `http://host123:8080/metrics`:
 
 ```yaml
@@ -57,6 +58,7 @@ scrape_configs:
 * BUGFIX: limit max memory occupied by the cache, which stores parsed regular expressions. Previously too long regular expressions passed in [MetricsQL queries](https://docs.victoriametrics.com/MetricsQL.html) could result in big amounts of used memory (e.g. multiple of gigabytes). Now the max cache size for parsed regexps is limited to a a few megabytes.
 * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly handle partial counter resets when calculating [rate](https://docs.victoriametrics.com/MetricsQL.html#rate), [irate](https://docs.victoriametrics.com/MetricsQL.html#irate) and [increase](https://docs.victoriametrics.com/MetricsQL.html#increase) functions. Previously these functions could return zero values after partial counter resets until the counter increases to the last value before partial counter reset. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2787).
 * BUGFIX: [MetricsQL](https://docs.victoriametrics.com/MetricsQL.html): properly calculate [histogram_quantile](https://docs.victoriametrics.com/MetricsQL.html#histogram_quantile) over Prometheus buckets with unexpected values. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2819).
+* BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly add service-level labels (`__meta_kubernetes_service_*`) to discovered targets for `role: endpointslice` in [kubernetes_sd_config](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config). Previously these labels were missing. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/2823).
 * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): make sure that [stale markers](https://docs.victoriametrics.com/vmagent.html#prometheus-staleness-markers) are generated with the actual timestamp when unsuccessful scrape occurs. This should prevent from possible time series overlap on scrape target restart in dynmaic envirnoments such as Kubernetes.
 * BUGFIX: [vmagent](https://docs.victoriametrics.com/vmagent.html): properly reload changed `-promscrape.config` file when `-promscrape.configCheckInterval` option is set. The changed config file wasn't reloaded in this case since [v1.69.0](#v1690). See [this pull request](https://github.com/VictoriaMetrics/VictoriaMetrics/pull/2786). Thanks to @ttyv for the fix.
 * BUGFIX: [VictoriaMetrics cluster](https://docs.victoriametrics.com/Cluster-VictoriaMetrics.html): assume that the response is complete if `-search.denyPartialResponse` is enabled and up to `-replicationFactor - 1` `vmstorage` nodes are unavailable. See [this issue](https://github.com/VictoriaMetrics/VictoriaMetrics/issues/1767).
diff --git a/docs/vmalert.md b/docs/vmalert.md
index 75166636d..e11cb61b7 100644
--- a/docs/vmalert.md
+++ b/docs/vmalert.md
@@ -394,6 +394,10 @@ Alertmanagers.
 To avoid recording rules results and alerts state duplication in VictoriaMetrics server
 don't forget to configure [deduplication](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#deduplication).
 The recommended value for `-dedup.minScrapeInterval` must be greater or equal to vmalert's `evaluation_interval`.
+If you observe inconsistent or "jumping" values in series produced by vmalert, try disabling `-datasource.queryTimeAlignment`
+command line flag. Because of alignment, two or more vmalert HA pairs will produce results with the same timestamps. 
+But due of backfilling (data delivered to the datasource with some delay) values of such results may differ, 
+which would affect deduplication logic and result into "jumping" datapoints.
 
 Alertmanager will automatically deduplicate alerts with identical labels, so ensure that
 all `vmalert`s are having the same config.
diff --git a/lib/promscrape/discovery/kubernetes/api_watcher.go b/lib/promscrape/discovery/kubernetes/api_watcher.go
index 63a266afb..a4afcd200 100644
--- a/lib/promscrape/discovery/kubernetes/api_watcher.go
+++ b/lib/promscrape/discovery/kubernetes/api_watcher.go
@@ -345,7 +345,7 @@ func (gw *groupWatcher) startWatchersForRole(role string, aw *apiWatcher) {
 		gw.startWatchersForRole("pod", nil)
 		gw.startWatchersForRole("service", nil)
 	}
-	if gw.attachNodeMetadata && role == "pod" {
+	if gw.attachNodeMetadata && (role == "pod" || role == "endpoints" || role == "endpointslice") {
 		gw.startWatchersForRole("node", nil)
 	}
 	paths := getAPIPathsWithNamespaces(role, gw.namespaces, gw.selectors)
@@ -803,8 +803,8 @@ func (uw *urlWatcher) maybeUpdateDependedScrapeWorksLocked() {
 			uwx.needRecreateScrapeWorks = true
 			continue
 		}
-		if attachNodeMetadata && role == "node" && uwx.role == "pod" {
-			// pod objects depend on node objects if attachNodeMetadata is set
+		if attachNodeMetadata && role == "node" && (uwx.role == "pod" || uwx.role == "endpoints" || uwx.role == "endpointslice") {
+			// pod, endpoints and enpointslices objects depend on node objects if attachNodeMetadata is set
 			uwx.needRecreateScrapeWorks = true
 			continue
 		}
diff --git a/lib/promscrape/discovery/kubernetes/endpointslice.go b/lib/promscrape/discovery/kubernetes/endpointslice.go
index 8e2244989..f4a2a07ac 100644
--- a/lib/promscrape/discovery/kubernetes/endpointslice.go
+++ b/lib/promscrape/discovery/kubernetes/endpointslice.go
@@ -38,8 +38,11 @@ func parseEndpointSlice(data []byte) (object, error) {
 //
 // See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpointslices
 func (eps *EndpointSlice) getTargetLabels(gw *groupWatcher) []map[string]string {
+	// The associated service name is stored in kubernetes.io/service-name label.
+	// See https://kubernetes.io/docs/reference/labels-annotations-taints/#kubernetesioservice-name
+	svcName := eps.Metadata.Labels.GetByName("kubernetes.io/service-name")
 	var svc *Service
-	if o := gw.getObjectByRoleLocked("service", eps.Metadata.Namespace, eps.Metadata.Name); o != nil {
+	if o := gw.getObjectByRoleLocked("service", eps.Metadata.Namespace, svcName); o != nil {
 		svc = o.(*Service)
 	}
 	podPortsSeen := make(map[*Pod][]int)


Test patch:
diff --git a/lib/promscrape/discovery/kubernetes/endpoints_test.go b/lib/promscrape/discovery/kubernetes/endpoints_test.go
index 845d92001..bbd7c296b 100644
--- a/lib/promscrape/discovery/kubernetes/endpoints_test.go
+++ b/lib/promscrape/discovery/kubernetes/endpoints_test.go
@@ -114,202 +114,250 @@ func TestParseEndpointsListSuccess(t *testing.T) {
 	}
 }
 
-func TestGetEndpointLabels(t *testing.T) {
+func TestGetEndpointsLabels(t *testing.T) {
 	type testArgs struct {
 		containerPorts map[string][]ContainerPort
 		endpointPorts  []EndpointPort
 	}
-	f := func(name string, args testArgs, wantLabels [][]prompbmarshal.Label) {
-		t.Run(name, func(t *testing.T) {
-			eps := Endpoints{
-				Metadata: ObjectMeta{
-					Name:      "test-eps",
-					Namespace: "default",
-				},
-				Subsets: []EndpointSubset{
-					{
-						Ports: args.endpointPorts,
-						Addresses: []EndpointAddress{
-							{
-								IP: "10.13.15.15",
-								TargetRef: ObjectReference{
-									Kind:      "Pod",
-									Namespace: "default",
-									Name:      "test-pod",
-								},
+	f := func(t *testing.T, args testArgs, wantLabels [][]prompbmarshal.Label) {
+		t.Helper()
+		eps := Endpoints{
+			Metadata: ObjectMeta{
+				Name:      "test-eps",
+				Namespace: "default",
+			},
+			Subsets: []EndpointSubset{
+				{
+					Ports: args.endpointPorts,
+					Addresses: []EndpointAddress{
+						{
+							IP: "10.13.15.15",
+							TargetRef: ObjectReference{
+								Kind:      "Pod",
+								Namespace: "default",
+								Name:      "test-pod",
 							},
 						},
 					},
 				},
-			}
-			svc := Service{
-				Metadata: ObjectMeta{
-					Name:      "test-eps",
-					Namespace: "default",
+			},
+		}
+		svc := Service{
+			Metadata: ObjectMeta{
+				Name:      "test-eps",
+				Namespace: "default",
+			},
+			Spec: ServiceSpec{
+				ClusterIP: "1.2.3.4",
+				Type:      "service-type",
+				Ports: []ServicePort{
+					{
+						Name: "test-port",
+						Port: 8081,
+					},
 				},
-				Spec: ServiceSpec{
-					Ports: []ServicePort{
-						{
-							Name: "test-port",
-							Port: 8081,
-						},
+			},
+		}
+		pod := Pod{
+			Metadata: ObjectMeta{
+				UID:       "pod-uid",
+				Name:      "test-pod",
+				Namespace: "default",
+			},
+			Spec: PodSpec{
+				NodeName: "test-node",
+			},
+			Status: PodStatus{
+				Phase:  "abc",
+				PodIP:  "192.168.15.1",
+				HostIP: "4.5.6.7",
+			},
+		}
+		node := Node{
+			Metadata: ObjectMeta{
+				Labels: []prompbmarshal.Label{
+					{
+						Name:  "node-label",
+						Value: "xyz",
 					},
 				},
-			}
-			pod := Pod{
-				Metadata: ObjectMeta{
-					Name:      "test-pod",
-					Namespace: "default",
+			},
+		}
+		for cn, ports := range args.containerPorts {
+			pod.Spec.Containers = append(pod.Spec.Containers, Container{Name: cn, Ports: ports})
+		}
+		var gw groupWatcher
+		gw.m = map[string]*urlWatcher{
+			"pod": {
+				role: "pod",
+				objectsByKey: map[string]object{
+					"default/test-pod": &pod,
 				},
-				Status: PodStatus{PodIP: "192.168.15.1"},
-			}
-			for cn, ports := range args.containerPorts {
-				pod.Spec.Containers = append(pod.Spec.Containers, Container{Name: cn, Ports: ports})
-			}
-			var gw groupWatcher
-			gw.m = map[string]*urlWatcher{
-				"pod": {
-					role: "pod",
-					objectsByKey: map[string]object{
-						"default/test-pod": &pod,
-					},
+			},
+			"service": {
+				role: "service",
+				objectsByKey: map[string]object{
+					"default/test-eps": &svc,
 				},
-				"service": {
-					role: "service",
-					objectsByKey: map[string]object{
-						"default/test-eps": &svc,
-					},
+			},
+			"node": {
+				role: "node",
+				objectsByKey: map[string]object{
+					"/test-node": &node,
 				},
-			}
-			var sortedLabelss [][]prompbmarshal.Label
-			gotLabels := eps.getTargetLabels(&gw)
-			for _, lbs := range gotLabels {
-				sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(lbs))
-			}
-			if !areEqualLabelss(sortedLabelss, wantLabels) {
-				t.Fatalf("unexpected labels:\ngot\n%v\nwant\n%v", sortedLabelss, wantLabels)
-
-			}
-		})
+			},
+		}
+		gw.attachNodeMetadata = true
+		var sortedLabelss [][]prompbmarshal.Label
+		gotLabels := eps.getTargetLabels(&gw)
+		for _, lbs := range gotLabels {
+			sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(lbs))
+		}
+		if !areEqualLabelss(sortedLabelss, wantLabels) {
+			t.Fatalf("unexpected labels:\ngot\n%v\nwant\n%v", sortedLabelss, wantLabels)
+		}
 	}
 
-	f("1 port from endpoint", testArgs{
-		endpointPorts: []EndpointPort{
-			{
-				Name: "web",
-				Port: 8081,
+	t.Run("1 port from endpoint", func(t *testing.T) {
+		f(t, testArgs{
+			endpointPorts: []EndpointPort{
+				{
+					Name:     "web",
+					Port:     8081,
+					Protocol: "foobar",
+				},
 			},
-		},
-	}, [][]prompbmarshal.Label{
-		discoveryutils.GetSortedLabels(map[string]string{
-			"__address__": "10.13.15.15:8081",
-			"__meta_kubernetes_endpoint_address_target_kind": "Pod",
-			"__meta_kubernetes_endpoint_address_target_name": "test-pod",
-			"__meta_kubernetes_endpoint_port_name":           "web",
-			"__meta_kubernetes_endpoint_port_protocol":       "",
-			"__meta_kubernetes_endpoint_ready":               "true",
-			"__meta_kubernetes_endpoints_name":               "test-eps",
-			"__meta_kubernetes_namespace":                    "default",
-			"__meta_kubernetes_pod_host_ip":                  "",
-			"__meta_kubernetes_pod_ip":                       "192.168.15.1",
-			"__meta_kubernetes_pod_name":                     "test-pod",
-			"__meta_kubernetes_pod_node_name":                "",
-			"__meta_kubernetes_pod_phase":                    "",
-			"__meta_kubernetes_pod_ready":                    "unknown",
-			"__meta_kubernetes_pod_uid":                      "",
-			"__meta_kubernetes_service_cluster_ip":           "",
-			"__meta_kubernetes_service_name":                 "test-eps",
-			"__meta_kubernetes_service_type":                 "",
-		}),
+		}, [][]prompbmarshal.Label{
+			discoveryutils.GetSortedLabels(map[string]string{
+				"__address__": "10.13.15.15:8081",
+				"__meta_kubernetes_endpoint_address_target_kind": "Pod",
+				"__meta_kubernetes_endpoint_address_target_name": "test-pod",
+				"__meta_kubernetes_endpoint_port_name":           "web",
+				"__meta_kubernetes_endpoint_port_protocol":       "foobar",
+				"__meta_kubernetes_endpoint_ready":               "true",
+				"__meta_kubernetes_endpoints_name":               "test-eps",
+				"__meta_kubernetes_namespace":                    "default",
+				"__meta_kubernetes_node_label_node_label":        "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label": "true",
+				"__meta_kubernetes_node_name":                    "test-node",
+				"__meta_kubernetes_pod_host_ip":                  "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                       "192.168.15.1",
+				"__meta_kubernetes_pod_name":                     "test-pod",
+				"__meta_kubernetes_pod_node_name":                "test-node",
+				"__meta_kubernetes_pod_phase":                    "abc",
+				"__meta_kubernetes_pod_ready":                    "unknown",
+				"__meta_kubernetes_pod_uid":                      "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":           "1.2.3.4",
+				"__meta_kubernetes_service_name":                 "test-eps",
+				"__meta_kubernetes_service_type":                 "service-type",
+			}),
+		})
 	})
 
-	f("1 port from endpoint and 1 from pod", testArgs{
-		containerPorts: map[string][]ContainerPort{"metrics": {{
-			Name:          "http-metrics",
-			ContainerPort: 8428,
-		}}},
-		endpointPorts: []EndpointPort{
-			{
-				Name: "web",
-				Port: 8081,
+	t.Run("1 port from endpoint and 1 from pod", func(t *testing.T) {
+		f(t, testArgs{
+			containerPorts: map[string][]ContainerPort{"metrics": {{
+				Name:          "http-metrics",
+				ContainerPort: 8428,
+				Protocol:      "foobar",
+			}}},
+			endpointPorts: []EndpointPort{
+				{
+					Name:     "web",
+					Port:     8081,
+					Protocol: "https",
+				},
 			},
-		},
-	}, [][]prompbmarshal.Label{
-		discoveryutils.GetSortedLabels(map[string]string{
-			"__address__": "10.13.15.15:8081",
-			"__meta_kubernetes_endpoint_address_target_kind": "Pod",
-			"__meta_kubernetes_endpoint_address_target_name": "test-pod",
-			"__meta_kubernetes_endpoint_port_name":           "web",
-			"__meta_kubernetes_endpoint_port_protocol":       "",
-			"__meta_kubernetes_endpoint_ready":               "true",
-			"__meta_kubernetes_endpoints_name":               "test-eps",
-			"__meta_kubernetes_namespace":                    "default",
-			"__meta_kubernetes_pod_host_ip":                  "",
-			"__meta_kubernetes_pod_ip":                       "192.168.15.1",
-			"__meta_kubernetes_pod_name":                     "test-pod",
-			"__meta_kubernetes_pod_node_name":                "",
-			"__meta_kubernetes_pod_phase":                    "",
-			"__meta_kubernetes_pod_ready":                    "unknown",
-			"__meta_kubernetes_pod_uid":                      "",
-			"__meta_kubernetes_service_cluster_ip":           "",
-			"__meta_kubernetes_service_name":                 "test-eps",
-			"__meta_kubernetes_service_type":                 "",
-		}),
-		discoveryutils.GetSortedLabels(map[string]string{
-			"__address__":                                   "192.168.15.1:8428",
-			"__meta_kubernetes_namespace":                   "default",
-			"__meta_kubernetes_pod_container_name":          "metrics",
-			"__meta_kubernetes_pod_container_port_name":     "http-metrics",
-			"__meta_kubernetes_pod_container_port_number":   "8428",
-			"__meta_kubernetes_pod_container_port_protocol": "",
-			"__meta_kubernetes_pod_host_ip":                 "",
-			"__meta_kubernetes_pod_ip":                      "192.168.15.1",
-			"__meta_kubernetes_pod_name":                    "test-pod",
-			"__meta_kubernetes_pod_node_name":               "",
-			"__meta_kubernetes_pod_phase":                   "",
-			"__meta_kubernetes_pod_ready":                   "unknown",
-			"__meta_kubernetes_pod_uid":                     "",
-			"__meta_kubernetes_service_cluster_ip":          "",
-			"__meta_kubernetes_service_name":                "test-eps",
-			"__meta_kubernetes_service_type":                "",
-		}),
+		}, [][]prompbmarshal.Label{
+			discoveryutils.GetSortedLabels(map[string]string{
+				"__address__": "10.13.15.15:8081",
+				"__meta_kubernetes_endpoint_address_target_kind": "Pod",
+				"__meta_kubernetes_endpoint_address_target_name": "test-pod",
+				"__meta_kubernetes_endpoint_port_name":           "web",
+				"__meta_kubernetes_endpoint_port_protocol":       "https",
+				"__meta_kubernetes_endpoint_ready":               "true",
+				"__meta_kubernetes_endpoints_name":               "test-eps",
+				"__meta_kubernetes_namespace":                    "default",
+				"__meta_kubernetes_node_label_node_label":        "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label": "true",
+				"__meta_kubernetes_node_name":                    "test-node",
+				"__meta_kubernetes_pod_host_ip":                  "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                       "192.168.15.1",
+				"__meta_kubernetes_pod_name":                     "test-pod",
+				"__meta_kubernetes_pod_node_name":                "test-node",
+				"__meta_kubernetes_pod_phase":                    "abc",
+				"__meta_kubernetes_pod_ready":                    "unknown",
+				"__meta_kubernetes_pod_uid":                      "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":           "1.2.3.4",
+				"__meta_kubernetes_service_name":                 "test-eps",
+				"__meta_kubernetes_service_type":                 "service-type",
+			}),
+			discoveryutils.GetSortedLabels(map[string]string{
+				"__address__":                                    "192.168.15.1:8428",
+				"__meta_kubernetes_namespace":                    "default",
+				"__meta_kubernetes_node_label_node_label":        "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label": "true",
+				"__meta_kubernetes_node_name":                    "test-node",
+				"__meta_kubernetes_pod_container_name":           "metrics",
+				"__meta_kubernetes_pod_container_port_name":      "http-metrics",
+				"__meta_kubernetes_pod_container_port_number":    "8428",
+				"__meta_kubernetes_pod_container_port_protocol":  "foobar",
+				"__meta_kubernetes_pod_host_ip":                  "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                       "192.168.15.1",
+				"__meta_kubernetes_pod_name":                     "test-pod",
+				"__meta_kubernetes_pod_node_name":                "test-node",
+				"__meta_kubernetes_pod_phase":                    "abc",
+				"__meta_kubernetes_pod_ready":                    "unknown",
+				"__meta_kubernetes_pod_uid":                      "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":           "1.2.3.4",
+				"__meta_kubernetes_service_name":                 "test-eps",
+				"__meta_kubernetes_service_type":                 "service-type",
+			}),
+		})
 	})
 
-	f("1 port from endpoint", testArgs{
-		containerPorts: map[string][]ContainerPort{"metrics": {{
-			Name:          "web",
-			ContainerPort: 8428,
-		}}},
-		endpointPorts: []EndpointPort{
-			{
-				Name: "web",
-				Port: 8428,
+	t.Run("1 port from endpoint", func(t *testing.T) {
+		f(t, testArgs{
+			containerPorts: map[string][]ContainerPort{"metrics": {{
+				Name:          "web",
+				ContainerPort: 8428,
+				Protocol:      "sdc",
+			}}},
+			endpointPorts: []EndpointPort{
+				{
+					Name:     "web",
+					Port:     8428,
+					Protocol: "xabc",
+				},
 			},
-		},
-	}, [][]prompbmarshal.Label{
-		discoveryutils.GetSortedLabels(map[string]string{
-			"__address__": "10.13.15.15:8428",
-			"__meta_kubernetes_endpoint_address_target_kind": "Pod",
-			"__meta_kubernetes_endpoint_address_target_name": "test-pod",
-			"__meta_kubernetes_endpoint_port_name":           "web",
-			"__meta_kubernetes_endpoint_port_protocol":       "",
-			"__meta_kubernetes_endpoint_ready":               "true",
-			"__meta_kubernetes_endpoints_name":               "test-eps",
-			"__meta_kubernetes_namespace":                    "default",
-			"__meta_kubernetes_pod_container_name":           "metrics",
-			"__meta_kubernetes_pod_container_port_name":      "web",
-			"__meta_kubernetes_pod_container_port_number":    "8428",
-			"__meta_kubernetes_pod_container_port_protocol":  "",
-			"__meta_kubernetes_pod_host_ip":                  "",
-			"__meta_kubernetes_pod_ip":                       "192.168.15.1",
-			"__meta_kubernetes_pod_name":                     "test-pod",
-			"__meta_kubernetes_pod_node_name":                "",
-			"__meta_kubernetes_pod_phase":                    "",
-			"__meta_kubernetes_pod_ready":                    "unknown",
-			"__meta_kubernetes_pod_uid":                      "",
-			"__meta_kubernetes_service_cluster_ip":           "",
-			"__meta_kubernetes_service_name":                 "test-eps",
-			"__meta_kubernetes_service_type":                 "",
-		}),
+		}, [][]prompbmarshal.Label{
+			discoveryutils.GetSortedLabels(map[string]string{
+				"__address__": "10.13.15.15:8428",
+				"__meta_kubernetes_endpoint_address_target_kind": "Pod",
+				"__meta_kubernetes_endpoint_address_target_name": "test-pod",
+				"__meta_kubernetes_endpoint_port_name":           "web",
+				"__meta_kubernetes_endpoint_port_protocol":       "xabc",
+				"__meta_kubernetes_endpoint_ready":               "true",
+				"__meta_kubernetes_endpoints_name":               "test-eps",
+				"__meta_kubernetes_namespace":                    "default",
+				"__meta_kubernetes_node_label_node_label":        "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label": "true",
+				"__meta_kubernetes_node_name":                    "test-node",
+				"__meta_kubernetes_pod_container_name":           "metrics",
+				"__meta_kubernetes_pod_container_port_name":      "web",
+				"__meta_kubernetes_pod_container_port_number":    "8428",
+				"__meta_kubernetes_pod_container_port_protocol":  "sdc",
+				"__meta_kubernetes_pod_host_ip":                  "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                       "192.168.15.1",
+				"__meta_kubernetes_pod_name":                     "test-pod",
+				"__meta_kubernetes_pod_node_name":                "test-node",
+				"__meta_kubernetes_pod_phase":                    "abc",
+				"__meta_kubernetes_pod_ready":                    "unknown",
+				"__meta_kubernetes_pod_uid":                      "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":           "1.2.3.4",
+				"__meta_kubernetes_service_name":                 "test-eps",
+				"__meta_kubernetes_service_type":                 "service-type",
+			}),
+		})
 	})
 }
diff --git a/lib/promscrape/discovery/kubernetes/endpointslice_test.go b/lib/promscrape/discovery/kubernetes/endpointslice_test.go
index a09f15841..f6c65a094 100644
--- a/lib/promscrape/discovery/kubernetes/endpointslice_test.go
+++ b/lib/promscrape/discovery/kubernetes/endpointslice_test.go
@@ -224,3 +224,281 @@ func TestParseEndpointSliceListSuccess(t *testing.T) {
 	}
 
 }
+
+func TestGetEndpointsliceLabels(t *testing.T) {
+	type testArgs struct {
+		containerPorts map[string][]ContainerPort
+		endpointPorts  []EndpointPort
+	}
+	f := func(t *testing.T, args testArgs, wantLabels [][]prompbmarshal.Label) {
+		t.Helper()
+		eps := EndpointSlice{
+			Metadata: ObjectMeta{
+				Name:      "test-eps",
+				Namespace: "default",
+				Labels: discoveryutils.GetSortedLabels(map[string]string{
+					"kubernetes.io/service-name": "test-svc",
+				}),
+			},
+			Endpoints: []Endpoint{
+				{
+					Addresses: []string{
+						"10.13.15.15",
+					},
+					Conditions: EndpointConditions{
+						Ready: true,
+					},
+					Hostname: "foo.bar",
+					TargetRef: ObjectReference{
+						Kind:      "Pod",
+						Namespace: "default",
+						Name:      "test-pod",
+					},
+					Topology: map[string]string{
+						"x": "y",
+					},
+				},
+			},
+			AddressType: "foobar",
+			Ports:       args.endpointPorts,
+		}
+		svc := Service{
+			Metadata: ObjectMeta{
+				Name:      "test-svc",
+				Namespace: "default",
+			},
+			Spec: ServiceSpec{
+				ClusterIP: "1.2.3.4",
+				Type:      "service-type",
+				Ports: []ServicePort{
+					{
+						Name: "test-port",
+						Port: 8081,
+					},
+				},
+			},
+		}
+		pod := Pod{
+			Metadata: ObjectMeta{
+				UID:       "pod-uid",
+				Name:      "test-pod",
+				Namespace: "default",
+			},
+			Spec: PodSpec{
+				NodeName: "test-node",
+			},
+			Status: PodStatus{
+				Phase:  "abc",
+				PodIP:  "192.168.15.1",
+				HostIP: "4.5.6.7",
+			},
+		}
+		node := Node{
+			Metadata: ObjectMeta{
+				Labels: []prompbmarshal.Label{
+					{
+						Name:  "node-label",
+						Value: "xyz",
+					},
+				},
+			},
+		}
+		for cn, ports := range args.containerPorts {
+			pod.Spec.Containers = append(pod.Spec.Containers, Container{Name: cn, Ports: ports})
+		}
+		var gw groupWatcher
+		gw.m = map[string]*urlWatcher{
+			"pod": {
+				role: "pod",
+				objectsByKey: map[string]object{
+					"default/test-pod": &pod,
+				},
+			},
+			"service": {
+				role: "service",
+				objectsByKey: map[string]object{
+					"default/test-svc": &svc,
+				},
+			},
+			"node": {
+				role: "node",
+				objectsByKey: map[string]object{
+					"/test-node": &node,
+				},
+			},
+		}
+		gw.attachNodeMetadata = true
+		var sortedLabelss [][]prompbmarshal.Label
+		gotLabels := eps.getTargetLabels(&gw)
+		for _, lbs := range gotLabels {
+			sortedLabelss = append(sortedLabelss, discoveryutils.GetSortedLabels(lbs))
+		}
+		if !areEqualLabelss(sortedLabelss, wantLabels) {
+			t.Fatalf("unexpected labels:\ngot\n%v\nwant\n%v", sortedLabelss, wantLabels)
+		}
+	}
+
+	t.Run("1 port from endpoint", func(t *testing.T) {
+		f(t, testArgs{
+			endpointPorts: []EndpointPort{
+				{
+					Name:     "web",
+					Port:     8081,
+					Protocol: "foobar",
+				},
+			},
+		}, [][]prompbmarshal.Label{
+			discoveryutils.GetSortedLabels(map[string]string{
+				"__address__": "10.13.15.15:8081",
+				"__meta_kubernetes_endpointslice_address_target_kind":                     "Pod",
+				"__meta_kubernetes_endpointslice_address_target_name":                     "test-pod",
+				"__meta_kubernetes_endpointslice_address_type":                            "foobar",
+				"__meta_kubernetes_endpointslice_endpoint_conditions_ready":               "true",
+				"__meta_kubernetes_endpointslice_endpoint_hostname":                       "foo.bar",
+				"__meta_kubernetes_endpointslice_endpoint_topology_present_x":             "true",
+				"__meta_kubernetes_endpointslice_endpoint_topology_x":                     "y",
+				"__meta_kubernetes_endpointslice_label_kubernetes_io_service_name":        "test-svc",
+				"__meta_kubernetes_endpointslice_labelpresent_kubernetes_io_service_name": "true",
+				"__meta_kubernetes_endpointslice_name":                                    "test-eps",
+				"__meta_kubernetes_endpointslice_port":                                    "8081",
+				"__meta_kubernetes_endpointslice_port_name":                               "web",
+				"__meta_kubernetes_endpointslice_port_protocol":                           "foobar",
+				"__meta_kubernetes_namespace":                                             "default",
+				"__meta_kubernetes_node_label_node_label":                                 "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label":                          "true",
+				"__meta_kubernetes_node_name":                                             "test-node",
+				"__meta_kubernetes_pod_host_ip":                                           "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                                                "192.168.15.1",
+				"__meta_kubernetes_pod_name":                                              "test-pod",
+				"__meta_kubernetes_pod_node_name":                                         "test-node",
+				"__meta_kubernetes_pod_phase":                                             "abc",
+				"__meta_kubernetes_pod_ready":                                             "unknown",
+				"__meta_kubernetes_pod_uid":                                               "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":                                    "1.2.3.4",
+				"__meta_kubernetes_service_name":                                          "test-svc",
+				"__meta_kubernetes_service_type":                                          "service-type",
+			}),
+		})
+	})
+
+	t.Run("1 port from endpoint and 1 from pod", func(t *testing.T) {
+		f(t, testArgs{
+			containerPorts: map[string][]ContainerPort{"metrics": {{
+				Name:          "http-metrics",
+				ContainerPort: 8428,
+				Protocol:      "foobar",
+			}}},
+			endpointPorts: []EndpointPort{
+				{
+					Name:     "web",
+					Port:     8081,
+					Protocol: "https",
+				},
+			},
+		}, [][]prompbmarshal.Label{
+			discoveryutils.GetSortedLabels(map[string]string{
+				"__address__": "10.13.15.15:8081",
+				"__meta_kubernetes_endpointslice_address_target_kind":                     "Pod",
+				"__meta_kubernetes_endpointslice_address_target_name":                     "test-pod",
+				"__meta_kubernetes_endpointslice_address_type":                            "foobar",
+				"__meta_kubernetes_endpointslice_endpoint_conditions_ready":               "true",
+				"__meta_kubernetes_endpointslice_endpoint_hostname":                       "foo.bar",
+				"__meta_kubernetes_endpointslice_endpoint_topology_present_x":             "true",
+				"__meta_kubernetes_endpointslice_endpoint_topology_x":                     "y",
+				"__meta_kubernetes_endpointslice_label_kubernetes_io_service_name":        "test-svc",
+				"__meta_kubernetes_endpointslice_labelpresent_kubernetes_io_service_name": "true",
+				"__meta_kubernetes_endpointslice_name":                                    "test-eps",
+				"__meta_kubernetes_endpointslice_port":                                    "8081",
+				"__meta_kubernetes_endpointslice_port_name":                               "web",
+				"__meta_kubernetes_endpointslice_port_protocol":                           "https",
+				"__meta_kubernetes_namespace":                                             "default",
+				"__meta_kubernetes_node_label_node_label":                                 "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label":                          "true",
+				"__meta_kubernetes_node_name":                                             "test-node",
+				"__meta_kubernetes_pod_host_ip":                                           "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                                                "192.168.15.1",
+				"__meta_kubernetes_pod_name":                                              "test-pod",
+				"__meta_kubernetes_pod_node_name":                                         "test-node",
+				"__meta_kubernetes_pod_phase":                                             "abc",
+				"__meta_kubernetes_pod_ready":                                             "unknown",
+				"__meta_kubernetes_pod_uid":                                               "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":                                    "1.2.3.4",
+				"__meta_kubernetes_service_name":                                          "test-svc",
+				"__meta_kubernetes_service_type":                                          "service-type",
+			}),
+			discoveryutils.GetSortedLabels(map[string]string{
+				"__address__":                                    "192.168.15.1:8428",
+				"__meta_kubernetes_namespace":                    "default",
+				"__meta_kubernetes_node_label_node_label":        "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label": "true",
+				"__meta_kubernetes_node_name":                    "test-node",
+				"__meta_kubernetes_pod_container_name":           "metrics",
+				"__meta_kubernetes_pod_container_port_name":      "http-metrics",
+				"__meta_kubernetes_pod_container_port_number":    "8428",
+				"__meta_kubernetes_pod_container_port_protocol":  "foobar",
+				"__meta_kubernetes_pod_host_ip":                  "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                       "192.168.15.1",
+				"__meta_kubernetes_pod_name":                     "test-pod",
+				"__meta_kubernetes_pod_node_name":                "test-node",
+				"__meta_kubernetes_pod_phase":                    "abc",
+				"__meta_kubernetes_pod_ready":                    "unknown",
+				"__meta_kubernetes_pod_uid":                      "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":           "1.2.3.4",
+				"__meta_kubernetes_service_name":                 "test-svc",
+				"__meta_kubernetes_service_type":                 "service-type",
+			}),
+		})
+	})
+
+	t.Run("1 port from endpoint", func(t *testing.T) {
+		f(t, testArgs{
+			containerPorts: map[string][]ContainerPort{"metrics": {{
+				Name:          "web",
+				ContainerPort: 8428,
+				Protocol:      "sdc",
+			}}},
+			endpointPorts: []EndpointPort{
+				{
+					Name:     "web",
+					Port:     8428,
+					Protocol: "xabc",
+				},
+			},
+		}, [][]prompbmarshal.Label{
+			discoveryutils.GetSortedLabels(map[string]string{
+				"__address__": "10.13.15.15:8428",
+				"__meta_kubernetes_endpointslice_address_target_kind":                     "Pod",
+				"__meta_kubernetes_endpointslice_address_target_name":                     "test-pod",
+				"__meta_kubernetes_endpointslice_address_type":                            "foobar",
+				"__meta_kubernetes_endpointslice_endpoint_conditions_ready":               "true",
+				"__meta_kubernetes_endpointslice_endpoint_hostname":                       "foo.bar",
+				"__meta_kubernetes_endpointslice_endpoint_topology_present_x":             "true",
+				"__meta_kubernetes_endpointslice_endpoint_topology_x":                     "y",
+				"__meta_kubernetes_endpointslice_label_kubernetes_io_service_name":        "test-svc",
+				"__meta_kubernetes_endpointslice_labelpresent_kubernetes_io_service_name": "true",
+				"__meta_kubernetes_endpointslice_name":                                    "test-eps",
+				"__meta_kubernetes_endpointslice_port":                                    "8428",
+				"__meta_kubernetes_endpointslice_port_name":                               "web",
+				"__meta_kubernetes_endpointslice_port_protocol":                           "xabc",
+				"__meta_kubernetes_namespace":                                             "default",
+				"__meta_kubernetes_node_label_node_label":                                 "xyz",
+				"__meta_kubernetes_node_labelpresent_node_label":                          "true",
+				"__meta_kubernetes_node_name":                                             "test-node",
+				"__meta_kubernetes_pod_container_name":                                    "metrics",
+				"__meta_kubernetes_pod_container_port_name":                               "web",
+				"__meta_kubernetes_pod_container_port_number":                             "8428",
+				"__meta_kubernetes_pod_container_port_protocol":                           "sdc",
+				"__meta_kubernetes_pod_host_ip":                                           "4.5.6.7",
+				"__meta_kubernetes_pod_ip":                                                "192.168.15.1",
+				"__meta_kubernetes_pod_name":                                              "test-pod",
+				"__meta_kubernetes_pod_node_name":                                         "test-node",
+				"__meta_kubernetes_pod_phase":                                             "abc",
+				"__meta_kubernetes_pod_ready":                                             "unknown",
+				"__meta_kubernetes_pod_uid":                                               "pod-uid",
+				"__meta_kubernetes_service_cluster_ip":                                    "1.2.3.4",
+				"__meta_kubernetes_service_name":                                          "test-svc",
+				"__meta_kubernetes_service_type":                                          "service-type",
+			}),
+		})
+	})
+}