Repository: prometheus/prometheus
Pull request #16365: refactor(endpointslice): use service cache.Indexer to achieve better iteration performance
Author: login: ryanwuer
Created: 2025-04-01T08:03:58Z | Merged: 2025-05-20T18:33:25Z

try to solve the TODO:

```
// TODO(brancz): use cache.Indexer to index endpointslices by
// LabelServiceName so this operation doesn't have to iterate over all
// endpoint objects.
for _, obj := range e.endpointSliceStore.List() {
	esa, err := e.getEndpointSliceAdaptor(obj)
	if err != nil {
		e.logger.Error("converting to EndpointSlice object failed", "err", err)
		continue
	}
	if lv, exists := esa.labels()[esa.labelServiceName()]; exists && lv == svc.Name {
		e.enqueue(esa.get())
	}
}
```


<!--
    Please give your PR a title in the form "area: short description".  For example "tsdb: reduce disk usage by 95%"

    If your PR is to fix an issue, put "Fixes #issue-number" in the description.

    Don't forget!

    - Please sign CNCF's Developer Certificate of Origin and sign-off your commits by adding the -s / --signoff flag to `git commit`. See https://github.com/apps/dco for more information.

    - If the PR adds or changes a behaviour or fixes a bug of an exported API it would need a unit/e2e test.

    - Where possible use only exported APIs for tests to simplify the review and make it as close as possible to an actual library usage.

    - Performance improvements would need a benchmark test to prove it.

    - All exposed objects should have a comment.

    - All comments should start with a capital letter and end with a full stop.
 -->



Diff:
diff --git a/discovery/kubernetes/endpointslice.go b/discovery/kubernetes/endpointslice.go
index 1002025128b..625601abc1f 100644
--- a/discovery/kubernetes/endpointslice.go
+++ b/discovery/kubernetes/endpointslice.go
@@ -32,6 +32,8 @@ import (
 	"github.com/prometheus/prometheus/util/strutil"
 )
 
+const serviceIndex = "service"
+
 // EndpointSlice discovers new endpoint targets.
 type EndpointSlice struct {
 	logger *slog.Logger
@@ -101,19 +103,14 @@ func NewEndpointSlice(l *slog.Logger, eps cache.SharedIndexInformer, svc, pod, n
 			return
 		}
 
-		// TODO(brancz): use cache.Indexer to index endpointslices by
-		// LabelServiceName so this operation doesn't have to iterate over all
-		// endpoint objects.
-		for _, obj := range e.endpointSliceStore.List() {
-			es, ok := obj.(*v1.EndpointSlice)
-			if !ok {
-				e.logger.Error("converting to EndpointSlice object failed", "err", err)
-				continue
-			}
-			// Only consider the underlying EndpointSlices in the same namespace.
-			if svcName, exists := es.Labels[v1.LabelServiceName]; exists && svcName == svc.Name && es.Namespace == svc.Namespace {
-				e.enqueue(es)
-			}
+		endpointSlices, err := e.endpointSliceInf.GetIndexer().ByIndex(serviceIndex, namespacedName(svc.Namespace, svc.Name))
+		if err != nil {
+			e.logger.Error("getting endpoint slices by service name failed", "err", err)
+			return
+		}
+
+		for _, endpointSlice := range endpointSlices {
+			e.enqueue(endpointSlice)
 		}
 	}
 	_, err = e.serviceInf.AddEventHandler(cache.ResourceEventHandlerFuncs{
diff --git a/discovery/kubernetes/endpointslice_test.go b/discovery/kubernetes/endpointslice_test.go
index 21d49defd40..9eea9abd7b0 100644
--- a/discovery/kubernetes/endpointslice_test.go
+++ b/discovery/kubernetes/endpointslice_test.go
@@ -1194,11 +1194,12 @@ func TestEndpointSliceInfIndexersCount(t *testing.T) {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
 			var (
-				n                    *Discovery
-				mainInfIndexersCount int
+				n *Discovery
+				// service indexer is enabled by default
+				mainInfIndexersCount = 1
 			)
 			if tc.withNodeMetadata {
-				mainInfIndexersCount = 1
+				mainInfIndexersCount++
 				n, _ = makeDiscoveryWithMetadata(RoleEndpointSlice, NamespaceDiscovery{}, AttachMetadataConfig{Node: true})
 			} else {
 				n, _ = makeDiscovery(RoleEndpointSlice, NamespaceDiscovery{})
diff --git a/discovery/kubernetes/kubernetes.go b/discovery/kubernetes/kubernetes.go
index 2c4829ca8d8..ebd2adb87a0 100644
--- a/discovery/kubernetes/kubernetes.go
+++ b/discovery/kubernetes/kubernetes.go
@@ -716,28 +716,41 @@ func (d *Discovery) newEndpointsByNodeInformer(plw *cache.ListWatch) cache.Share
 
 func (d *Discovery) newEndpointSlicesByNodeInformer(plw *cache.ListWatch, object runtime.Object) cache.SharedIndexInformer {
 	indexers := make(map[string]cache.IndexFunc)
+	indexers[serviceIndex] = func(obj interface{}) ([]string, error) {
+		e, ok := obj.(*disv1.EndpointSlice)
+		if !ok {
+			return nil, errors.New("object is not an endpointslice")
+		}
+
+		svcName, exists := e.Labels[disv1.LabelServiceName]
+		if !exists {
+			return nil, nil
+		}
+
+		return []string{namespacedName(e.Namespace, svcName)}, nil
+	}
 	if !d.attachMetadata.Node {
 		return d.mustNewSharedIndexInformer(plw, object, resyncDisabled, indexers)
 	}
 
 	indexers[nodeIndex] = func(obj interface{}) ([]string, error) {
+		e, ok := obj.(*disv1.EndpointSlice)
+		if !ok {
+			return nil, errors.New("object is not an endpointslice")
+		}
+
 		var nodes []string
-		switch e := obj.(type) {
-		case *disv1.EndpointSlice:
-			for _, target := range e.Endpoints {
-				if target.TargetRef != nil {
-					switch target.TargetRef.Kind {
-					case "Pod":
-						if target.NodeName != nil {
-							nodes = append(nodes, *target.NodeName)
-						}
-					case "Node":
-						nodes = append(nodes, target.TargetRef.Name)
+		for _, target := range e.Endpoints {
+			if target.TargetRef != nil {
+				switch target.TargetRef.Kind {
+				case "Pod":
+					if target.NodeName != nil {
+						nodes = append(nodes, *target.NodeName)
 					}
+				case "Node":
+					nodes = append(nodes, target.TargetRef.Name)
 				}
 			}
-		default:
-			return nil, errors.New("object is not an endpointslice")
 		}
 
 		return nodes, nil


Reviews:
user: login: machine424
body: thanks for this.
Some comments.
We'll need to give this a try on a real kube cluster, let's discuss that once we're happy with the code.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-08T21:57:17Z
user: login: ryanwuer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T08:38:15Z
user: login: ryanwuer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T08:38:58Z
user: login: ryanwuer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T08:48:24Z
user: login: machine424
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T21:58:57Z
user: login: ryanwuer
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T02:07:45Z
user: login: machine424
body: lgtm, thanks again!
just some nits.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-20T08:12:40Z
user: login: machine424
body: lgtm, thanks again!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-20T18:31:51Z

Review comments:
diff_hunk: @@ -724,6 +724,19 @@ func (d *Discovery) newEndpointsByNodeInformer(plw *cache.ListWatch) cache.Share
 
 func (d *Discovery) newEndpointSlicesByNodeInformer(plw *cache.ListWatch, object runtime.Object) cache.SharedIndexInformer {
 	indexers := make(map[string]cache.IndexFunc)
+	indexers[serviceIndex] = func(obj interface{}) ([]string, error) {
+		var services []string
+		switch e := obj.(type) {
path: discovery/kubernetes/kubernetes.go
user: login: machine424
body: let's check the type and error early
created_at: 2025-04-08T21:43:46Z
author_association: MEMBER
diff_hunk: @@ -724,6 +724,19 @@ func (d *Discovery) newEndpointsByNodeInformer(plw *cache.ListWatch) cache.Share
 
 func (d *Discovery) newEndpointSlicesByNodeInformer(plw *cache.ListWatch, object runtime.Object) cache.SharedIndexInformer {
 	indexers := make(map[string]cache.IndexFunc)
+	indexers[serviceIndex] = func(obj interface{}) ([]string, error) {
+		var services []string
+		switch e := obj.(type) {
+		case *disv1.EndpointSlice:
+			if serviceName, exists := e.ObjectMeta.Labels[disv1.LabelServiceName]; exists {
+				return []string{serviceName}, nil
path: discovery/kubernetes/kubernetes.go
user: login: machine424
body: the service name in the label doesn't seem to be namespaced.
we should use `namespacedName()` to ensure uniqueness (two services in two different namespaces can have the same name)

if you could add add a `TestEndpointSliceDiscoveryWithUnrelatedServiceUpdate` where we update an unrelated service with the same name, in another namespace and check the no-op, it'd be a great regression test.
created_at: 2025-04-08T21:49:04Z
author_association: MEMBER
diff_hunk: @@ -724,6 +724,19 @@ func (d *Discovery) newEndpointsByNodeInformer(plw *cache.ListWatch) cache.Share
 
 func (d *Discovery) newEndpointSlicesByNodeInformer(plw *cache.ListWatch, object runtime.Object) cache.SharedIndexInformer {
 	indexers := make(map[string]cache.IndexFunc)
+	indexers[serviceIndex] = func(obj interface{}) ([]string, error) {
+		var services []string
+		switch e := obj.(type) {
path: discovery/kubernetes/kubernetes.go
user: login: ryanwuer
body: Fixed at the new commit
created_at: 2025-04-09T08:38:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -724,6 +724,19 @@ func (d *Discovery) newEndpointsByNodeInformer(plw *cache.ListWatch) cache.Share
 
 func (d *Discovery) newEndpointSlicesByNodeInformer(plw *cache.ListWatch, object runtime.Object) cache.SharedIndexInformer {
 	indexers := make(map[string]cache.IndexFunc)
+	indexers[serviceIndex] = func(obj interface{}) ([]string, error) {
+		var services []string
+		switch e := obj.(type) {
+		case *disv1.EndpointSlice:
+			if serviceName, exists := e.ObjectMeta.Labels[disv1.LabelServiceName]; exists {
+				return []string{serviceName}, nil
path: discovery/kubernetes/kubernetes.go
user: login: ryanwuer
body: `TestEndpointSliceDiscoveryWithUnrelatedServiceUpdate` is added at the new commit.
created_at: 2025-04-09T08:38:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -724,6 +724,19 @@ func (d *Discovery) newEndpointsByNodeInformer(plw *cache.ListWatch) cache.Share
 
 func (d *Discovery) newEndpointSlicesByNodeInformer(plw *cache.ListWatch, object runtime.Object) cache.SharedIndexInformer {
 	indexers := make(map[string]cache.IndexFunc)
+	indexers[serviceIndex] = func(obj interface{}) ([]string, error) {
+		var services []string
+		switch e := obj.(type) {
+		case *disv1.EndpointSlice:
+			if serviceName, exists := e.ObjectMeta.Labels[disv1.LabelServiceName]; exists {
+				return []string{serviceName}, nil
path: discovery/kubernetes/kubernetes.go
user: login: ryanwuer
body: And there's no need to use `namespacedName()` because the `NewEndpointSlice` constructor func is called for each namespace. The outcome of the unit test can prove that. 

The key code is here:
```
func (d *Discovery) Run(ctx context.Context, ch chan<- []*targetgroup.Group) {
	d.Lock()

	namespaces := d.getNamespaces()

	switch d.role {
	case RoleEndpointSlice:
		for _, namespace := range namespaces {
			var informer cache.SharedIndexInformer
			e := d.client.DiscoveryV1().EndpointSlices(namespace) 

			elw := &cache.ListWatch{
				ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {
					options.FieldSelector = d.selectors.endpointslice.field
					options.LabelSelector = d.selectors.endpointslice.label
					return e.List(ctx, options)
				},
				WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {
					options.FieldSelector = d.selectors.endpointslice.field
					options.LabelSelector = d.selectors.endpointslice.label
					return e.Watch(ctx, options)
				},
			}

			informer = d.newEndpointSlicesByNodeInformer(elw, &disv1.EndpointSlice{})
                        ....
			eps := NewEndpointSlice(
				d.logger.With("role", "endpointslice"),
				informer,
				d.mustNewSharedInformer(slw, &apiv1.Service{}, resyncDisabled),
				d.mustNewSharedInformer(plw, &apiv1.Pod{}, resyncDisabled),
				nodeInf,
				d.metrics.eventCount,
			)
```
created_at: 2025-04-09T08:48:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -629,6 +629,109 @@ func TestEndpointSliceDiscoveryWithServiceUpdate(t *testing.T) {
 	}.Run(t)
 }
 
+func TestEndpointSliceDiscoveryWithUnrelatedServiceUpdate(t *testing.T) {
path: discovery/kubernetes/endpointslice_test.go
user: login: machine424
body: I thought we agreed here https://github.com/prometheus/prometheus/pull/16365#issuecomment-2804668786 that such a test isn't really needed/useful, has anything changed in the meantime?
created_at: 2025-05-19T21:58:57Z
author_association: MEMBER
diff_hunk: @@ -629,6 +629,109 @@ func TestEndpointSliceDiscoveryWithServiceUpdate(t *testing.T) {
 	}.Run(t)
 }
 
+func TestEndpointSliceDiscoveryWithUnrelatedServiceUpdate(t *testing.T) {
path: discovery/kubernetes/endpointslice_test.go
user: login: ryanwuer
body: Nope. Let me remove this. The PR is updated.
created_at: 2025-05-20T02:07:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1194,11 +1194,12 @@ func TestEndpointSliceInfIndexersCount(t *testing.T) {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
 			var (
-				n                    *Discovery
-				mainInfIndexersCount int
+				n *Discovery
+				// service indexer is enabled by default for better iteration performance
+				mainInfIndexersCount = 1
 			)
 			if tc.withNodeMetadata {
-				mainInfIndexersCount = 1
+				mainInfIndexersCount = 2
path: discovery/kubernetes/endpointslice_test.go
user: login: machine424
body: nit
```suggestion
				mainInfIndexersCount++
```
created_at: 2025-05-20T08:03:46Z
author_association: MEMBER
diff_hunk: @@ -1194,11 +1194,12 @@ func TestEndpointSliceInfIndexersCount(t *testing.T) {
 		t.Run(tc.name, func(t *testing.T) {
 			t.Parallel()
 			var (
-				n                    *Discovery
-				mainInfIndexersCount int
+				n *Discovery
+				// service indexer is enabled by default for better iteration performance
path: discovery/kubernetes/endpointslice_test.go
user: login: machine424
body: ```suggestion
				// service indexer is enabled by default
```
created_at: 2025-05-20T08:04:15Z
author_association: MEMBER

PR comments:
user: login: ryanwuer
created_at: 2025-04-01T08:13:27Z
body: @brancz Hi, could you please take a look at this PR? Does it meet expectations?
author_association: CONTRIBUTOR
user: login: ryanwuer
created_at: 2025-04-01T09:03:36Z
body: CI / More Go tests failed because of the error below, which looks like has nothing to do with my PR?

```
--- FAIL: TestQueryLog (0.00s)
    --- FAIL: TestQueryLog/api_queries,_127.0.0.1:36843,_enabled_at_start (0.64s)
        query_log_test.go:322: time=2025-04-01T08:07:57.583Z level=INFO source=main.go:660 msg="No time or size retention was set so using the default time retention" duration=15d
            time=2025-04-01T08:07:57.583Z level=INFO source=main.go:711 msg="Starting Prometheus Server" mode=server version="(version=, branch=, revision=unknown)"
            time=2025-04-01T08:07:57.583Z level=WARN source=main.go:713 msg="This Prometheus binary has not been compiled for a 64-bit architecture. Due to virtual memory constraints of 32-bit systems, it is highly recommended to switch to a 64-bit binary of Prometheus." GOARCH=386
            time=2025-04-01T08:07:57.583Z level=INFO source=main.go:716 msg="operational information" build_context="(go=go1.23.7, platform=linux/386, user=, date=, tags=unknown)" host_details="(Linux 6.8.0-1021-azure #25-Ubuntu SMP Wed Jan 15 20:45:09 UTC 2025 x86_64 fd3d6ffe3ac4 (none))" fd_limits="(soft=1048576, hard=1048576)" vm_limits="(soft=unlimited, hard=unlimited)"
            time=2025-04-01T08:07:57.584Z level=INFO source=main.go:792 msg="Leaving GOMAXPROCS=4: CPU quota undefined" component=automaxprocs
            time=2025-04-01T08:07:57.589Z level=INFO source=web.go:654 msg="Start listening for connections" component=web address=127.0.0.1:368[43](https://github.com/prometheus/prometheus/actions/runs/14190417155/job/39753627384?pr=16365#step:7:44)
            time=2025-04-01T08:07:57.589Z level=ERROR source=main.go:1018 msg="Unable to start web listener" err="listen tcp 127.0.0.1:36843: bind: address already in use"
            
        query_log_test.go:116: 
            	Error Trace:	/__w/prometheus/prometheus/cmd/prometheus/query_log_test.go:116
            	            				/__w/prometheus/prometheus/cmd/prometheus/query_log_test.go:341
            	            				/__w/prometheus/prometheus/cmd/prometheus/query_log_test.go:479
            	Error:      	Received unexpected error:
            	            	Get "http://127.0.0.1:36843/api/v1/query_range?step=5&start=0&end=3600&query=query_with_api": dial tcp 127.0.0.1:36843: connect: connection refused
            	Test:       	TestQueryLog/api_queries,_127.0.0.1:36843,_enabled_at_start
FAIL
```
author_association: CONTRIBUTOR
user: login: ryanwuer
created_at: 2025-04-03T02:34:39Z
body: @machine424 Please help confirm if this PR makes any sense when you got some free time.
author_association: CONTRIBUTOR
user: login: ryanwuer
created_at: 2025-04-09T13:17:42Z
body: @machine424 After `TestEndpointSliceDiscoveryWithUnrelatedServiceUpdate` executed, no service label is patched into endpointslice's labels, which means update of other namespace's service won't make any effect to endpointslice discovery.
```
Labels: model.LabelSet{
	"__meta_kubernetes_endpointslice_address_type":                            "IPv4",
	"__meta_kubernetes_endpointslice_name":                                    "testendpoints",
	"__meta_kubernetes_endpointslice_label_kubernetes_io_service_name":        "testendpoints",
	"__meta_kubernetes_endpointslice_labelpresent_kubernetes_io_service_name": "true",
	"__meta_kubernetes_endpointslice_annotation_test_annotation":              "test",
	"__meta_kubernetes_endpointslice_annotationpresent_test_annotation":       "true",
	"__meta_kubernetes_namespace":                                             "default",
}
``` 

I  debug locally and find `len(endpointSlices)` is 0 when service in default2 namespace is update. So  `namespacedName()` is not necessary. Please help confirm the logic.
```
endpointSlices, err := e.endpointSliceInf.GetIndexer().ByIndex(serviceIndex, svc.Name)
```
author_association: CONTRIBUTOR
user: login: ryanwuer
created_at: 2025-04-10T03:55:08Z
body: For best practice considerations, i'm still willing to use `namespacedName()`.
author_association: CONTRIBUTOR
user: login: machine424
created_at: 2025-04-15T11:01:40Z
body: I should have been more explicit.

It's true that the service informer is namespaced https://github.com/prometheus/prometheus/blob/b4526c0735a07feba666b056249c4980346fed4f/discovery/kubernetes/kubernetes.go#L411, but only when the config itself is namespaced; Prometheus could be given cluster wide permissions and we'd end up with cluster wide informers https://github.com/prometheus/prometheus/blob/b4526c0735a07feba666b056249c4980346fed4f/discovery/kubernetes/kubernetes.go#L271

So, for some setups, a change to a service with the same name but from another namespace could still enqueue https://github.com/prometheus/prometheus/blob/b4526c0735a07feba666b056249c4980346fed4f/discovery/kubernetes/endpointslice.go#L114 the endpointslice to do unnecessary work, during processing, the labels of the appropriate service in the same namespace will end up being used, but we'll still be doing unnecessary work.

having services with the same name across different namespaces may not be that common, but it's always good to avoid that unnecessary work.

The test `TestEndpointSliceDiscoveryWithUnrelatedServiceUpdate` would need to make sure no enqueue/no-op processing happens when an unrelated service is updated AND the SD is configured to watch resources cluster wide (via `n, c := makeDiscovery(RoleEndpointSlice, NamespaceDiscovery{}, makeEndpointSliceV1())` ). The problem is that, I don't think, we currently have test tools to make sure of that.

So let's do this in two steps:
- Make `if lv, exists := es.Labels[v1.LabelServiceName]; exists && lv == svc.Name {` namespaced in one PR, and maybe add a test if you could think about a check (no need to spend time on this if not)
- Introduce the indexer in another PR.
author_association: MEMBER
user: login: ryanwuer
created_at: 2025-04-15T12:50:21Z
body: @machine424 Really appreciate your explanation and suggestions. I've make another PR to Make `if lv, exists := es.Labels[v1.LabelServiceName]; exists && lv == svc.Name {` namespaced. Pls help take a look.

https://github.com/prometheus/prometheus/pull/16433
author_association: CONTRIBUTOR
user: login: machine424
created_at: 2025-05-09T08:43:19Z
body: could you rebase?
author_association: MEMBER
user: login: ryanwuer
created_at: 2025-05-09T09:34:06Z
body: @machine424 Rebase is done. I just made another commit to make service indexer namesapced. Pls take a look to see if it makes sense.
author_association: CONTRIBUTOR
user: login: ryanwuer
created_at: 2025-05-20T08:28:58Z
body: Suggestions applied. Thanks.
author_association: CONTRIBUTOR
user: login: machine424
created_at: 2025-05-20T18:32:38Z
body: Well, I even approved it twice, there couldn't be anything wrong with it :)
author_association: MEMBER