Repository: elastic/elasticsearch
Pull request #127697: Add projectsDelta method to ClusterChangedEvent
Author: login: ywangd
Labels: name: >non-issue
name: :Distributed/Cluster Coordination
description: Cluster formation and cluster state publication, including cluster membership and fault detection.
name: auto-merge-without-approval
description: Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!)
name: serverless-linked
description: Added by automation, don't add manually
name: Team:Distributed Coordination (obsolete)
description: Meta label for Distributed Coordination team. Obsolete. Please do not use.
name: v9.1.0
Created: 2025-05-05T07:03:03Z | Merged: 2025-05-20T06:33:29Z

Similar to nodesDelta, the projectsDelta provides a centrailzed method to tell the added and removed projects so that cluster listeners/appliers do not need to compute it individually.

Note the delta currently does not have updated. This needs to wait till we stop eagerly creating ProjectMetadata.Builder in Metadata.Builder.




Diff:
diff --git a/server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java b/server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
index f1f3b7354055c..8285900f0b00f 100644
--- a/server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
+++ b/server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
@@ -13,9 +13,11 @@
 import org.elasticsearch.cluster.metadata.IndexGraveyard.IndexGraveyardDiff;
 import org.elasticsearch.cluster.metadata.IndexMetadata;
 import org.elasticsearch.cluster.metadata.Metadata;
+import org.elasticsearch.cluster.metadata.ProjectId;
 import org.elasticsearch.cluster.metadata.ProjectMetadata;
 import org.elasticsearch.cluster.node.DiscoveryNodes;
 import org.elasticsearch.cluster.routing.IndexRoutingTable;
+import org.elasticsearch.common.util.set.Sets;
 import org.elasticsearch.gateway.GatewayService;
 import org.elasticsearch.index.Index;
 
@@ -41,6 +43,8 @@ public class ClusterChangedEvent {
 
     private final DiscoveryNodes.Delta nodesDelta;
 
+    private final ProjectsDelta projectsDelta;
+
     public ClusterChangedEvent(String source, ClusterState state, ClusterState previousState) {
         Objects.requireNonNull(source, "source must not be null");
         Objects.requireNonNull(state, "state must not be null");
@@ -49,6 +53,7 @@ public ClusterChangedEvent(String source, ClusterState state, ClusterState previ
         this.state = state;
         this.previousState = previousState;
         this.nodesDelta = state.nodes().delta(previousState.nodes());
+        this.projectsDelta = calculateProjectDelta(previousState.metadata(), state.metadata());
     }
 
     /**
@@ -237,6 +242,13 @@ public boolean nodesChanged() {
         return nodesRemoved() || nodesAdded();
     }
 
+    /**
+     * Returns the {@link ProjectsDelta} between the previous cluster state and the new cluster state.
+     */
+    public ProjectsDelta projectDelta() {
+        return projectsDelta;
+    }
+
     /**
      * Determines whether or not the current cluster state represents an entirely
      * new cluster, either when a node joins a cluster for the first time or when
@@ -336,4 +348,32 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata == currentMetadata
+            || (previousMetadata.projects().size() == 1
+                && previousMetadata.hasProject(ProjectId.DEFAULT)
+                && currentMetadata.projects().size() == 1
+                && currentMetadata.hasProject(ProjectId.DEFAULT))) {
+            return ProjectsDelta.EMPTY;
+        }
+
+        final Set<ProjectId> added = Collections.unmodifiableSet(
+            Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())
+        );
+        final Set<ProjectId> removed = Collections.unmodifiableSet(
+            Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet())
+        );
+        // TODO: Enable the following assertions once tests no longer add or remove default projects
+        // assert added.contains(ProjectId.DEFAULT) == false;
+        // assert removed.contains(ProjectId.DEFAULT) == false;
+        return new ProjectsDelta(added, removed);
+    }
+
+    public record ProjectsDelta(Set<ProjectId> added, Set<ProjectId> removed) {
+        private static final ProjectsDelta EMPTY = new ProjectsDelta(Set.of(), Set.of());
+
+        public boolean isEmpty() {
+            return added.isEmpty() && removed.isEmpty();
+        }
+    }
 }
diff --git a/server/src/test/java/org/elasticsearch/cluster/ClusterChangedEventTests.java b/server/src/test/java/org/elasticsearch/cluster/ClusterChangedEventTests.java
index c08c82e27fa8b..8c04ddd6163ff 100644
--- a/server/src/test/java/org/elasticsearch/cluster/ClusterChangedEventTests.java
+++ b/server/src/test/java/org/elasticsearch/cluster/ClusterChangedEventTests.java
@@ -14,7 +14,9 @@
 import org.elasticsearch.cluster.metadata.IndexGraveyard;
 import org.elasticsearch.cluster.metadata.IndexMetadata;
 import org.elasticsearch.cluster.metadata.Metadata;
+import org.elasticsearch.cluster.metadata.ProjectId;
 import org.elasticsearch.cluster.metadata.ProjectMetadata;
+import org.elasticsearch.cluster.metadata.ReservedStateMetadata;
 import org.elasticsearch.cluster.node.DiscoveryNode;
 import org.elasticsearch.cluster.node.DiscoveryNodeRole;
 import org.elasticsearch.cluster.node.DiscoveryNodeUtils;
@@ -520,6 +522,65 @@ public void testChangedCustomMetadataSetMultiProject() {
         assertEquals(Set.of(IndexGraveyard.TYPE, project2Custom.getWriteableName()), event.changedCustomProjectMetadataSet());
     }
 
+    public void testProjectsDelta() {
+        final var state0 = ClusterState.builder(TEST_CLUSTER_NAME).build();
+
+        // No project changes
+        final var state1 = ClusterState.builder(state0)
+            .metadata(Metadata.builder(state0.metadata()).put(ReservedStateMetadata.builder("test").build()))
+            .build();
+        ClusterChangedEvent event = new ClusterChangedEvent("test", state1, state0);
+        assertTrue(event.projectDelta().isEmpty());
+
+        // Add projects
+        final List<ProjectId> projectIds = randomList(1, 5, ESTestCase::randomUniqueProjectId);
+        Metadata.Builder metadataBuilder = Metadata.builder(state1.metadata());
+        for (ProjectId projectId : projectIds) {
+            metadataBuilder.put(ProjectMetadata.builder(projectId));
+        }
+        final var state2 = ClusterState.builder(state1).metadata(metadataBuilder.build()).build();
+        event = new ClusterChangedEvent("test", state2, state1);
+        assertThat(event.projectDelta().added(), containsInAnyOrder(projectIds.toArray()));
+        assertThat(event.projectDelta().removed(), empty());
+
+        // Add more projects and delete one
+        final var removedProjectIds = randomNonEmptySubsetOf(projectIds);
+        final List<ProjectId> moreProjectIds = randomList(1, 3, ESTestCase::randomUniqueProjectId);
+        metadataBuilder = Metadata.builder(state2.metadata());
+        GlobalRoutingTable.Builder routingTableBuilder = GlobalRoutingTable.builder(state2.globalRoutingTable());
+        for (ProjectId projectId : removedProjectIds) {
+            metadataBuilder.removeProject(projectId);
+            routingTableBuilder.removeProject(projectId);
+        }
+        for (ProjectId projectId : moreProjectIds) {
+            metadataBuilder.put(ProjectMetadata.builder(projectId));
+        }
+
+        final var state3 = ClusterState.builder(state2).metadata(metadataBuilder.build()).routingTable(routingTableBuilder.build()).build();
+
+        event = new ClusterChangedEvent("test", state3, state2);
+        assertThat(event.projectDelta().added(), containsInAnyOrder(moreProjectIds.toArray()));
+        assertThat(event.projectDelta().removed(), containsInAnyOrder(removedProjectIds.toArray()));
+
+        // Remove all projects
+        final List<ProjectId> remainingProjects = state3.metadata()
+            .projects()
+            .keySet()
+            .stream()
+            .filter(projectId -> ProjectId.DEFAULT.equals(projectId) == false)
+            .toList();
+        metadataBuilder = Metadata.builder(state3.metadata());
+        routingTableBuilder = GlobalRoutingTable.builder(state3.globalRoutingTable());
+        for (ProjectId projectId : remainingProjects) {
+            metadataBuilder.removeProject(projectId);
+            routingTableBuilder.removeProject(projectId);
+        }
+        final var state4 = ClusterState.builder(state3).metadata(metadataBuilder.build()).routingTable(routingTableBuilder.build()).build();
+        event = new ClusterChangedEvent("test", state4, state3);
+        assertThat(event.projectDelta().added(), empty());
+        assertThat(event.projectDelta().removed(), containsInAnyOrder(remainingProjects.toArray()));
+    }
+
     private static class CustomClusterMetadata2 extends TestClusterCustomMetadata {
         protected CustomClusterMetadata2(String data) {
             super(data);
diff --git a/x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/inference/persistence/TrainedModelCacheMetadataServiceTests.java b/x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/inference/persistence/TrainedModelCacheMetadataServiceTests.java
index b333b3596aba5..387c6f36af1ee 100644
--- a/x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/inference/persistence/TrainedModelCacheMetadataServiceTests.java
+++ b/x-pack/plugin/ml/src/test/java/org/elasticsearch/xpack/ml/inference/persistence/TrainedModelCacheMetadataServiceTests.java
@@ -74,6 +74,7 @@ public void testRefreshCacheVersionOnMasterNode() {
         ClusterState clusterState = mock(ClusterState.class);
         when(clusterState.clusterRecovered()).thenReturn(true);
         when(clusterState.nodes()).thenReturn(clusterNodes);
+        when(clusterState.metadata()).thenReturn(Metadata.EMPTY_METADATA);
 
         modelCacheMetadataService.clusterChanged(new ClusterChangedEvent("test", clusterState, ClusterState.EMPTY_STATE));
 
diff --git a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/support/QueryableBuiltInRolesSynchronizerTests.java b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/support/QueryableBuiltInRolesSynchronizerTests.java
index eca0e875ffbf1..5d3430cbef640 100644
--- a/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/support/QueryableBuiltInRolesSynchronizerTests.java
+++ b/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/support/QueryableBuiltInRolesSynchronizerTests.java
@@ -330,8 +330,8 @@ public void testSecurityIndexDeleted() {
 
         synchronizer.clusterChanged(event(currentClusterState, previousClusterState));
 
-        verify(previousClusterState, times(1)).metadata();
-        verify(currentClusterState, times(1)).metadata();
+        verify(previousClusterState, times(2)).metadata();
+        verify(currentClusterState, times(2)).metadata();
         verifyNoMoreInteractions(nativeRolesStore, featureService, taskQueue, reservedRolesProvider, threadPool, clusterService);
     }
 


Reviews:
user: login: nielsbauman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T07:09:51Z
user: login: ywangd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-05T07:13:46Z
user: login: ywangd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-05T07:23:03Z
user: login: nielsbauman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T09:08:38Z
user: login: nielsbauman
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T09:22:18Z
user: login: pxsalehi
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-05T13:42:40Z
user: login: nielsbauman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T14:20:51Z
user: login: ywangd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T05:40:00Z
user: login: ywangd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T05:43:23Z
user: login: ywangd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T06:47:44Z
user: login: nielsbauman
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T06:59:46Z
user: login: pxsalehi
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-06T07:07:24Z
user: login: ywangd
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T06:30:35Z
user: login: nielsbauman
body: LGTM, thanks for the iterations
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T06:37:30Z

Review comments:
diff_hunk: @@ -49,6 +53,7 @@ public ClusterChangedEvent(String source, ClusterState state, ClusterState previ
         this.state = state;
         this.previousState = previousState;
         this.nodesDelta = state.nodes().delta(previousState.nodes());
+        this.projectsDelta = calculateProjectDelta(previousState.metadata(), state.metadata());
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: nielsbauman
body: The logic itself seems fine to me, but what do your expected use cases of this delta look like?

I'm addressing this because I want to avoid a situation similar to `ClusterChangedEvent#changedCustomProjectMetadataSet`, where we compute a set of changes even though all use cases only want to know if one specific entry changed or not, and computing the whole set is thus wasted effort. Since we're eagerly computing the detla here, that waste would be even bigger.
created_at: 2025-05-05T07:09:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -49,6 +53,7 @@ public ClusterChangedEvent(String source, ClusterState state, ClusterState previ
         this.state = state;
         this.previousState = previousState;
         this.nodesDelta = state.nodes().delta(previousState.nodes());
+        this.projectsDelta = calculateProjectDelta(previousState.metadata(), state.metadata());
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: ywangd
body: Yeah please see the linked serverless PR for an use case. Similar use case will be needed for snapshot repositories and likely more.
created_at: 2025-05-05T07:13:46Z
author_association: MEMBER
diff_hunk: @@ -49,6 +53,7 @@ public ClusterChangedEvent(String source, ClusterState state, ClusterState previ
         this.state = state;
         this.previousState = previousState;
         this.nodesDelta = state.nodes().delta(previousState.nodes());
+        this.projectsDelta = calculateProjectDelta(previousState.metadata(), state.metadata());
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: ywangd
body: > waste would be even bigger

The ProjectsDelta contains only added or removed projectIds, i.e. two Sets operations. So I am not sure why the waste would be bigger?
created_at: 2025-05-05T07:23:02Z
author_association: MEMBER
diff_hunk: @@ -49,6 +53,7 @@ public ClusterChangedEvent(String source, ClusterState state, ClusterState previ
         this.state = state;
         this.previousState = previousState;
         this.nodesDelta = state.nodes().delta(previousState.nodes());
+        this.projectsDelta = calculateProjectDelta(previousState.metadata(), state.metadata());
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: nielsbauman
body: Ok, on second look, I think it's fine to compute this eagerly. I think it's worth it.
created_at: 2025-05-05T09:08:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: nielsbauman
body: I'd prefer to construct the set differences manually, as we can construct both with one loop if we do it manually. You'll probably think this is a premature optimization, but I think the cost of the optimization is low and it's one less thing to consider optimizing later on.
created_at: 2025-05-05T09:22:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
+
+    }
+
+    private static final ProjectsDelta EMPTY_PROJECT_DELTA = new ProjectsDelta(Set.of(), Set.of());
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: pxsalehi
body: nit: why not move this into the record?
created_at: 2025-05-05T13:40:12Z
author_association: MEMBER
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: pxsalehi
body: +1. Also can we simplify that logic for `DEFAULT`? It seems looping through them and skipping default project in the loop is more readable.
created_at: 2025-05-05T13:42:34Z
author_association: MEMBER
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: nielsbauman
body: I think we still want that shortcut logic for single-project mode to avoid any performance impact in stateful. This wouldn't have a big impact anyway, but every bit counts IMO.
created_at: 2025-05-05T14:20:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
+
+    }
+
+    private static final ProjectsDelta EMPTY_PROJECT_DELTA = new ProjectsDelta(Set.of(), Set.of());
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: ywangd
body: Addressed in fbe4b8188ae5a33de20e868d7c249454477f38d7
created_at: 2025-05-06T05:40:00Z
author_association: MEMBER
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: ywangd
body: Updated in fbe4b8188ae5a33de20e868d7c249454477f38d7. It should be a little more efficient since due to less call to Set#contains and less set constructions. But it essentially still has two 2 loops, 1st to initialize the removed set and 2nd to populate the added set. Please let me know whether it works for you or if you have better ideas.

> we still want that shortcut logic for single-project mode

Yeah. I kept that. We don't need to skip the default project in the main loop since it should never be added or removed. I added assertions for it.
created_at: 2025-05-06T05:43:23Z
author_association: MEMBER
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: ywangd
body: The assertions caused some test failures because they can create unreleasitc cluster state without the default project. Pushed e02d5d57f801243dd029b76cc0348dcc97d16109 for fix. Let's see if there are more issues.
created_at: 2025-05-06T06:47:44Z
author_association: MEMBER
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: nielsbauman
body: Ah yeah I had something else in mind but I just realized that doesn't work. You're right, we're still essentially looping twice over the data. Then I think I'd prefer your initial solution... :sweat_smile: That uses less memory as we need to copy the whole key set here. In the end it's just going to come down to memory vs. runtime, I guess. I think I'd be leaning towards saving some memory to avoid some GC work, but the differences will be minimal, so I'd be fine either way I think.

I think we can also add a shortcut for `previousMetadata == currentMetadata`, to skip some cluster state updates.
created_at: 2025-05-06T06:59:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -336,4 +348,26 @@ private List<Index> indicesDeletedFromTombstones() {
             .toList();
     }
 
+    private static ProjectsDelta calculateProjectDelta(Metadata previousMetadata, Metadata currentMetadata) {
+        if (previousMetadata.projects().size() == 1
+            && previousMetadata.hasProject(ProjectId.DEFAULT)
+            && currentMetadata.projects().size() == 1
+            && currentMetadata.hasProject(ProjectId.DEFAULT)) {
+            return EMPTY_PROJECT_DELTA;
+        }
+
+        return new ProjectsDelta(
+            Collections.unmodifiableSet(Sets.difference(currentMetadata.projects().keySet(), previousMetadata.projects().keySet())),
+            Collections.unmodifiableSet(Sets.difference(previousMetadata.projects().keySet(), currentMetadata.projects().keySet()))
+        );
path: server/src/main/java/org/elasticsearch/cluster/ClusterChangedEvent.java
user: login: ywangd
body: OK I reverted it to the original approach. Also added check for `previousMetadata == currentMetadata`. I had to comment out the assertion for no add/remove for default project since there are quite some tests don't do the right thing and fixing them feels a bit overwhelming for this PR.
created_at: 2025-05-09T06:30:35Z
author_association: MEMBER

PR comments:
user: login: elasticsearchmachine
created_at: 2025-05-05T07:03:27Z
body: Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)
author_association: COLLABORATOR
user: login: ywangd
created_at: 2025-05-20T04:18:46Z
body: @elasticmachine update branch
author_association: MEMBER