Change:
My cluster contains about 4w endpointslices. There are 3 related metrics (endpointslicecontrollerendpointsaddedpersynccount, endpointslicecontrollerendpointsremovedpersynccount, endpointslicecontrollerchanges) to indicates endpoint add remove and update. I found that the apiserver periodically receives lots of endpointslice update requests. In terms of the number of requests, the endpointslice has been fully updated. But from the endpointslice resource itself, it seems that it has not been updated, because "" annotation didn't change It seems that there is no need to update the endpointslice with no really changes. On the other hand, a large number of ep slice update requests will stress the apiserver. In order to make the phenomenon more obvious, the min-resync-period parameter of controller-manager can be adjusted to 1 minute. Create one or more empty endpointslices, and then you can find in the apiserver logs that requests to update these endpointslice every one to two minutes. For clusters with lots of empty endpointslices, this bug may affect the stability of it. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
there were different improvements in this area, but since 1.20 is going EOL in 8 days it will be good to know if this happens in 1.21 /cc
I'm glad to recieve your comment. I'm not sure because I only have cluster with version 1.20.10. Expect someone who could provide more detail about it. It is actually happening in my cluster, but I don't know how it happens.
I've seen this in 1.20 clusters as well, but have not seen it in 1.21. My theory was that this is mostly or entirely fixed by Prior to that, any change to EndpointSlice labels would not trigger an EndpointSlice controller sync loop, but the resync period would result in those EndpointSlices being updated at that point. more recent fix () seems like it could also be relevant, but I don't think it matches what's described here. In that case, I believe we're unnecessarily updating EndpointSlices if something irrelevant in the Pod changes (resource version), but I don't believe there's anything special there that would mean we weren't updating EndpointSlices until the resync period. So with all of that said, I believe this is fixed in 1.21+, but let me know if anyone's able to recreate this in those versions and/or if is actually more relevant than it appears on the surface.
Thanks for your reply I will check it.
I confirmed that the periodic updates in my cluster are all empty endpointslices. There are l lot of IPv6 endpointslice in my cluster. But it shouldn't be updated when endpointslice-controller couldn't list pods by service label selector.
/assign
I found that when endpointslice's endpoint array is empty, endpointslice-controller updates it every periodic resync, although it didn't change. I consider it a bug and would like to fork and fix it. The code below could fix the issue of updating empty endpointslices, but I'm not sure if it will introduce new issues. for i := 0; i < len(slicesToDelete); { if len(slicesToCreate) == 0 { break } sliceToDelete := slicesToDelete[i] slice := slicesToCreate[len(slicesToCreate)-1] // Only update EndpointSlices that are owned by this Service and have // the same AddressType. We need to avoid updating EndpointSlices that // are being garbage collected for an old Service with the same name. // The AddressType field is immutable. Since Services also consider // IPFamily immutable, the only case where this should matter will be // the migration from IP to IPv4 and IPv6 AddressTypes, where there's a // chance EndpointSlices with an IP AddressType would otherwise be // updated to IPv4 or IPv6 without this check. if sliceToDelete.AddressType == slice.AddressType && ownedBy(sliceToDelete, service) { slicesToCreate = slicesToCreate[:len(slicesToCreate)-1] // We must consider the situation that an empty unchanged endpointslice no need to update. If the one to be deleted and the one to be created are the same endpointslice which is empty, skip the update step. // eg: If there are many headless svc with single stack ipv4 for example, a lot of ipv6 empty // endpointslice update could be avoid. if slice.Name != sliceToDelete.Name len(slice.Endpoints) != 0 { slice.Name = sliceToDelete.Name slicesToUpdate = append(slicesToUpdate, slice) } slicesToDelete = append(slicesToDelete[:i], slicesToDelete[i+1:]...) } else { i++ } /cc
Hey I'm not sure I completely understand this fix, but thanks for volunteering to take this on! As I recall, this specific section of the reconciler code is part of the "recycling" logic, where an EndpointSlice marked for creation reuses the name of when marked for the deletion, and the operation becomes an update. I'd appreciate a PR if you can also include a test that recreates the buggy behavior + shows that this fixes it.
Thanks for your reply. I will try more test and show this fix does work.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I understand the bug now - it looks the root issue is here: Essentially we'd consider the previous placeholder EndpointSlice no longer desired, mark it for deletion, and then that would mean no EndpointSlices would be left, triggering a new placeholder slice to be marked for creation. The recycling logic you linked above essentially merges creates and deletes into updates, resulting in the behavior you described of empty EndpointSlices always being updated during the resync period. I think this should be relatively straightforward to fix with some changes to the code I referenced above.
/assign
GitHub didn't allow me to assign the following users: spencerhance. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign

Discussion:
assert.Nil(t, err, "Expected no error syncing service") // The EndpointSlice marked for deletion should be ignored by the controller, and thus <del> // should not result in more than one action from the client (an update to the non-terminating // EndpointSlice removing the trigger time annotation.) if len(client.Actions()) != numActionsBefore+1 { t.Errorf("Expected 1 more action, got %d", len(client.Actions())-numActionsBefore) </del> <ins> // should not result in any action. if len(client.Actions()) != numActionsBefore { t.Errorf("Expected 0 more actions, got %d", len(client.Actions())-numActionsBefore) </ins> } }

discovery "k8s.io/api/discovery/v1" "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/conversion" </ins> "k8s.io/apimachinery/pkg/types" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/apimachinery/pkg/util/sets"

// When no endpoint slices would usually exist, we need to add a placeholder. if len(existingSlices) == len(slicesToDelete) && len(slicesToCreate) < 1 { <ins> // Check for existing placeholder slice outside of the core control flow </ins> placeholderSlice := newEndpointSlice(service, &endpointMeta{Ports: []discovery.EndpointPort{}, AddressType: addressType}) <del> slicesToCreate = append(slicesToCreate, placeholderSlice) spMetrics.Set(endpointutil.NewPortMapKey(placeholderSlice.Ports), metrics.EfficiencyInfo{ Endpoints: 0, Slices:    1, }) </del> <ins> if len(slicesToDelete) == 1 && placeholderSliceCompare.DeepEqual(slicesToDelete[0], placeholderSlice) { // We are about to unnecessarily delete/recreate the placeholder, remove it now. slicesToDelete = slicesToDelete[:0] } else { slicesToCreate = append(slicesToCreate, placeholderSlice) spMetrics.Set(endpointutil.NewPortMapKey(placeholderSlice.Ports), metrics.EfficiencyInfo{ Endpoints: 0, Slices:    1, }) } </ins> } metrics.EndpointsAddedPerSync.WithLabelValues().Observe(float64(totalAdded))

return r.finalize(service, slicesToCreate, slicesToUpdate, slicesToDelete, triggerTime) } <ins> // placeholderSliceCompare is a conversion func for comparing two placeholder endpoint slices. // It only compares the specific fields we care about. var placeholderSliceCompare = conversion.EqualitiesOrDie( func(a, b metav1.OwnerReference) bool { return a.String() == b.String() }, func(a, b metav1.ObjectMeta) bool { if a.Namespace != b.Namespace { return false } for k, v := range a.Labels { if b.Labels[k] != v { return false } } for k, v := range b.Labels { if a.Labels[k] != v { return false } } return true }, ) </ins> // finalize creates, updates, and deletes slices as specified func (r *reconciler) finalize( service *corev1.Service,

"time" "github.com/stretchr/testify/assert" corev1 "k8s.io/api/core/v1" discovery "k8s.io/api/discovery/v1" apiequality "k8s.io/apimachinery/pkg/api/equality"

} } <del> // given an existing endpoint slice and no pods matching the service, the existing // slice should be updated to a placeholder (not deleted) </del> <ins> // given an existing placeholder endpoint slice and no pods matching the service, the existing // slice should not change the placeholder </ins> func TestReconcile1EndpointSlice(t *testing.T) { <del> client := newClientset() setupMetrics() </del> namespace := "test" <del> svc, endpointMeta := newServiceAndEndpointMeta("foo", namespace) endpointSlice1 := newEmptyEndpointSlice(1, namespace, endpointMeta, svc) </del> <ins> svc, epMeta := newServiceAndEndpointMeta("foo", namespace) emptySlice := newEmptyEndpointSlice(1, namespace, epMeta, svc) emptySlice.ObjectMeta.Labels = map[string]string{"bar": "baz"} </ins> <del> _, createErr := client.DiscoveryV1().EndpointSlices(namespace).Create(context.TODO(), endpointSlice1, metav1.CreateOptions{}) assert.Nil(t, createErr, "Expected no error creating endpoint slice") </del> <ins> testCases := []struct { desc        string existing    *discovery.EndpointSlice wantUpdate  bool wantMetrics expectedMetrics }{ { desc:        "No existing placeholder", wantUpdate:  true, wantMetrics: expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 1, numUpdated: 0, numDeleted: 0, slicesChangedPerSync: 1}, }, { desc:        "Existing placeholder that's the same", existing:    newEndpointSlice(&svc, &endpointMeta{Ports: []discovery.EndpointPort{}, AddressType: discovery.AddressTypeIPv4}), wantMetrics: expectedMetrics{desiredSlices: 0, actualSlices: 0, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 0, numUpdated: 0, numDeleted: 0, slicesChangedPerSync: 0}, }, { desc:        "Existing placeholder that's different", existing:    emptySlice, wantUpdate:  true, wantMetrics: expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 0, numUpdated: 1, numDeleted: 0, slicesChangedPerSync: 1}, }, } </ins> <del> numActionsBefore := len(client.Actions()) r := newReconciler(client, []*corev1.Node{{ObjectMeta: metav1.ObjectMeta{Name: "node-1"}}}, defaultMaxEndpointsPerSlice) reconcileHelper(t, r, &svc, []*corev1.Pod{}, []*discovery.EndpointSlice{endpointSlice1}, time.Now()) assert.Len(t, client.Actions(), numActionsBefore+1, "Expected 1 additional clientset action") actions := client.Actions() assert.True(t, actions[numActionsBefore].Matches("update", "endpointslices"), "Action should be update endpoint slice") </del> <ins> for _, tc := range testCases { t.Run(tc.desc, func(t *testing.T) { client := newClientset() setupMetrics() </ins> <del> slices := fetchEndpointSlices(t, client, namespace) assert.Len(t, slices, 1, "Expected 1 endpoint slices") </del> <ins> existingSlices := []*discovery.EndpointSlice{} if tc.existing != nil { existingSlices = append(existingSlices, tc.existing) _, createErr := client.DiscoveryV1().EndpointSlices(namespace).Create(context.TODO(), tc.existing, metav1.CreateOptions{}) assert.Nil(t, createErr, "Expected no error creating endpoint slice") } </ins> <del> assert.Regexp(t, "^"+svc.Name, slices[0].Name) assert.Equal(t, svc.Name, slices[0].Labels[discovery.LabelServiceName]) assert.EqualValues(t, []discovery.EndpointPort{}, slices[0].Ports) assert.EqualValues(t, []discovery.Endpoint{}, slices[0].Endpoints) expectTrackedGeneration(t, r.endpointSliceTracker, &slices[0], 1) expectMetrics(t, expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 0, addedPerSync: 0, removedPerSync: 0, numCreated: 0, numUpdated: 1, numDeleted: 0, slicesChangedPerSync: 1}) </del> <ins> numActionsBefore := len(client.Actions()) r := newReconciler(client, []*corev1.Node{{ObjectMeta: metav1.ObjectMeta{Name: "node-1"}}}, defaultMaxEndpointsPerSlice) reconcileHelper(t, r, &svc, []*corev1.Pod{}, existingSlices, time.Now()) var numUpdates int if tc.wantUpdate { numUpdates = 1 } wantActions := numActionsBefore + numUpdates assert.Len(t, client.Actions(), wantActions, "Expected %d additional clientset actions", numUpdates) slices := fetchEndpointSlices(t, client, namespace) assert.Len(t, slices, 1, "Expected 1 endpoint slices") if !tc.wantUpdate { assert.Regexp(t, "^"+svc.Name, slices[0].Name) assert.Equal(t, svc.Name, slices[0].Labels[discovery.LabelServiceName]) assert.EqualValues(t, []discovery.EndpointPort{}, slices[0].Ports) assert.EqualValues(t, []discovery.Endpoint{}, slices[0].Endpoints) if tc.existing == nil { expectTrackedGeneration(t, r.endpointSliceTracker, &slices[0], 1) } } expectMetrics(t, tc.wantMetrics) }) } } func TestPlaceHolderSliceCompare(t *testing.T) { testCases := []struct { desc string x    *discovery.EndpointSlice y    *discovery.EndpointSlice want bool }{ { desc: "Both nil", want: true, }, { desc: "Y is nil", x:    &discovery.EndpointSlice{}, want: false, }, { desc: "X is nil", y:    &discovery.EndpointSlice{}, want: false, }, { desc: "Both are empty and non-nil", x:    &discovery.EndpointSlice{}, y:    &discovery.EndpointSlice{}, want: true, }, { desc: "Only ObjectMeta.Name has diff", x: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ Name: "foo", }}, y: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ Name: "bar", }}, want: true, }, { desc: "Only ObjectMeta.Labels has diff", x: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ "foo": "true", }, }}, y: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{ "bar": "true", }, }}, want: false, }, { desc: "Creation time is different", x: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ CreationTimestamp: metav1.Unix(1, 0), }}, y: &discovery.EndpointSlice{ObjectMeta: metav1.ObjectMeta{ CreationTimestamp: metav1.Unix(2, 0), }}, want: true, }, { desc: "Different except for ObjectMeta", x:    &discovery.EndpointSlice{AddressType: discovery.AddressTypeIPv4}, y:    &discovery.EndpointSlice{AddressType: discovery.AddressTypeIPv6}, want: false, }, } for _, tc := range testCases { t.Run(tc.desc, func(t *testing.T) { got := placeholderSliceCompare.DeepEqual(tc.x, tc.y) if got != tc.want { t.Errorf("sliceEqual(%v, %v) = %t, want %t", tc.x, tc.y, got, tc.want) } }) } </ins> } // when a Service has PublishNotReadyAddresses set to true, corresponding