Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: was closed as not a problem with kubectl, but with the server What you expected to happen: 'kubectl replace' with no change on a service should work. How to reproduce it (as minimally and precisely as possible): reproduction included in this issue Anything else we need to know?: Seems to me like this comment sums up the options I got here from this issue Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
After a long day of reading and experimenting with the Kubernetes API i found a work-around I want to share with you. Problem in my case: I'm using the Java Client K8 API, especially I got the exact same error like written in and here in this issue summary: I send an update of a service WITHOUT specifying the spec.clusterip within the V1Service object. The request gets then rejected with the reason . Workaround: First load the existing service that contains the current clusterIP. Set the old clusterIp to the updated V1Service. Also load the old metadata.resourceVersion value and set it to the new V1Service object, because leaving the value out using the Java API Client results in an error: Please do not close this issue due to inactivity again, like it was done with . Even if there are some workarounds, this issue will pop up again and again, because there really is a problem with the API restrictions or at least the messages you will get when interacting with the API. Constructive Suggestion: e.g. instead of throwing the error you could throw something like and for the resourceVersion:
Seriously, this has been going on for years. is over 2 years old and there have been dozens of bug reports/feature requests (in Kubernetes, Kubectl, Helm, and elsewhere) showing the problems the current behavior is causing. (List available on request.) is from over 3 years ago. is from 2015. The old justifications for keeping the current behavior ("works as designed") have proven to be insufficient. It is time for a redesign. The API should accept as meaning "assign a clusterIP" on creation and "keep the current clusterIP" on update. Since, when you set on creation, the resource does not end up with , this means a resource cannot validly have , and since the field is immutable, could never be a correct value for modification under the current API. While this could be considered a breaking change, I doubt there is anyone who would complain about it. Rejecting as a modification is not helping anyone. It is not rejecting an accidental change or avoiding confusion because the meaning is unclear, and it is hard for me to imagine a use case where someone is relying on the current behavior and would be upset about the proposed new behavior (with the possible exception of people writing compliance tests). It is just causing problems for the sake of some kind of theoretical purity that is in the eye of the beholder. Treating as meaning "system should replace this with a valid setting" is equally pure.
cc long-term-issue (note to self)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
keepalive
Is there something I can do to move this forward?
/remove-sig cli /sig network /assign this is sort of the inverse of the issue fixed in Using with a static manifest on an object that has spec fields populated by the API server (like clusterIP in service) or controller (like replicas in deployment) is an anti-pattern. is a better match for this usage. Alternatively, a controller that did a 'read -modify -replace' could be ok, since it would be starting with the current API server state.
This is not impossible to fix in the PUT path (), but it basically turns PUT into PATCH. We have distinct PATCH operations for a reason, and that is almost certainly what you want in this case. should "just work" as a replacement for . Is there a reason why you can't use PATCH or apply? - while I continue to argue that is wrong here, maybe we should just cave and make it quietly patch the value of clusterIP in this specific case? I really dislike it, not just for purity reasons, but because I have a hard time seeing if that will bite us later. Thoughts?
I really dislike it as well. "replace all spec content" is a bad thing to teach clients to do on resources that have fields set by different actors.
Agree, and while we might be able to quietly patch this SPECIFIC case, we can't do that in general. - My inclination is to close this. I know it's not what you want, but I do think you and most users will be better served by patch or apply in almost all cases (even knowing that Service has an ambiguity around changing ports that overlap each other which is not easily fixed because it is breaking). Is there a reason why you can't use patch or apply?
This really seems related to bug in Helm, as far as what I have read, and some recent (Jan2021) posts suggest that was fixed. Have not tested this out myself. Agree that adhering to expected PUT/PATCH api behavior would be best. Thanks for the followup!
I reject the idea that this is a bug in Helm, even though Helm is working hard (and failing) to handle the situation. I agree with you when you "Agree that adhering to expected PUT/PATCH api behavior would be best". My point is that the expected "PUT" behavior is that if a "PUT" request succeeds, the exact same "PUT" request given a second time should succeed as well. That is what is broken and I am asking to have fixed. Is there a better reason why I should be prohibited from using replace on a Service, when I can use it for every other resource? Why should Service require special-case handling by the user? I ask you to think of this in the larger context of using Infrastructure as Code (IaC) to manage resources, and the automation around that. In the IaC definition of a Service, the has to be set to in order to get a system-assigned address. Semantically, says "make the resource exactly like this". It can be, and is, used to ensure there is not "drift" in the configuration, i.e. that the configuration stored in the IaC code base is live. It is the correct thing to use when you want to say "here is the configuration that should be active", and it is what automation uses to reset the configuration after it has drifted. Users should not have to treat Services specially, they should be able to any resources with the same spec that created them. Stating this another way, (A user should be able to the same thing repeatedly, successfully, and without causing changes after the first accepted .) By not accepting for in , you are breaking the contract of making idempotent. (As a matter of practicality, the Helm maintainers are trying to make up for the fact that is not idempotent, but that is not an excuse for Kubernetes to leave this API in its current state.) As I stated above, I am not even advocating for this case to be handled as a . As I see it, the value of is handled specially but not idempotently. The value of currently means one thing if the resource does not exist and something else if it does exist. I am asking instead that it consistently, idempotently be taken as meaning "any valid address: the existing one if one is set, otherwise one randomly assigned by the controller".
Agree, and while we might be able to quietly patch this SPECIFIC case, we can't do that in general. Unfortunately, to make PUT idempotent in this case, it would need to turn into a variant of PATCH. I'm not even saying I totally disagree, but that is what it means. As for PUT being idempotent, I don't think that's ever really been a hard objective. In this case (and look, I am not saying I am happy about it) that means reading existing state and patching it into the PUT. We can do that (probably) but I can't say for sure that there are no other APIs where this will happen (there probably are not). Note that to do this we would need to fixup , , and probably the newer dual-stack-related fields. Can someone help me understand the "bug"? Is it about using PUT instead of apply? I could see two paths to fixing this, if we want to (and I am still not 100%). 1) Change clusterIP and nodePort fields to pointers. If not-specified on PUT AND the object exists AND the old warrants a value AND the new warrants a value, copy the existing values. This is a bit tricky for nodePort we'd have to at least specify what happens if the list is reordered or otherwise modified. For example, if the existing object has: and the update specifies: Do I track the move by name? What if the names changed, does that lose the tracking? We've had innumerable issues here because the API-specified merge-key was (is) ambiguous. The only way to get exactly what you want in ALL cases is to RMW the resource. Explicitly setting "" and 0 would work on initialization (compat) but would not be auto-patched. 2) Leave the fields as non-pointers and accept "" and 0 as "DWIM" mode. I dislike this because it makes it less obvious when a user mistakenly tries to clear a field - it used to fail and will now "succeed" but silently change the value. I am not sure I see a clever 3rd option. The only reason I entertain this at all is because of the issues around apply/patch and ports. NOTE: My willingness to think it thru is not a commitment to implement it. I am swamped right now :)
wrote I am shocked that you would think that, although I don't really know what you mean by "hard objective". The doc certainly implies should be idempotent, via such statements as and Moving on to how to do make it so, the idea of is that at the end of the operation, the object is in the state described by the action. If you PUT ports P and Q and the existing object has ports R and S, you delete R and S and add P and Q. If adding or deleting a port is not allowed, then you reject the PUT. I think to some extent you are overthinking/overdesigning a solution here. In Kubernetes, every PUT is to some extent a patch because there are some fields that are not specifiable via PUT, like all the managed fields, or , and fields like and are special in a whole different way. But that stuff should already be handled by the more generalized resource handling architecture. The general rule about PUT is that PUT on an existing resource should adjust that resource to be in the same working configuration as if you used that PUT to create a new resource. Calling PUT a second time with the same input should be acceptable and should not change the resource. I believe that Kubernetes generally has a canonical ordering of all fields in a resource (and if there is no clear way to order a list, the canonical order is simply the order in which the list was originally specified on resource creation), so what you do is enumerate the fields of the existing resource in canonical order, filter out the fields like that PUT cannot specify, and then compare it to the PUT resource, after normalizing it into canonical order, and add/delete/replace field by field. Clearly something like this is already happening in handling PUT and PATCH. This should not be a lot of new work. The ongoing community complaint is that when PUT says and it is matched against the exiting the API rejects it, saying you can't change the . All we are asking for is that this case be interpreted differently and allowed. is documented as an allowable value in the input, but it is never an allowable value in the realized resource, so it cannot possibly mean to be interpreted as "force this value on the resource". Its meaning is that the system should assign a valid value and the user does not care what it is. So on PUT, when is matched against the exiting , he API should just confirm that is a valid value and, if so, accept the input.
As for PUT being idempotent, I don't think that's ever really been a hard objective. What I meant is it's just never been a primary question in reviewing APIs. Service is a weird case, far outside the norm and I'm not excusing it - just explaining what I meant. Moving on to how to do make it so, the idea of is that at the end of the operation, the object is in the state described by the action. If you PUT ports P and Q and the existing object has ports R and S, you delete R and S and add P and Q. If adding or deleting a port is not allowed, then you reject the PUT. For historical reasons, the "key" for ports was not the name but the port value, then the value + protocol. So we can key off that, I think. I think to some extent you are overthinking/overdesigning a solution here. In Kubernetes, every PUT is to some extent a patch because there are some fields that are not specifiable via PUT, like all the managed fields, or , and fields like and are special in a whole different way. Those are all outside of . I don't know of any resource that PUT-PATCH inside the spec. Hilariously, I just noticed that nodePort is even worse - it accepts the non-value and then re-allocates the nodePort. I am going to call this a bug, but now we have a compatibility discussion to have. If a user does not specify a value, we can reasoinably argue that we should retain existing values. But if they specify , do we stay compatible and re-allocate the node port, or do we say that was always a bug? Is it a useful "feature" to be able to re-allocate? We disallow it explicitly gfor ClusterIPs. Clearly something like this is already happening in handling PUT and PATCH. This should not be a lot of new work. You haven't spent much time in the Service REST code, I guess. I have a large rework of that on-going, but this was not part of it. :) The ongoing community complaint is that when PUT says and it is matched against the exiting the API rejects it, saying you can't change the . Given the previous point about nodePort, would it be fair to modify your statement as: "when PUT says (or does not specify at all) ..." With the corollart that if you actually specify you will get the current (immutable error) behavior. This seems like it would satisfy helm and other users who are not specifying literal "" - they are not specifying any value at all. Thoughts?
At this point I am willing to call it a bug, albeit an unpleasant one.
wrote I have had regular run-ins with this issue for years, which is why I'm being insistent on getting it fixed properly. I do not recall having similar problems with and so I am less informed on those issues. I don't know what means in the spec or in practice, so I don't have a specific opinion as to what it should do to an existing resource. I will simply restate the general principle which should apply to just like everything else: what ever you allow as a setting on creation should be allowed in any and should not cause a change if repeatedly applied. The very practical reason for this is so that I can record some place (like a checked into a VCS) that "this is how the resource should be configured" and use that (1) to create the resource and (2) to return the resource to expected configuration after some external operation changed it, using a standard API while otherwise treating the resource as opaque.
/cc
/cc (i know i am going to regret this)
It is a bug. Not because it does not match rest api semantics. But because it is not consistent with the rest of the apis. In other words, If user can do on other resources they should be able to do it on as well. It is hard (and weird) to ask users to do one thing for all resources, but then do a totally different thing for Services. As discussed over sig-network the problem will be user intent. Some ideas for a solution: If irrespective of changes in other we can assume that the user is replacing and does not care about ClusterIP value, hence copying to new should be fine. It is basically like the user saying so copying old=new should be Ok. If that is a failure case, because it breaks ClusterIP immutability rules ( we should really, really remove ClusterIP immutability but i digress). We will still require the validation to work as is. e.g. replacing with with a type that does not require ClusterIP and not setting it to should fail. I can look into building a POC for this.
I am probably missing some subtlety here, but as in I suspect that Helm is behaving properly and the problem is in way Kubernetes handles s: the of a resource should never be rewritten with variable environmental information; that is what is for. In other words could be set in the unusual case that you wished to pick an IP yourself but would normally be null; would reflect the service’s current value, expected to be retained across unrelated patches.
nailed it on the head with his last comment. I do agree that we should not be re-defining the behaviour of a PUT request. But the inconsistency of how a Service updates the field compared to other resources results in inconsistencies in the PUT behaviour. Once a resource defines fields in the object, normally Kubernetes leaves it unchanged. My understanding is that Kubernetes uses this field to use as a reference point to be the "ideal state of the world", using the field to record the "current state of the resource". The only offending case I've seen in the wild is a , as it updates both the field AND the field. The end result is that a with a defining results in differing behaviour than the rest of the resource APIs. From Helm's point of view, a chart author's only course of action is to a) omit from the manifest and all subsequent updates, or b) if you happened to define in your first install, find the value Kubernetes provided and inject it into your template so that the value Kubernetes provided is preseved. Failing to do so results in a failed PUT () request. I wrote this behaviour down in-depth in this Helm comment: I'd propose an alternative solution than solutions: instead of updating after an IP has been assigned to the Service, use the API, i.e. . It would be a breaking change to the API, but it should bring Services back in line with the rest of the resource APIs, leaving as a user-defined field.
I agree
I'm afraid that breaking an API is not a feasible option
That document does allow for breaking changes under the apiVersion. Instead of making a breaking change to v1 services, you can introduce Service v2. Existing v1 services migrate their spec.clusterIP’s existing value (auto-generated or not) to the new version. spec.clusterIP and status.clusterIP would now match their corresponding relationships, and Kubernetes retains backwards/forwards compatibility.
(If it's not clear whether the API changes document does or doesn't allow the change from last comment, let's raise a separate improvement request against that document).
Does the case is important ? Because here I can read or ...
I have a patch. I'll try to get it out for review tonight.
Has anyone yet checked that the purported fix actually addresses the issue in Helm?
Please do!
I can still see the error but I'm on GKE 1.18, will this only apply to Kubernetes 1.21/22?
It will be part of 1.22 or the next 1.21.X (I don't remember now what is the latest stable)
Can anyone confirm that this issue was actually fixed in k8s version 1.21.4 or 1.22.1 ? I do not see it listed in the release notes/change log
The change was released in the 1.22.0-beta.2, so yes it should be available in 1.22. See I do not believe it was backported to 1.21.
Thanks for confirming and providing the details

Discussion:
} } <ins> // SetExternalTrafficPolicy sets the externalTrafficPolicy field for a Service. func SetExternalTrafficPolicy(policy api.ServiceExternalTrafficPolicyType) Tweak { return func(svc *api.Service) { svc.Spec.ExternalTrafficPolicy = policy } } </ins> // SetAllocateLoadBalancerNodePorts sets the allocate LB node port field. func SetAllocateLoadBalancerNodePorts(val bool) Tweak { return func(svc *api.Service) {

} } <ins> func TestServiceRegistryUpdateUnspecifiedAllocations(t *testing.T) { testCases := []struct { name  string svc   *api.Service // Need a clusterIP, NodePort, and HealthCheckNodePort allocated tweak func(*api.Service) }{{ name: "single-port", svc: svctest.MakeService("foo", svctest.SetTypeLoadBalancer, svctest.SetExternalTrafficPolicy(api.ServiceExternalTrafficPolicyTypeLocal)), tweak: nil, }, { name: "multi-port", svc: svctest.MakeService("foo", svctest.SetTypeLoadBalancer, svctest.SetExternalTrafficPolicy(api.ServiceExternalTrafficPolicyTypeLocal), svctest.SetPorts( svctest.MakeServicePort("p", 80, intstr.FromInt(80), api.ProtocolTCP), svctest.MakeServicePort("q", 443, intstr.FromInt(443), api.ProtocolTCP))), tweak: nil, }, { name: "shuffle-ports", svc: svctest.MakeService("foo", svctest.SetTypeLoadBalancer, svctest.SetExternalTrafficPolicy(api.ServiceExternalTrafficPolicyTypeLocal), svctest.SetPorts( svctest.MakeServicePort("p", 80, intstr.FromInt(80), api.ProtocolTCP), svctest.MakeServicePort("q", 443, intstr.FromInt(443), api.ProtocolTCP))), tweak: func(s *api.Service) { s.Spec.Ports[0], s.Spec.Ports[1] = s.Spec.Ports[1], s.Spec.Ports[0] }, }} for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { ctx := genericapirequest.NewDefaultContext() storage, server := NewTestREST(t, []api.IPFamily{api.IPv4Protocol}) defer server.Terminate(t) svc := tc.svc.DeepCopy() obj, err := storage.Create(ctx, svc.DeepCopy(), rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) if err != nil { t.Fatalf("Expected no error: %v", err) } createdSvc := obj.(*api.Service) if createdSvc.Spec.ClusterIP == "" { t.Fatalf("expected ClusterIP to be set") } if len(createdSvc.Spec.ClusterIPs) == 0 { t.Fatalf("expected ClusterIPs to be set") } for i := range createdSvc.Spec.Ports { if createdSvc.Spec.Ports[i].NodePort == 0 { t.Fatalf("expected NodePort[%d] to be set", i) } } if createdSvc.Spec.HealthCheckNodePort == 0 { t.Fatalf("expected HealthCheckNodePort to be set") } // Update from the original object - just change the selector. svc.Spec.Selector = map[string]string{"bar": "baz2"} svc.ResourceVersion = createdSvc.ResourceVersion obj, _, err = storage.Update(ctx, svc.Name, rest.DefaultUpdatedObjectInfo(svc.DeepCopy()), rest.ValidateAllObjectFunc, rest.ValidateAllObjectUpdateFunc, false, &metav1.UpdateOptions{}) if err != nil { t.Fatalf("Expected no error: %v", err) } updatedSvc := obj.(*api.Service) if want, got := createdSvc.Spec.ClusterIP, updatedSvc.Spec.ClusterIP; want != got { t.Errorf("expected ClusterIP to not change: wanted %v, got %v", want, got) } if want, got := createdSvc.Spec.ClusterIPs, updatedSvc.Spec.ClusterIPs; !reflect.DeepEqual(want, got) { t.Errorf("expected ClusterIPs to not change: wanted %v, got %v", want, got) } portmap := func(s *api.Service) map[string]int32 { ret := map[string]int32{} for _, p := range s.Spec.Ports { ret[p.Name] = p.NodePort } return ret } if want, got := portmap(createdSvc), portmap(updatedSvc); !reflect.DeepEqual(want, got) { t.Errorf("expected NodePort to not change: wanted %v, got %v", want, got) } if want, got := createdSvc.Spec.HealthCheckNodePort, updatedSvc.Spec.HealthCheckNodePort; want != got { t.Errorf("expected HealthCheckNodePort to not change: wanted %v, got %v", want, got) } }) } } </ins> func TestServiceRegistryUpdateDryRun(t *testing.T) { ctx := genericapirequest.NewDefaultContext() storage, server := NewTestREST(t, []api.IPFamily{api.IPv4Protocol})

oldService := old.(*api.Service) newService.Status = oldService.Status <ins> patchAllocatedValues(newService, oldService) </ins> NormalizeClusterIPs(oldService, newService) dropServiceDisabledFields(newService, oldService) dropTypeDependentFields(newService, oldService)

return nil } <ins> // patchAllocatedValues allows clients to avoid a read-modify-write cycle while // preserving values that we allocated on their behalf.  For example, they // might create a Service without specifying the ClusterIP, in which case we // allocate one.  If they resubmit that same YAML, we want it to succeed. func patchAllocatedValues(newSvc, oldSvc *api.Service) { if needsClusterIP(oldSvc) && needsClusterIP(newSvc) { if newSvc.Spec.ClusterIP == "" { newSvc.Spec.ClusterIP = oldSvc.Spec.ClusterIP } if len(newSvc.Spec.ClusterIPs) == 0 { newSvc.Spec.ClusterIPs = oldSvc.Spec.ClusterIPs } } if needsNodePort(oldSvc) && needsNodePort(newSvc) { // Map NodePorts by name.  The user may have changed other properties // of the port, but we won't see that here. np := map[string]int32{} for i := range oldSvc.Spec.Ports { p := &oldSvc.Spec.Ports[i] np[p.Name] = p.NodePort } for i := range newSvc.Spec.Ports { p := &newSvc.Spec.Ports[i] if p.NodePort == 0 { p.NodePort = np[p.Name] } } } if needsHCNodePort(oldSvc) && needsHCNodePort(newSvc) { if newSvc.Spec.HealthCheckNodePort == 0 { newSvc.Spec.HealthCheckNodePort = oldSvc.Spec.HealthCheckNodePort } } } </ins> // NormalizeClusterIPs adjust clusterIPs based on ClusterIP.  This must not // consider any other fields. func NormalizeClusterIPs(oldSvc, newSvc *api.Service) {