jiebi/CodeConvo huggingface dataset:
Code change:
for _, epSvcPair := range connectionMap { if svcInfo, ok := proxier.serviceMap[epSvcPair.ServicePortName]; ok && svcInfo.GetProtocol() == v1.ProtocolUDP { endpointIP := utilproxy.IPPart(epSvcPair.Endpoint) <del> err := conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) </del> <ins> nodePort := svcInfo.GetNodePort() var err error if nodePort != 0 { err = conntrack.ClearEntriesForPortNAT(proxier.exec, endpointIP, nodePort, v1.ProtocolUDP) } else { err = conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) } </ins> if err != nil { klog.Errorf("Failed to delete %s endpoint connections, error: %v", epSvcPair.ServicePortName.String(), err) }

Related discussion:
text: from /etc/os-release): tried both: and Kernel (e.g. ): and Install tools: Others:
/sig network
We also ran into this in kubernetes v1.7.6 using OpenShift 3.7:
~I believe this is the same as ~If so, it is fixed in 1.10: These are about HostPort, not NodePort, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any other known workaround for this besides updating conntrack rules on the kube nodes? I tried deleting and recreating the service, but this didn't update the rules. When I restart the stream on the client side data flows correctly again (perhaps it's perceived as a new connection), but I have to restart the stream for each sender.
/remove-lifecycle stale
FWIW, this is not fixed in 1.10 - that fix was for HostPort, and this is still a bug for NodePort.
I am seeing this issue in 1.11. I am using LoadBalancer type.
has been working on a fix for this, FYI.
This issue is extremely impactful for telephony protocols that rely on static source ip and source port. Any update on a possible workaround? I was thinking a potential workaround might be to reduce the UDP timeout to something really short so the stale conntrack entries go away faster. Or maybe there is a way to flush conntrack on the event of pod deletion? Edit: Removed stupid idea of a workaround to disable conntrack for UDP. Can't disable conntrack for UDP because it's used for DNAT from node to container.
The workaround is to not use NodePorts. Other kinds of services are already correct.
not sure what you mean. I am seeing the issue with a LoadBalancer type so it does not affect NodePorts only. As a workaround I have created a small python script that watches the K8s API for pod deletions and clears the related conntrack entries when that happens. I can share it when I get to work on Monday if you want to try it out.
Code change:
for _, epSvcPair := range connectionMap { if svcInfo, ok := proxier.serviceMap[epSvcPair.ServicePortName]; ok && svcInfo.GetProtocol() == v1.ProtocolUDP { endpointIP := utilproxy.IPPart(epSvcPair.Endpoint) <del> err := conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) </del> <ins> nodePort := svcInfo.GetNodePort() var err error if nodePort != 0 { err = conntrack.ClearEntriesForPortNAT(proxier.exec, endpointIP, nodePort, v1.ProtocolUDP) } else { err = conntrack.ClearEntriesForNAT(proxier.exec, svcInfo.ClusterIPString(), endpointIP, v1.ProtocolUDP) } </ins> if err != nil { klog.Errorf("Failed to delete %s endpoint connections, error: %v", epSvcPair.ServicePortName.String(), err) }

Related discussion:
text: I've my script . Note that it's not tested much so don't use it in a production environment. Also it cleans up all conntrack entries for a deleted pod, not only the UDP related ones.
We are also seeing this issue with type loadbalancer. We are on 1.11. Our use-case is the following: multiple devices are sending to an aggreagator machine. This machine forwards the UDP packages to a loadBalancer in gcp. When debugging the service, I noticed the following: The traffic is assigned to one pod, and the packages are not loadbalanced (client and server are the same cfr use-case) When redeploying a new cluster, Conntrack is cleared on termination signal of the old pod. (client receives connection refused) red/black and highlander deployment strategies have the same effect. How would I expect this to be working then? New cluster is . Conntrack is cleared Data is routed to NEW pods old pods are terminated Best Regards, Tim
I am seeing this exact same issue on our cluster which has k8s v1.14.5 . We have endpoint nginx pods backing a UDP service with nodeport. Suddenly one day we see traffic not hitting into nginx pod but reaching the node hosting the nginx pod. We could simulate the problem by restarting the nginx pods. I am assuming with this issue marked as closed; and having made its way into v1.14; we should not be seeing this. Would also like to point out; the one thing, in addition we have in the service is ExternalTrafficPolicy set to Local. Can anyone please guide/help on why we are seeing this in v.1.14.5 of our cluster.
Seeing this with k8s v1.16.5
Code change:
AttachDisk(isManagedDisk bool, diskName, diskURI string, nodeName types.NodeName, lun int32, cachingMode compute.CachingTypes) error // DetachDiskByName detaches a vhd from host. The vhd can be identified by diskName or diskURI. DetachDiskByName(diskName, diskURI string, nodeName types.NodeName) error <del> // GetDiskLun finds the lun on the host that the vhd is attached to, given a vhd's diskName and diskURI. GetDiskLun(diskName, diskURI string, nodeName types.NodeName) (int32, error) // GetNextDiskLun searches all vhd attachment on the host and find unused lun. Return -1 if all luns are used. GetNextDiskLun(nodeName types.NodeName) (int32, error) // DisksAreAttached checks if a list of volumes are attached to the node with the specified NodeName. DisksAreAttached(diskNames []string, nodeName types.NodeName) (map[string]bool, error) </del> <ins> // GetDataDisks gets a list of data disks attached to the node. GetDataDisks(nodeName types.NodeName) ([]compute.DataDisk, error) </ins> }

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: VirtualMachineScaleSetVM and VirtualMachine are different data structures (because of different API versions), so the disk attach operation seems duplicate between vm and vmss. That means although disk operations are in same logic, some duplication are still required now. They should use same api version in the future and should be merged into same code base, e.g. move sharing logic to Refer What you expected to happen: vmss and vm disk operations should share same code logic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have checked with currently we can’t merge, e.g. following two structs are different since they using different sdk-for-azure versions. While in the end when sdk-for-azure version is upgraded, above two versions should be same and at that time, we should merge functions like ‘AttachDisk’, ‘DetachDiskByName’, 'GetDiskLun', GetNextDiskLun, DisksAreAttached
Code change:
} var clientBuilder controller.ControllerClientBuilder if s.UseServiceAccountCredentials { <del> if len(s.ServiceAccountKeyFile) > 0 { </del> <ins> if len(s.ServiceAccountKeyFile) == 0 { </ins> // It's possible another controller process is creating the tokens for us. // If one isn't, we'll timeout and exit when our client builder is unable to create the tokens. glog.Warningf("--use-service-account-credentials was specified without providing a --service-account-private-key-file")

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: During startup shows the following warning: However is set and refers to a valid X.509 private key file. All service accounts seem to be created despite the warning. How to reproduce it (as minimally and precisely as possible): start controller-manager with Anything else we need to know?: Environment: Kubernetes version (use ): v1.8.0 OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Thanks for the report. The condition for printing the warning is backwards, but has no effect on behavior. Fixed in
Still present in 1.8.3 , can it be ~backported~ cherry-picked to 1.8 branch please?
See for proposed cherry-pick
Code change:
// Start the reconciler to fill ASW. stopChan, stoppedChan := make(chan struct{}), make(chan struct{}) go func() { <ins> defer close(stoppedChan) </ins> reconciler.Run(stopChan) <del> close(stoppedChan) </del> }() waitForMount(t, fakePlugin, volumeName, asw) // Stop the reconciler.

Related discussion:
text: <!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit : (PR with no go code changes, should not have failed any unit tests) links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assigne
/assign
/assign tried to fix it in
Code change:
t.Errorf("Expected %v, got %v", e, a) } } <ins> func TestCacheParallelNoDeadlocksNoDoubleCalls(t *testing.T) { // Make 50 random keys keys := []string{} fuzz.New().NilChance(0).NumElements(50, 50).Fuzz(&keys) // Data structure for tracking when each key is accessed. type callTrack struct { sync.Mutex accessTimes []time.Time } calls := map[string]*callTrack{} for _, k := range keys { calls[k] = &callTrack{} } // This is called to fill the cache in the case of a cache miss // or cache entry expiration. We record the time. ff := func(key string) T { ct := calls[key] ct.Lock() ct.accessTimes = append(ct.accessTimes, time.Now()) ct.Unlock() // make sure that there is time for multiple requests to come in // for the same key before this returns. time.Sleep(time.Millisecond) return key } cacheDur := 10 * time.Millisecond c := NewTimeCache(RealClock{}, cacheDur, ff) // Spawn a bunch of goroutines, each of which sequentially requests // 500 random keys from the cache. runtime.GOMAXPROCS(16) var wg sync.WaitGroup for i := 0; i < 500; i++ { wg.Add(1) go func(seed int64) { r := rand.New(rand.NewSource(seed)) for i := 0; i < 500; i++ { c.Get(keys[r.Intn(len(keys))]) } wg.Done() }(rand.Int63()) } wg.Wait() // Since the cache should hold things for 10ms, no calls for a given key // should be more closely spaced than that. for k, ct := range calls { if len(ct.accessTimes) < 2 { continue } cur := ct.accessTimes[0] for i := 1; i < len(ct.accessTimes); i++ { next := ct.accessTimes[i] if next.Sub(cur) < cacheDur { t.Errorf("%v was called at %v and %v", k, cur, next) } cur = next } } } </ins>

Related discussion:
text: Failed due to an apparently unrelated PR: https://travis-
Problem here is that I left a small chance of a double-call in my parallelized cache. Didn't realize this was being tested. Will fix.
Code change:
} } <ins> func TestWithMaxWaitRateLimiter(t *testing.T) { limiter := NewWithMaxWaitRateLimiter(DefaultControllerRateLimiter(), 500*time.Second) for i := 0; i < 100; i++ { if e, a := 5*time.Millisecond, limiter.When(i); e != a { t.Errorf("expected %v, got %v", e, a) } } for i := 100; i < 5100; i++ { if e, a := 500*time.Second, limiter.When(i); e < a { t.Errorf("expected %v, got %v", e, a) } } for i := 5100; i < 5200; i++ { if e, a := 500*time.Second, limiter.When(i); e != a { t.Errorf("expected %v, got %v", e, a) } } } </ins>

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Currently when using it adds as one of the limiters. This does not work as intended because there is no immediate sleep following the call to the limiter as seen in which does work correctly. The issue with the in its current usage is there is no forget nor a sleep immediately following the call so if is called thousands of times in a row, the timeout grows very quickly with no max. For example, if a clusters has 10k secrets and those are all enqueued using the the will end up with a 16 min wait before the tail end of objects can be enqueued and processed. I would expect some max backoff to take affect. This can be seen at
/sig api-machinery
/cc /triage accepted
it possible to set maxWaitingTime for BucketRateLimiter? when r.Limiter.Reserve().Delay() maxWaitingTime, the return of BucketRateLimiter When() is maxWaitingTime. defined is 1000ms / r.limit * MAXQPS(5000) this issue example (r.tokens = 100, r.limit = 10), if call When() times, the tail end of the object is enqueued waiting for 8.33 min. i'm not sure how to set the value of MAXQPS please help review it
The fix merged to close this issue does not fix the underlying issue with the DefaultControllerRateLimiter. The bucket is not being used correctly so this does not work as intended, please read my issue description again as to what is actually happening.
is another fix needed to solve this issue?
A token bucket rate limiter will only refill at the defined rate. If enough requeues are , it will take a long time for the refill rate to clear the queue. If a max wait is introduced, then requeues will be inserted into the queue faster than the refill rate of the token bucket rate limiter. A max wait will violate the token bucket rate limiter and we want token bucket rate limiting behavior.
Code change:
inferredHost = host } // Make a copy to avoid polluting the provided config <del> tlsConfigCopy, _ := utilnet.TLSClientConfig(transport) </del> <ins> tlsConfigCopy := utilnet.CloneTLSConfig(tlsConfig) </ins> tlsConfigCopy.ServerName = inferredHost tlsConfig = tlsConfigCopy }

Related discussion:
text: When the master dials nodes, it is mutating the tlsConfig held by the master kubelet client, to add a server name. That means if two different nodes are dialed from a master started with , x509 verification will fail for the second one. Introduced in
this is a regression from 1.3 and breaks pod logs/exec/attach and node proxying on installations that use
We probably need to review that PR and look to see if any other deliberate copies were flagged incorrectly by govet.
Installations that depend on secure trust from externally provided kubelet server certs are the highest security rings. If we don't fix this we need to tell anyone using this flag not to upgrade until we release 1.4.1.
the rest of the changes in the govet cleanup PR looked fine to me. fix for this issue is in
Since we are telling folks they have to upgrade to kubectl 1.4.0 for rollingupdate to work it would definitely be a confusing message to say other folks should not upgrade to 1.4.0 at all.
If the fix is merged into master by this afternoon, lets discuss whether or not we should cherrypick it - e.g. risks involved, how we can manually qa to verify nothing else was broken, etc.
Risk involved: If you are using a secure cluster (defined as ensuring no one can MITM your nodes and masters) and you upgrade to 1.4.0 and it does not include this fix - you are broken.
What is the likelihood of interactions with other code? Is it possible this fix could introduce a breaking change to something else? Has the fix been manually verified? Are we confident that this will fully fixes the issue (vs other edge cases exist for secure clusters that this doesn't catch). Ideally I would like to see this cherrypicked into 1.4 this evening and then cut a new beta that includes it.
None. The single line that is currently broken was changed. The rest is a new function (with no side effects), and tests. No, it restores the original intent of the TLS config copy, and tests cover the cases we are using this with (secure, insecure, custom dialer) Yes. I swept for the same issue elsewhere and didn't find any issues. We test with a fully secured master-kubelet setup in Openshift and got a clean test run with this fix applied.
SGTM. Thanks for fixing this.
Code change:
migrated := getMigratedStatusBySpec(volumeToDetach.VolumeSpec) if err != nil { <del> // On failure, add volume back to ReportAsAttached list actualStateOfWorld.AddVolumeToReportAsAttached( logger, volumeToDetach.VolumeName, volumeToDetach.NodeName) </del> <ins> // On failure, mark the volume as uncertain. Attach() must succeed before adding the volume back // to node status as attached. uncertainError := actualStateOfWorld.MarkVolumeAsUncertain( logger, volumeToDetach.VolumeName, volumeToDetach.VolumeSpec, volumeToDetach.NodeName) if uncertainError != nil { klog.Errorf("DetachVolume.MarkVolumeAsUncertain failed to add the volume %q to actual state after detach error: %s", volumeToDetach.VolumeName, uncertainError) } </ins> eventErr, detailedErr := volumeToDetach.GenerateError("DetachVolume.Detach failed", err) return volumetypes.NewOperationContext(eventErr, detailedErr, migrated) }

Related discussion:
text: After KCM restart, any Pod that reuses a volume that failed to detach gets stuck ContainerCreating with message: User "system:node:ip-10-0-243-" cannot get resource "volumeattachments" in API group "" at the cluster scope: no relationship found between node 'ip-10-0-243-' and this object See steps to reproduce below, it's quite a complex condition. It happens when a pod is restarted, a CSI driver "busy" (i.e. times out) + KCM restarted at a wrong time. Pods start just fine. a Pod that uses an attachable CSI volume. the Pod. Make sure it unmounts successfully, but any fails with a transient error, such as timeout. The external attacher continues to detach the volume at this point. kube-controller-manager (KCM) and start a new one. Wait for any leader election etc - it's important for the new KCM to populate A/D controller with no pod using the detaching volume. Let the A/D controller to observe detach error. a new Pod on the same node that uses the same volume. the CSI driver to detach the volume (i.e. stop injecting any transient errors). At this point: The external attacher succeeds detaching the volume, VolumeAttachment is deleted. The volume is marked as attached in node status (!!!) despite no VolumeAttachment exists. A/D controller does not even try to create a new VolumeAttachment, the volume is marked as attached in its Actual State of World (ASW). Kubelet tries to mount the volume, but WaitForAttach fails with: , basically forever. I created . The script expects that the cluster is created by , because it restarts KCM in the middle. It uses my hacked hostpath CSI driver to inject detach errors. Sources are at / the last commit(s) there. There are two issues that together lead to this situation: fails, , i.e. puts it back to . With the volume in , kubelet may try to mount it, while the external attacher tries to detach it. It is not as bad as it sounds, because kubelet can't read VolumeAttachment with the deletion timestamp and it will not call / of un-attached volume.
Code change:
migrated := getMigratedStatusBySpec(volumeToDetach.VolumeSpec) if err != nil { <del> // On failure, add volume back to ReportAsAttached list actualStateOfWorld.AddVolumeToReportAsAttached( logger, volumeToDetach.VolumeName, volumeToDetach.NodeName) </del> <ins> // On failure, mark the volume as uncertain. Attach() must succeed before adding the volume back // to node status as attached. uncertainError := actualStateOfWorld.MarkVolumeAsUncertain( logger, volumeToDetach.VolumeName, volumeToDetach.VolumeSpec, volumeToDetach.NodeName) if uncertainError != nil { klog.Errorf("DetachVolume.MarkVolumeAsUncertain failed to add the volume %q to actual state after detach error: %s", volumeToDetach.VolumeName, uncertainError) } </ins> eventErr, detailedErr := volumeToDetach.GenerateError("DetachVolume.Detach failed", err) return volumetypes.NewOperationContext(eventErr, detailedErr, migrated) }

Related discussion:
text: KCM starts, all VolumeAttachments that don't have a Pod (that's good), but also (that's probably bad). When the newly started KCM re-tries to detach the volume and gets an error again, it marks the volume as fully attached (same as the previous point) and adds it to . Now the volume is both mounted + attached in ASW and A/D controller actually stops here and waits forever for kubelet to unmount the volume. <detailsToday's k/k master, </details<detailsNone </details<detailsN/A, it's KCM issue. </details<details, but seen in a downstream Kubernetes distro too. </details<details</details<details</details
/sig storage
/assign
/triage accepted
I have the same problem: Seems restart of Kubelet solves this problem. Kubernetes v1.25.13
Code change:
// ResourceConfiguration stores per resource configuration. type ResourceConfiguration struct { <del> // resources is a list of kubernetes resources which have to be encrypted. </del> <ins> // resources is a list of kubernetes resources which have to be encrypted. The resource names are derived from `resource` or `resource.group` of the group/version/resource. // eg: pandas.awesome.bears.example is a custom resource with 'group': awesome.bears.example, 'resource': pandas) </ins> Resources []string `json:"resources"` // providers is a list of transformers to be used for reading and writing the resources to disk. // eg: aesgcm, aescbc, secretbox, identity.

Related discussion:
text: Update the godoc on the encryption config API on how to specify group/resources to be encrypted ref: godoc has the details na No response_ na na <details</details<details</details<details</details<details</details
/sig auth /triage accepted
/assign Cool-Coder174
Thanks for volunteering for this LMK if you need any help with it.
Just for clarification, I just need to add documentation on how to configure encryption on groups/resources?
You can update godoc to mention that custom resources can now be to the list of to be encrypted as well and how to add the of the resource to the list. e.g. from this test
/assign /unassign Release v1.26 deadlines are here so we need to get this completed ASAP.
/milestone v1.26
I apologize for the slow actions. I know time and contributions are valuable. May I request a walk through. This is my first time ever contributing and I really want to learn. I have been reading some of the code base on my spare time.
this feature is on the harder side of things to get started with. Reading the blog post and the associated documentation is probably the best starting point:
& what is the status on this issue? 1.26 is coming up! cc
/milestone v1.27
Code change:
return err } <del> // Record some output. outputSetLock.Lock() defer outputSetLock.Unlock() outputSet.Insert(key) </del> <ins> // Report this deletion. deletionCounter <- key </ins> } return nil },

Related discussion:
text: From shippable, I noticed there are several failed cases, for example, E0413 17:34:38. ] watch of *api.Pod ended with: Stop requested --- FAIL: Example (0.61s) got: want: a-hello b-controller c-framework FAIL coverage: 85.9% of statements FAIL 7.652s cc/
another example:
ah-- this is a different thing. I will fix this, too.
I'm seeing this from an unrelated change
Yeah I think that's the one I expected to see.