jiebi/CodeConvo huggingface dataset:
Code change:
return nil } <del> // DeleteTCPLoadBalancer is an implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. func (gce *GCECloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> // EnsureTCPLoadBalancerDeleted is an implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. func (gce *GCECloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> op, err := gce.service.ForwardingRules.Delete(gce.projectID, region, name).Do() if err != nil && isHTTPErrorCode(err, http.StatusNotFound) { glog.Infof("Forwarding rule %s already deleted. Continuing to delete target pool.", name)

Related discussion:
text: The target pool leak in the code might be fixed, but the forwarding rule + target pool leak from deleting GKE clusters too quickly is certainly still happening. Reopening for the sake of that half of the issue.
Can we just add a short sleep to jenkins after the last test has run before it deletes anything?
Yes, you could add a sleep to Ginkgo's AfterEachSuite(): But I'd be concerned about what the upper bound on the sleep period is. Can we get the tests and code in sync with a 1 minute latency here? (e.g. if the controller garbage collects every 60 seconds, then the test only needs to sleep for 60 sec?) And add a BEEG comment to the test's sleep config noting that increasing the sleep time will not improve things, otherwise I'm pretty sure that someone noting flakes in this test in future will think "maybe I should just increase the sleep period a little bit."
Actually, thinking about this a little more, it would be better to put the "sleep" logic in the k8s master teardown procedure (i.e. not in the e2e tests or in Jenkins), because otherwise users are going to run into the same resource leakage problems as we are. So master deletion blocks on cleanup of backend cloud resources.
There's no way to block master deletion from the master itself. The test framework (or user) just pulls the rug out from underneath the master. See
Robby's right -- is the real solution to this, but until we have that the best we can do is to have the wait happen elsewhere. The options seem to be: GKE itself Ginkgo's post-testing logic Jenkins' teardown logic I don't like putting it in GKE, because arbitrarily slowing down our API makes for a poor experience for the majority of people that don't still have resources being deleted. I'm not eager to put it in Gingko either, since won't that sleep also take effect whenever anyone runs an e2e test locally against a dev cluster? Putting it in the Jenkins logic makes the most sense to me. I'll send a PR and cc you.
We may also want to change the kube-down logic to bring down the master and then nodes in parallel to both speed it up and more closely resemble GKE's teardown logic.
I've deleted the old forwarding rules and target pools from the project.
Code change:
return nil } <del> // DeleteTCPLoadBalancer is an implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. func (gce *GCECloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> // EnsureTCPLoadBalancerDeleted is an implementation of TCPLoadBalancer.EnsureTCPLoadBalancerDeleted. func (gce *GCECloud) EnsureTCPLoadBalancerDeleted(name, region string) error { </ins> op, err := gce.service.ForwardingRules.Delete(gce.projectID, region, name).Do() if err != nil && isHTTPErrorCode(err, http.StatusNotFound) { glog.Infof("Forwarding rule %s already deleted. Continuing to delete target pool.", name)

Related discussion:
text: I'll periodically check to see whether anything is leaked after this point. I apologize if the deletions affected any running tests - everything looked to only be from GKE and parallel e2e runs.
Code change:
package events <del> import ( "k8s.io/api/core/v1" clientv1 "k8s.io/api/core/v1" ) </del> const ( // Container event reason list CreatedContainer        = "Created"

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub
Code change:
gomega.Expect(stdout).To(gomega.Equal(mountName), msg) } else { // We *expect* cat to return error here <del> gomega.Expect(err).To(gomega.HaveOccurred(), msg) </del> <ins> framework.ExpectError(err, msg) </ins> } } }

Related discussion:
text: What happened: After fixing golint failures under (now still in progress), we need to put for each and . Then common error checking code has become like: We can replace this code with which is more readable and easy to be understood what the code does. Now we have following e2e test directories which can be replaced: [x] test/e2e/apimachinery/ () [x] test/e2e/apps/ () [x] test/e2e/auth/ () [x] test/e2e/common/ () [x] test/e2e/node/ () [x] test/e2e/storage/ () [x] () [x] () [x] () [x] () [x] () [x] () [x] [x] test/e2e/storage/persistentvolumes- [x] () [x] () [x] [x] [x] [x] [x] ()
/sig testing /kind cleanup
/assign PR will be submitted soon.
Fix the following in
Fix in
Fix in
Hi, maybe I could take files beneath test/e2e/storage/vsphere/
oops, I'm finished and prepare for PR.
Thank you That's OK
Fix the following in
/assign
All related PRs were merged. I think this ticket can be closed. BTW, should we also replace to ?
Thanks for your work. thing is managed with We are almost finishing the work also :-)
/remove-kind bug
I found in these files. I'm working on :
I can work to the following item.
Code change:
vmPowerStateStopped      = "stopped" vmPowerStateDeallocated  = "deallocated" vmPowerStateDeallocating = "deallocating" <ins> // nodeNameEnvironmentName is the environment variable name for getting node name. // It is only used for out-of-tree cloud provider. nodeNameEnvironmentName = "NODE_NAME" </ins> ) var (

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If computer name prefix does not match VMSS name, then Azure cloud provider will use VMSS API instead of using metadata service data. This leads to throttling, because every affected kubelet is making API requests. What you expected to happen: Azure cloud provider should use metadata service data instead of calling VMSS API when instance is current. How to reproduce it (as minimally and precisely as possible): Set computer name prefix of the VMSS to something different than default. You should observe Azure provider making requests to the VMSS API rather than using metadata service, which leads to throttling if you have multiple kubelets with this bug. Anything else we need to know?: This is bug was introduced by - returns false even if instance is current, because it expects node name to match VMSS name + instance id, however node name is derived from the computer-name-prefix which can be different than VMSS name. Node name should be matched against , not against . Environment: Kubernetes version (use ): 1.18.x and higher Cloud provider or hardware configuration: azure /sig cloud-provider
/sig cloud-provider
/area provider/azure
CC
/remove-sig cloud-provider /sig cloud-provider (trying to remove the label)
/sig cloud-provider
Thanks for the reporting. Unfortunately, computer name prefix is not included in IMDS, so it can't be used without VMSS API for constructing the NodeName. Would check what could we do to workaround the issue.
Can you memorize the computer name prefix in the first request to the Azure API and then use it for the isCurrentInstance check?
opened to fix the issue. Would you like to have a look?
I can confirm that your change should fix the problem I reported. Do you know if your fix will be to 1.18.x?
is working on the cherry pick
Thank you for quick fix!
Code change:
"containers": [ { "name": "kubernetes-pause", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> } ], "restartPolicy": "Never",

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the pause-amd64:3.0 image on has an empty schema2 manifest, which prevents is from being pulled by proxy/caches which only support V2 registries. What you expected to happen: Being able to pull ` using a V2 manifest compatible client. Anything else we need to know?: The proxy/cahce is requesting this
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After talking with , he mentionned the following
AFAIK, does not support well. After inspecting the image, you can find is still using manifest v1.
/cc FYI
Im not sure which exact version x schéma X is needed. What I know is that we can download through the proxy/cache all other Kubernetes kube-system images, only the pause image is giving us trouble. Therefore, I would say that having the image available using the same format as the other kube-system images would be the request here.
Yeah, we haven't updated the pause image in a while, since it doesn't really ever change. I'm not sure whether rebuilding + pushing over the existing tag is safe, or if we should bump to or something. thoughts?
If we do rebuilding/updating the image, that would help and a bunch of similar issues.
it's better/save to go with new tag
I didn't realize there were changes to the pause binary that were never actually deployed. It seems like we should probably just do that (and bump the tag).
that sounds great. Any idea when the existing 3.0 tag could be pushed again?
We shouldn't push over the 3.0 tag; we should push a 3.1 or 4.0 tag instead. I'm not sure who owns the pause container who should drive this. Maybe sig-node and/or
It seems like it should be sig-node. I'll look into it.
The new container has been built, can you test with
Its working! thanks! I was able to pull it through our artifactory proxy , Lee Verberne <:
excellent, thanks for checking. /assign
Code change:
"containers": [ { "name": "kubernetes-pause", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> } ], "restartPolicy": "Never",

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: pause does not reap zombie processes using the default pause container provided by What you expected to happen: should include the up to date pause code to reap zombie processes when using docker = 1.13.1 found here How to reproduce it (as minimally and precisely as possible): a.out will leave a zombie process that is not cleaned up Anything else we need to know?: Building the pause container from master solves this issue Related: Environment: Kubernetes version (use ): 1.7.0+ Cloud provider or hardware configuration**: N/A OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13+ Install tools: Others: docker = 1.13.1
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
i get a simliar problem when i run crond service in pod container, every day it makes some [anacron] zombie progresses UID PID PPID C STIME TTY TIME CMD root 1 0 0 Nov14 ? 00:00:00 /pause root 7 0 0 Nov14 ? 00:00:00 /usr/local/bin/dumb-init /usr/lo root 13 7 0 Nov14 ? 00:00:00 nginx: master process /usr/local nobody 21 13 0 Nov14 ? 00:00:01 nginx: worker process nobody 22 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 23 13 0 Nov14 ? 00:00:07 nginx: worker process nobody 24 13 0 Nov14 ? 00:00:05 nginx: worker process root 77 1 0 Nov14 ? 00:00:00 [anacron] root 1 0 00:01 ? 00:00:00 [anacron] <defunctroot 1 0 01:01 ? 00:00:00 [anacron] <defunctroot 1 0 02:01 ?
Code change:
"containers": [ { "name": "kubernetes-pause", <del> "image": "gcr.io/google_containers/pause-amd64:3.0" </del> <ins> "image": "gcr.io/google_containers/pause-amd64:3.1" </ins> } ], "restartPolicy": "Never",

Related discussion:
text: 00:00:00 [anacron] <defunctroot 1 0 03:01 ? 00:00:00 [anacron] <defunctroot 0 0 11:49 ? 00:00:00 /bin/bash root 0 1 19:15 ? 00:00:00 /bin/bash root 0 19:15 ? 00:00:00 ps -ef !
Do we know why this was never resolved / was never deployed?
I just fought through this issue with a rabbitmq helm chart. I had to custom build a pause container from master to fix it. The rabbitmq helm chart uses for its liveness checks. It's a thin wrapper around a call to the rabbitmq-server daemon, which is not PID 1 (/pause is PID 1). The rabbitmq-server process forks some children to do the work and then they exit. Since the pause-amd64:3.0 is not reaping, those processes stick around as zombies. The liveness check runs every 5 seconds. After about 4 hours, there are about 27K zombies and the node begins failing to launch new containers because it's starved for resources. The pause container built from current master fixes that behavior, no zombies at all in the container.
The plan is to build a new version of pause-amd64 and then update the image referenced by Kubernetes. This means the default image will change in the next Kubernetes release. We can make the new image available once the freeze lifts and we can merge .
Looks like this made it into 1.10
Code change:
registry = "k8s.gcr.io/build-image", repository = "debian-iptables", # Ensure the digests above are updated to match a new tag <del> tag = "buster-v1.4.0",  # ignored, but kept here for documentation </del> <ins> tag = "buster-v1.5.0",  # ignored, but kept here for documentation </ins> ) def etcd_tarballs():

Related discussion:
text: The based container images for are shipped with the wrong architecture within the manifest if not amd64: Refers to We updated the images a few minutes ago to buster-v1.3.0, which are now ready to be used as base for the multi-architecture images. If the images contain the right architectures, then it may magically happen that the kube-proxy images are fixed, too. This is because we already use the build syntax for it: /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
Code change:
if err != nil { return probe.Unknown, err } <del> if strings.ToLower(string(data)) != defaultHealthyOutput { </del> <ins> if !strings.HasPrefix(strings.ToLower(string(data)), defaultHealthyOutput) { </ins> return probe.Failure, nil } return probe.Success, nil

Related discussion:
text: As far as I can tell, container liveness probes of type exec flat out don't work against clusters built at head. I had the kubelet always log the output of the exec'ed command, and it's always empty no matter what I exec, causing the probe to fail since it requires the output to be equal to the string "ok". If you want to test it, try adding a liveness probe of this form to any container: I also tried these lovely forms, to more closely match the liveness example under examples/: On the plus side, this may be an opportunistic time to switch over to
The reason that the e2e test for exec liveness probes hasn't caught this is that it only tests that containers are restarted when they fail liveness probes. It doesn't test that containers aren't restarted when their liveness checks should be succeeding. The e2e test's exec'ed commands also only ever produced empty output, rather than the expected "fail" output.
Is it possible to include this in a new pre-release, since it fixes a bug in health checking? I see it merged in master and I'm wondering what the policy is of releasing new code in a separate (0.16.*) release.
Code change:
// Otherwise we will return an error. func findSecurityGroupForInstance(instance *ec2.Instance, taggedSecurityGroups map[string]*ec2.SecurityGroup) (*ec2.GroupIdentifier, error) { instanceID := aws.StringValue(instance.InstanceId) <del> var best *ec2.GroupIdentifier </del> <ins> var tagged []*ec2.GroupIdentifier var untagged []*ec2.GroupIdentifier </ins> for _, group := range instance.SecurityGroups { groupID := aws.StringValue(group.GroupId) if groupID == "" { glog.Warningf("Ignoring security group without id for instance %q: %v", instanceID, group) continue } <del> if best == nil { best = group continue </del> <ins> _, isTagged := taggedSecurityGroups[groupID] if isTagged { tagged = append(tagged, group) } else { untagged = append(untagged, group) </ins> } <ins> } </ins> <del> _, bestIsTagged := taggedSecurityGroups[*best.GroupId] _, groupIsTagged := taggedSecurityGroups[groupID] if bestIsTagged && !groupIsTagged { // best is still best } else if groupIsTagged && !bestIsTagged { best = group } else { // We create instances with one SG // If users create multiple SGs, they must tag one of them as being k8s owned return nil, fmt.Errorf("Multiple security groups found for instance (%s); ensure the k8s security group is tagged", instanceID) </del> <ins> if len(tagged) > 0 { // We create instances with one SG // If users create multiple SGs, they must tag one of them as being k8s owned if len(tagged) != 1 { return nil, fmt.Errorf("Multiple tagged security groups found for instance %s; ensure only the k8s security group is tagged", instanceID) </ins> } <ins> return tagged[0], nil </ins> } <del> if best == nil { glog.Warningf("No security group found for instance %q", instanceID) </del> <ins> if len(untagged) > 0 { // For back-compat, we will allow a single untagged SG if len(untagged) != 1 { return nil, fmt.Errorf("Multiple untagged security groups found for instance %s; ensure the k8s security group is tagged", instanceID) } return untagged[0], nil </ins> } <del> return best, nil </del> <ins> glog.Warningf("No security group found for instance %q", instanceID) return nil, nil </ins> } // Return all the security groups that are tagged as being part of our cluster

Related discussion:
text: reported hitting this error: The instance had 4 security groups, and only one of them was tagged. However, the logic around picking the security group is incorrect.. if we get two non-tagged SGs before we see the first tagged SG, it throws an error. We'll want to cherry-pick this to 1.2
I think this one was a victim of flakiness in the auto-assigner when the author is the first-listed maintainer. Assigning to you, :-)
Code change:
} if latency := time.Since(now); latency > longThrottleLatency { <del> klog.V(4).Infof("Throttling request took %v, request: %s:%s", latency, r.verb, r.URL().String()) </del> <ins> klog.V(3).Infof("Throttling request took %v, request: %s:%s", latency, r.verb, r.URL().String()) </ins> } return err

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : Report request throttling by the kubelet by default (i. e. V=3) Why is this needed: Request throttling can have serious consequences, but it is reported (in ) at V=4, so not by default. While attempting to probe limits of how many pods per node we could run, we hit a situation where the test pods would go Pending and eventually fail. After investigation, it turned out that the pod was being started by the kubelet, but that status update was not provided to the apiserver in a timely manner, resulting in the pod-to-pod network not coming up, so a pod attempting to connect would time out and fail. The delay was root caused to throttling by the kubelet; raising the KubeAPIBurst and KubeAPIQPS limits resolved the issue. The throttling was caused by fetching secrets for each pod, which was in its own namespace. The subjects of pods/node, namespaces/node, and throttling parameters are complex and I do not propose to solve them here (those are scalability issues that have to be approached carefully). However, being able to easily identify that throttling is taking place without having to restart the kubelet at higher verbosity would help administrators identify the possibility of this issue. My test attempted to create 500 pods across 3 nodes (OpenShift allows 250 pods/node). Running at V=5, I captured a histogram of the throttling delay in seconds. There were 7881 throttling events reported total over perhaps a 45 minute period, with a somewhat bimodal (in this case) distribution:
/sig node /sig scalability
Thanks for filing the pr to fix :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Code change:
hash := hash(nodeUpdate.name()) select { case <-stopCh: <ins> tc.nodeUpdateQueue.Done(item) </ins> break case tc.nodeUpdateChannels[hash%workers] <- nodeUpdate: } <ins> tc.nodeUpdateQueue.Done(item) </ins> } }(stopCh)

Related discussion:
text: We're seeing an issue with large memory growth in NoExecuteTaintManager when pods with tolerations are rapidly created/deleted. The item to the workqueue contains the full pod objects received from the informer: That means those pod objects are retained until they are processed. Additionally, it looks like the processing loop never removed the keys from the workqueue when it was done. It also means that multiple updates for the same pod result in multiple items in the workqueue, rather than enqueuing the pod to be processed once, like other controllers do. Simplifying this controller to enqueue pod updates only by name/namespace, and operate on the latest pod obtained from the store (like other controllers do) would speed up the processing and relieve the memory burden of the workqueue. /sig scheduling /cc
/assign
Thanks for finding this Jordan - looking into your fixes.
the memory issue was fixed by /close