jiebi/CodeConvo huggingface dataset:
Code change:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

Related discussion:
text: import ( "fmt" "net/http" <ins> "reflect" </ins> "sync" "sync/atomic" "time"
Code change:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.

Related discussion:
text: // Injectable for testing. Send the event down the outgoing channel. emit func(watch.Event) <ins> // HighWaterMarks for performance debugging. incomingHWM HighWaterMark outgoingHWM HighWaterMark </ins> cache etcdCache }
Code change:
https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

Related discussion:
text: } // errorBadPodsStates create error message of basic info of bad pods for debugging. <del> func errorBadPodsStates(badPods []api.Pod, desiredPods int, ns string, timeout time.Duration) string { errStr := fmt.Sprintf("%d / %d pods in namespace %q are NOT in the desired state in %vn", len(badPods), desiredPods, ns, timeout) </del> <ins> func errorBadPodsStates(badPods []api.Pod, desiredPods int, ns, desiredState string, timeout time.Duration) string { errStr := fmt.Sprintf("%d / %d pods in namespace %q are NOT in %s state in %vn", len(badPods), desiredPods, ns, desiredState, timeout) </ins> // Pirnt bad pods info only if there are fewer than 10 bad pods if len(badPods) > 10 { return errStr + "There are too many bad pods. Please check log for details."
Code change:
https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

Related discussion:
text: // pods have been created. func WaitForPodsSuccess(c *client.Client, ns string, successPodLabels map[string]string, timeout time.Duration) error { successPodSelector := labels.SelectorFromSet(successPodLabels) <del> start, badPods := time.Now(), []api.Pod{} </del> <ins> start, badPods, desiredPods := time.Now(), []api.Pod{}, 0 </ins> if wait.PollImmediate(30*time.Second, timeout, func() (bool, error) { podList, err := c.Pods(ns).List(api.ListOptions{LabelSelector: successPodSelector})
Code change:
https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

Related discussion:
text: return true, nil } badPods = []api.Pod{} <ins> desiredPods = len(podList.Items) </ins> for _, pod := range podList.Items { if pod.Status.Phase != api.PodSucceeded { badPods = append(badPods, pod)
Code change:
https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

Related discussion:
text: }) != nil { logPodStates(badPods) LogPodsWithLabels(c, ns, successPodLabels) <del> return fmt.Errorf("Not all pods in namespace %q are successful within %v", ns, timeout) </del> <ins> return errors.New(errorBadPodsStates(badPods, desiredPods, ns, "SUCCESS", timeout)) </ins> } return nil }
Code change:
https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:

Related discussion:
text: logPodStates(badPods) return false, nil }) != nil { <del> return errors.New(errorBadPodsStates(badPods, desiredPods, ns, timeout)) </del> <ins> return errors.New(errorBadPodsStates(badPods, desiredPods, ns, "RUNNING and READY", timeout)) </ins> } wg.Wait() if waitForSuccessError != nil {
Code change:
Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:

Related discussion:
text: } // Create the container. <del> fClock.SetTime(time.Now()) </del> <ins> fClock.SetTime(time.Now().Add(-1 * time.Hour)) </ins> *expected.CreatedAt = fClock.Now().Unix() id, err := ds.CreateContainer("sandboxid", config, sConfig) // Set the id manually since we don't know the id until it's created.
Code change:
Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:

Related discussion:
text: assert.Equal(t, expected, status) // Advance the clock and stop the container. <del> fClock.SetTime(time.Now()) </del> <ins> fClock.SetTime(time.Now().Add(1 * time.Hour)) </ins> *expected.FinishedAt = fClock.Now().Unix() *expected.State = runtimeApi.ContainerState_EXITED *expected.Reason = "Completed"
Code change:
Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:

Related discussion:
text: ) func newTestDockerSevice() (*dockerService, *dockertools.FakeDockerClient, *clock.FakeClock) { <del> c := dockertools.NewFakeDockerClient() </del> fakeClock := clock.NewFakeClock(time.Time{}) <ins> c := dockertools.NewFakeDockerClientWithClock(fakeClock) </ins> return &dockerService{client: c}, c, fakeClock }
Code change:
Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.11", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-23T22:53:01Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Environment: AWS - CentOS 7.2 x8664 with cloud-init (HVM) (ami-) OS (e.g. from /etc/os-release): NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" IDLIKE="rhel fedora" VERSIONID="7" PRETTYNAME="CentOS Linux 7 (Core)" ANSICOLOR="0;31" CPENAME="cpe:/o:centos:centos:7" HOMEURL="" BUGREPORTURL="" CENTOSMANTISBTPROJECT="CentOS-7" CENTOSMANTISBTPROJECTVERSION="7" REDHATSUPPORTPRODUCT="centos" REDHATSUPPORTPRODUCTVERSION="7" Linux ip-172-31-35-103 3.10.0-327.el7.x8664 SMP Thu Nov 19 22:10:57 UTC 2015 x8664 x8664 x86_64 GNU/Linux Install tools: Others: What happened: when selinux is enabled i was getting following error on discovery container What you expected to happen: i expected node to be able to join master, but nothing was running on 9898 port of master i.e. discovery due the error mentioned above How to reproduce it (as minimally and precisely as possible): use the api which i gave you and do normal kubeadm init Anything else do we need to know: This issue got fixed with setenforce 0
This issue got fixed with setenforce 0 but want to track it
cc
BTW the issue was on latest Centos iso too downloaded from here
cc
this is to do with secrets, and kubelet log is the place to look for more info, could you please grab and paste it in a gist (it's gonna be fairly large)? Also, the kernel audit log would be handy, do you know how to get this (I forgot)?
not sure if this will help
the lines are incomplete, it's best if you redirect it to a file and upload it somewhere with a link here.
could you also provide the output of ?

Related discussion:
text: // `HostIP: s.API.AdvertiseAddrs[0]`, if there is only one address` {Name: "http", ContainerPort: 9898, HostPort: 9898}, }, <ins> SecurityContext: &api.SecurityContext{ SELinuxOptions: &api.SELinuxOptions{ // TODO: This implies our discovery container is not being restricted by // SELinux. This is not optimal and would be nice to adjust in future // so it can read /tmp/secret, but for now this avoids recommending // setenforce 0 system-wide. Type: "unconfined_t", }, }, </ins> }}, Volumes: []api.Volume{{ Name: kubeDiscoverySecretName,
Code change:
Just saw this warning today: Not sure if this is only limited to kubemark, or if there are other places we need to fix, too.
Sorry if this sounds stupid, but as expected, won't show this warning - what needs to be fixed here?
we have a bunch of scripts and Makefiles which are omitting the currently. We need to fix these by adding the missing s.

Related discussion:
text: docker build -t $(PREFIX)/logs-generator:$(TAG) . push: <del> gcloud docker push $(PREFIX)/logs-generator:$(TAG) </del> <ins> gcloud docker -- push $(PREFIX)/logs-generator:$(TAG) </ins> clean: rm -f logs-generator  No newline at end of file
Code change:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray

Related discussion:
text: RUN mkdir -p ${K8S_PATCHED_GOROOT}  && curl -sSL https://github.com/golang/go/archive/go${K8S_PATCHED_GOLANG_VERSION}.tar.gz | tar -xz -C ${K8S_PATCHED_GOROOT} --strip-components=1 <ins> # We need a patched go1.7.1 for linux/arm (https://github.com/kubernetes/kubernetes/issues/29904) # We need go1.7.1 for all darwin builds (https://github.com/kubernetes/kubernetes/issues/32999) </ins> COPY golang-patches/CL28857-go1.7.1-luxas.patch ${K8S_PATCHED_GOROOT}/ RUN cd ${K8S_PATCHED_GOROOT}  && patch -p1 < CL28857-go1.7.1-luxas.patch  && cd src  && GOROOT_FINAL=${K8S_PATCHED_GOROOT} GOROOT_BOOTSTRAP=/usr/local/go ./make.bash  <del> && for platform in linux/arm; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done </del> <ins> && for platform in linux/arm darwin/386 darwin/amd64; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done </ins>
Code change:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray

Related discussion:
text: <del> v1.6.3-7 </del> <ins> v1.6.3-8 </ins>
Code change:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray

Related discussion:
text: if [[ ${platform} == "linux/arm" ]]; then export CGO_ENABLED=1 export CC=arm-linux-gnueabi-gcc <ins> # See https://github.com/kubernetes/kubernetes/issues/29904 </ins> export GOROOT=${K8S_PATCHED_GOROOT} elif [[ ${platform} == "linux/arm64" ]]; then export CGO_ENABLED=1
Code change:
See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray

Related discussion:
text: elif [[ ${platform} == "linux/ppc64le" ]]; then export CGO_ENABLED=1 export CC=powerpc64le-linux-gnu-gcc <ins> elif [[ ${platform} == "darwin/"* ]]; then # See https://github.com/kubernetes/kubernetes/issues/32999 export GOROOT=${K8S_PATCHED_GOROOT} </ins> fi fi }
Code change:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Master Environment: Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: What happened: now can update a LB instead of deleting it and recreating it. However the security group code is still set to try and create security group rules, and if it gets an error, delete the LB and start over. What you expected to happen: Updating a service does not delete the LBaaS LB How to reproduce it (as minimally and precisely as possible): Enable in the section of Restart a Anything else do we need to know:
you're absolutely right, this is a critical issue (if you enable security group management). The dangerousness of aggressive cleanups has already been discussed in detail in this thread: I hope you guys can fix this.
Yup - I have a larger refactor, that ensures this, and a few other edge cases we have found in our development. It is in massive need of a rebase though, so I will need a day or 2 of uninterrupted time to get it together
Are you still working on this? If not, I can take a look.
ping :)
/close
We just ran into this where Kubernetes recreated an LB with a new Floating IP because the OpenStack API was not available for a few seconds while the LB was updated. Could this issue be re-opened? Since this is a major bug for us, we will start working on a fix for this today.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Related discussion:
text: if lbaas.opts.ManageSecurityGroups { err := lbaas.ensureSecurityGroup(clusterName, apiService, nodes, loadbalancer) if err != nil { <del> // cleanup what was created so far _ = lbaas.EnsureLoadBalancerDeleted(ctx, clusterName, apiService) return status, err </del> <ins> return status, fmt.Errorf("Error reconciling security groups for LB service %v/%v: %v", apiService.Namespace, apiService.Name, err) </ins> } }
Code change:
Kubernetes version (use ): HEAD Environment: GCE What happened: When I set up I can see the following message: What you expected to happen: master should not be registered
cc
Looks like the env var isn't being respected any longer by the master scripts (but the logic for # of expected nodes did adjust correctly).
/cc I think this may have broken in because it which unconditionally sets the field that tells a kubelet to register with the master outside of the guard that checks .
Fix in progress: . There was a problem caused by unconditional setting on the master in .

Related discussion:
text: create-static-ip "${MASTER_NAME}-ip" "${REGION}" MASTER_RESERVED_IP=$(gcloud compute addresses describe "${MASTER_NAME}-ip"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') <del> KUBELET_APISERVER="${MASTER_RESERVED_IP}" </del> <ins> if [[ "${REGISTER_MASTER_KUBELET:-}" == "true" ]]; then KUBELET_APISERVER="${MASTER_RESERVED_IP}" fi </ins> KUBERNETES_MASTER_NAME="${MASTER_RESERVED_IP}" create-certs "${MASTER_RESERVED_IP}"
Code change:
From . During the firewall e2e test we need to change the service type from to to avoid leaking LB resources. But found it is not working because the field. Error I got as below: If don't delete this field, got instead another error:
Just saw the TODO comment . Working on it.

Related discussion:
text: } // TODO(freehan): allow user to update loadbalancerSourceRanges <del> allErrs = append(allErrs, ValidateImmutableField(service.Spec.LoadBalancerSourceRanges, oldService.Spec.LoadBalancerSourceRanges, field.NewPath("spec", "loadBalancerSourceRanges"))...) </del> <ins> // Only allow removing LoadBalancerSourceRanges when change service type from LoadBalancer // to non-LoadBalancer or adding LoadBalancerSourceRanges when change service type from // non-LoadBalancer to LoadBalancer. if service.Spec.Type != api.ServiceTypeLoadBalancer && oldService.Spec.Type != api.ServiceTypeLoadBalancer || service.Spec.Type == api.ServiceTypeLoadBalancer && oldService.Spec.Type == api.ServiceTypeLoadBalancer { allErrs = append(allErrs, ValidateImmutableField(service.Spec.LoadBalancerSourceRanges, oldService.Spec.LoadBalancerSourceRanges, field.NewPath("spec", "loadBalancerSourceRanges"))...) } </ins> allErrs = append(allErrs, validateServiceFields(service)...) allErrs = append(allErrs, validateServiceAnnotations(service, oldService)...)
Code change:
The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc

Related discussion:
text: return errStr + buf.String() } <del> // check if a Pod is controlled by a Replication Controller in the List func hasReplicationControllersForPod(rcs *api.ReplicationControllerList, pod api.Pod) bool { for _, rc := range rcs.Items { selector := labels.SelectorFromSet(rc.Spec.Selector) if selector.Matches(labels.Set(pod.ObjectMeta.Labels)) { return true } } return false } </del> // WaitForPodsSuccess waits till all labels matching the given selector enter // the Success state. The caller is expected to only invoke this method once the // pods have been created.
Code change:
The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc

Related discussion:
text: // WaitForPodsRunningReady waits up to timeout to ensure that all pods in // namespace ns are either running and ready, or failed but controlled by a <del> // replication controller. Also, it ensures that at least minPods are running // and ready. It has separate behavior from other 'wait for' pods functions in // that it requires the list of pods on every iteration. This is useful, for </del> <ins> // controller. Also, it ensures that at least minPods are running and // ready. It has separate behavior from other 'wait for' pods functions in // that it requests the list of pods on every iteration. This is useful, for </ins> // example, in cluster startup, because the number of pods increases while // waiting. // If ignoreLabels is not empty, pods matching this selector are ignored and