jiebi/CodeConvo huggingface dataset:
Change:
Logf("Waiting up to %v for all pods (need at least %d) in namespace '%s' to be running and ready", timeout, minPods, ns) if wait.Poll(poll, timeout, func() (bool, error) { <del> // We get the new list of pods in every iteration beause more pods come // online during startup and we want to ensure they are also checked. </del> <ins> // We get the new list of pods and replication controllers in every // iteration because more pods come online during startup and we want to // ensure they are also checked. rcList, err := c.ReplicationControllers(ns).List(labels.Everything()) if err != nil { Logf("Error getting replication controllers in namespace '%s': %v", ns, err) return false, nil } replicas := 0 for _, rc := range rcList.Items { replicas += rc.Spec.Replicas } </ins> podList, err := c.Pods(ns).List(labels.Everything(), fields.Everything()) if err != nil { Logf("Error getting pods in namespace '%s': %v", ns, err) return false, nil } <del> nOk, badPods := 0, []api.Pod{} </del> <ins> nOk, replicaOk, badPods := 0, 0, []api.Pod{} </ins> for _, pod := range podList.Items { if res, err := podRunningReady(&pod); res && err == nil { nOk++ <ins> if hasReplicationControllersForPod(rcList, pod) { replicaOk++ } </ins> } else { <del> badPods = append(badPods, pod) </del> <ins> if pod.Status.Phase != api.PodFailed { Logf("The status of Pod %s is %s, waiting for it to be either Running or Failed", pod.ObjectMeta.Name, pod.Status.Phase) badPods = append(badPods, pod) } else if !hasReplicationControllersForPod(rcList, pod) { Logf("Pod %s is Failed, but it's not controlled by a ReplicationController", pod.ObjectMeta.Name) badPods = append(badPods, pod) } //ignore failed pods that are controlled by a replication controller </ins> } } Logf("%d / %d pods in namespace '%s' are running and ready (%d seconds elapsed)", nOk, len(podList.Items), ns, int(time.Since(start).Seconds())) <del> if nOk == len(podList.Items) && nOk >= minPods { </del> <ins> Logf("expected %d pod replicas in namespace '%s', %d are Running and Ready.", replicas, ns, replicaOk) if replicaOk == replicas && nOk >= minPods && len(badPods) == 0 { </ins> return true, nil } logPodStates(badPods)

Discussion:
On a healthy test cluster the health check looks like this: On a failed cluster it looks like this:
Possibly related to ?
this has elasticsearch logging failing.
As per discussion with I think that the solution here is to ignore failed pods for these test cluster health checks. The failed pods were all correctly restarted by the relevant replication controllers, so the tests should have been able to proceed in this and all the other similar cases that we looked at. Why the pods failed is a separate question, which will hopefully be detected and fixed elsewhere.
It might be due to
Just waiting for oncall to merge
Change:
test/e2e/common/init_container.go: "should invoke init containers on a RestartAlways pod" test/e2e/common/init_container.go: "should not start app containers if init containers fail on a RestartAlways pod" test/e2e/common/init_container.go: "should not start app containers and fail the pod if init containers fail on a RestartNever pod" <ins> test/e2e/common/kubelet.go: "should print the output to logs" test/e2e/common/kubelet.go: "should have an terminated reason" test/e2e/common/kubelet.go: "should be possible to delete" test/e2e/common/kubelet.go: "should write entries to /etc/hosts" test/e2e/common/kubelet.go: "should not write to root filesystem" </ins> test/e2e/common/kubelet_etc_hosts.go: "should test kubelet managed /etc/hosts file" test/e2e/common/lifecycle_hook.go: "should execute poststart exec hook properly" test/e2e/common/lifecycle_hook.go: "should execute prestop exec hook properly"

Discussion:
What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. To my eyes they may meet the criteria for conformance. There's definitely nothing about them that is node specific. They should be running as part of our regular CI, and they should probably be running as presubmits. Porting them to test/e2e/common would cause this to happen. /kind cleanup /sig node /area test /assign
GitHub didn't allow me to assign the following users: mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. /sig node /area test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Change:
"context" "fmt" "io" <ins> "sort" </ins> "github.com/spf13/cobra" authenticationv1alpha1 "k8s.io/api/authentication/v1alpha1"

Discussion:
Unit tests failed with , see TestWhoAmIRun/successtest None No response No response_ /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I think it was caused by
When we loop the golang Map, it's unordered.
I can work on this one.
Thanks.
/priority critical-urgent this is impacting CI,
Change:
// on its parent directory. func (prober *flexVolumeProber) addWatchRecursive(filename string) error { addWatch := func(path string, info os.FileInfo, err error) error { <del> if info.IsDir() { </del> <ins> if err == nil && info.IsDir() { </ins> if err := prober.watcher.AddWatch(path); err != nil { glog.Errorf("Error recursively adding watch: %v", err) }

Discussion:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a new file is introduced in the FlexVolume driver directory then quickly deleted, a race condition would occur causing segmentation violation and crashing kubelet. What you expected to happen: No crashes. How to reproduce it (as minimally and precisely as possible): It is a race condition. A file needs to be created then deleted many times until the kubelet crashes. Anything else we need to know?: Patch: crash log: Environment: Kubernetes version (use ): 1.8.7 - applies to all Cloud provider or hardware configuration: Azure ACS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig storage
the proposed solution looks good to me, can you please submit a pull request?
can you submit this patch if does not get to it.
ACK
I'm on it
Associated PR approved:
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Change:
batchv1 "k8s.io/api/batch/v1" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/kubernetes/test/e2e/framework" )

Discussion:
Which jobs are flaking: pull-kubernetes-e2e-kind Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig apps
Failure at
Hi I looked into the failures and think it was because the test required the job succeeds 3 times with pseudorandom 50% failure chance within 15 minutes, while there is an exponential back-off delay (10s, 20s, 40s …) capped at 6 minutes before recreating failed pods. As 7 consecutive failures (1/128 chance) could exceed the timeout, the test failed intermittently because of "timed out waiting for the condition". For example, in the failure of , the job pods succeeded and failed in the following orders: After the last failure at , the delay has increased to 640s so no pod was created any more before the test failed at . I think it can be fixed by using a deterministic approach to make the job pods succeed and fail alternately, and its executing time can be reduced to tens of seconds stably. Assigning myself. /assign
Change:
} func TestProxyUpdateDelete(t *testing.T) { <del> port, err := echoServer(t, "127.0.0.1:0") if err != nil { t.Fatal(err) } </del> lb := NewLoadBalancerRR() lb.OnUpdate([]api.Endpoints{{JSONBase: api.JSONBase{ID: "echo"}, Endpoints: []string{net.JoinHostPort("127.0.0.1", port)}}})

Discussion:
https://travis-
Looking.
Another instance, or a different thing? https://travis-
Change:
ts := httptest.NewServer(WithPanicRecovery(handler, resolver)) defer ts.Close() <del> _, err = http.Get(ts.URL) </del> <ins> _, err := http.Get(ts.URL) </ins> if err == nil { t.Fatal("expected to receive an error") } <del> capturedOutput := readStdErr() </del> <ins> klog.Flush() klog.SetOutput(&bytes.Buffer{}) // prevent further writes into buf capturedOutput := buf.String() </ins> // We don't expect stack trace from the panic to be included in the log. if isStackTraceLoggedByRuntime(capturedOutput) {

Discussion:
/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?
Change:
) func TestGetLoadBalancer(t *testing.T) { <del> t.Parallel() </del> vals := DefaultTestClusterValues() gce, err := fakeGCECloud(vals) require.NoError(t, err)

Discussion:
Since merged on Friday night, my PR () has failed several retests of pull-kubernetes-bazel-test. My PR is unrelated to the load balancer so I suspect that introduced a flake. I see several runs of that test against master that aren't failing, so clearly the test works sometimes (and the original PR passed it several times to before it merged). Perhaps the test harness sometimes runs problem tests in parallel and other times it doesn't. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: My PR had one good run of pull-kubernetes-bazel-test before merged, but has failed several times since with no code changes to my PR. What you expected to happen: The tests should consistently pass unless related problem code is introduced.
/cc /cc /sig gcp /kind flake
My PR started failing recently too. See https://k8s-
Hey, thanks for identifying the issue. contains tests for - but the race condition was caused when tests with modified a singleton variable in a mock. The short-term fix is removing the calls, but the long-term fix is to not use a singleton. Fix is .
Change:
} // Complete completes all the required options. <del> func (o *Options) Complete() error { </del> <ins> func (o *Options) Complete(fs *pflag.FlagSet) error { </ins> if len(o.ConfigFile) == 0 && len(o.WriteConfigTo) == 0 { o.config.HealthzBindAddress = addressFromDeprecatedFlags(o.config.HealthzBindAddress, o.healthzPort) o.config.MetricsBindAddress = addressFromDeprecatedFlags(o.config.MetricsBindAddress, o.metricsPort)

Discussion:
introduced configuration of additional logging aspects (e.g. format) through command line flags and config file. This was merged for Kubernetes 1.28. Now the following invocation doesn't work as expected when the config file doesn't configure logging: It should run with verbosity 4. Instead it runs with verbosity 0, from the config. Run with this config file: No response 28 rc n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is caused by this line: At that point, is 4 (from the command line flag), and then it gets overwritten with == 0. The normal approach would be: set up config with defaults modify config through flags modify config further by parsing a config file into it. Can kube-proxy do this? It has the downside that a config file can only overwrite fields if they are in the file. A config field doesn't get reset to defaults just by parsing the file. /cc
it looks like kube-proxy special-cases the to let a flag override config... can/should it do the same thing for verbosity? are there any other flags like this?
I don't think that's the normal approach... it's more normal to make flags and config file mutually exclusive. That's difficult for things which started as flags and moved into config
/sig network /sig instrumentation /kind bug /kind regression /milestone v1.28
let's go with and warn it and comment as much as possible, I was going with something like but Patrick is working on it so I assign to him the issue
/assign
Hey folks, v1.28 Bug Triage Lead here Just wanted to know if this is already tracked by SIG leads and someone already working actively fixing it and set a priority?
Perhaps I should have said "this is how component-base/logs was meant to be used". Somehow it worked for kubelet.

Here's my preliminary solution: It needs more testing, but I wanted to share it early because I am running out of time today.
it is , I can take over Patrick's work in case is needed
It would be nice if the logs API could distinguish between and , how about make to a pointer? But this change is supposed to be a breaking change, I'm not sure of the scope of the impact. For side, if set and at the same time, should the command flag override the config file or warn it and be discarded? I think it should be overridden.
xref:
Change:
func (dswp *desiredStateOfWorldPopulator) ReprocessPod( podName volumetypes.UniquePodName) { <del> dswp.deleteProcessedPod(podName) </del> <ins> dswp.markPodProcessingFailed(podName) </ins> } func (dswp *desiredStateOfWorldPopulator) HasAddedPods() bool {

Discussion:
It happens during the "delete" phase in the load test. ! In perf-dash we can see that in total there are ~20M requests like that during the load test. ! It started in Sep 2019 (sic!), so probably when we enabled PVs in our load tests. We haven't detected it so far as, strangely, it doesn't seem to be impacting our scalability SLIs. We should investigate this. /sig scalability
I checked and we create 35 PersistentVolumeClaims per 100 nodes. So in case of 5K node test there will be 1750 PVCs in the cluster. This is close to the 2K of GET PVC requests we see.
Let's analyze the last run of 5K node test in more depths - Grafana: ! As I wrote already somewhere here, it happens during the delete phase. We reach the peak of ~2K QPS around 2:30 AM UTC in that run. Let's check the logs (api-server, kubelets) around that time.
Looking at the api-server logs one can easily tell that something is wrong. It's swamped by GET persistentvolumeclaim requests:
Restricting the logs to a single stateful set on a single node (single client ip) - it also doesn't look right. Total such requests from this IP address: So it looks like a single kubelet issued almost 20K GET requests for this PVC within ~75min, which gives us ~4 QPS from a single kubelet...
From the scheduler logs it looks like the statefulset pod for this PVC was bound on the node Let's check the kubelet logs there
Kubelet logs: Bingo, around the time when the issue happens, it's full of the following logs:
Looks like the errors disappeared exactly when the pod was deleted So, what happens here is partially caused by our test as it may happen that we'll first delete PVC and won't delete the corresponding pod for the next O(1h). Nevertheless, the code in the should be fixed so it doesn't cause kubelet to send ~4QPS of GET PVC requests in such case.
/sig node /sig storage
There is a pending discussion about ways to fix it in
Thanks for fixing that! I know how much work was it and I really appreciate!
Change:
} } <ins> // checkLeftoverEndpoints lists all currently existing endpoints and adds their // service to the queue. This will detect endpoints that exist with no // corresponding service; these endpoints need to be deleted. We only need to // do this once on startup, because in steady-state these are detected (but // some stragglers could have been left behind if the endpoint controller // reboots). func (e *EndpointController) checkLeftoverEndpoints() { list, err := e.client.Endpoints(api.NamespaceAll).List(labels.Everything()) if err != nil { glog.Errorf("Unable to list endpoints (%v); orphaned endpoints will not be cleaned up. (They're pretty harmless, but you can restart this component if you want another attempt made.)", err) return } for i := range list.Items { ep := &list.Items[i] key, err := keyFunc(ep) if err != nil { glog.Errorf("Unable to get key for endpoint %#v", ep) continue } e.queue.Add(key) } } </ins> func findDefaultPort(pod *api.Pod, servicePort int, proto api.Protocol) int { for _, container := range pod.Spec.Containers { for _, port := range container.Ports {

Discussion:
So we'll leak them and/or reuse them perhaps inappropriately. I imagine this could be pretty hard on kube-proxy after a while. Discovered while working on . was suggesting implementing a generic delete-when-parent-is-deleted mechanism, so I may not try to fix this 100% with my current work on .
Generic mechanism proposal was here:
I think it should be enough to check if endpoints match services once just after we initialize, right?
Yeah that should be enough to close this. Although the general mechanism would be nice, too.
Change:
return list } <del> // Contains returns a util.StringSet containing all IDs of stored the items. </del> <ins> // ContainedIDs returns a util.StringSet containing all IDs of stored the items. </ins> // This is a snapshot of a moment in time, and one should keep in mind that // other go routines can add or remove items after you call this. <del> func (c *FIFO) Contains() util.StringSet { </del> <ins> func (c *FIFO) ContainedIDs() util.StringSet { </ins> c.lock.RLock() defer c.lock.RUnlock() set := util.StringSet{}

Discussion:
The client/cache interface has a method which is supposed to return a of the IDs in a store. This is a bit misleading and should probably be renamed to 'Contents' or 'ContainedIDs'.
Thoughts
I tend to agree. contains to me should always return a bool and take an input key. Sent from my iPhone
Yeah, I agree, both of the suggested names would be an improvement.
Change:
t.Fatalf("Failed to create Pod %q: %v", pod.Name, err) } <del> // first binding try should fail. err := wait.PollUntilContextTimeout(ctx, 200*time.Millisecond, wait.ForeverTestTimeout, false, testutils.PodSchedulingError(cs, ns, pod.Name)) </del> <ins> // 1. first binding try should fail. // 2. The pod should be enqueued to activeQ/backoffQ without any event. // 3. The pod should be scheduled in the second binding try. // Here, waiting until (3). err := wait.PollUntilContextTimeout(ctx, 200*time.Millisecond, wait.ForeverTestTimeout, false, testutils.PodScheduled(cs, ns, pod.Name)) </ins> if err != nil { <del> t.Fatalf("Expect pod-1 to be rejected by the bind plugin: %v", err) </del> <ins> t.Fatalf("Expect pod-1 to be scheduled by the bind plugin: %v", err) </ins> } <del> // The pod should be enqueued to activeQ/backoffQ without any event. // The pod should be scheduled in the second binding try. err = wait.PollUntilContextTimeout(ctx, 200*time.Millisecond, wait.ForeverTestTimeout, false, testutils.PodScheduled(cs, ns, pod.Name)) if err != nil { </del> <ins> // Make sure the first binding trial was failed, and this pod is scheduled at the second trial. if fakeBind.counter != 1 { </ins> t.Fatalf("Expect pod-1 to be scheduled by the bind plugin in the second binding try: %v", err) } }

Discussion:
pull-kubernetes-integration: scheduler TestRequeueByBindFailure Running with stress reproduces the flake Not sure, seeing a lot of retests on pull-kubernetes-integration Bind logic in scheduler integration tests just changed 3 days ago in , not sure if that's related No response No response /sig scheduling /priority important-soon cc
I doesn't look related to the latest change you were also looking at Bind. Any ideas?
It's a little strange because the error reports but since the bind-plugin will fail for the first calling time and there's no kubelet, that means the pod status will be kept since. I created another PR to help output the error message, which maybe helpful for analysis.
This test is very straightforward: the first try to bind "pod-1", the Pod is rejected by the binding plugin. () scheduling queue puts back this "pod-1" to activeQ/backoffQ directly, and "pod-1" will soon come back to the scheduling cycle. the second try to bind "pod-1", it's successful. And, means that (1) doesn't reject "pod-1" correctly. And I don't have any idea why it can happen. The only scenario I can think of is that there is somehow another Pod created before pod-1. When is executed , is it possible that any other Pods (like system ones) are created automatically?
/triage accepted current top flake of integration
/assign I've got one possible scenario, will have a quick fix.
Change:
replicas := 2 // Create a replication controller for a service <del> // that serves its hostname on port 8080. </del> <ins> // that serves its hostname. </ins> // The source for the Docker containter kubernetes/serve_hostname is // in contrib/for-demos/serve_hostname By(fmt.Sprintf("Creating replication controller %s", name))

Discussion:
Some of the e2e tests specify HostPort in their pod configurations. This can cause issues running those tests if the port is already in use.
We've been removing these over the last week and a half and I think that may be the only one left remaining.
We need to decide if HostPort is a supported use case for Kubernetes -- and if it's supported, then it needs e2e test coverage to ensure it remains functional. Note that supported and not-preferred are orthogonal. My understanding is that it's supported-but-not-preferred and that there are reasonable use cases for using it.