jiebi/CodeConvo huggingface dataset:
kubeadm-kinder-latest ci-kubernetes-e2e-kubeadm-kinder-latest.Overall E2EKubeadm suite.[sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role 2021-11-09 23:32 CET () E2EKubeadm suite: [sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role This issue seems to be linked to FYI: /cc /sig cluster-lifecycle /area kubeadm
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fix PR is here, needs LGTM:
Also failing on Thanks !
kubeadm-kinder-latest-on-1-22 E2EKubeadm suite.[sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role ci-kubernetes-e2e-kubeadm-kinder-latest-on-1-22.Overall task-06-e2e-kubeadm 2021-11-09 23:31 CET : E2EKubeadm suite: [sig-cluster-lifecycle] [area-kubeadm] proxy addon kube-proxy ServiceAccount should be bound to the system:node-proxier cluster role No response /sig cluster-lifecycle
/assign
/triage accepted will have a look and send a fix if it's kubeadm related.
/area kubeadm
fix PR is here, needs LGTM:
/sig api-machinery /kind bug /cc this makes any updates on a "Local" apiservice failing by kubectl (unless you disable the client-side validation). a quick repro: (edit whatever a local apiservice) a label "foo: bar" will fail on the schema validation # "" was not valid: # * : Invalid value: "The edited file failed validation": ValidationError(): missing required field "service" in client-side v1.15.0 server-side v1.15.2 referencing the published openapi v2 for apiservice#spec below:
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can on this if no one else is currently working on it.
sure, feel free to tag me to request reviewing..
awesome, I will /assign
are you still working on this one?
In the function, there is a that always runs and closes the channel when returning. This means that instead of running periodically, the image garbage collection will be terminated immediately when the function returns. The bug was introduced by in 1.11 /cc
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
BTW, when the threshold is set to 100% (the use case that the PR originally wanted to fix), the code'd try sending to a closed channel and triggers a panic.
i will check it soon, will submit one patch immediately, thanks
does this bug applies to v1.11.2 ? or v.1.11.2 includes the fix?
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
The test failed because it read nothing back from the key for the "member with index 2". Looking at , I notice that it's writing to the "member with index 0" and expecting to read it back from the "member with index 2". I'm curious, is each index supposed to represent a separate redis instance? What is the consistency model here?
/cc
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
It's master slave redis, you should be able to create the same petset with this: , fyi
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
https://k8s- Failed: [] PetSet [Slow] [Feature:PetSet] [] Deploy clustered applications [Slow] [Feature:PetSet] should creating a working redis cluster [Feature:PetSet] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This issue proposes the writing of a new test for the API endpoints , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining ConfigMap endpoints which are untested. Create a ConfigMap with a static label Patch the ConfigMap with a new Label and updated data Get the ConfigMap to ensure it's patched List all ConfigMaps in all Namespaces find the ConfigMap(1) ensure that the ConfigMap is found and is patched Delete Namespaced ConfigMap(1) via a Collection with a LabelSelector This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents: Display endpoints hit by the test: Show the change in coverage after the mock test: If a test with these calls gets merged, Conformance coverage will go up by 2 points /sig testing /sig architecture /area conformance
Bad to list all secrets without a label selector... look at label usage in secret test. Approved for work on 14th January Meeting.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/retitle Write CoreV1ConfigMap test+promote - +2 Coverage
please close this Issue in favour of and place in In Progress Issues on org dashboard
/close Close this Issue in favour of and place in In Progress Issues on org dashboard
Closing this issue. <detailsIn response to : /close Close this Issue in favour of and place in In Progress Issues on org dashboard Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close close this Issue in favor of and place in In Progress Issues on org dashboard
We ran into an issue with an old version of k8s 1.18 that we were running where subnet for cluster IP is given as . The control plane comes up just fine but then when you try to create a k8s service, it fails due to a broken behavior with the IP Allocator Since this is not really a supported version of the k8s, I went around digging up a bit more to see what is the regular behavior and found this to be very inconsistent across a few release branches. bash go test -timeout 30m -run ^TestAllocateNextFC$ -v === RUN TestAllocateNextFC ? --- PASS: TestAllocateNextFC (0.00s) PASS ok 0.566s go func TestAllocateNextFC(t *testing.T) { , cidr, err := netutils.ParseCIDRSloppy("fc::/112") if err != nil { t.Fatal(err) } // Need to be replaced with the suitable handler for that version of the branch r, err := newTestAllocator(cidr) if err != nil { t.Fatal(err) } defer r.Destroy() ip, err := r.AllocateNext() if err != nil { t.Fatalf("%s", err.Error()) } fmt.Println(ip.String()) } Add the above Test case under the IP allocator test case No response_ <detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details<detailsNA </details
/sig network /assign /kind bug
Thanks for helping me with this issue.
/triage accepted /assign
Unit tests on our CI have apparently been failing for 2 weeks and no one noticed (oops!)... The PR that seems to have caused the failures is . The failing test is - . Full Test Log: I'm not sure why this is failing on our CI and not on shippable. Has anyone else seen something similar? That test only seems to fail on the second run through with . Our unit tests run in a container built specifically for testing k8s-mesos: The Dockerfile for the env: Is anyone else able to repro with the above env or their own?
/cc
that test was around a week ago, it's doesn't seem possible to me that it was failing since 2 weeks ago. Anyway, we have few duplications of this issue: this one And the PR where is working on: It's a little bit difficult to me to follow all the conversations in all the differente places about the same bug, could it be possible to unificate some? Since you provided the environment I will try and take a look to reproduce it locally. Thanks!
I think I've got a working patch in
Closing in favor of &
reports the following failure after syncing with latest code: --- FAIL: TestSetUpAtInternal (0.00s) Error Trace: Error: Not equal: "expected-to-be-set-properly" (expected) != "" (actual) FAIL FAIL 0.067s cc/
CC
Taking a look. What do you mean with failed sometimes? Is it a random failure?
Can you please provide more information. This test seems to be consistently green in our CI runs.
The test is failing on this assertion: but as you can see on the previous line I am checking that there was no error on that call. So the function is returning for the error checking but it's not setting . Only place where it can happen is here: Is the not been clean between runs?
When I said "the only place where it can happen" it was because the other s are being returned after a check. I think that there is no other way to get there.
PS: I run the test suite 100 times and I didn't get any error: (That branch is based on the branch)
can you test to see if the patch in fixes things for you?
See for reproducible test environment and failure logs.
Based on the solution in it sounds like this test creates files the first time through which causes the tests to fail on the second time through.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
There are 3 different issues in this bug: high resource consumption (or too strict limits on our side) - should fix it to remove namespace: [I looked into it, and it was because events etcd didn't even come up in this test. The problem here is that we are not collecting logs from etcd in kubemark - I will send a PR fixing it later today hopefully. (that I didn't have time to look into yet).
So actually, the non-started events etcd was the pulling image problem: So we may want to retry in case of failures also.
The first one should be fixed by The second one will be fixed by The third problem is actually related to pod-nanny, which is frequently sending list requests to apiserver. It should be rewritten to use watch instead.
Since the main issue is fixed, I'm decreasing the priority.
For the third problem, I've just sent out: for review. Once that one is merged, we should build the new image and bump tags in our config files.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
So the remaining things here to do are: kubernetes/contrib + update version of pod-nanny in heapster manifests tun kubemark-5 resource constraints
What we should also do, is to add "kubemark-scale" to list of non-blocking suites of merge-queue, so that bugs will be automatically filed for that suite.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
There are three remaining things to do here: tweak limits for kubemark-5 merge add kubemark-scale to non-blocking suites of merge-queue can you please take care of the first one? I will take care of the other two.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
It seems that this PR broke scalability: I'm reverting it and will try to debug it offline.
https://k8s- Failed: [] Density [Feature:Performance] should allow starting 30 pods per node {Kubernetes e2e suite}
All the scalability failures from last day are because of - that PR was reverted, and I fixed the problem in and built a new image. (I did some 100-node cluster tests locally and it worked fine after the change).
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Followup to In 1.15, we can merge this commit: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: https://k8s- What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig testing /area build-release
A lot of tests in are very flaky. From and , and are the top 3 flaky tests among all kubernetes tests. /assign
fix double close channel panic related to https://k8s-
Is this issue ready to be closed? Did that PR fix the reason for flakes in all these tests?
This is still a problem, these tests are still the flakiest https://k8s-
My fix is double close channel panic. This one is not fixed. May be I shoud edit the title.
we also need a new issue then, I'm a bit overloaded ATM could one of open a new one with the garbagecollector failures?
OK
I'll try to debug some of these.
I created a new issue
Thanks!
In our custom plugin, we want to move pods that is unscheduled by the plugin to activeQ/backoffQ when a new Pod is created. So, we define like this: But, even when a new Pod is created, the Pod that was unscheduled by our custom plugin isn't moved to backoffQ/activeQ. when a new Pod is created, the Pod unscheduled by our custom plugin is moved to backoffQ/activeQ. a custom plugin that has pod events in EventsToRegister. a new Pod that the custom plugin will reject. another new Pod that makes the Pod created in (1) schedulable. can see the Pod created in (1) isn't scheduled immediately (it should've been scheduled immediately after (3) because (3) should move the Pod created in (1) to activeQ/backoffQ.) If I understand correctly and if it's not expected behavior, this bug has a non-small impact, especially in v1.24 because we change to 5 min. v1.23.4 but I think latest master has the same issue. <details</details<details</details<details</details<details</details<details</details
/assign /sig scheduling
Maybe it's only on a newly created unscheduled Pod. We've moved correctly for events related to assigned Pod.
And deleting Node events has the same issue...? I guess we need to run when some Nodes are deleted. In in-tree plugins, only pod topology spread uses the deleting Node event and I guess this bug affects its performance. (Even when some Nodes are deleted, the Pods, that were unscheduled by Pod topology spread, aren't moved to activeQ/backoffQ by that deleting events.)
Hm, it may be a problem that events related to Node and Pod are not always called when an event occurs. (Even if it's intended behavior, at least we need to write this on the comment of .) /retitle scheduler: events in related to Pod and Node don't always work
To summarize: The problem is "not all events related to Pod or Node don't always make move from unschedulableQ to active/backoffQ and it may not be intended for custom-plugin developers because this behavior isn't even documented." I think we should move all unschedulable Pods with all Pod and Node events.
If concerned about performance degradation, we can extend or , for example, to have the function like ? This problem has a non-small impact, especially in v1.24+ because we change PodMaxUnschedulableQDuration to 5 min. Let's see the details... ↓ We move the unscheduled Pods (rejected by the plugin that has Pod add/update events in ) in or But, the target Pods against this moving are only unschedulable pods which have any affinity term that matches "pod". So,,,, users cannot move from unschedulableQ to activeQ/backoffQ with all assignedPod events? For example, for Pod Topology Spread (it uses Pod All event in ), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of whether that unschedulable Pods have any affinity term that matches a newly created assigned pod or not. Of course this is not just a problem for Pod Topology Spread, it could be a problem for user's custom-plugins. I guess we should pass all unscheduled Pods to . Or at least we should document this behavior. No issue on this. We don't move the unschedulable Pods with those events at all. That's a problem... Even when a plugin uses Pod add/update/delete event in and wants to move the rejected Pods with non-scheduled Pod's add/update/delete events, scheduling queue doesn't move the unschedulable Pods with those events. (In our company, we have the custom plugin for and the plugin wants the scheduler queye to move the Pods rejected by the plugin from unschedulableQ to active/backoffQ when a new non-scheduled Pod is created.) We move only unscheduled Pods that pass . checks the Node can run that unscheduled Pod from the points of view of some in-tree filter plugins like noderesources/nodeport/nodeAffinity/nodename. The problem that arises here is similar to "Assigned Pod's add/update events". For example, for Pod Topology Spread (it uses Node Add event in ), the new Node creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of the results of against that Node. (In , we change the skew calculation of Pod Topology Spread to exclude Nodes that don't match the node affinity.
But, after is implemented, the calculation can be configured to include those Nodes.) We don't move the unschedulable Pods with Node delete events... That's a problem. Even when the plugin use Node delete event in , scheduling queue doesn't move the unschedulable Pods with any Node delete events.)
/cc Could you please take a look
These are all intended optimizations. These basic checks prevent the scheduler from retrying unschedulable pods over and and over. So far we haven't had the need to make these checks configurable as they check well established scheduling features. +1 on documenting the behavior, but I'm not sure about changing it or making it configurable (that adds complexity that might not be justified). What is it that you are trying to do?
What is it that you are trying to do? We are trying to remove flushing in queue in , right? Without improving like I suggest, some Pods in unschedulableQ cannot move back to activeQ forever in some cases if we remove flushing. (Actually, our company's custom plugin needs to wait flush because non-assigned Pod's events don't let scheduling queue move unschedulable Pods rejected by that plugin.) So, I think we should simply inform all events to queue and let queue move unschedulable Pods with that events. This can happen in some custom plugins like our company's one and also even in in-tree plugins. For example, Pod Topology Spread (after is implemented). When a user uses and , the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result.
cc who implemented NodeInclusionPolicy. Can you provide a concrete example? The policy just refers to the topology spread calculations. It doesn't mean that the node doesn't have to match the nodeaffinity to be able to accept the pod. I'm asking what is your custom plugin trying to do. Depending on whether it makes sense, we need to evaluate if it's worth the complexity.
For example, for Pod Topology Spread (it uses Pod All event in EventsToRegister), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable This is actually a problem here, 5 mins would be too long for background flush. But it doesn't mean that we should simply shorten the interval time.