jiebi/CodeConvo huggingface dataset:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)
We had rather strange case today when after rolling update one of the kubelet's started wrong container. For this specific replicationcontroller we are using What happened was that after rolling-update when i started to debug the issue i discovered that one of the kubelet's did not pull the image: When looking into what image it actually started i saw: When checking from the actual node i saw this It seems that instead of starting myimage with tag 2208 kubelet started image tagged as 1393 since the docker image id starts with 2208.
TestRunOneVolumeDetachFailNodeWithReadWriteOnce 4 times since 03-19 23:00 PDT, intermittently ! https://k8s- Still investigating whether this is a tolerable flake or not with more history than testgrid view, or otherwise would appreciate input from anyone with expertise in this test, especially as regards to whether this should be a blocker for tomorrows beta cut. maybe: /sig storage /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is already open for this, looks like it is attempting to fix it
thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. /close
Closing this issue. <detailsIn response to : thank you!!! closing this in favor of that one, which I also to our CI signal project board for visibility. For completeness: since it's a longer standing issue, well identified as a flake and not a failure and also actively in progress, NOT considering it a blocker for the go/nogo signal for the beta tomorrow. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
pull-kubernetes-unit TestRunOneVolumeDetachFailNodeWithReadWriteOnce Jan 7th, 2022 Check volume <fake-plugin/volume-nameis reported as attached to node <fail-detach-node, got false, expected true No response_ /sig storage
/cc
this is the top unit test flake according to
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
I think this test is no longer flaky by checking the test grid, close it now.
Unfortunately this does still seem to be flaky: Here was the output from : /reopen
Reopened this issue. <detailsIn response to : Unfortunately this does still seem to be flaky: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
still the case
How about adding some logging to help troubleshoot this? I looked at it a little, but it is hard to tell which part of the test is failing. PR
/
This is still happening
/triage accepted
From the logs, in the OneVolumeDetachFailNodeWithReadWriteOnce test, succeeded, but the next verification failed. In verifyVolumeAttachedToNode, the attachState is from calling and within asw, it calls . In verifyVolumeReportedAsAttachedToNode, the volumes are obtained from calling , it seems that the node is not in asw.nodesToUpdateStatusFor at this step. Cannot reproduce locally. I tried to run the test 100 times locally with debug.Stack()... 3 * 100 times all passed.
Tried to run stress test on my linux machine as well but still cannot reproduce. Command: Result: Also tried running n process in parallel. Command: Result:
It seems like it happens more frequently.
But I may have no time to dig into the cause. Let's see.
cc as the test author.
The problem is that this test has a lot of dependency on the sleeps that are hardcoded with a very narrow scope, but our CI seems to be able to stress it much more. As soon as you slightly change one of the sleep duration it starts to flake locally The test can not make an assumption that timing is perfect, even more in the sub second range can someone in sig-storage review this test and make it less dependant on those tight sleeps?
According to most failures, it failed at Line 660. Let me run some stress locally. with 250ms sleep with 100ms sleep
I agree this test case is too brittle due to its specific timing dependencies. To fix this, I suggest reworking the test case to: 1) Add retry with timeout to wait for final conditions 2) Remove the intermediate timing-based state transition checks. Instead, we can keep track of state transitions in the Fakes and validate those stats. wdyt?
Friendly ping from 1.27 CI Signal that this test is still a noisy flake on the master-blocking dashboard as CI Signal has flagged it in and
introduced somethng that really should be e2e tested.
Would be nice to get this in in the 2 weeks we have after 1.2 branch cut, if no more flakes crop up (but I'm not too optimistic). Bumping to next-release.
/sig network I'm assuming sig-network owns kube-proxy
I thought I noted a change related to this recently, but I may be mistaken.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/lifecycle frozen /remove-lifecycle stale
/remove-lifecycle rotten
Going to try to take this on.
thanks!
I've had some trouble constructing a test for this. should I determine the timeout? There doesn't appear to be an existing mechanism for waiting for service creation. seem to time out rather than be refused - I'm confused by this, as it seems to be the opposite of the intended behavior.
can look at the endpoints + wait for longer than kube-proxy sync time. refused means packets got to their destination but there was nothing listening there. If traffic is blocked, then you will just get a timeout.
I created a service with no matching pods to test manually outside of the e2e suite. # curl test-api curl: (7) Failed to connect to athena-api port 80: Connection timed out # curl --connect-timeout 8 test-api curl: (28) Connection timed out after 8000 milliseconds Looking back at original example in I see that I'm getting the "before" behaviour. An iptables REJECT should send a denial, not a timeout. Am I looking at the wrong thing here? To me this looks like a regression, but I don't think that's likely.
I have a cluster from HEAD as of 2 weeks ago: So indeed it seems to not work. If I hit that same service IP from the VM itself I immediately get curl: (7) Failed to connect to 10.0.199.54 port 80: Connection refused. If I insert an iptables jump to KUBE-SERVICES at the front of FORWARD, it correctly rejects. This seems in accordance with the iptables flowchart. I'll do some spelunking to see if there's a reason NOT to reject in this case. Vallery Lancey <:
Can you check IPVS mode?
This is why we need tests. Sigh.
Proposed fix is - still needs your e2e :)
Thanks Tim, if I have time I’ll get a PR up tonight.
Sure will check IPVS mode.
Base on my test, IPVS has no such issue.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --/kind bug /sig cloud-provider /area provider/azure Azure in-tree CCM fails to delete a load balancer: The corresponding logs: The clouprovider config is: The Nodes are actually Azure VMs that belong to a VMO (aka vmss with flexible orchestration mode) - regular VMs and managed independently of the vmss. So those VMs are managed on Azure as . LoadBalancer deletion works with the very same setup with v1.21.1 of the in-tree Azure CCM. After recent cherry-picks, with v1.21.4 we observe the failure above. a cluster using VMs that belong to a VMO a Service of type LoadBalancer the Service of type LoadBalancer sure that the deletion fails with: Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Gardener Network plugin and version (if this is a network-related bug): Others:
/assign /triage accepted
The recently merged PR a version cache in dockertools, which makes the test  flaky. The reason is that injected error into fakedockerclient and then expected to see the error from right after that (See ). However, the error was consumed by the goroutine which periodically updates the version cache (See ). The docker daemon version is rarely updated, and we only need version once for each container creating, during which we also do container create, start, inspect #etc. 1) Do we care about the cost of the call so much that we have to introduce the complexity? 2) Kubelet periodically calls , can we rely on that to update the cache? Anyhow, at least, we should disable the cache or have a fake one in the unit test. :) /cc
I once considered move this cache to runtime level, while thus may need to change runtime interface. What if only cache apiversion?
We can only get via , so there will still be the data race. :) One option is to update the version cache in , now kubelet periodically call to check the runtime status. Another better option is to cache the version in layer, now is under our control, we can refactor it with an additional argument indicating whether to use cached value or get directly from the daemon. If we cache the version only for performance, I prefer the second option which is cleaner and more efficient. :)
OK, will look into this tonight
The unit test itself should not start a goroutine to update the cache (unless you're actually testing the goroutine). We should disable it anyway, as we often do in our unit tests. It also sounds like we don't need a goroutine dedicated for this. If we can tolerate the additional overhead from time to time, we can set a expiration time (e.g., 1 minute), and only call the runtime if the entry expires.
Totally agree. :) Maybe a new function in wrapping the update logic, or wrapping the logic in the .
expiration time sounds good in this case
This is happening quite frequently - increasing priority to P0.
The compute project and zone in the prerequisites check are incorrectly calculated: !!! Error in hack/make-rules/test-e2e- Error in hack/make-rules/test-e2e- 'tee -i "${artifacts}/build-"' exited with status 1 0 Call stack: 1: hack/make-rules/test-e2e- main(...) Exiting with status 1 blocked by the following errors in each case: 1- Could not find gcloud compute/zone when running: .... 2- Could not find gcloud project when running: ..... 1- gcloud config unset compute/zone 2- gcloud config unset project 3- make test-e2e-node REMOTE=true LIST_IMAGES=true gcloud --version Google Cloud SDK 453.0.0 alpha 2023.10.27 beta 2023.10.27 bq 2.0.98 core 2023.10.27 gcloud-crc32c 1.0.0 <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign
Currently, it could not delete multiple pods at once, need to delete them one by one, it's very inconvenient if I have tons of pods to delete. Unlike docker, docker can multiple stop/rm containers/images at once.
This is possible using label selectors as of , which is better if you have tons of pods.
I agree that using label selector is better to delete all of pods under the label. But to delete multile pods at once is still usable: I just want to delete 1/2 pods under the label, label selector will not work selector could not span over other selectors or pods Another reason is docker rm/stop x y z works, I will wonder why kubectl cannot. This is not urgent, but I hope it can be implemented.
I agree. Thanks for the suggestion.
Would this be a suitable task for someone new to k8s development? If so, I'd be willing to take a look at this, if someone could provide a couple of hints where to start.
Further to my last message, I have a functional work in progress I would like to share, if anyone is interested?
PRs are welcome. See
Yes, this is a reasonable place to start. Since its in the client, you won't get much orientation to the core server code, but it will be a good place to orient to the client side API. There us also prior art that you can model off of. The kubectl create ... command already supports multiple resources. A good command to adapter would be the 'label' command. You can find it in Many thanks in advance! Brendan On Feb 19, 2015 8:21 AM, "Eric Tune" wrote:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-new-master-upgrade-cluster-new-parallel Which test(s) are failing: Varies, but all subpath failures, including: [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath with backstepping is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath directory is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath-v0] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should fail if subpath file is outside the volume [Slow] ... and pretty much every other subpath test, but never all of them at once. There's also a few other storage tests failing, such as: [sig-storage] Volume expand [Slow] Verify if editing PVC allows resize [sig-storage] Detaching volumes should not work when mount is in progress Since when has it been failing: 11/22 Testgrid link: https://k8s- Reason for failure: These flakes started around the time that merged, but doesn't match up with the exact merge stamp, so it's probably coincidental. The subpath test failures seem to be mostly timeouts: ... so possibly this is just GCE fail. Anything else we need to know: This test job has always been flaky, with around a 40% failure rate. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig storage /priority important-soon cc
can you please help investigate this. It looks to me like the tests are flaking for and . I wonder if the v0 tests are racing with the non-v0 tests for something in this test?
The other tests including for the other CSI driver and non-CSI in-tree plugins are solid green which also makes me think this is a issue with the hostpath tests or driver.
(just adding additional similar flakes here for tracking) Similar flakes in :
adding to the milestone for triage since the flakes look new
These all 4 subpath tests call and it fails in finding a specific failed event by using . As uses to check if the specific error event happens, and , I guess that tests may flake if other failed event happens before the expected one. Also, already wait for the event, we might be able to delete  in . I will create a PR to fix above.
/priority critical-urgent /remove-priority important-soon /milestone v1.13
while we should update the test with your PR, I'm not sure if it's going to fix the failure. Looking at one failing test log, these are the events I see on the failing pod: So it seems to be failing before even reaching the subpath logic.
Timeline of events: Test pod started getting processed by kubelet: It took a minute after the test pod was created for the csi driver to finish registering and mount to succeed, and then subpath event to trigger: So should address the issue. I was initially confused because the test logs didn't show the subpath event for some reason
I opened up to investigate if 1 minute for csi plugin registration is expected
I've a tracker issue for the complete list of storage flakes: This is so that we can track whether we're addressing most of the flakes with the subpath fixes, or if additional fixes are required.
keeping open until flakes are verified cleaned up
/reopen Reopening this issue until we can verify the fix in CI
As of this morning we are seeing CSI "subpath should fail" flakes in the following job Multiple "subpath should fail" flakes in this slow job alone Multiple "subpath should fail" flakes in the 1.13 slow job, similar to master. The following 2 tests are flaky only in this upgrade job can you help us determine if these are cases are flaky tests that you are fixing or is there an underlying CSI bug that needs addressing?
root cause of this is a delay in an external sidecar containers and the tests not waiting long enough to accommodate the delay. There are no changes required in (outside of test) to handle this. These should not be 1.13 blocking. We'll continue to look in to getting them non-flaky today.
In the meantime, we can mark these tests as flaky to reduce the noise, if you'd like. looks like the same issue () is still being hit. See : And logs from (): We need to make the tests handle this condition more gracefully. The suggestion here is one possible solution.
Would be good to capture this as a known issue in release notes.
to followup on capturing this in Release notes
per , can you create a targeted fix increasing the timeout? Looks like marking these tests as flaky will be difficult for 1.13, but if that doesn't fix it we will need to do that.
Short term fix to bump the test timeout:
As per discussion in release burndown meeting, this is not a 1.13 blocker. please move this issue to 1.14 if there are no pending issues you are concerned about
Ok looks like the test still flaked. Looking at , the mount succeeded after a minute and proceeded to subpath processing, which (correctly) failed. However, for some resaon, the test is not seeing the subpath event and still failed after 4 minutes. On kubelet, subpath processing failed after a minute and generated the event: But the test doesn't see it for some reason, and still times out after 4 more minutes:
I saw the same on CSI volume mount succeeded at: Kubelet correctly failed the pod because of subpath But test still failed: Because the test failed to find the pod event. This looks like a test issue. Not a 1.13 blocker.
can you PTAL at this
Is there a way to check if events are overloaded and not able to handle new events? The MountVolume retries while the driver is not registered is very frequent and could have overloaded the event server.
Checking for the existence of an event seems like the wrong way to test this. Would checking pod status alone be insufficient? Ultimately that is what we really care about.
Checking for the specific event guarantees that we are failing with the correct error. But perhaps events are not reliable enough to depend on. Checking if the Pod never came up running would probably reduce the flakes but could potentially mask false negatives. cc for any thoughts.
Tests are still flaking and actually it seems like in general storage flakes in slow are flaking more since 11-21 or so: https://k8s- My theory is that the driver registration delay is causing a flood of events, which causes tests that check for events to fail. I think this is revealing multiple issues in both the tests and product, and we should work on fixing them all, although any one of the fixes should mitigate the test flakes: Driver registrar can sometimes take a minute to come up: Volume manager operation executor doesn't seem to be throttling retries or backing off: Tests should not rely on events: Tests should wait for CSI driver to be registered on all nodes before creating test pods:
and have been merged and will address the biggest issues. has been backported to 1.13 already, backport is still pending.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details