jiebi/CodeConvo huggingface dataset:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Data race in client-go with below sample code: Data race: What you expected to happen: No data race How to reproduce it (as minimally and precisely as possible): Using above given sample code Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): OSX
/sig api-machinery
I'd like to take it. /assign
/remove-sig api-machinery /sig auth cli
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then?
Another reminder here that code freeze is in less than two weeks (Nov. 14). We will need to move this issue out of the milestone if isn't going to be resolved before then.
During debugging , I noticed that one of services pods: monitoring and logging pods stuck at pending sometimes forever: I logged into the node, and found the container is running happily on the node: Looks like status is never reported from kubelet. I checked kubelet log, then I found: Why we cannot delete mirror pod? and Created a new one? Because this regression, once we run into such state, the rest of test won't be triggered, and eventually timeout / failed.
checks those service pods in ready state, and prevent the test to run once it is pending. cc/ I think this is the real issue I talked to you earlier about PodStatus.
I spoke to offline. For some reason, kubelet thinks the spec of the mirror pod does not match the static pod locally. It then decides to delete the mirror pod and recreate it, repeatedly. As a result of that, it slows down (due to the QPS limit?) reporting statuses of other pods. This can be reproduced by running the e2e "Shell" test. I am looking into why the specs would suddenly mismatch. Did any recent PR modified the pod spec in apiserver?
The recently introduced default cpu share limit was not there in the static pod, causing the difference between its spec and the mirror pod's.
Seeing this as well. Makes the whole cluster slow and unusable for me. Sent from my iPhone
Is there a desired fix for this?
said she would add default resource limits to the static pod manifest files. If we go for that, I think it's reasonable to fail (not run the static pod) if resource limit is not specified. Also, the resource limits set in the mirror pods are not meaningful because kubelet creates containers based on the spec of the static pod. In general, static pods do not subject to any defaulting values applied at the apiserver.
Pretty sure that this is fixed by . Certainly testing against head since then works well for me. What version are you testing?
was fixing node status, and the problem had been around for a while. This issue is about pod status, which is still unfixed on head. These two are very different, and my cluster did include the fix :) This issue will slow down kubelet at times, but if you get lucky and the pod status (running) get sent out in time, the tests will still pass.
There is no way the issue is fixed by that PR. I am still seeing the issue on HEAD. The system gets worse the longer its up. I agree with the analysis of the issue in source PR Sent from my iPhone
I am on this one, and here is proposed solution in my mind so far (not final yet since I haven't put too much thoughts into it yet): 1) For all our static pods I am going to update them with some initial cpu limit for now. This should quickly stop flakiness caused by this particular issue. 2) Next step is that for static pods, if there is no initial cpu limit, kubelet should refuse to run. Before I do this, I need to send an announcement to our container-vm users about this potential breakage for the in the future. Another potential fix is we can ignore resource changes when comparing if a mirror pod is updated or not. I think is proposed a similiar one earlier. But I had a concern with it: this is defeat the initial purpose we created mirror pods and sync back to master. So if one made some resource changes on the static pods, you have to always ssh-ed into the node to see if the new value applied or not. Also eventually (post 1.0) those mirrored pod's resource limit is going to be used by scheduler when calculating if user pods are fit or not, we need them to be accurated. Another concern is in the code implementation itself to distinuguish mirror pods future from rest of pods. I guess this shouldn't be the big issue here. Any more suggestions here? One puzzle I had is that why master component pods doesn't have issue at all. I manually restart kube-apiserver pod, still doesn't have the issue.
Depending on time we could just remove the default limit from the limit range as a stop gap. Sent from my iPhone
Can you just bypass defaulting at the apiserver when you create mirror pods? I can't think of any other solution that works in general and is easily maintainable. Approach 2 you described in the previous entry works if we are defaulting cpu limit, but breaks as soon as we default anything else.
Same problem with ignoring resource differences when comparing mirror pod to static pod -- as long as defaulting only touches resources, that works, but if we change it to touch other stuff, then it breaks.
Commit I had can quickly fix current bleeding issue. Currently we only have master component pods and fluentd pod are file based by default. Master component pods doesn't create any issue now because mirror pods aren't created at the first place (due to recent change made by Also file-based pods will have issue only when its spec doesn't have initial value, and need to update (delete miss matched mirror pod, and recreate a new one). Please note that with my commit, everything is working even I manually update its cpu limit from 100m to 200m, there is no issue at all, and query pod shows accurate result with the new limit too: How badly we request all users using file-based pod feature providing default cpu limit for now? Eventually we want to get rid of file-based pods completely, also encouraging the user provides the initial resource limit is aligning with our direction in a long run anyway.
Just to clarify, what I was suggesting was for a "real fix" not the temporary measure to stop the bleeding (which you already have in your PR). But what do you think of what I suggested as the longer-term fix? My point is that today, we default CPU, but in the future, we may default anything. It seems that the most maintainable fix is to just not do defaulting when creating mirror pod from static pod. Is that possible? I wonder if the reason manual update works (what you described in your last comment) is because we don't go through the defaulting when you do an update, only a create? (Though that sounds like a bug.) IMO we don't need CPU default right now, especially since we already have kubelet flag for limiting pods per machine (though unfortunately it's currently set to 100 because we haven't reconfigured the tests that intentionally run lots of pods per node, to override the flag. we can change the default to 30 and add an override mechanism if this is important.)
Comparing a config file with the live state and expecting them to match is doomed. The recommended approach is described by . Kubelet could store the config as an annotation in the mirror pod and compare with that.
It's useful to have a helm chart that expose a value like . However, the annotation simply ignores , as seen in It's just checking that the value is not empty, so even if it's false - it'll try to create internal LB.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: mcrute. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Service loadbalancer source range don't work in ipvs proxy mode. Which result in gci-gce-ipvs CI failed. http://k8s- How to reproduce it (as minimally and precisely as possible): In gce environment. a service with backend. service type to service util service external IP is specified. the service from pod inside and pod out of the . Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: cluster/kube- Others:
/kind bug /sig network /label ipvs
/area ipvs
The test seems to be flaky now. Can you take a look please?
I don't know why, but I think this flaky have the similiar reason with other normal service-test cases, like Those flaky are pretty annoying, problem is, I have run those cases in my own environment many times, never fail... : ( But still, I'll do my best to find the reason.
I'm using K8S 1.10 and have a specific use case where I'm trying to use an emptyDir to share mounts between two containers in a pod. I'm using the MountPropagation feature successfully, but while debugging another issue I noticed that the node was running a very long running du command. Looking into the source I noted: Which runs du on any emptyDir, do measure disk usage. I recommend that in this case we should add the -x flag that will skip any mounted directories as they do not count against the file usage and would make these scans a lot faster. I could of course add a simple pull request for this but first wanted to see if this was known and on purpose. To reproduce bind mount/loop mount/device mount any large/slow directory into an emptyDir volume and watch as the du process slogs away Environment: Kubernetes 1.10 GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it makes sense to ignore submounts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
do you still want to submit a PR to fix this? Thanks!
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .
/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
[X] Two weeks soak end date: Nov 18 According to this APIsnoop query, there are still a remaining Service endpoint which is untested. A service resource didn&rsquo;t support endpoint until was merged. Create a watch to track service events Create a service with a static label. Confirm via the watch that the service was created. Delete the service via the service with the dynamic client with . Confirm via the watch that the service was deleted. The above steps have been implemented in the follwing . This query shows all the endpoints hit within a short period of running the above e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig network /area conformance
/triage accepted
Waiting on 2 week soak before promoting the test to conformance /reopen
Reopened this issue. <detailsIn response to : Waiting on 2 week soak before promoting the test to conformance /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
During
any chance this is related to pass: https://k8s- fail: https://k8s- https://k8s-
This is because for some reason the submit queue rebuild didn't catch the flake before merging. Protobuf landed first, merging a new api object like petset should have failed.
Quick fix is to rebuild the proto marshalers and push.
Also, hack/verify-generated- is broken, which probably is the root cause.
Looking at it now
Three problems: -generated-protobuf wasn't updated when we made update-generated-protobuf use docker, so it didn't fail out script was verifying the old API groups, not any of the new API groups that have been merged without protobuf serializations (although pkg/api/serialization_test should have failed, possibly the job didn't build with the latest master?), which caused the tests to fail
Something about the merge bot is broken - it merged without all the tests being green: . That's why this broke -
Spawned since there is something wrong with update-generated-protobuf - will merge that to fix the failing test and then debug the generator in the morning.
The bot can race with pushes between when it starts testing and when it merges...
What happened: states that Container.SecurityContext applies to a Pod, which can be misleading since we already have PodSecurityContext. What you expected to happen: Documentation states that it applies to container How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment:
/sig api-machinery /kind documentation
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Not stale.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
The test pod gets deleted by the test upon failure and so there is not enough information about the state of the test pod. I'm changing the test to not delete the pod and instead let the namespace deletion logic in the test framework handle deletions. I'm also increasing the pod lifetime from to a longer duration to avoid any possible race conditions in the test. The test will also retry if the logs output from the test pod does not contain non-space characters.
I'm lowering the priority of this test once my changes go in and see if the flake re-appears.
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
I suspect a race between container starting up and the test querying the logs. I'm adding a retry in the test to avoid the race. Lowering the priority since I still don't suspect any kubelet bugs.
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite}
https://k8s- Failed: [] Pods should support retrieving logs from the container over websockets {E2eNode Suite} Happened on a presubmit run in .
The pod security policy capability test is using wrong CAP format with prefix. In our document, we explicitly say: It fails for cri-containerd. I'll find out why it doesn't fail for Docker, maybe docker has handled it. But we should fix the test anyway.
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
[ ] OWNERSALIASES:581: # conformance aliases [x] // More info: ~~ [x] // See [x] fs.MarkDeprecated("redirect-container-streaming", "Container streaming redirection has been removed from the kubelet as of v1.20, and this flag will be removed in v1.22. For more details, see ") [x] // More info: [x] // More info: ~~ [x] [x] [x] CHANGELOG/CHANGELOG-[scalability validation report]: ~~ [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [x] ALPHA stability level ) [ ] link: [ ] link: Some needs to be regenerated
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help
/kind documentation
/remove-kind bug
/good-first-issue
/assign I shall fix .
/assign i'm working in
i have create a pr for
i see , you need
/assign fixing
/assign
/assign CHANGELOG/CHANGELOG-
/assign CHANGELOG/CHANGELOG- Edit : Link is already resolved, unassigning
/assign CHANGELOG/CHANGELOG-
/assign ,111
Sorry, already assigned
/assign i will take
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
  is merged. Most KEP links are fixed. You may scan it again. fixed most of them.
/sig architecture
/assign I will work on and
/remove-help