jiebi/CodeConvo huggingface dataset:
kube-proxy regularly stops running with the below panic / stack-trace. This is when configured with vagrant. (It is also running on a laptop, so my unsupported-by-any-evidence theory is that it happens when the laptop resumes after sleep.) Restarting the service causes the proxy to continue working happily. (Should systemd be restarting the service?) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: panic: runtime error: invalid memory address or nil pointer dereference Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: [signal 0xb code=0x1 addr=0x0 pc=0x431b51] Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 22 [running]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x6b7c00, 0xa51de8) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/panic.c:266 +0xb6 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720, 0x77359400, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x841 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210066600, 0xc210037660, 0xc210037720,
0x77359400) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x165 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x1b3 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 1 [select (no cases)]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x47a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 3 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (loggingT).flushDaemon(0xa57340) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x50 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by ·1 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x276 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 4 [syscall]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: () Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /usr/local/go/src/pkg/runtime/proc.c:1394 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 6 [sleep]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: time.Sleep(0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: +0x31 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0x7aca18,
0x12a05f200) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x57 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x99 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 7 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077120, 0x7ff2602627f0, 0xc210077060, 0xc210037540) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 8 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc210077210, 0x7ff260262840, 0xc210077150, 0xc2100375a0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x196 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 9 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (Mux).listen(0xc2100770c0, 0x718260, 0x4,
0xc210037600) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x3a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x43 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·001() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x40 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: (0xc21000a920, 0x0) Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x49 Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: created by (*Mux).Channel Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x20a Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: goroutine 10 [chan receive]: Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: ·004() Aug 13 12:04:06 kubernetes-minion-1 kube-proxy[]: /var/src/kube- +0x4d Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: kube-proxy.service: main process exited, code=exited, status=2/INVALIDARGUMENT Aug 13 12:04:06 kubernetes-minion-1 systemd[1]: Unit kube-proxy.service entered failed state.
I looked at this one a little bit and I think it's because of This line isn't present at , but I'm not sure which is newer. Presumably results in the channel being closed (and receiving a zero value that it then tries to dereference) on error.
Diagnostic: This line doesn't check for a closed channel, and it needs to. Yeah, how are we supposed to know if our client is going to close the channel or not?
I think the behavior has changed at least once already. This is part of why I'm trying to (for our code, anyway) standardize all of our watches with a watch.Interface. I took the liberty of assigning this to you since you're more familiar with this code, but feel free to unassign if you can't take it right now.
Got it
can you give a try?
after i do the minion watch for kubelet config i'll come back and gut this and replace it.
Thanks - with merged I haven't seen this happening. Thanks for the super-quick fix!
For 1.25 clusters with PodDIsruptionConditions enabled pods continue to run if the node is tainted with untolerated NoExecute taint. Also, the error is not logged. The pod running on the tainted node should be terminated (prep) Create a 1.25 cluster with the enabled (tested on kind) a job with a single Pod running long enough for the test (say 3min), the node with a custom taint (so that it is not tolerated by the Pod) with the NoExecute effect (nodeName No response <details</details<detailsreproducible on kind </details<details</details<details</details<details</details<details</details
/sig apps /triage accepted /priority important-soon
/assign
Hello there, I did search around and didn't find a report for this already, I hope it's not a dupe. What happened: In an HTTP probe definition (readyness or liveness), the custom host header is not set properly after a redirect occurs. What you expected to happen: Seeing that states that redirects are weird but allowed, I would expect the custom headers to be maintained after the redirect occurs. How to reproduce it (as minimally and precisely as possible): Setup an HTTP service that expects a specific Host header to be used (using Django here) and would return a 400 if the Host doesn't match (for a more detail view at how Django deals with this ). Then setup a probe like: You should now see in your logs that the first call to the service succeeded with a 301 (or 3XX) which means the Host header was set properly, but that the call to the redirected url will fail with a 400 which means the Host header was not set anymore and makes the probe raise an error. Anything else we need to know?: In the above test, setting the "path" to the redirected url works and the probe is green. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: intel NUC OS (e.g. from /etc/os-release): Ubuntu 16.04.5 LTS Kernel (e.g. ): Linux nuc 4.4.0-138-generic -Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Minikube Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node /sig network
We are calling Go's HTTP client here - is it possible that this is "normal" behavior for Go? I don't know...
Seems like an old bug indeed: - not sure where to go from there...
want to try attempting a patch using ? code that mentioned is in
I can definitely try to submit a PR with that change! Thanks for pointing in the right direction
I just tested this on GKE and also have a unit test confirming that this is not an issue anymore, I guess HTTP go client has been fixed.
Awesome! looks like losing track of my commitment here was a good investment thx for the feedback !
Awesome. Looking forward to more tests :)
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: A NetworkPolicy test in , not run in Kubernetes CI. Which test(s) are flaking: Testgrid link: Reason for failure: Its failure rate is 6.66% according to 30 latest tests in (a Kubernetes CNI) CI. The reason is, the test doesn't wait for the server Pod to be ready before testing traffic via its service, then even the NetworkPolicy allows it, the SYN packets will be rejected by iptables because the service has no endpoints at that moment. e2e log: server pod status: kube-proxy log: In short: 14:35:58.874 server Pod was created 14:36:00.916 server Pod became running 14:36:00.916 The traffic test against Service started, there was no endpoints for it at this moment so it was always rejected 14:36:10.921 The traffic test failed and terminated 14:36:10 server Pod became ready, Endpoints is updated 14:36:10.547 kube-proxy received Endpoints update and configured nat table The other tests don't have this problem because they all wait for server Pods to be ready, this test should do it too to avoid flakes.
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry. ‍
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
These tests have flaked consistently since merged: in many cases, they are the only failure in a test run: Either the functionality introduced by that PR is not reliable, or the test is faulty. /sig node /priority critical-urgent /assign
/milestone v1.19
/assign
Looks like this is failing ~5% of pull-kubernetes-node-e2e runs
FYI
Looks like there might still be a race somewhere looking at the kubelet logs Success case: Fail case:
Looking at it more, it seems like is unexpectedly empty here Success: Fail:
Stacks of interest I think we might be dealing with a stale If there is a call before the container is to the Pod in the runtime and the is called within the 2 second default cache period, it seems possible that the in the could have no containers.
what do you think of this analysis? If so, any obvious solutions occur to you?
If I'm correct, didn't introduce this; just the e2e test exposed an existing bug in the kubelet.
That all looks correct. I'm taking a look at the proposed solutions
This is wrong, cluster scoped resources should never have a NAMESPACE column. We need to fix and add a test to ensure this doesn't regress.
The presence of the flag is what triggers this.
I would help fix it.
I mean to add a test after modification, but found that current test framework could not support the case: To cover the scene, we need a real NewHumanReadablePrinter in test case, to ensure the output format, and modify the related funcs in FakeFactory to support pass in param withNamespace. I feel this not like behavior of unit test to some extent. The loss may outweighs the gain to have such a big impact on the whole. What's your opinion
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is securit pty related, please disclose irivately via --What happened: There is a Windows minion node in my k8s cluster, I have specified "--pod-infra-container-image" for kubelet which is pointing to a private docker repository. The kubelet running as a Windows service, service account is LocalSystem: I have login to the repository with Administrator, and then copied from Administrator's profile folder to "C:windowssystem32configsystemprofile.docker". When I schedule pod on Windows node, kubelet cannot pull infra container image from the private repository with error: I can see there is a PR solved this issue, but it seems not work on Windows. What you expected to happen: Able to pull infra container image from a private repository on Windows node. How to reproduce it (as minimally and precisely as possible): Run kubelet as a Windows service and specify --pod-infra-container-image to point to a private repository with basic auth. Anything else we need to know?: Environment: Kubernetes version (use ): Linux node: Windows node: Cloud provider or hardware configuration: OS (e.g: ): Windows Server core 1709 Kernel (e.g. ): 10.0..125 Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Your private repo only provides http service? does it need authentication?
the private repo built by Nexus, it requires basic authentication
/assign
Hello, First of all, the PR you mentioned was closed, but not merged. Secondly, why do you need to use a private repo for the [2]. Even if you opt to use a private repo, if you pull it after you've authenticated, it would fix your problem. Finally, I can confirm that kubelet is not able to pull the configured private image case). [1] [2] [3]
Ok, so, I have a couple of updates regarding this.
First of all, indeed, the PR mentioned in the issue was closed without merging, but another one was merged in its stead: [1] Secondly, it seems that kubelet is trying multiple credential providers, including for azure, aws, gcp, kubernetes secrets, and local configurations, as it can be seen here: [2]. I actually ended up getting it to work by setting up the local configrations properly. From what I can see in the code, it could still be a Windows related issue [3] (the env var name should be ), but it can be avoided, since it will check multiple locations for the configurations: I0513 16:22:05. ] looking for at C:varlibkubelet I0513 16:22:05. ] looking for at C:k I0513 16:22:05. ] looking for at C:k.docker I0513 16:22:05. ] looking for at C:.docker I0513 16:22:05. ] looking for .dockercfg at C:varlibkubelet.dockercfg I0513 16:22:05. ] looking for .dockercfg at C:k.dockercfg I0513 16:22:05. ] looking for .dockercfg at C:k.dockercfg I0513 16:22:05. ] looking for .dockercfg at C:.dockercfg I0513 16:22:05. ] Unable to parse Docker config file: couldn't find valid .dockercfg after checking in [/var/lib/kubelet /] So, basically, I just had to copy the file to one of those locations. [1] [2] [3]
Yes, %HOMEDRIVE%+%HOMEPATH% is the right profile path: HOMEDRIVE=C: HOMEPATH=UsersAdministrator
Hm, keep in mind that you're trying to run the Kubelet as a service, under the ` (probably).
sure, just to show the name of environment variables
this has a PR open, so adding /milestone v1.15
when deploying a container that has runAsNonRoot but runs as root this error appears, but for pods with lots of containers it can be confusing which container failed, especially if there are 3-rd party webhooks that inject containers please add the container name to the message, making it
/sig api-machinery
/assign I can try this fix
I just submitted a pr, can you review it for me, is it as you expected?
thx, looks good!
/remove-sig api-machiner /sig node /area kubelet
Those labels are not set on the issue: <detailsIn response to : /remove-sig api-machiner /sig node /area kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig api-machinery
once the we hit a beta release, we can reset the default value for strict handling to false to prep for 1.26. /kind cleanup /priority important-soon /milestone v1.26 /assign /cc
/sig api-machinery
/triage accepted
Looks like Nov 10 ( ) <img width="946" alt="image" src="https://user-"
/milestone v1.26
please don't drop the milestone from issues without asking first... this is why CI is failing on master currently (my fault for not remembering to resolve this between beta and rc, sorry)
I removed all with code freeze and the ones back that got approved as exception request / are big fixes. Missed this one; sorry.
There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help
We sorta fixed this-- now it's a real status error, but we left the status code 500, so it still produces stack traces and potentially confuses clients. we should figure out the correct status code.
I think there is a lot more going on related to this in . Clients are re-listing after the error but not rewatching.
/sub
It turned out this was not causing job e2e flakes. Just sending 500s that were confusing. So lowering priority.
How about a NewBadRequest error?
that certainly sounds like an improvement.
Hi, We have a device (router in this case) that sends long living stream of udp packets to our kubernetes cluster. We then create a service and pods that expose a node port to receive the incoming udp packets. However, packets were not reach our containers. Further digging and found that because UDP packets already stream into our host (therefore conntrack entry already created) BEFORE our service is created (iptable rules ), the iptable rule didn't take effect. Since our UDP stream is long living, the conntrack entry never get timeout neither. To workaround the issue, we have to stop the UDP stream for sometime after the service is created so that the conntrack entry times out. If we use "conntrack -D" to clear all entries after service is deployed also workaround the issue. Though it is the behavior of the linux iptable that new rules doesn't affect already existing connection, we are trying to see if kubernetes proxy have any config that can clear the related conntrack entries when a new service is deployed?
I am interested in your problem. Can you give some more details how did you add iptables rules before service started. Did you use the PREROUTING chain to forward port to pod ?
OK I was able to reproduce the problem
I think the issue is more problematic and complex. It is explained here:
Yes I think it is the same conntrack issue as in the one reported in docker issue. But I also think that the kubernetes might have a better control over it, since it knows the service up/down. I think k8s already deleting the conntrack entries when removing services. Just wondering if the same can apply to adding services?
Yes you are right. Kubernetes as a Docker manager should take care of this. Deleting the conntrack entries is done in very simple way: Just call the application
But there is another problem because I removed conntrack entries and then I created service and still have problem. I had to clear conntrack entries after I started my service. I will try add this to source code and then try it.
In code it looks much better. I assume you used for your service to use some particular port. The fix is in place when node port is binded. Do you think there is any other scenario where conntrack entries should be cleared?
When I create PR I will let you know
The PR created: If you want verify it you can easily do it using local cluster Then clone kube-deploy project to create local k8s cluster with image you already created with fixes
Thanks for the fix. This is what we did manually to workaround so it should be good enough for us. It would be even better if you can clear the conntrack entry only for the affected entry, such as destination port, to minimizes the impact. Wondering if that possible? Will try out as well. Thanks.
I wanted do it like you mentioned but unfortunately conntract command doesn't give that possibilities. In parameters you can only pass the source and destination IP addresses.
Sorry didn't get back for a while. I tried the conntrack tool and seems like you can use --dport option: conntrack -D -s 172.27.209.165 -p udp --dport 53 udp 17 63 src=172.27.209.165 dst=171.70.168.183 sport= dport=53 src=171.70.168.183 dst=172.27.209.165 sport=53 dport= [ASSURED] mark=0 use=1 udp 17 33 src=172.27.209.165 dst=171.70.168.183 sport= dport=53 src=171.70.168.183 dst=172.27.209.165 sport=53 dport= [ASSURED] mark=0 use=1 conntrack v1.4.3 (conntrack-tools): 2 flow entries have been deleted.
Thanks for info I will try it
I've updated my PR for flag It works well
It's not clear to me why there would be a conntrack rule for your service port before we add any rules?
We have a router, which configured to export statistics using UDP to our host. That UDP stream is long live. If our host restarted, or we re-deploy our cluster (mainly during development), then udp packets received by host before K8S is provisioned.
But why do you have conntrack rules - simply receiving packets will not cause conntrack to create records.. , dachulin <:
https://k8s- /sig apps /priority critical-urgent /assign PR that the test: /cc
GitHub didn't allow me to assign the following users: mortent. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : https://k8s- /priority critical-urgent /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I'm looking into this.
The test is still flaky, but with different symptom: An example flake: https://k8s- It's likely because the collision count was updated in a previous sync loop, but the in the cache store hasn't been updated and we used a stale collision count while calculating the new hash: /assign
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: after granting sp access to azure ACR , pull image from ACR would fail, and after wait about 15-30min, pull image would success. Root cause is that needs to be refreshed when doing , otherwise it will always return empty registry list. Related issue: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.7 - v1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign I will implement that function.
Thanks -- can we add a milestone label?
+1. We should fix this within v1.12 release and cherry pick to old releases. /priority critical-urgent /milestone v1.12
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I'm not sure if this is related, but I'm seeing nodes (after they've been working fine for a few days) randomly start to fail pulling images with the error . Edit: I see this as the culprit: , after which all image pulls start to fail.
this is not related, original issue is that after granting sp access to azure acr, pull image still fails with .