jiebi/CodeConvo huggingface dataset:
What happened: Deployment of node-local-dns in an ipv6-only cluster fails with the following iptables-related error Does node-local-dns support binding to IPv6 addresses? If not, can node-local-dns be used in an ipv6-only cluster? What you expected to happen: node-local-dns is up and running normally and binding to an ipv6 link-local address How to reproduce it (as minimally and precisely as possible): Create an ipv6-only cluster using nodes with single ipv6 stack Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: Openstack VMs OS (e.g: ): SLES 15 SP1 Kernel (e.g. ): 4.12.14-197.45-default Install tools: kubeadm Network plugin and version (if this is a network-related bug): calico v3.12.0 Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
seems that node-local-dns only supports IPv4
looks like whatever's calling might need to have a log msg b4 it
/assign
Looks like there's already a logging statement for running iptables checks: Perhaps the log level was too low for that log message to get tripped?
hi Matt, ok so your looking at the logs? If you want you can into one of your kind nodes. Then edit run and if you wanna hack around in there, and open inside that container, which will allow you to play with the logging verbosity of Kubelet. If you decide its a good idea to surface this info then you can test there (). There are probably ways to plumb Kubelet args directly into kind also but that's just my off the cufff answer :)
/assign
/assign /assign
Not yet, node-local-dns is not supported to run on ipv6-only clusters.
/remove-triage unresolved
Thank you for letting me know, and thank you all for your efforts following through with the issue, I'll keep watch for any future updates on the feature.
I put together a patch to to make it more clear in this case what the issue is. Do you think it's worth moving forward with adding that log statement?
(I know I'd have to redo the patch differently, I did it a really dumb way and sure there's a much much better way :) ) Happy to do the leg work on this! Thanks for the guidance! Edit: forgot to mention, this is only supposed to be a stopgap short-term solution before node-local-dns gets support for IPv6!
so , is what pramesh is suggesting as simple as running something in to figure out if there is an ipv4 address... and if not, ... fail ?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign I have a PR ready, but we need to update the KEP, the docs and add some testing, ... targeting 1.21
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): cluster-info, dump, "server rejected our request for an unknown reason" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: kubeadm Others: What happened: kubectl cluster-info dump fails to get logs from pods with multiple containers. It exits at the first such pod it encounters (in lexical order). The kube-dns pod invariable triggers this. What you expected to happen: should gracefully handle multi-container pods, fetching logs for all of containers. a dump fail, kubectl should continue dumping as much of the information from remaining resources as possible. How to reproduce it (as minimally and precisely as possible): Run with a kube-dns pod running. Anything else we need to know:
I can reproduce the error by using the latest code, let me take some investigation on this first.
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The help for needs to be updated to reflect the latest change. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc I will work on this together with
What happened: I do . I see the last 10 lines of output of each pod matching . What you expected to happen: I was expecting to see only the last line of output of each pod. How to reproduce it (as minimally and precisely as possible): The image prints the date every second. Normally we should just see two lines of output; instead we see 20. Anything else we need to know?: This seems to be a regression in Kubernetes 1.12, as this worked correctly from 1.8 to 1.11. (I observed this problem starting with k8s 1.12.0 and it's still present in 1.12.2) Environment: Kubernetes version (use ): 1.12.2 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu Kernel (e.g. ): 4.4.0 Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
It looks like the issue was introduced by commit , specifically . If I understand that code correctly, it forces the number of lines to (10) even if is set. Since that seems to be in : /sig cli
/assign
didn't that commit just move the code from ?
Not quite; the old code was: The new code is: I don't know why it was changed, but the new logic reads as "if the user specified a selector and a number of lines different from -1, then set the number of lines to 10".
What happened: when running integration tags were missing on files What you expected to happen: add the needed tags on test/integration files How to reproduce it (as minimally and precisely as possible): add an integration test package under cc
/sig testing /priority important-soon /area build-release
/cc
if we want to assume that all tests in a certain directory are integration tests this is very easy to automate, here's an example that we do now to mark go tests and binaries as
Is the assumption that all tests in should be tagged as integration? I'd be happy to add this to the script, but would prefer to get in without conflicts first
/assign /lifecycle active
Is this a BUG REPORT or FEATURE REQUEST?: What happened: After upgrade of Kubernetes 1.11.3 to 1.12.1 via kubeadm in bare-metal setup I see these errors in kubelet logs: What you expected to happen: Since this is a bare-metal setup I expect no messages concerning Azure cloud How to reproduce it (as minimally and precisely as possible): Do an upgrade from 1.11.3 to 1.12.1 Anything else we need to know?: kubelet-1.12.1- was downloaded from Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Oracle Linux Server 7.4 Kernel (e.g. ): 62 Install tools: Others:
Lines from kubelet log which say that no cloud provider was specified (that is a bare-metal setup): ..
/sig azure
have the same issue related to the azure.. also a bare-metal cluster.. these messages are flooding my centralized logging elasticsearch (though they aren't really problematic more than being an annoyance). i also upgraded from 1.11.3 to 1.12.1 on a 3-master HA set up.
i also noted (perhaps worthy of a separate issue) that the two of my three control-planes that are showing the azure error also previously had been having cert authority errors from the api-server container, when having used kubeadm upgrade.. (on the second two control plane systems, with the issue) (and on the first control without the issue) (and i needed to kubeadm reset and then kubeadm init all over again with 1.12.1 on the 2nd two control-plane nodes).. the first control-plane node that didn't need to do a reset/init didn't have the azure errors (nor the cert errors). so i kind of suspect it might not be an kubeadm upgrade issue but a general issue going from 1.11.3 to 1.12.1 ?
Same problems +1
Same problem on brand new bm 1.12.1 cluster built with kubeadm and latest 1alpha3 config based off of customized kubeadm config print-default example
/assign
Same problems +1
Oct 9 11:11:32 localhost kubelet: E1009 11:11:32. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:11:42 localhost kubelet: E1009 11:11:42. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:11:52 localhost kubelet: E1009 11:11:52. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:02 localhost kubelet: E1009 11:12:02. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:12 localhost kubelet: E1009 11:12:12. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:23 localhost kubelet: E1009 11:12:23. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:33 localhost kubelet: E1009 11:12:33.
3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:43 localhost kubelet: E1009 11:12:43. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:12:53 localhost kubelet: E1009 11:12:53. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1 Oct 9 11:13:03 localhost kubelet: E1009 11:13:03. 3342 ] failed to get azure cloud in GetVolumeLimits, kubernetes-1
how often did you hit this error log: I am planning to make this log as , is that ok for you? Usually V(2) is the default info(see ) log for k8s on azure I am not sure what the default info log by kubeadmin installation.
It is logged every 10 seconds. I disagree. At least previous incarnation of GetVolumeLimits in version 1.11.3 : would have logged an error "No cloudprovider present". Which I didn't see in kubelet log. In fact, I didn't see any message in kubelet logs with 'azure' substring at all. I think it means that AzureDataDisk plugin was not instantiated at all in 1.11.3. So the question is: why azuredd is instantiated now in 1.12.1 in bare-metal setup (that is without cloud provider)? Do you mind checking this idea? (Besides your method of lowering message priority won't work as I said this is a bare-metal setup where kubelet systemd unit file has no --v option at all )
That's because feature reaches beta in v1.12, in v1.11.3, even the code is there, it won't be invoked since feature is still alpha in v1.11
And in v1.12, aws, gce, and azure disk plugins are all loaded, take aws ebs for example, it will return error directly: And logging as V(4): I suppose V(4) logs won't display in logs. Let me make the same change as aws and gce.
also tries to address this error.
The same mistake, I also encountered NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 172.16.16.13 Ready <none5m27s v1.12.1 172.16.16.13 <noneCentOS Linux 7 (Core) 4.18.12- docker://18.6.0 failed to get azure cloud in GetVolumeLimits, 172.16.16.13 failed to get azure cloud in GetVolumeLimits, 172.16.16.13 failed to get azure cloud in GetVolumeLimits, 172.16.16.13
I have cherry picked the fix to v1.12
I understand that the feature Node-specific Volume Limits has now reached beta in 1.12, but for those of us running Kubernetes on-premises, this log error is pretty useless. In fact, I would even want to disable this plugin alltogether, since it will never be used on-prem. Is there a way to stop the plugin from loading, or in the very least make sure the log does not appear? Right now, it is polluting the system log file quite a lot.
I second that. That was the key message of my bug report. That is we must not even instantiate cloud-specific plugins if no cloud provider is configured.
I'm having exactly the same problem, doing an installation from scratch on DigitalOcean
+1
+1
Hi I have the same issue. It would be really nice to know how to disable this error message that floods the logs for no reason.
For those using logstash it was sufficient to just temporarily add this to your syslog filter: if [message] =~ "" { drop { } }
After a fresh installation of Raspbian and kubernetes, same issue is occurring in my 4 node cluster.
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/arm"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/arm"} flood of this: epsilon kubelet[2412]: E1024 12:25:58. 2412 ] failed to get azure cloud in GetVolumeLimits, epsilon
disable in controller plane should work: "--feature-gates=AttachVolumeLimit=false"
I have tried "--feature-gates=AttachVolumeLimit=false" and it's not working. I still see messages kubelet[]: E1024 22:51:02. ] failed to get azure cloud in GetVolumeLimits,
I have the same problem here.., ] failed to get azure cloud in GetVolumeLimits,
+1 same case here, also tried with --feature-gates=AttachVolumeLimit=false
Will this be disabled by default in the next release?
It's already fixed in v1.12.2 and v1.13.0. do you know how to disable in kubelet in v1.12? Would add in kubelet command arguments work?
I haven't updated to 1.12.2 yet (still on 1.12.1) but just to confirm, disabling that feature works. For those that are also running on CentOS and/or RHEL, just to the following: Edit file on your master server (control plane), and add the following: If you already have extra arguments in there, just add this one at the end of the line. Then restart kubelet on your master server:
Yes that will work.
--feature-gates=AttachVolumeLimit property doesn't exist in the documentation
Update tho 1.12.2 resolve for me Em qui, 1 de nov de 2018 9:22 PM, Abhilash Pallerlamudi < escreveu:
add follow line to "your kubelet.service" file. like this: [root ~]# systemctl daemon-reload [root ~]# sudo systemctl restart kubelet then it's ok
So to get rid of a bug we should disable an entire feature? Says it is supported by Azure-Disk. Why isn't node reporting the limit? It is getting instance metadata okay as all my nodes are tagged with SKU. Being that SKU determines attach limit as well, I do not understand why this cannot be made working? I am on 1.14 in Azure on Ubuntu.
how do you deploy k8s on Azure? To get volume limit, kubelet on agent node needs azure identity, e.g. service principal, usually it's under file
I deployed with kubespray. I have cloudconfig set as such: KUBELETCLOUDPROVIDER="--cloud-provider=azure --cloud-config=/etc/kubernetes/cloudconfig" Service principle is set in cloudconfig file.
I triple checked and location was misspelled on some of the nodes for the cloud provider. Thanks!
the get vm size list call depends on location, so now it works?
Yes it does. Thanks!
In our scalability tests it sometimes happens that all tests are passing, but the run is marked red as "failure". This is because uploading logs to GCS sometimes failed with error: I've seen 2 examples during last 24 hours: For now it's just misleading. However, since scalability suite is now pretty stable, I would like to add it to list of projects that our merge bot requires to be green to merge anything. Unfortunately this looks like a blocker for it. Any thoughts how to fix that? cc
cc
Another example of it:
Yes, I think this is a good idea. It should be relatively trivial to mask the uploading error from Jenkins (in the script). We do ideally need some sort of warning to alert us if/when the uploads are failing repeatedly (which could be done in a separate PR). Ideally it would email the oncall, or automatically (re-)open and assign a github ticket. do you have time to look at this next week? Anyone else? If not, I will. Tentatively assigning to myself, although hoping that someone else will pick it up.
We're using a Jenkins plugin, not the e2e scripts, to upload build results to GCS. There's no way that I can find to make it not fail the build if an upload fails. Side note: the actual error message is a 401 error from GCS which seems to suggest an actual issue in the underlying infrastructure somewhere, which probably shouldn't be overlooked, but I have no idea where to start to try to figure which team to notify.
That said, it should be pretty straightforward to write a script to upload the artifacts and build log, rather than relying on the plugin.
Yes - that would be great if you could do it...
Just bumping it up - it happened again few times over the weekend and also today:
I looked at the existing plugin that we're using, and it does indeed retry (5 times!) Also searched the logs and found a lot of GCS errors: $ grep GoogleJsonResponseException: uniq -c 50 401 Unauthorized 22 500 Internal Server Error 12 503 Service Unavailable So I don't really think that there is much more that we can do here other than help GCS to track down their errors, and increase the retry count (although 5 with exponential backoff already seems rather excessive).
Thanks for looking into it and ! Is it easy to increase the retry count? Can we do it (I agree that 5 doesn't seem low, but we still observe errors). Did you talk to GCS people? I wouldn't expect the number of errors to be such big...
No, the retry count is unfortunately not configurable. I have not yet spoke to the GCS people, but plan to.
^
did we ever follow up with the GCS folks? I'm working on a PR to upload test artifacts/logs all within itself.
sorry, no.
- why did you close this? IIUC the PR is still not merged.
Yes, sorry, you're right. I'll assign this one to to close once the PR is merged.
I enabled the new script for -scalability, but it will take a few hours to verify that it works as intended. I haven't disabled the plugin yet, but assuming everything works as intended, I'll do that during the day PDT Friday.
- thanks!
- this doesn't work at all - in fact the cluster doesn't even start after this change
- I reverted your change Can you please experiment on something different than -scalability - if this doesn't work it blocks merge bot for ~2.5 hours which is how long a single run lasts. Can you please experiment on something that is not blocking merge-bot first? FYI:
Ugh, sorry about that. It'd been working fine on a number of builds, but all of those cleaned their workspaces before running tests. should fix the issue, and I'll test it on one of the flaky builds first.
I saw this in the kubelet logs: and were working from the host. describe node showed that it had stopped posting node status: tcpdump showed requests to the IAM metadata endpoint, but it showed no requests to the EC2 local-ipv4 / public-ipv4 endpoints. Restarting kubelet caused it to recover, and we saw requests to the local-ipv4 / public-ipv4 endpoints again.
Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.
do you want this in v1.7? is this the right sig to own this issue?
I a richer log message in to help diagnostics. I'm not sure if this behaviour is wrong actually. The kubelet goes offline, the node gets evicted etc. The only challenge is if the metadata service goes down cluster-wide. We don't want to just shut-down. When we have sig/cloud we can talk about it there I guess.
/remove-priority P0 /priority critical-urgent (I'm not actually sure this is the right priority, just trying to remove the old priority/PN labels)
Thanks - punting to 1.8 as I don't think this behaviour is necessarily wrong, though it is suboptimal if (for some reason) the metadata service went down AWS-wide.
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close Taking an educated guess that this isn't going to get fixed given that we've seen no additional reports of this happening here. Please /reopen if this is still happening
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign
We should solve this problem in a better way
cc :))
/sig contributor-experience release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/assign
http://k8s- /priority failing-test /kind bug /status approved-for-milestone /sig cluster-lifecycle the job is on the master-blocking dashboard and the e2e test has been failing, please take a look cc
testgrid change suggests
(author of the change in that range) -- any idea whether might have broken reboot tests?
I reverted the change as it may lead to this regression. The other change is really small () and unlikely related to this. Still the continues to say otherwise, I think we can reopen this issue.
The reboot test is still failing.
Just opened to achieve the revert.
Is the test passing now? Can this issue be closed?
looks green now