jiebi/CodeConvo huggingface dataset:
SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp: lookup kubernetes-master on 169.254.169.254:53: no such host 46m 46m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: i/o timeout 45m 45m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedMount MountVolume.SetUp failed for volume "" (spec.Name: "default-token-kjoqg") pod "-b5bb-11e6-bab0-" (UID: "-b5bb-11e6-bab0-") with: Get https://kubernetes-master/api/v1/namespaces/default/secrets/default-token-kjoqg: dial tcp 10.128.0.2:443: getsockopt: connection refused 11m 11m 2 {controller-manager } Warning FailedMount Failed to attach volume "test" on node "kubernetes-minion-group-yxra" with: error getting instance "kubernetes-minion-group-yxra" 9m 9m 1 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig. This is most likely due to lack of PodCIDR] 9m 9m 2 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: network is not ready: [Kubenet does not have netConfig.
This is most likely due to lack of PodCIDR] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedMount Unable to mount volumes for pod "test-ipj93_default(-b5bb-11e6-bab0-)": timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] 6m 14s 4 {kubelet kubernetes-minion-group-yxra} Warning FailedSync Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "test-ipj93"/"default". list of unattached/unmounted volumes=[data] What you expected to happen: GCE PD to attach and mount successfully post the upgrade How to reproduce it (as minimally and precisely as possible): Running the upgrade steps in the doc for upgrade of manual test from 1.4 to 1.5 Anything else do we need to know:
Ran the upgrade again. This time the .
kube-controller-
Current theory is because kubelet deletes and recreates the node API object during upgrade (because externalID changed), see the new object does not contain the correct status (it is written by the Attach/Detach controller to the old object but not subseqently updated). If this proves to be the issue, the proposed fix is to force the attach/detach controller to update the node status when the node's external ID changes. is working to verify this theory by executing this code path (without a full upgrade). Once he has validated the theory, he will work on the fix, verify it, merge it to master, and get it cherry-picked to 1.5.
: is expected to fix this problem?
: Yes
Tests in are using image for gRPC probe tests: should not be restarted with a GRPC liveness probe should be restarted with a GRPC liveness probe We should use image for these tests as it is much smaller size. /sig node /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue KEP:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi can I work on this issue? I'm new to Kubernetes and want to contribute!
Should I change to in this file?
Yes.
/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.
Kubernetes should resize filesystem on a volume when it's created as a clone of a smaller volume / smaller snapshot is restored into it. Example: takes a snapshot of 1 GiB volume with ext3 filesystem on it. restores the snapshot into 2 GiB volume. 2.1 CSI driver restores the snapshot into a bigger volume, but ext3 filesystem still uses only 1 GiB of space. driver should make sure that the filesystem on the volume has been resized before the volume is mounted into a pod. There are few steps needed: [x] Clarify behavior of CSI call, [ ] Update CSI drivers to the new behavior. [x] Move filesystem resize code from kubernetes/kuberenetes to kubernetes/mount-utils, so all CSI drivers can re-use the same code [x] Enhance the code above to compare the volume size from block device perspective and from filesystem perspective. See for proof of concept. [ ] File issues in all community CSI drivers that we care about & fix them. [ ] Update csi-tests to (optionally?) check for volume size when cloning / restoring snapshot to a bigger size.
/assign /sig storage /priority important-soon
True, considering the CSI spec allows restore to a bigger size volume, we should have the support from CO.
This is hard, Kubernets PV controller overwrites on binding. Therefore the external-provisioner cannot create PVC as "marked for resize on node", i.e. with = = the real size of the new volume) and with = size of the original volume. We either need to change the binding mechanism not to overwrite or add a flag (annotation?) that marks the volume for resize on node, even when .
Talked to The external-provisioner could mark for resize on node: Set to the original volume size. Add condition to the PVC. As result, the extenal-provisioner needs to modify , which needs new RBAC and is a breaking change. The PV controller could respect , if set. We discussed and rejected adding a new flag / annotation to PV by the external-provisioner for kubelet to call NodeExpandVolume on the first mount. Such a flag would need to be cleared by kubelet and kubelet can't modify PVs.
From latest discussion last week, I believe the conclusion was to have external-provisioner set to the smaller size.
Then we just let the rest of the normal resize flow continue. No need to have provisioner set the conditions.
we agreed to set conditions too because we want to be as close as possible to external-resizer. If there is some tooling that depends on condition for resizing pending on node, we don't want to break any such tooling. currently kubelet does not depend on condition for node expansion though.
Why can't we let resize controller set the condition when it sees pvc size != pv size?
hmm good point. I think that should work but we may have to verify that flow.
I think we can do that, but only with a change to the CSI spec wording, because the current implication is that should not be necessary.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
can you summarize the current conclusion/state of this? /remove-lifecycle rotten
Updated the issue with TODO list.
should this issue stay open to keep track of remaining issues?
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle stale /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The issue has been marked as an important bug and triaged. Such issues are automatically marked as frozen when hitting the rotten state to avoid missing important bugs. Please send feedback to sig-contributor-experience at . /lifecycle frozen
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted /priority long-term what is the latest status on this?
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accepted /priority long-term what is the latest status on this? Instructions for interacting with me using PR comments are available .
If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-long-term
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority important-long-term Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I've checked what is the current status of the drivers that were involved in this effort and I think we might be able to close this as all upstream work is done. Please take a look at the table below and let me know if anything is missing.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
container manager setup node routine should ensure that cpu and memory cgroups for container runtime are of the same value, and warn if they are equal to . For background, on systemd environments, the default unit file for has CPU and Memory accounting not enabled, so the /proc/.../cgroup file reports as cpu cgroup.
/cc - I will send a bug-fix for 1.3 for this in next couple days.
/cc
Looking forward to the fix
The PR that fixes this issue was prematurely merged by the merge bot so re-opening.
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In , the controller add NodeNetworkUnavailable to the node conditions, and use to update it. But node status can not be updated by , it doesn't work because replace the node status with old status. To fix this bug, we should use to update node status. /sig cloud-provider /sig cluster-lifecycle /area cloudprovider What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I agree this is a bug. As a matter of interest though, how are you exercising this bug? It seems specific to cloud controller on gce. However that controller isn't actually enabled on gce. It would be good for me to understand your use case, to make sure we don't break it later.
I built my kubernetes cluster with external cloudprovider, but I found if I stop the route controller, pod will be scheduled on node even if the route is not for that node. I read the code and found that the node condition would not be add when using other cloudproviders. At first, I change "gce" in that line code to the name of my cloudprovider, but the problem also exists. Then I found that didn't take effect actually.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Service with clusterIP and a domain , the domain resolve to ingress-nginx. Anothor service will request , when use domain it works fine. I think using clusterIP directly will get better performance because it saves some time for dns resolve and ingress-nginx proxy, so I change domain to clusterIP. But the result was not expected, it got worse, very low throughput and high latency. What you expected to happen: On the above situation, clusterIP should have better performance than ingress-nginx. How to reproduce it (as minimally and precisely as possible): mode, do test with ab Anything else we need to know?: I install an ipvs enviroment manually and do some test, and found it possiblly because ipvs will use iptables to realize ! Environment: Kubernetes version (use ): 1.11.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): centos 7.2 Kernel (e.g. ): 3.10 Install tools: binary Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I try to change some kernel param and found is related to this issue , default . change to then complete about 3000k request, pretty good performance. So, should I change the param in production? Does it have any side effects
/area ipvs
Can you look at it?
, default 1. change to 0then complte about 3000k request, pretty good performance. Yes, seems you find the key to the problem - it's set to 0 in my environment and I have not found any issue so far(it's about IPVS module behavior and we will not find parameter in old kernels ). BTW, you can also try to set the ipvs conntrack() to 0 - it will improve performance as well.
Thanks for reply.
I am using kubernetes , and after set , clusterIP can not be reachable. Which version do I need ?
oops, have you ever configured Cluster-CIDR or iptables masq-all for kube-proxy?
Yes, I configure Cluster-CIDR
Sorry, Cluster IP still needs the IPVS conntrack and nodeport does not rely on it any more - we are trying to figure it out for Cluster IP.
Got it, thanks!
cc Should we tune the kernel parameter in kube-proxy? Given that many users complain about the performance issue, probably the answer is YES.
There is a nice article describing even more kernel parameters to tune ipvs:
Side effect like this comment: <img width="790" alt="hpaprize" src="https://user-"Note: time in picture below is UTC+8. ! Abviously every time scaledown , response time will increase. Any idea for this?
Thanks for reporting. Have you ever read the article posted by In this article, some ipvs parameters can be tuned, e.g.
yes, I had test these parameters not found in centos 7.2(kernel 3.10) will not affect the result is similar with , will be low throughput According to this issue ,I guess it may be the root cause, so I upgrade my setup from 1.11.1 to 1.12.2. But did not solve this problem, and found a new problem, rs delete failed after scaledown. I had open a new issue to this problem:
/assign
Hi, , I raised a pr for this issue. Please check. : ) Looking forward for your advise.
Could you please check out the HEAD to see if your issues have been resolved?
It's not a big surprise this benchmark performs better with . uses http/1.0 so is creating a new connection for every single request. With IPVS will reuse the existing connection entry if source port matches - sending repeated requests to the same real server rather than actually load balancing. So my guess is you will see an unbalanced load with using and , but it makes the benchmark look better. I'm not sure this benchmark is a realistic scenario. I suggest using a tool like instead which uses persistent connections and see what happens.
Apart from the load balancing issue, it's also possible the overhead of is bad when creating connections in a short period of time.