jiebi/CodeConvo huggingface dataset:
After KCM restart, any Pod that reuses a volume that failed to detach gets stuck ContainerCreating with message: User "system:node:ip-10-0-243-" cannot get resource "volumeattachments" in API group "" at the cluster scope: no relationship found between node 'ip-10-0-243-' and this object See steps to reproduce below, it's quite a complex condition. It happens when a pod is restarted, a CSI driver "busy" (i.e. times out) + KCM restarted at a wrong time. Pods start just fine. a Pod that uses an attachable CSI volume. the Pod. Make sure it unmounts successfully, but any fails with a transient error, such as timeout. The external attacher continues to detach the volume at this point. kube-controller-manager (KCM) and start a new one. Wait for any leader election etc - it's important for the new KCM to populate A/D controller with no pod using the detaching volume. Let the A/D controller to observe detach error. a new Pod on the same node that uses the same volume. the CSI driver to detach the volume (i.e. stop injecting any transient errors). At this point: The external attacher succeeds detaching the volume, VolumeAttachment is deleted. The volume is marked as attached in node status (!!!) despite no VolumeAttachment exists. A/D controller does not even try to create a new VolumeAttachment, the volume is marked as attached in its Actual State of World (ASW). Kubelet tries to mount the volume, but WaitForAttach fails with: , basically forever. I created . The script expects that the cluster is created by , because it restarts KCM in the middle. It uses my hacked hostpath CSI driver to inject detach errors. Sources are at / the last commit(s) there. There are two issues that together lead to this situation: fails, , i.e. puts it back to . With the volume in , kubelet may try to mount it, while the external attacher tries to detach it. It is not as bad as it sounds, because kubelet can't read VolumeAttachment with the deletion timestamp and it will not call / of un-attached volume.
KCM starts, all VolumeAttachments that don't have a Pod (that's good), but also (that's probably bad). When the newly started KCM re-tries to detach the volume and gets an error again, it marks the volume as fully attached (same as the previous point) and adds it to . Now the volume is both mounted + attached in ASW and A/D controller actually stops here and waits forever for kubelet to unmount the volume. <detailsToday's k/k master, </details<detailsNone </details<detailsN/A, it's KCM issue. </details<details, but seen in a downstream Kubernetes distro too. </details<details</details<details</details
/sig storage
/assign
/triage accepted
I have the same problem: Seems restart of Kubelet solves this problem. Kubernetes v1.25.13
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In the test failed due to namespace leakage: we created e2e-tests-load-30-nodepods-1-pfn9m, but never got response from api server (got 0-length response instead) we retried and created e2e-tests-load-30-nodepods-1-m9tq5 e2e-tests-load-30-nodepods-1-m9tq5 to f.namespacesToDelete and leaked -pfn9m next test case failed because e2e-tests-load-30-nodepods-1-pfn9m wasn't deleted. What you expected to happen: While it's unclear why we got 0-length response instead ( investigates that), we always can get error while the namespace was created (e.g. connection reset, timeout, io error). We should handle this case in retrying correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.0-alpha.0.793+ Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
Hello, the SRC folks asked that I file this publicly: In the client-go repo, , function creates a private key and stores it into a file with mode : Typically keys are created with more restrictive modes to prevent other users or services on the system from being able to copy them before the user fixes the permissions. Of course, if a more restrictive umask is in place, this may not be a big deal, but it's an extremely fickle piece of the environment to rely upon for security. It's better to be explicit. Thanks
/kind bug Not sure if this is /sig api-machinery or /sig auth
is this related to us?
Hi IMO, mode make sense for the key file. I would like to work on this, waiting for Thanks!!
/triage accepted
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/lifecycle frozen /good-first-issue I think changing the code to use should be fine. Anyone is free to pick this up. Would be good to include a unit test.
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /lifecycle frozen /good-first-issue Instructions for interacting with me using PR comments are available .
attemting to run throws and error with ERROR: Error: - does not specify a valid ReplicationController
The documentation is a little ambiguous, as the second example suggests that - should work but the Synopsis section does not. Could you either fix the documentation or the command?
Calvin, can you post the contents of , and the schema of the original rc? There are some for the schema when providing a target from file, which may not be met.
running works, I was interpreting the error message that it though - was a rc
will work for any object schema file. I.e, that is the same command you use to create a podreplicationcontroller from file. The type being created is determined by the contents of the file. Looking back at the code, the error you saw indicates that does not contain a replicationcontroller spec. If you can create it, that must mean it's something else; can you post/link the contents?
yes I need to clean it up first
That looks like a valid rc spec, but the labels may be insufficient for rollingupdate. In order to do a rollingupdate you need to have a label to control the update. Typically this would be a "version" label. Example Basically the selector for the old and new rc must share at least one label key with a differing value. By the way, rollingupdate is currently getting a simplified version, which generates the new rc spec for you. This is wip that will be available in next week's GKE release. See
ah I see, I actually have a dynamic version in there so that will be easy to do thanks Jeff Lowdermilk wrote:
so with an updated config still does not work while does work with the first one still erroring out with so likely my missing selector issue was an unrelated issue
From your last comment, seems this is resolved. If you are still having trouble, lmk and I will reopen.
still not fixed because I can't do
I'm pretty sure that's a piping issue with the wrapper command that likely doesn't occur if you use kubectl directly. Try Note that steps 1,2 will be unnecessary with the next gcloud release; you will be able to use kubectl directly after creating a cluster.
With cloudsdk 0.9.60, standalone kubectl works by default with GKE clusters.
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Currently when using it adds as one of the limiters. This does not work as intended because there is no immediate sleep following the call to the limiter as seen in which does work correctly. The issue with the in its current usage is there is no forget nor a sleep immediately following the call so if is called thousands of times in a row, the timeout grows very quickly with no max. For example, if a clusters has 10k secrets and those are all enqueued using the the will end up with a 16 min wait before the tail end of objects can be enqueued and processed. I would expect some max backoff to take affect. This can be seen at
/sig api-machinery
/cc /triage accepted
it possible to set maxWaitingTime for BucketRateLimiter? when r.Limiter.Reserve().Delay() maxWaitingTime, the return of BucketRateLimiter When() is maxWaitingTime. defined is 1000ms / r.limit * MAXQPS(5000) this issue example (r.tokens = 100, r.limit = 10), if call When() times, the tail end of the object is enqueued waiting for 8.33 min. i'm not sure how to set the value of MAXQPS please help review it
The fix merged to close this issue does not fix the underlying issue with the DefaultControllerRateLimiter. The bucket is not being used correctly so this does not work as intended, please read my issue description again as to what is actually happening.
is another fix needed to solve this issue?
A token bucket rate limiter will only refill at the defined rate. If enough requeues are , it will take a long time for the refill rate to clear the queue. If a max wait is introduced, then requeues will be inserted into the queue faster than the refill rate of the token bucket rate limiter. A max wait will violate the token bucket rate limiter and we want token bucket rate limiting behavior.
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
Failed: https://k8s- Run so broken it didn't make JUnit output!
this appears to have always been broken, I'm going to disable the job for now. https://k8s-
Failed: https://k8s- Run so broken it didn't make JUnit output!
What happened: After creating Kubernetes service on a Kubernetes cluster built on Azure IaaS (not AKS), nodes which are not master nodes but are also not worker nodes and which are labeled with are included in the backend pool for the load balancer. What you expected to happen: All nodes labeled with should be excluded from the load balancer's backend pool. How to reproduce it (as minimally and precisely as possible): Create a node with a type other than and label it with Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-05-20T14:00:52Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-05-20T13:51:56Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Azure OS (e.g: ): CentOS Linux release 7.8.2003 (Core) Kernel (e.g. ): Linux master- 3.10.0-1127.8.2.el7.x8664 SMP Tue May 12 16:57:42 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Custom terraform stack Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry. ‍
/priority important-soon /sig cloud-provider /area provider/azure
/priority critical-urgent
I followed the document to create an e2e cluster However, doesn't work after the cluster is up. I got the following error: /cc
/cc
where are you trying to ssh from? Can you ssh to a Linux instance in GCE? If you run do you see ssh keys there? Do any of them match a public key that you have under ? There could be a delay the very first time you try to ssh; we noted this in , but we could add it to the README too. If you try a second time after a couple minutes does it still fail?
Yeah I can ssh to a linux instance in the same cluster. It still fails after a long. I've removed all my ssh keys from cloud metadata, and recreating a cluster. Let me see how it goes this time.
I still got the same error after recreating the cluster. I can ssh into linux nodes in the same cluster just fine.
Keep running but not crashing Have a long volume attach path This corresponds to these lines We are using k0s <details</detailsbare metal <details</details<details</details<details</details<details</details
I turned to use the JSON logger, it is gorgeous no wonder it failed
related
/sig node storage
/cc
VolumeToAttach contains Pod pointers, so yes, I think we should not log the entire struct, and only log important parts to it. /triage accepted /help-wanted
/priority important-longterm /assign
As planned for 1.12 in kubernetes/features: After , the next step is to add the logic to the reconciler to detach the volumes when the nodeShutdown taint is applied. /assign /sig storage /kind feature looping
How do you know that node is Shutdown? is it a planned shutdown or a unplanned shutdown?
AFAIK regardless of whether it's planned or not, it's detected by the cloud provider. In case of bare-metal node it doesn't happen automatically, as there is no universal way to reliably determine if a node is powered off, so the node must be tainted as by some external source.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
any update on this issue?
, I am currently working with sig-node team on handling node shut down. I will make a proposal and discuss in sig-node meeting next Tuesday. Will post the update soon.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /kind bug /priority critical-urgent /assign /sig node
Those labels are not set on the issue: <detailsIn response to : /remove-kind feature Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
any update on this? is it going to be part of 1.15?
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .
Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/priority important-longterm
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/triage accepted /kind feature
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this can be closed now as we have non-graceful node shutdown.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s- Which test(s) are failing: should unmount if pod is force deleted while kubelet is down [Disruptive][Slow] Since when has it been failing: 11/07/18 15:02 Testgrid link: https://k8s- Reason for failure: The test case stops and starts kubelet, and it seems like kubelet is unable to start again Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
cc
Thanks for opening this Looking at , looks like could be the cause? /assign can you please look into this as priority. This is failing critical release blocking tests. /cc /sig cli /priority critical-urgent /milestone v1.13
It looks like kubelet keeps restarting. The error right before restarting is:
/assign
This bug could have been exposed by the newly tests in
Hmm, so I see two errors: Which is a Fatal error and would prevent Kubelet from starting anyways. In which case this error is handled correctly by the pluginwatcher, but I'm not sure preventing Kubelet from starting is the desired behavior. Should we log this error and skip it? There also seem to be a cleanup issue here on that file is this expected? This one is clearly a bug and not really handled correctly. It's due to the fact that we encounter an error at start, cleanup but don't wait for the gorountines to finish before cleaning up. I can send in a fix pretty easily for the second issue, basically add a waitgroup in the . For the first error, what do we think is the expected behavior?
This was mentioned in as well. /cc
Is this related to the issue where plugin watcher is recursively scanning for sockets?
Also, it should be noted that the path it's erroring on is not a socket type.
That's a bit different, as it is intended to scan recursively. Also note that the error here is produced by . What should be our behavior in such a case? Ignore that error?
I'm not sure if we want to be recursively scanning. The volume directories are mount points, which means there could be lots of files/directories existing in those volumes.
maybe we can put a limit on depth of plugin directory for on the same PR:
Ideally the directory that the plugin sockets are dropped in should be different from the directory where mounts happens. Currently the plugin socket directory is: ) The mounts directory is (so far so good). Problem is the "global mount" directory is also :( So options are: recursively scanning like suggested, and say plugins will can only be directly under This will break existing plugins, for example, the GCP plugin is at (see ). recursively scanning but change the plugin socket directory to ? This will also break existing plugins, like the GCP one pointed out above. the plugin dir the same, and leave recursion in, but put a limit on depth of plugin directory as suggested by This is ugly but may be the cleanest solution.
This conversation is completely different from the bug surfaced by this issue. Let's move that to the corresponding issue: I have a fix ready to go out for the panic, we should figure out what behavior we expect Kubelet to have when encountering this error from filepath.Walk: Should we ignore it?
Ignoring the error (with a warning maybe) instead of failing all of kubelet seems reasonable to me. If the plugin did not setup their registration socket correctly, then the end result should be that the plugin fails to register, but all other plugins should continue to work correctly.
the latest runs of this job no longer timesout - Instead I am seeing a new CSI related failure Is this expected. Do we want to track that failure in this bug now or do we have a separate issue?
the latest test logs indicate some test-infra issue:
the same shows up as a timeout on that particular test. Lets wait for the next run. but somehow the timeout seems to have gone. Are we still planning to merge and associated with this issue?
should be merged as it fixes a bug. can you help with approval? Thanks!
are you missing the right PR# in the comment above?
this is the correct PR -
/reopen Keeping it open until we can see the job pass a couple of rounds
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.