jiebi/CodeConvo huggingface dataset:
</details
I see just one CSI test timing out in serial job Is that something the fixes here attempted to resolve? if not I can open a new issue if it persists and close this one out. Thanks.
The test mentioned in my comment above was a flake.. closing this issue now. will open a new issue for the flake
I'm getting the following related error on a busy Node: In the above case it's a full filesystem, but I imagine the same error would pop up with a RO filesystem as well. Looking at the patch, I'd expect kubelet would still be failing to start when it tries to watch it. (above was from a v1.12.3 cluster) , what do you reckon?
plugin watcher fixes for 1.12 are expected to be released with 1.12.5:
, I understand that I'm missing that patch. The patch code, however, would not make a difference in this case, if I read it correctly. It would not prevent the recursive dive into mounted volumes, as the errors happening at the deeper levels of that dive.
The patch blacklists the plugin watcher from searching the directory
OK, I was looking at the wrong patch after all (). Thanks for the clarifications.
master-informing: capz-windows-master ! 07-10 17:46 IST No response /sig cli cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test
Looking at the failure and tracing the recent code changes, the test was recently modified for the regex pattern match as part of this PR which comes with deprecation of the golang-based short version information in favor of making the --short the new default. But the output of kubectl version is still reflecting the older long format which is causing this test to fail.
/assign
Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: If you are using a mac and GCE and you specify NODELOCALSSDS=0 when using GCE via kube-up or e2e --up, then you end up getting 2 local SSDs. What you expected to happen: I would expect to get 0 local SSDs by default and when I specify NODELOCALSSDS=0 explicitly. How to reproduce it (as minimally and precisely as possible): On mac execute: NODELOCALSSDS=0 go run -- -v --up Anything else we need to know?: I've discovered that this issue is due to If NODELOCALSSDS=0, this evaluates to $(seq 0) From man seq on mac osx This was run on mac with the seq manpage indicating it comes from BSD Feb 19 2010. Note: some unix/linux variants, e.g. a test on ubuntu with coreutils 8.25 and a test on centos-7 with coreutils 8.22, had a different behavior that does not demonstrate this bug. Environment: Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): mac osx
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
If the namespace controller gets an error when doing a sync and the error is not a error the error will not be handled. Ran in to this when we found When deleting a namespace it received a error that wasn't reported anywhere and the namespace stuck in because a finalizer was left.
Other things noticed in the namespace controller: need a crash handler in sweep for all error return cases and determine whether should be returned determine what to do with non- errors at ... I think that currently it just hangs termination of that namespace until a resync
I will take a look. Thanks for the heads up.
reported a variety of failure modes in .
+1
I figured I'd add to this issue rather than create a new one. I attempted to create a vagrant cluster today (dev tree is up to date) with 1 Master and 1 Minion and the cluster creation is stuck in the 'Validating master' loop ( for over 10 minutes in the validating loop ). salt-master and salt-minion logs seem to be empty ( they don't appear to be running, in fact ).
I still have the cluster running in case anybody wants additional information.
/cc please let me know if there is any other information about my vagrant setup that I need to post here.
I have been working to get more stability in this environment over last few days. Can you try again from HEAD after this merges:
Can you also verify that you built a release prior to running cluster/Kube-up?
yes, I started from scratch and built a release before running cluster/kube- - cluster/kube- seems to fail fairly early in case there is no release build available.
It looks like got merged. I'll try again and get back to you.
The vagrant cluster seems to come up ( completes master and minion validation ). However, there is new credentials related error that is showing up at the end of running cluster/kube- :
I'll add one more problem I've encountered. Runing from head:
New to me. Will investigate more tomorrow. Sent from my iPhone
Also new to me :-) - will see if I can reproduce either. Sent from my iPhone
Your credendials and certificates are not working with new environment. Just delete your .kubernetesvagrant* files and recreate the cluster. It would be nice to automate this task, if you tear down the cluster.
I am able to reproduce the 502 Bad Gateway error and am working on determining what changed in order to provide a resolution over the last couple days. Please assign this issue to me.
The issue appears to me that the kube-apiserver is not getting provisioned by salt correctly. Parsing Salt logs to determine why.
Looks like the file is not getting laid down by salt for some reason ;-)
The Jinja template was changed as part of identifying the cluster. Fixing up now, and will send a PR to close this issue.
that was it, thanks!
What happened: In the PMEM-CSI CI, we rapidly deploy and remove a custom resources which then owns all objects created for that resource. Deletion of those objects is handled through ownership references, i.e. deleting the CR is expected to cascade to all objects. This worked fine up to Kubernetes 1.17. When starting to test with 1.18.2, we pretty reliably get into a state where the CR and the service objects owned by it are marked for deletion, but don't go away. Debugging with the garbage collector's dependency graph (see ) showed that endpointslices were blocking the deletion of the services and thus the CR. In the graph dump (= a snaphot in time), they were not even marked for deletion. A closer look showed that endpointslices for the services were getting created and deleted continuously for over 15 minutes. Eventually, the garbage collector won and managed to delete endpointslices and services before new endpointslices were created. What you expected to happen: There seems to be a missing check in if the service is marked for deletion (i.e. has a deletionTimestamp), then the controller should not create new endpointslices. How to reproduce it (as minimally and precisely as possible): I tried to boil it down to a script, but it just ran fine. The "easiest" way to reproduce it is to run PMEM-CSI's E2E testing, which is probably not easy enough :disappointed: Anything else we need to know?: Disabling the EndpointSlice feature makes the problem go away. Environment: Kubernetes version (use ): 1.18.2 Cloud provider or hardware configuration: QEMU virtual machines OS (e.g: ): Fedora Install tools: kubeadm /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
cc for endpointslice
Thanks for the detailed bug report! Will work on getting a fix in soon. /assign
/remove-triage unresolved
Can you describe the general process to reproduce this bug? Would it involve hundreds of EndpointSlices for a Service? I've tried to recreate this in a , but unfortunately I haven't been able to get anything to break. Any ideas for what I could try?
There's one endpointslice per service. Our operator creates two services.
One peculiarity may be that we don't actually want the pods to start during these tests because we only want to verify object creation, not the usage of the result, and therefore all pods reference non-existent images. All objects immediately get deleted once the test has verified that they exist, so the timing is probably very different from "normal" usage patterns. I suspect that it is more important to create and delete services quickly than it is to have many endpointslices per service.
I had tried to come up with a script that triggers the problem, without success. That was before I realized that endpointslices are involved. On Monday I can have another look.
That was because I thought that (the default) enables foreground deletion. It doesn't, and there's also no option to select it. After explicitly selecting it via curl, the following script failed for me eventually: Output: Note that for the endpointslice is greater than the of its service.
What happened: One pod stuck at CreateContainerError and can't get recovered automatically. Kubelet log shows that the requested name (let's say Container2) is already in use. shows the Container2 Running, and it's created before Container1 This should never happen. How to reproduce it (as minimally and precisely as possible): I wish i knew. Anything else we need to know?: I know this can be caused by time shifting, but I didn't find any time shifting sign in system log and chrony log, and there is no rebooting between the create timestamp of Container1 and Container2, so this should be caused by something else. This happens intermittently in our env, like 3 times in half a year, and I didn't find any ime shifting sign each time. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:20:18Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-08-28T11:18:26Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): Linux node-1 3.10.0-957.21.3.el7.x8664 SMP Wed Jul 17 06:57:50 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig node
Any update on this issue, we are also facing the same issue. EKS version : 1.14.9 Docker version: 19.03.6-ce AWS CNI: 1.6.2 Intermittently we are also facing this , what might the issue, could please guide us here? Below are the kubelet logs. )"
with CreatePodSandboxError: "CreatePodSandbox for pod "-f0fe-11ea-9872-)" failed: rpc error: code = Unknown desc = failed to create a sandbox for pod "aiqx-server-dep--b9lmv": operation timeout: context deadline exceeded" Sep 07 11:40:40 kubelet[2899]: W0907 11:40:40. 2899 ] No ref for container "docker://7b9c86b529530f15d941f8e68973a202c8f348ab5db864a8371e8c82ddbf9dc4" (r(-e845-11ea-bd96-) Sep 07 11:40:43 kubelet[2899]: W0907 11:40:43. 2899 ] Unable to create pod sandbox due to conflict. Attempting to remove sandbox "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8" Sep 07 11:40:43 kubelet[2899]: E0907 11:40:43. 2899 ] Failed to remove the conflicting container "a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8": Error: No such container: a73bd9a03d56bc7855e4e89e163e13c5e7173397f10edcc200fb686e1b6885c8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .
/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is here to satisfy PR requirements of issue association. A lot of times we need to do version bumps for minor bug fixes that don't deserve a separate issue in the kubernetes/kubernetes repo. /kind bug /sig node
/lifecycle frozen
all the linked PRs are emrged and no activity since more than 3 years /close
Closing this issue. <detailsIn response to : all the linked PRs are emrged and no activity since more than 3 years /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
committed a changed version of go into godeps. Was this deliberate? I don't see any discussion of it ( I've almost done this by mistake as well, so I'm inclined to think it was just an oversight. (I think this is giving me errors building, but I'm not yet 100% sure this is the cause)
Hmm, this is a good point. It does look to be accidental. I don't know. I think and would have context though I believe they are out— do you know offhand whether this is probably an accident? I guess were talking specifically about in the .
I believe it's an accident.
Thanks, The linked PR should fix this (assuming I don't have to do anything else with Godeps).
This was almost certainly an accident. Whenever I've updated godeps it's changed automatically to 1.3 and I've had to adjust it back manually.
Actually, jeff lowdermilk was super helpful and showed me how to update the deps on this PR. it was his commit which I merged into my PR, so not sure it it was accidental or not. Fortunately, now that k8petstore is in the go build, we will know if it breaks anything when you revert the version back. Thanks for noticing this. , Alex Robinson wrote:
If I have probe and then all good. Previous version of the pod is terminated after startup probe success, but if wasn't specified the pod becomes ready immediately. It would be nice to add new state (for example starting) which would have pods until a is successful. Also, it would mean that container not ready for connection. Thank you for attention!
/sig cluster-lifecycle
/remove-sig cluster-lifecycle /sig apps try asking in #kubernetes-users on k8s slack.
they are silent in the chat. Who is involved in the development of startup probe?
i think this topic better fits SIG Node. try asking in their channel on slack , but be patient ;) /sig node /remove-sig apps
Those labels are not set on the issue: <detailsIn response to : i think this topic better fits SIG Node. /sig node /remove-sig apps </details
I think this is a feature request, not support, so I changed labels. On its face it SEEMS reasonable, but I am not close enough to the details to say. This is definitely sig-node.
Adding a new pod lifecycle state has been determined to be intractable in the past. Is the proposed change keeping the pod in !Ready until the first success from ?
Where do you usually discuss this? Just wondering. Yes, I want that pod considered not ready until will not be successful. If readiness probe doesn't present then a pod can become ready. How can a pod be ready if it is not running?
how long wait?
Looking at this issue, I can confirm it, at least superficially - a pod with a perma-failing is "ready" if there is no readinessGate defined and "not ready" if it is defined. The question I have is whether we think that is "correct" or not? Initially I thought it was correct, but the more I think about it the less convinced I am. If I have defined a startupProbe and it is not passing, my app should not be ready. Does anyone disagree?
Also, the app should not be ready in status. It's funny, but without a readiness probe, pods in the final state have the status ready and accept requests. This is unexpected.
Whether a pod is ready while terminating is an entirely different question, IMO.
Apps that server network traffic probably need to "drain" and to do that they need to stay "ready" for a while as upstream LBs de-program. Let's not conflate issues, please.
The fix seems simple - lets see what CI says.
Sorry, I don't spend much time on slack... you could have checked the release notes in 1.18 ;-) Tim's fix seems good, thanks!
/reopen Until we merge all related PRs...
Reopened this issue. <detailsIn response to : /reopen Until we merge all related PRs... Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All merged. /close
Closing this issue. <detailsIn response to : All merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When service is set with ILB annotation and empty string subnet, e.g. The service would be pending forever with following error: What you expected to happen: ILB should be provisioned successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug /assign
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When attempting to upgrade kubernetes using kubeadm upgrade apply, kubeadm fails to parse an etcd imagetag containing build metadata in the form of as a valid semver. An example upgrade produces an error similar to: Note this will occur even if the etcd version is unchanged. The +` character. This pattern is used to support custom kubernetes images, for instance kube-apiserver:v1.20.4+patch.0 will be converted to imagetag v1.20.4_patch.0 when running make quick-release-images. Add the following to kubeadm- Install kubernetes. Then upgrade kubernetes with kubeadm upgrade apply. I have tested a fix for this issue and will PR shortly. Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: kubeadm Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cluster-lifecycle
/area kubeadm
/triage accepted
In asked the following question. While the user-facing documentation answers this question clearly enough, John was reading the HELP string in the Prometheus scrapes. That thing only says the following. The metric HELP strings should be clear enough. Scrape or read the code (). No response <detailsThis problem started when we generalized to allow width other than 1 and got worse when we introduced borrowing. </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
/assign
Actually, this metric should be removed. It is a duplicate of but with an inferior name.
Looks good (Sorry I never got around to logging this issue myself. Have been super busy. And your wording is better anyway :-) )
/triage accepted Thank you.
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I visited the Kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. The server returned a 404. What you expected to happen: I expected to navigate to the article on resizing persistent volumes in Kubernetes. How to reproduce it (as minimally and precisely as possible): Visit the release announcement blog post (), scroll down to the "4 Day Features Blog Series" section, and click on the link to the article on resizing persistent volumes. Anything else we need to know?: The link currently points to: The correct link is: Environment: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A
Reiterating the mentions to trigger a notification: <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! --/kind bug /kind feature What happened: I visited the Kubernetes 1.11 release blog post and clicked on the link to the article on resizing persistent volumes. The server returned a 404. I expected to navigate to the article on resizing persistent volumes in Kubernetes. Visit the release announcement blog post (), scroll down to the "4 Day Features Blog Series" section, and click on the link to the article on resizing persistent volumes. The link currently points to: The correct link is: Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It should be reported to kubernetes/website, but doesn't matter.
is failing with: Examples:
This is the failing line of code...
PASS: FAIL: Seems like this disappeared: Also this is flipped: we run the following
my leading theory is this is something in
The issue is that the above PR changed a bunch of to be and thus we no longer wait for getArchive() to complete before starting tests. And unfortunately the previous PR cannot be reverted
After merging, kubelet-gce-e2e-ci still failed with the same issue. reopen the issue. Currently the failure blocked the merge-bot.
did stop the bleeding. Now kubelet-gce-e2e-ci is green: and merge-bot is unblocked.
I think the "go arc" piece was a red herring. I suspect the issue is that the same PR tries to copy the binary into the archive directory, but PR stops building ginkgo, so the copy fails. That is why this line then fails...