jiebi/CodeConvo huggingface dataset:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When kube-apiserver's livenessProbe suggests that pod is unhealthy, kubelet tries to kill its container with 30 seconds grace period. The container is killed ungracefully with SIGKILL when grace period passes. This happens each time. What you expected to happen: I expected that SIGTERM will be passed to kube-apiserver which will try to gracefully stop before 30 seconds pass (at least in some cases). How to reproduce it (as minimally and precisely as possible): Inject kube-apiserver failure (e.g. ) Watch kubelet logs for container's death You will see that container is killed after 30 seconds each time Anything else we need to know?: I believe the problem is that in kube-apiserver.manifest uses '/bin/sh -c CMD' as a PID1 process which doesn't pass SIGTERM signals to child process. See Environment: Kubernetes version (use ): v1.7.11
/sig node
Looks like without the pipe command will have to be removed which will make the logs hard to track. There is indeed some which makes kube-apiserver run in background and trapping SIGTERM then send a SIGTERM to apiserver's PID.
I think this is the case for all our components, not just kube-apiserver. - FYI
How about using 'exec'-based solution suggested here: In our case it means prepending 'exec': I did changed this in my cluster and the process seems to meet requirements: fd/1 and fd/2 are redirected to /var/log/kube- container is killed instantaneously If everyone is fine with such a solution, I can prepare PR adding 'exec ' in front of all kubernetes/cluster/saltbase/salt/kube-apiserver/*.manifest using '/bin/sh -c CMD' pattern.
Not sure if understand what you mean. Before there was no logs, so /bin/sh -c wasn't needed. -based solution seems to have all required properties. What is your concern about using it?
Umm. just ignore my previous comment. I just tried prepending and it works fine for me.
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 3 Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: All of the salt scripts bundled in kubernetes (for example, ) start up the component in a broken way. They use which means the shell will intercept all signals. So these daemons are not gracefully shutdown when the pod is stopped. This causes serious problems since when apiserver is killed abruptly it doesn't seem to close watches correctly. This was observed in for example. What you expected to happen: Running is incorrect. It should run the binary directly. Otherwise, it needs to perform an so the binary correctly replaces the shell script process. How to reproduce it (as minimally and precisely as possible): Run kube-apiserver inside a container with and try to stop it. You can't - eventually docker will force kill it. Anything else we need to know:
It looks like we changed from running the binaries directly to using in (from a few years ago to enable us to capture logs. or may recall why we had to use a shell since they reviewed that PR.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.
The scheduler has a for the bind operation to succeed; if we don't get a response within 30s, the in-memory assignment of pod to node in the . A race condition will happen in the follow case: 1 is assigned to a node, scheduler cache is updated with the assignment, bind operation issued to apiserver. the apiserver is under huge pressure, bind takes more than 30s, scheduler expires the cached pod-to-node assignment. eventually succeeds, but because the apiserver is under huge pressure, the pod update with the node name takes a long time to propagate to the scheduler. the pod update took a long time to propagate and the cache entry expired, the scheduler is not aware that the assignment actually happened, and so it had no problem assigning a second pod to the same node that would otherwise not fit if the scheduler was aware that the first pod was eventually assigned to the node. On the scheduler side, what we need to do is make the 30s longer for large clusters, and ideally adaptable to cluster state. /sig scheduling No race conditions. Create a large scale cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
IIRC, the 30s timeout only starts running after we receive a 200 response for the binding. But yes, if it takes longer than 30s to receive the update, we would invalidate the cache. I don't know if there is particular reason to invalidate the cache at all. I suppose it was meant to guard against missing Pod deletion events. We need to improve that regardless. I would be in favor of dropping the timeout altogether.
ok, so the deadline is effectively for receiving the update. Perhaps making it longer, like 15min, is good first step before completely removing it just so we can easily revert in case something else goes wrong.
I sent to increase the timeout to 15min.
We should backport this.
We should double check the history, but I think that in the past we were: asynchronously sending bind request updating the cache before sending the request and then the timeout was actually protecting us from the binding request actually not succeeding. So I agree with Aldo - that we should double check if this timeout is at all needed currently.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
We are probably good to remove this timeout altogether at this point. /remove-lifecycle stale
/good-first-issue
I see this issue was labeled as "good first issue" and I would like to start working on it!
I would like to take this task
Plz assign to yourself to notify that you"re working on this.
/assign
/assign
done. thanks!
/assign
Could you please guide for this Issue are you suggesting we should not check cache before assigning the pod and need other way around?
FTR, I'm considering assigned to this issue, as they were the first to get assigned. You can see that in we increased the timeout. We want to get rid of the timeout. Then, we can just get rid of the logic that uses the timeout to remove some pods from the cache.
/reopen We still need to remove the code in 1.27
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/unassign
one more release before we can remove this code.
do you have plan to cherry pick for previous versions, e.g 1.23, 1.24?
did you mean 1.13 and 1.14?
oh, sorry my bad, I meant for 1.23 and 1.24
It is fixed in 1.23 , so it would be fixed in 1.24 too.
my understanding is the previous fix to increase the duration only reduce the chance of issue, but not completely avoid it, that's why the other 2 fixes and are needed, right?
Hi, is this issue still available?
yes, but since the likelihood that an issue happens is very low, I don't think it's worth the risk of cherry-picking at this point.
According to , we can remove the TTL code entirely in 1.27 I'll give the chance to to confirm if they can still do this. In the meantime: /unassign /unassign /remove-help
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /unassign as we haven't heard from them.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
I want to help fix this /assign
Thanks. What is left is to remove the timeout logic altogether.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned".
<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen remove the expire logic
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen remove the expire logic Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we can leave this closed. Entirely removing the timeouts has proven to be close to impossible without introducing regressions. We might need a few more iterations. Maybe we can reconsider when scheduling hints graduates to Stable.
I keep seeing these server rejected event errors every now and then. My guess is it is trying to update an event that was expired in etcd. However I don't expect an error, since there is specific logic in to handle that use case. Any ideas why this may happening? I am using kubernetes 1.2. Here is an example, from a host getting to a cluster:
Found the problem - the recordEvent logic in was expecting the wrong status code. I've submitted a PR for it. This could be reproduced by: 1 - Update load balancers by remove a node, then adding it back 2 - Remove all events manually from etcd for that namespace 3 - Remove node and add it back again - this results in server rejected event error
What would you like to be : A comment in that includes the possible values of . I got the possible values from . The should contain the further explanation available in the linked code that explains what each condition type means. Why is this needed: Currently the output of is incomplete: It is important to include the values of because that's the value used to . It is not user-friendly to require users to guess the possible string values for condition or require them to read the code to deduce what the possible values are.
/sig cli
Please let me do it~
oops, I just notice your message. I already submitted a PR for this.
It's ok~
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: param is not validated in func Validate What you expected to happen: validate admission plugins, if not registered, return err to prevent going on until call . How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig auth
/kind bug
from the user's perspective, a reasonable error is returned
On AWS we (will) now run our bootstrap script on every boot. The function to initialize ephemeral volumes doesn't tolerate being re-run, and we probably shouldn't re-run it anyway for speed. This does preserve the contents, which may be a good thing to save user's data (it doesn't belong on ephemeral drives, but...). On the other hand, if the ephemeral drives are corrupted, we lose the opportunity to reformat them and recover. Arguably, if the ephemeral storage is corrupted the right thing to do is to terminate the instance entirely.
https://k8s- https://k8s- https://k8s- https://k8s- https://k8s- ... A lot of them /cc
/assign
Still see this failure after is merged, although less frequently: https://k8s- https://k8s- Reopen.
Looks like the client doesn't differentiate between 500 and 503 here: It should. Will open a fix shortly
opened
fix merged, will monitor https://k8s-
Thanks! :)
haven't seen this in the 12 hours since the fix merged, was seeing it every 2-3 hours before. /close
I have an extension apiserver, and I was trying to upgrading the dependency . In our code, we have a serval resources has its , and we expect the storage version is their . for example, we have a resource in both and group, and they are wired up by method, and expected encoded as to storage. However, this seems broken after upgrade to v1.31, so I did some invertgation by myself, and I find the storage version is changed by here: In old behavior honor , but now don't. storage version semantic of would be honored. there is not easy approach, need to create an extension api server and find out. No response <detailsClient Version: v1.30.1 Kustomize Version: v5.0.4-0.- Server Version: v1.31.0-beta.0.25+-dirty </details<detailsNONE </details<detailsKind Cluster run on Darwin Darwin ReficuldeMacBook- 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:34 PDT 2023; root:xnu-.1.13~1/RELEASEARM64_T8103 arm64 </details<detailskind </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
PTAL cc
I noticed that kube-controller-manager terminated with a concurrent map access: No concurrent map access, and no fatal error. No reproducer. I noticed this in an internal CI run. This appears to be an obvious (in retrospect) logic error with a straightforward solution, so I assume that it isn't necessary to provide a reproducer. This is from an OpenShift cluster based on Kubernetes 1.27. AWS. Red Hat Enterprise Linux CoreOS 414.92.-0. No response No response No response
/sig network
/assign
/triage accepted
Is this a BUG REPORT or FEATURE REQUEST?: feature request What happened: As far, GPU and hugepages are particular resources which reqiure limits must equal to requests. But this rule may be broken when creating LimitRange, LimitRange with different values in default and defaultRequest can be created successfully, which would cause error in creating pods afterwards. We should not leave this error to pod creation but tell the users when creating the LimitRange. What you expected to happen: Validate if values are proper for GPU and hugepages when creating LimitRange. How to reproduce it (as minimally and precisely as possible): LimitRange with the following yaml: pods with the following yaml: would prompt error like: maybe we should let this error happen in creating LimitRange. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.1.1879+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-01T07:53:05Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.1.1879+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-01T07:50:07Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others: /cc would this a possible feature?
/sig node
I have committed PR fixing this issue, please check when you are free :)
Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.
I am running kubernetes 0.5.2 + and I've noticed that the number of entries in iptables increases every few seconds. I started with a clean install, no pods or replicationControllers, there is now two services by default (kubernetes-ro and kubernetes). I ran and got . Next, I turned up 7 services and 3 replicationControllers which resulted in 3 pods also being turned up. I started a watch... I saw "38", "43, "50, "64", "66", "68", "70", ..., "115" and on until I stopped . It appears the proxy continues to add REDIRECT's for the services, for example the lines that keep being look like: A similar line for each service gets every second or so, it appears to all be in the table. I'm thinking the newly function might be the culprit, since calls and in the log I see every few seconds (looks like does a sync every 5 seconds).
I noticed that the 0.5.3 and 0.5.4 releases don't include the , though it's in master. Perhaps the issue I've filed is already known and I missed it when searching the issues.
I have not heard this issue. Can you paste a listing of iptables-save -t nat when you are seeing this problem? Alternatively, I can cut you a branch with some extra debug instrumentation if you're willing to test it out today...
I'm not able to test it out today. As far as the iptables output, I'd have to type it in (can't copy/paste -- different physical network). However, the output has lines, like the one given in the issue, which are every 5 seconds to the "nat" table. It seems like the check to see if the rule already exists is perhaps not functioning properly. I am working on a fix for (because it uses the iptables -C flag which isn't supported in older iptables), in that fix I created a function to check if an existing rule already exists: Example call: Using may not be ideal, perhaps breaking up the output of the iptables command into a slice and checking each line would be better(?) sure yet.
Fix is pending - would be great if someone could verify it against an actually old iptables (I just faked the version string).