jiebi/CodeConvo huggingface dataset:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example in , I created a pod with 3 sysctls specified in the pod's security context. Setting the sysctl resulted in the pod failing to start. What you expected to happen: I expected the pod to start with no issues. If there is something missing in the docs, I appreciate updating There is also a discussion at that suggests that kernel does not namespace certain net.* sysctls. If that's the case, can we document which sysctls are supported? I assumed is a supported sysctl because it's in the official k8s docs. How to reproduce it (as minimally and precisely as possible): Following the pod creation, the pod doesn't start: Once I delete the following two lines from the pod securityContext, the pod starts fine: Anything else we need to know?: You can see that I've enabled the unsafe sysctls with kubelet: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VMware Fusion OS (e.g: ):Kernel (e.g. ):Install tools: A single-node k8s cluster created with kubeadm Network plugin and version (if this is a network-related bug): Calico Others:
/sig node
Thank you for the incredibly detailed issue report :) I did some poking around, and if is accurate and still current, then I believe the issue is that is not namespaced (i.e. cannot be set on a per pod/container level). If I'm understanding correctly, we'd need to make changes to the Linux kernel in order for it to be namespaced. As a disclaimer, I'm not positive if this parameter became namespaced in new kernels... it could be interesting to investigate if this still happens on a machine with the latest kernel. W.r.t. fixes... I would agree with you that short term, we need to update this documentation so that it doesn't include . do you have any interest in doing so? If not, I'm happy to.
Long term, I think it would be great to have better documentation re which sysctl parameters are namespaced (and thus supported) and which ones are not, and also a more descriptive error message if you try and utilize a non-supported parameter.
This historic issue also seems relevant: Perhaps we were premature to close it.
Matt, thanks for confirming the problem! I found the following comments very helpful: Looking at , there doesn't seem to be any difference between Linux 4.15 and 5.1. I'm happy to remove the references to in the docs, but before doing so, should we confirm sysctl worked at some point in the past? As you said, we should enhance the error message and the docs with the supported sysctls. I understand documenting the namespaced sysctls can be a moving target, but not everyone likes to check the Linux source code to see what's namespaced or not. :)
It's probably better to change the docs to set a namespaced sysctl like or . A good way to figure out what network sysctls are namespaced is to pass to kubelet and see all the sysctls listed under inside the pod.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
Agree with all your statements here :) I'm also surprised that ever worked, but perhaps we could sync with the original author of the docs? +1 to in the meantime changing the docs to a namespaced sysctl.
I don't think ever worked as it's hardly to believe a sysctl got unnamespacified in the kernel (it's always the other way around). is a good example of namespaced networking sysctl. However, the above suggested is not in some cases - it's unnamespaced in kernel 4.14 and became namespaced in 4.15. In general, this is hard to document because there are hundreds of parameters and some of them is kernel version dependent and you always need to lookup the kernel code for answers. I think what we should do now is to with in the example in the doc. in the doc to something like . I think this should be very close ( is an exception) - I couldn't verify all of them as there are so many sysctls and they can also be by loading kernel modules.
SGTM, can you drive that?
Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): HEAD Since v1.7.0-alpha.3, control plane images are only pushed for amd64. Compare: to This is a release-blocker and should be fixed ASAP. I guess the actual regression code-wise happened in test infra. cc
/assign
also cc
/remove-sig testing
Still looking, but "Only tag release containers if the name is different." makes me suspicious.
oh, this is a bug introduced by . fix should be easy.
I can open a new issue as well to track what I sig-testing for but I want us to set up the required testing to make sure this never happens again
the removal was under the guise of "sig testing doesn't control the cutting of bits", if there are tests that need to be put in place to ensure all the right things are cut/released, I would expect sig-release to track that eg: if there's code that needs to be changed in kubernetes/release, it's a sig-release issue; if there's code that needs to be changed in kubernetes/test-infra, it might be a sig-testing issue (or we can triage to the right group)
Sounds good
Details about the exact bug and the fix are in . The reason we didn't notice this sooner is that anago only if the required image doesn't exist - it does so because we don't support all arches for all release versions, and we didn't want to hard-code a list of expected versions into the release repo. (Also, anago automatically , so we were still pushing the legacy versions - we were only dropping the non-amd64 images with this bug.)
Can we add a after the warns? Otherwise even is the user of anago sees the warning, there's nothing he can do about it.
Anago is moving toward being less interactive. If any checks need to be done, they should be done at build time. 1) Determine what should be built 2) Stop right there if the build fails for whatever reason. Adding redundant checks comes with the cost of maintaining hardcoded lists in multiple places. The push step (and release infra in general) should be agnostic and just push what was built.
But in this issue, the build was successful, but some images were removed due to the bug.
I'm worried that when building beta.2 this Wednesday, if a similar problem happens, i won't be able to stop anago from pushing the incomplete image sets.
I'm reopening this unless we are really sure that it actually works...
How can we create the testing required to stop this from happening again?
I'm doing a dry run today.
anago could have some sort of sanity checking, though I'm not sure how we'd do that without maintaining hardcoded lists, as notes. I guess the "real" fix is having some automated testing somewhere to consume the non-x86 docker images.
After anago dry run, i saw these images on my machine, so it looks fixed.
Thanks and :tada:!
The replicas is 3 but just create 1 pod 'web-0' and one pvc, then kube-controller-manager panic happend. Should create 3 pod and 3 pvc add '--feature-gates=StatefulSetAutoDeletePVC=true' arg for kube-controller-manager, then create statefulset with 2 replicas and volumeClaimTemplates. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig apps
I can reproduce the issue and want to fix it. It seems like variables cause . /assign
/triage accepted
Great find! Does this only happen on an upgraded cluster? I'll spin up a cluster and see if the e2e test catches this problem.
The e2e tests are still passing on a 1.24 cluster, so I suspect this is an upgrade problem. I wonder if there are other holes here.
I installed the Kubernetes 1.25 cluster directly using and still reproduced this issue, so it was probably not caused by upgrading the cluster.
Interesting. I thought that would set the policy to be non-nil. Let's keep this bug open after you submit your PR as I think something else is going on here---I don't think we should be seeing the null policy in the first place.
I suspect that somewhere accidentally set it to nil, and I'll keep looking at this code.
I tried the following (and also with parallel pod management), and it didn't seem to cause the panic. This was a 1.24 cluster, maybe this is only happening on head versions?
I just tried this on a kind cluster running on k8s head, and the problem didn't seem to reproduce. But this was setting the feature flag default in I wonder----does the api server need to have the feature gate as well? That would explain why defaulting wasn't happening.
As far as I'm concerned,I just set the in kube-controller-manager manifests to reproduce this issue. I repeat the yaml used as follows: I tested that in statefulset yaml, setting or not would cause the component to panic. The panic logs are the same as above, so ignore them here.
statefulset with : The environment
I believe you---this doesn't seem to be well-documented, but it's my understanding that you need to set the feature gate on all of the system components. Obviously failing to do so shouldn't cause a panic, so thank you again for finding & fixing this bug! Let me ask around to see what the expected behavior is. I know there may be skew between kubelets and master versions, but I didn't expect to have to deal with skew between master components.
I read the code, and now I understand what you mean. This feature gate must be enabled in both the and components. We need to describe it in the to prevent someone from ignoring the component, which also needs to enable the feature gate.
I don't think we need all the system components, like the component, right? I read the original KEP documentation. This feature gate only involves components and :
I had a bit of discussion about this with others. Because upgrades don't occur atomically, any feature gate has to work regardless of how the other system components are configured (things shouldn't panic!). I suppose if the cluster administrator wants to track what needs to be enabled on each feature they can try to sequence which component has which feature gate. If it were me I'd try to make all the components consistent though.
This is certainly a method, but it is a bit more cumbersome for the administrator. It is best to have a place for the document to be queried by the administrator and avoid confusion. I don't think everyone is very familiar with the k8s mechanism. I suggest adding a partial description to this section of the : Before: After: I can submit this PR on .
This is going to be true of any api change. When you tried to use it with only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?).
I've seen other feature gates that say what components are needed to enable it to use the new feature. For example: If it's not necessary, then I close .
only adding the controller manager feature gate, I think you would be able to set the retain policy, but there wouldn't be any defaulting or validation (?). I remember just enabling it in kube-controller-manager.
it using the retain policy, but the statefulset does not have the policy from output of (It should be the apiserver authentication forbids it)
Yeah I think that makes sense. Thanks for finding this issue! After your PR with the panic fix is submitted, I think we can close this?
Yes. I agree. /close
Closing this issue. <detailsIn response to : Yes. I agree. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Follow up from: See details there. When is , there is no need to discover the NUMA topology. /sig node /priority backlog /kind feature /help-wanted
/cc
/triage accepted
I would like to take this up also can I get more brief information about it? code review comments from asking that inside the NewManager() implementation, we need to skip NewPolicyOptions and NewNUMAInfo but when the NumaTopology comes kubelet issue also comes. I need some more information to start with this!
/cc
When there is no topology manger policy specified, NewManager should be a no-op. Just skip all checks.
Exactly. Just to add: we may need few iterations over the change to get a clean flow in the call.
/assign
Event for a pod: The cni plugins do exist at /opt/cni/bin. They cannot be executed and bash is complaining:
cc
I'm guessing this is a shared lib problem, since /usr/grte/v4/lib64/ld-linux-x86- doesn't exist on my gce vm. We should update docs to make it clear how someone would install manually (if they run into such problems again) and statically link (if the problem is indeed the shared lib) with something like Unsure how we'd support different GOOS (vend different tarballs vs checking in and building the plugin inline with our make scripts)
uggh. These binaries need to be ultra-static or else we need to containerize them.
Essentially doing would've prevented this bug :)
And see for the CNI_ARGS problems
also , re ... however, I'm not too sure if your comment was mean to be on this ticket.
cc
I have a working version. I uploaded my cni binary with bug fix on GCS. change source to change sourcehash to Welcome to give it a try.
Also see problem:
<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: kubernetes/node-problem-detector/node-problem-detector-ci: ci-npd-e2e-node See the Which test(s) are failing: [] NodeProblemDetector [NodeFeature:NodeProblemDetector] [] SystemLogMonitor should generate node condition and events for corresponding errors Since when has it been failing: Dec 03. was at 19:50 PST, was at 21:51 PST. Testgrid link: https://k8s- Reason for failure: Apparently is the culprit. Anything else we need to know: This is likely caused by some library upgrade. Nothing big.
/sig node
/assign
Seems like the failure is caused by . /cc /cc /cc /cc
/priority critical-urgent
The particular bug I care about has been fixed in . I'm gonna still work on submitting , as the same error should also be affecting a few other tests.
PR introduced an annotation for fluentd migration from static pod to the daemonset to avoid two instances of fluentd on a single node during the transition stage. I didn't read the pr carefully, but based on the code at , every node object has one label be default: "". I think this is wrong. 1) Not everyone use fluentd as the logging agent. 2) It is hard-coded in Kubelet code as part of kubeletnodestatus. No matter running fluentd as a static pod or a daemonset on the node, Kubelet shouldn't treat it different from other pods. The right approach is introducing an external controller to apply this cloud-provider / kubernetes vendor (i.e. GKE) specific label to the node. cc/
The string "fluentd" shouldn't appear anywhere in our code. Please remove it ASAP.
Absolutely agree with you, thanks for bringing this up! Is it possible to apply labels to the node as part of the node startup script/salt?
To clarify, I'm thinking about possible ways to provide a fix for that ASAP. Writing external controller is a best solution, but here we have a release coming and this label is intended to be only a temporary measure to perform upgrade from using fluentd as static pods to using daemon set.
There are many ways to achieve this outside the core system. But I believe GKE already has an existing mechanism to add this kind of labels to the node, and the consistency is important for GKE management. cc/
OK, thanks! However, I believe we should be sure not to break everyone's upgrade process, i.e. after master upgrade to 1.6 on any platform, there should be only one fluentd agent on each node. I propose changing the configure scripts and salt to set up needed labels in kubelet's flag (). WDYT?
It seems to me like upgrades could survive without the explicit dependency on fluentd: I'm not sure if upgrades call initialNode or not, my own ignorance ? If they do, and this is removed, can't an external controller just do the work of re--adding that label if its dropped?
thanks for raising this. Let's sync f2f
cc
WDYT about exposing kubelet version as a label and have a selector in fluentd ds config targeting version 1.6+?
and I talked about this issue a bit for migrating kube-proxy. One idea we came up with was that of running a controller that will identify kubelet versions and apply taints to nodes running static pods. The daemonset version would not tolerate that taint and that way we keep daemon set pods off of nodes running static pods. Have you considered that approach?
It doesn't differ from a controller applying labels. I'd like to avoid necessity of introducing such controller, since it's not clear what is it's responsibility (generic controller for applying labels on nodes based on some rules? a custom controller just for fluentd purposes). In GKE we can do it on GKE control plane side, but we need a story for open source.
Move applying labels to startup scripts/salt seems to be the least intrusive and easiest to implement way.
Actually what I proposed in is not an option since nodeSelector in PodSpec is old style selector.
I'm not a big fan of approach with salt, but it seems there is no better option.
Re: Instead of exposing Kubelet version as a label, why not using field selector. Kubelet version is part of NodeStatus already. I just simply don't want to have a duplicate info for the same object.
is it possible to schedule based on fieldselector? I'm only aware of using field selector while listing objects.
Also node-labels option is in alpha
But it's present since 1.2 What is the status of this flag? Can we use it for fluentd purposes or better avoid this?
Maybe it does make sense to implement a temporary ad-hoc controller? It's the easiest to deploy solution, just another in fluentd addon.
I know. It totally doesn't make sense to introduce anything custom. I'll use just the flag. It's rather hacky but unfortunately I can't see anything better.
Cool thing about controller is that you can deploy it into any cluster and it will work, with CA also. You don't have to set up your node labels manually then or restart the cluster.
There is a number of open question with having a new controller: it run on the master or a node? we guarantee that it will be always running? it have any resource requirements? I'd like to avoid discussing it right before the release.
Having a controller will help with kube-proxy transition too in the future.
Files storing latest versions for minors are not updated since infix was to the version string. Example Similar issues can be found in 1.25 and 1.24 builds (last 1.23 build infix so it has not happened there yet): I expect the files to be overwritten with newer version of the build. Manually run build job for one of the minors. No response Not related. Not related. No response No response No response No response
I believe this may be also responsible for breaking kubemark tests: since they fail due to version mismatch
/sig release
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority high Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for raising the issue! /cc who is working on before
I assume this will be fixed automatically with the next patches, because then we would compare with . Anyway, I think we should fix that intermediately by manually updating the version markers to match their correct version. I did that now for the bucket
We had to update the versions markers again, because those were pointing to two commits earlier and didn't include the Go updates. The latest version markers are: v1.24.12-35+ v1.25.8-37+ v1.26.3-41+
latest-1.27 has been affected by this as well. I updated it manually to v1.27.0-9+. See this thread for more details:
/triage accepted
This should be fixed now, closing it. Feel free to reopen if we missed anything.
This issue is still present, see this Slack thread for more details: /reopen
Reopened this issue. <detailsIn response to : This issue is still present, see this Slack thread for more details: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a similar situation with latest-1.28 which is currently pointing to: . This is one commit before the release commit: ! It's mostly like that the release step didn't update version markers, we should look into this.
As of now, I'm going to manually update this version marker to unblock 1.28.1 patch release. I'll also take a look into this. /assign
The PR that fixed this issue got reverted so this is still relevant: /reopen
Reopened this issue. <detailsIn response to : The PR that fixed this issue got reverted so this is still relevant: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're running into this issue again with stable-1.29 marker. It's currently pointing to which is It should instead point to I'm going to manually change it following the same procedure as the last time. Version marker updated to
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Same issue for 1.30, where I had to update the version marker manually again.
is a bash script linter which could help catch common problems in our (numerous) bash scripts. I know we'd like to reduce our use of bash scripts, but until then I think being stricter about script quality would be helpful. /cc
cc you were talking about this earlier. It has to only check modified lines.
False positives are a problem. For instance, not being able to follow a call through an environment variable causes an error. I would be much more comfortable with a non-blocking way to point out these issues, and leave to it human judgement.
Yeah, I've been interested in doing this for a while, but can be pretty persnickety and I haven't yet taken the time to configure it to be less noisy (or figure out if it can be). It's good, though.
Yeah, it would definitely need to be configured to ignore a lot of types of errors, but I still think it could be used to check for some types of errors. We could start with a pretty minimal set.
I would settle for something that just produced a quick-fix format artifact in the GCS bucket, and then the PR munger reading from it and commenting on the PR. We could re-use that for other linters too.
has been pretty accurate historically.
shellcheck, not spellcheck
reading too many issues in a on
/sig testing /priority awaiting-more-evidence Might be more noise than it's worth, but we do pylint our .py scripts now
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .