jiebi/CodeConvo huggingface dataset:
with timeout is used for graceful termination of 'running pods' for feature. We should not re-schedule pod to not-ready unless tolerating it explicitly.
reopen this for more discussion :)
Let me extracted the mixed problems in this issue :-) First one: These two default tolerations attached to the pod by means: if the node is tainted as or , wait for 300s before the pod is evicted from the node. While according to the issue description, is working as expected (pods are not evicted immediately). This part should be closed. Second one: The discussion raised is actually about taint during eviction: if TaintNodesByCondition is enabled, we should also taint node with NotReady:NoSchedule. My thoughts is: it's reasonable to taint node with as well for consistency consideration (but I'm hesitate if only for eliminating confusion as pods are expected to pending on NotReady nodes) I think it should be clearer now.
I open a PR If TaintNodesByCondition is enable, taint node with NotReady:NoSchedule and Unreachable:NoSchedule.
, we did not catch 1.11 release, so this issue will be handled in 1.12 :)
in 1.9.7enable the TaintNodesByCondition and taint nodee.g. node01 with NotReady:NoSchedulethen pod also can not be scheduled to node01 even when node01 is healthy. What you expected to happen: The pod should be scheduled to node01 when it is healthy node and not be scheduled to node01 when it is not ready. Thanks
How did you do that? by kubectl? The fix is not ready until 1.12.
YesI tried "kubectl taint nodes node01 ",and also tried "kubectl taint nodes node01 "
this feature did not handle taint manually
Ok, thanks a lot! So I can not solve the problem manually now and what I should do is just waiting for the version 1.12, right?
This issue proposes the writing of a new test for the API endponts , , , , , and with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these query results from APIsnoop, we are lacking in tests for the following endpoints related to Events We chose the following 5 endpoints for this mock test: - - - Test outline: Event in namespace default with custom fields all events in all namespaces find event created in in response of the content of the event created in events by name in namespace default ensure that the event updated from event by name all events in namespace ensure event in namespace default was deleted This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents Display endpoints hit by the test Show the change in coverage after the mock test If a test with these calls gets merged in, conformance will go up by 5 endpoints. /sig architecture /area conformance
lgtm according to the SIG-Arch conformance office hours 17th December 2019.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen can you put the checkboxes at the top for your apisnoop org pr, the test, and link to the promotion PR? I think we are ready.
/open
Reopened this issue. <detailsIn response to : /reopen can you put the checkboxes at the top for your apisnoop org pr, the test, and link to the promotion PR? I think we are ready. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Promote CoreV1Event test - +5 coverage
/assign
This test was written before the tickets were stored in APISnoop branches. Unsure what to do with it in regards to the APISnoop PR
After , we observed pods moving from phase to . Presents on master and release-1.8 as it was picked in test script has an command for basically doing a CLI watch or you can look in the kubelet log to see the phase transition from to Thanks to for helping isolate this :+1: /sig node
/assign
taking a look. although at first glance it is strange this didnt happen with evicted pods prior to this.
Thanks. FYI, I did confirm that reverting the mentioned PR on master corrected the issue.
This showed up breaking a job controller that depended on the pod not transitioning back to pending. It's possible this can cause applications to fail / run multiple times in potentially dangerous ways. I'm not sure we should allow a status update that moves back to a non terminal state via validation. It's not a legal transition as we have defined phase. We should separately put a guard in place that will report a clear error to clients.
after thinking on it, I think that the situation with eviction is a little different in that the containers are terminated but they are not immediately cleaned up. It is the proximity between pod termination time and container removal time that is uncovering the issue, I think. AFAICT, some path is attempting to get the container status after the container has been removed. When this fails, something in the status manager gets messed up and it under the impression that the pod is new and the status needs to be reinitialized.
Here are the relevant logs showing the bug: Still trying to figure out what caused the status update.
Ok, I figured out why this is happening. When we compute a container's status, we assume that . So deleting a pod's containers before the pod object is removed causes the state to return to pending.
Maybe we should expand this that I to prevent init container status reset to include all containers with an oldStatus
Would that prevent updating a container that crashed and restarted? Edit: nvm, I think that might work
I a comment at
i also commented here that it would be good if syncPod in status manager protected against an invalid transfer of phase... see: can we do that in this same PR?
My hope is to cherrypick this to 1.8. Is that check something we want to cherrypick as well?
Istio folks have a iptables image here that is quite small In the past, we did talk about this, for example wrote it well what we could try: We could even do for running the hack script we have for switching the symlinks when vs : kube-proxy image is definitely one of the main causes of the cost we incur in our GCR repository, so we should definitely give this a shot again. /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is a sig/network-only issue. A gives; The looks like this (narroved): If I don't misinterpret this, K8s uses a unified server-image build and it should not be different for . The size of the binary for amd64 in is; It is stripped and can't easily be "made smaller" I guess.
We (Istio) would definitely be interested in sharing ownership (hopefully by upsteaming to distroless mainline) in the image we have if there is interest. Some other projects have expressed similar needs as well
!
kube-proxy make up 9% of current image data traffic
I threw together this little repo (based on our next-gen distroless infrastructure): It pushed which on x64 is roughly another 5x smaller (musl vs. glibc): If folks are interested in this, I'd be happy to create a repo under which is where we've been standing up new images, or you can create your own from pretty easily.
debian based please!! we have KEPs and stuff written that lay down that direction:
For a possible approach + above in that issue and for "distroless" debian based images with arbitrary C/C++ binaries + deps. Note that we need things besides just the distro binaries (see comment linked). and is pretty straightforward for one way to handle the debian binaries part.
I don't even know how we build the kube-proxy image. Where does that happen and form which dockerfile?
Note we need iptables, iptables-save and iptables-restore but also ipvsadm, shell, conntrack, ipset, and maybe other tools
It used to be in kubernetes/kubernetes under build/images, it's now in (Dockerfile etc there)
That's the base - where is the logic to turn that into a kube-proxy image?
Turning that image into a kube-proxy image is just adding the kube-proxy binary with a dockerfile and entirely too much bash which ultimately just does: (except api-server, where we do a setcap in )
/triage accepted Will take a look between this and next week
It doesn't even need bash; the version of the script in intentionally avoids bash-specific features (and gets tested against Alpine where /bin/sh is busybox (which I guess is "ash"?)). And the version in kubernetes/release appears to be just a stripped-down version of that script. (But actually, you should update the script to be based on the latest version anyway, to get the new logic in , in preparation for .)
Probably per one of the things we don't want to do right now is jump into a musl based image like Alpine, right? So I would take the case that we want to use distroless, and copy all the binaries we need to inside that container (conntrack, ipvs, iptables-*, etc etc). per iptables-wrapper, you said it's not bash specific so maybe we can use distroless + the smaller available shell? I will check some approaches here before jumping into changing the release/dockerfiles/whatever scripts, and check if I can run some comformance tests in a distroless k/proxy. Maybe starting with a good binary list is a way to cross check the binaries + libraries required for this distroless container. I did a list with the installed packages per kproxy needs, and some ldd on it: Will start working on something to assemble an image just with the binaries and required libraries, and check size + libraries
I agree no musl/alpine On Sun, Apr 17, 2022, 11:38 AM Ricardo Katz
If you identify the list of binaries, has logic to handle ldd for a binary (e.g. ignore vdso) + keep the copyright and package version info via dpkg.
We probably need a few more not in the list above for the other proxy modes?
Thanks this is gonna be very helpful! I will work on it using the script you provided during the week!
/cc
From script and some changes: There's a pretty good reduction of size (50%). But this is without kproxy binary, that is almost 40Mb. I will open a Draft PR in k/release so we can discuss over there the approach. BTW, anyone knows why kube-proxy binary is that big? Never digged into that, but curious about it.
https://dr- !
(updated with conntrack-tools) Why? The binary is statically linked so it does not need or use , it only needs some binaries to be executable and it doesn't matter if they are built with musl. They just have to work. I have sucessfully built and executed my own tests, both with proxy-mode and , using an image built with this : The image size on cluster including the binary (~40MB) is; (I use a private local registry) Compare to;
what about the different iptables-nft vs iptables-legacy binaries?
I have assumed that takes care of that (executed by the Dockerfile).
Re size: Are DWARF2 symbols stripped? Ideally k8s should produce artifacts without DWARF2 symbols and distribute them separatelly. Or produce the optional -debug artifacts. This can cut GCR/GCS costs, but introduce some debug/profiling complexity. The Go symbol table should be always included. Related
IIRC wrapper in prod needs both binaries existing in the container, and will use this in a way to check if the host machine is using iptables-legacy or iptables-nft.
Both exist in Alpine; The first example in is Alpine. can you please confirm that it works with Alpine?
I have altered my testenv so I can switch between legacy and nft and verified that the tests work with both and that shows the same in the container and the K8s node, e.g;
Side remark; to my suprise is not needed in the kube-proxy image
Tested the with K8s e2e;
There are / we're legal reasons to avoid Alpine. Have those been resolved? I haven't heard anything.
On Tue, Apr 19, 2022, 11:31 AM Lars Ekman
i don't think so. it's status quo on Alpine which is to avoid it totally.
Can I read about those somewhere? Are the legal reasons about Alpine? Or about musl?
Not my personal decision, and I'm not a lawyer let alone an OSS license expert but ... sharing what I've heard on the subject: IIRC for example Alpine does not make available the source code (AFAICT Alpine publishes source code for the packaging scripts, but not the package sources), or even the LICENSE of binary packages, which is problematic for compliance with licenses ... (Various common license require including the license text, and / or making source code available). Normal alpine images don't appear to install these doc packages: So those images don't even ship with license / copyright info files for packages which is ... not a great start. I'm not sure where a good public reference is for this, Google OSPO has some relevant docs internally about following license requirements when producing container images. So far the best public doc on the topic I've found is
Thanks!
TL:DR;
I wasn't suggesting switching to Alpine, I was just saying that I unit test the iptables-wrappers scripts against Alpine, which doesn't use bash, and so I know the scripts don't require bash. Yeah, I don't know the details of how distroless works, but if we're talking about debian packages then it looks like is 1/10 the size of (212k vs 2285k)
In Alpine (and Debian, and probably most distros other than Fedora), the "iptables" package includes both the legacy and nft binaries, so that's the only dependency.
Made a comment on bandwidth that may be better in this issue rahter than the PR;
Want to give a quick update on this (EDIT: I wrote this was going to be quick update, but ended up not being! sorry!): The image generation is working "fine", see: This process generates a temporary that can be used later with and/or changed in Kubernetes code to trigger the tests (see ) Due to the way iptables-wrapper-installer works, it calls iptables-legacy and iptables-nft to check if they are working fine.
We need to skip it (via --skip-integrity-check) in arch not amd64, as those are really specific kernel features and when called from arm64, as an example the command ends in an error. We need /bin/sleep to be present in distroless as well, due specifically to this image being used in this test: cloud-build timeout needs to be increased, it takes a long time for arm/s390/etc to build this The image can be optimized a bit more, removing some steps like linking the iptables binaries (as pointed by and removing some binaries like wc (as pointed by ) but right now I didn't wanted to change the image build AND the iptables-wrapper script. My take here would be: let's see now with sig-release if the build process is fine, get this image promoted and used before v1.25 All the tests but verify ran fine (verify due to dependency check, etc etc) - Here are the main tests with the distroless image: kind - kind ipv6 - gce ipvs - performance 100 nodes - fyi. I can bring this to next sig-net meeting, wanted to check: from sig/release perspective, is the PR fine? I will make some more changes here, but wanted even to check if this should be called debian-iptables:distroless or something like from sig/net if we want to move forward migrating the iptables-wrapper to shell script this release or not. I tend of agree with Rob that maybe we are getting too many changes in one release, and we can plan this for v1.26 (the PR is already there, I need to check Dan's comments only) Thanks
/assign
/reopen let's close it once we switch to the new image
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Could you suggest if it's ok to use already? That would be super cool to use the image as a base for NodeLocalDNS (as it uses iptables). I already did manual smoke test in cluster - all is working correctly. Thanks!
Let's introduce it gradually, I suggest to wait until the image is promoted for kube-proxy
I've lost track of this, what do we need to do to promote it for kube-proxy?
I think Ricardo was on it But if someone is busy and others have time, let's sync next monday on slack and organize the work, so someone can get this to the finish line, there is no need to overload one person if there are other wanting to help
I didn’t saw there was a comment there. I can fix that, and wanted to plan with you folks next steps. I agree with Antonio and don’t want to block this on my availability :) I will check suggestion tomorrow and as we get this merged I’m more than happy to handle to someone else! On Sat, 16 Jul 2022 at 07:21 Antonio Ojea
I've opened a new PR to do the proper promotion. As soon as this is promoted I think we can start testing and migrating to distroless :) From here I think I can handle to someone else as I wont have much time and attention to this specific effort and I know this is important :) I'm still willing to migrate iptables-wrapper to go code (we started it in kubernetes-sigs/iptables-wrapper) so I will probably finish fixing Dan's reviews and we can discuss the distroless v0.2.0 being based entirely in Go :)
we should probably nudge a few of the subprojects to migrate, we just migrated KIND but I'm sure we have at least a few more out there like ip-masq-agent
I'm migrating nodelocaldns. I've got a couple of questions: nodelocaldns uses registry . Should I switch it to ? Is considered sort of a prod image registry? a multi-arch image? Because currently we use is it better to keep up with the new tags for ? So that we could quickly upgrade base image version as well? Thanks for the work!
YES! Make happy! This registry is actually a "proxy" that sig-k8s-infra is maintaining so people get directed to the closest Image registry, based on their cloud provider. As an example, with this users in AWS wont need to go to GCR to pull their image Yes, it is. It follows the same build pattern of debian-iptables. See Well, this is a good question.
AFAIK we don't have an event / notification system that sends a message (in Slack, as an example) whenever a base image gets promoted. It would be cool to have at least for debian-base, debian-iptables and distroless-iptables, probably it's worth to question sig-release on this :) ( maybe know)
Thanks for the answer! Regarding the - we use it as a compile image for dnsmasq. Will this image still be supported or should I find something other to replace this compile image?
Short answer: I imagine/suppose there is no deprecation plan for this image right now, as it is used as the compilation base for a bunch of other things (and it is base of etcd image) Long answer: In case of distroless-iptables the rationale was: we need to reduce costs (and by reducing costs, reducing image size) we need to use something with a smaller footprint and reduce the amount of packages, and as a result the amount of CVEs in image package. kube-proxy exists in all nodes (unless you use a CNI that implements its features) so it is a good candidate I guess right now debian-base will still be used (I don't remember on the top of my head if other kubernetes components are already running using distroless or compiled based on debian-base, as far as I can see at least etcd image depends on debian-base () so probably any effort on this image should be announced.
Node 2's kubelet logs: controller-manager side: I have a cluster to hand off if someone wants to debug. It's not a transient state: node 2 just failed to ever post a valid status. cc cc (who looked at it briefly with me)
I will take a look soon.
Ok, why kubelet on node 2 talks to ip address of master instead of hostname? root ps -ef | grep kubelet root 1 3 17:35 ? 00:10:45 /usr/local/bin/kubelet --apiservers=https://10.240.49.47 --config=/etc/kubernetes/manifests --allowprivileged=False --v=2 --clusterdns=10.51.240.10 -- It looks like a cluster up issue to me, not a node problem.
It looks like failed to translate the master name into the fqdn (which is done ).
Fascinating. I think we're seeing instance-DNS flakiness, because that name is supposed to get converted ASAP. But what's weirder is ... that's not even the master:
The is definitely right:
Okay, this is messed up. Here's node 1, which is happy: Here's our unhappy node .... that decided to bring itself as a master:
haha... I checked kube-env and did suspect unhappy node treat itself as master node based on those variables. But I didn't check salt state :-)
I ... just can't get there. By the time we get here: , node-2 has set: My best guess is that flaked here: . I think we're seeing a systemic issue where we're racing with GCE network configuration.
I'm going to work up a PR that hopefully improves the reliability of this section to some extent.
<a href=""<img src="" title="made at "/</a
My working theory is that we raced with the configuration done normally by the GCE NetworkManager, leaving us in a bogus state when we made that check.
Why are we relying on inferring the master/node status from the hostname. It should be obvious from the variables set in the kube-env if you are the master or not. Masters don't get passed so why not just check that? Making sure DNS is working is probably also good for reliability...
On GKE, we actually pass the same kube-env on both sides for simplicity.
Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.