jiebi/CodeConvo huggingface dataset:
Code change:
Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Related discussion:
text: Note we need iptables, iptables-save and iptables-restore but also ipvsadm, shell, conntrack, ipset, and maybe other tools
It used to be in kubernetes/kubernetes under build/images, it's now in (Dockerfile etc there)
That's the base - where is the logic to turn that into a kube-proxy image?
Turning that image into a kube-proxy image is just adding the kube-proxy binary with a dockerfile and entirely too much bash which ultimately just does: (except api-server, where we do a setcap in )
/triage accepted Will take a look between this and next week
It doesn't even need bash; the version of the script in intentionally avoids bash-specific features (and gets tested against Alpine where /bin/sh is busybox (which I guess is "ash"?)). And the version in kubernetes/release appears to be just a stripped-down version of that script. (But actually, you should update the script to be based on the latest version anyway, to get the new logic in , in preparation for .)
Probably per one of the things we don't want to do right now is jump into a musl based image like Alpine, right? So I would take the case that we want to use distroless, and copy all the binaries we need to inside that container (conntrack, ipvs, iptables-*, etc etc). per iptables-wrapper, you said it's not bash specific so maybe we can use distroless + the smaller available shell? I will check some approaches here before jumping into changing the release/dockerfiles/whatever scripts, and check if I can run some comformance tests in a distroless k/proxy. Maybe starting with a good binary list is a way to cross check the binaries + libraries required for this distroless container. I did a list with the installed packages per kproxy needs, and some ldd on it: Will start working on something to assemble an image just with the binaries and required libraries, and check size + libraries
I agree no musl/alpine On Sun, Apr 17, 2022, 11:38 AM Ricardo Katz
If you identify the list of binaries, has logic to handle ldd for a binary (e.g. ignore vdso) + keep the copyright and package version info via dpkg.
Code change:
Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Related discussion:
text: We probably need a few more not in the list above for the other proxy modes?
Thanks this is gonna be very helpful! I will work on it using the script you provided during the week!
/cc
From script and some changes: There's a pretty good reduction of size (50%). But this is without kproxy binary, that is almost 40Mb. I will open a Draft PR in k/release so we can discuss over there the approach. BTW, anyone knows why kube-proxy binary is that big? Never digged into that, but curious about it.
https://dr- !
(updated with conntrack-tools) Why? The binary is statically linked so it does not need or use , it only needs some binaries to be executable and it doesn't matter if they are built with musl. They just have to work. I have sucessfully built and executed my own tests, both with proxy-mode and , using an image built with this : The image size on cluster including the binary (~40MB) is; (I use a private local registry) Compare to;
what about the different iptables-nft vs iptables-legacy binaries?
I have assumed that takes care of that (executed by the Dockerfile).
Re size: Are DWARF2 symbols stripped? Ideally k8s should produce artifacts without DWARF2 symbols and distribute them separatelly. Or produce the optional -debug artifacts. This can cut GCR/GCS costs, but introduce some debug/profiling complexity. The Go symbol table should be always included. Related
IIRC wrapper in prod needs both binaries existing in the container, and will use this in a way to check if the host machine is using iptables-legacy or iptables-nft.
Both exist in Alpine; The first example in is Alpine. can you please confirm that it works with Alpine?
I have altered my testenv so I can switch between legacy and nft and verified that the tests work with both and that shows the same in the container and the K8s node, e.g;
Side remark; to my suprise is not needed in the kube-proxy image
Tested the with K8s e2e;
There are / we're legal reasons to avoid Alpine. Have those been resolved? I haven't heard anything.
Code change:
Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Related discussion:
text: On Tue, Apr 19, 2022, 11:31 AM Lars Ekman
i don't think so. it's status quo on Alpine which is to avoid it totally.
Can I read about those somewhere? Are the legal reasons about Alpine? Or about musl?
Not my personal decision, and I'm not a lawyer let alone an OSS license expert but ... sharing what I've heard on the subject: IIRC for example Alpine does not make available the source code (AFAICT Alpine publishes source code for the packaging scripts, but not the package sources), or even the LICENSE of binary packages, which is problematic for compliance with licenses ... (Various common license require including the license text, and / or making source code available). Normal alpine images don't appear to install these doc packages: So those images don't even ship with license / copyright info files for packages which is ... not a great start. I'm not sure where a good public reference is for this, Google OSPO has some relevant docs internally about following license requirements when producing container images. So far the best public doc on the topic I've found is
Thanks!
TL:DR;
I wasn't suggesting switching to Alpine, I was just saying that I unit test the iptables-wrappers scripts against Alpine, which doesn't use bash, and so I know the scripts don't require bash. Yeah, I don't know the details of how distroless works, but if we're talking about debian packages then it looks like is 1/10 the size of (212k vs 2285k)
In Alpine (and Debian, and probably most distros other than Fedora), the "iptables" package includes both the legacy and nft binaries, so that's the only dependency.
Made a comment on bandwidth that may be better in this issue rahter than the PR;
Want to give a quick update on this (EDIT: I wrote this was going to be quick update, but ended up not being! sorry!): The image generation is working "fine", see: This process generates a temporary that can be used later with and/or changed in Kubernetes code to trigger the tests (see ) Due to the way iptables-wrapper-installer works, it calls iptables-legacy and iptables-nft to check if they are working fine.
Code change:
Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Related discussion:
text: We need to skip it (via --skip-integrity-check) in arch not amd64, as those are really specific kernel features and when called from arm64, as an example the command ends in an error. We need /bin/sleep to be present in distroless as well, due specifically to this image being used in this test: cloud-build timeout needs to be increased, it takes a long time for arm/s390/etc to build this The image can be optimized a bit more, removing some steps like linking the iptables binaries (as pointed by and removing some binaries like wc (as pointed by ) but right now I didn't wanted to change the image build AND the iptables-wrapper script. My take here would be: let's see now with sig-release if the build process is fine, get this image promoted and used before v1.25 All the tests but verify ran fine (verify due to dependency check, etc etc) - Here are the main tests with the distroless image: kind - kind ipv6 - gce ipvs - performance 100 nodes - fyi. I can bring this to next sig-net meeting, wanted to check: from sig/release perspective, is the PR fine? I will make some more changes here, but wanted even to check if this should be called debian-iptables:distroless or something like from sig/net if we want to move forward migrating the iptables-wrapper to shell script this release or not. I tend of agree with Rob that maybe we are getting too many changes in one release, and we can plan this for v1.26 (the PR is already there, I need to check Dan's comments only) Thanks
/assign
/reopen let's close it once we switch to the new image
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Could you suggest if it's ok to use already? That would be super cool to use the image as a base for NodeLocalDNS (as it uses iptables). I already did manual smoke test in cluster - all is working correctly. Thanks!
Code change:
Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Related discussion:
text: Let's introduce it gradually, I suggest to wait until the image is promoted for kube-proxy
I've lost track of this, what do we need to do to promote it for kube-proxy?
I think Ricardo was on it But if someone is busy and others have time, let's sync next monday on slack and organize the work, so someone can get this to the finish line, there is no need to overload one person if there are other wanting to help
I didn’t saw there was a comment there. I can fix that, and wanted to plan with you folks next steps. I agree with Antonio and don’t want to block this on my availability :) I will check suggestion tomorrow and as we get this merged I’m more than happy to handle to someone else! On Sat, 16 Jul 2022 at 07:21 Antonio Ojea
I've opened a new PR to do the proper promotion. As soon as this is promoted I think we can start testing and migrating to distroless :) From here I think I can handle to someone else as I wont have much time and attention to this specific effort and I know this is important :) I'm still willing to migrate iptables-wrapper to go code (we started it in kubernetes-sigs/iptables-wrapper) so I will probably finish fixing Dan's reviews and we can discuss the distroless v0.2.0 being based entirely in Go :)
we should probably nudge a few of the subprojects to migrate, we just migrated KIND but I'm sure we have at least a few more out there like ip-masq-agent
I'm migrating nodelocaldns. I've got a couple of questions: nodelocaldns uses registry . Should I switch it to ? Is considered sort of a prod image registry? a multi-arch image? Because currently we use is it better to keep up with the new tags for ? So that we could quickly upgrade base image version as well? Thanks for the work!
YES! Make happy! This registry is actually a "proxy" that sig-k8s-infra is maintaining so people get directed to the closest Image registry, based on their cloud provider. As an example, with this users in AWS wont need to go to GCR to pull their image Yes, it is. It follows the same build pattern of debian-iptables. See Well, this is a good question.
Code change:
Containers: []v1.Container{ { Name:            "e2e-net-exec", <del> Image:           imageutils.GetE2EImage(imageutils.DebianIptables), </del> <ins> Image:           imageutils.GetE2EImage(imageutils.DistrolessIptables), </ins> ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"sleep", "600"}, SecurityContext: &v1.SecurityContext{

Related discussion:
text: AFAIK we don't have an event / notification system that sends a message (in Slack, as an example) whenever a base image gets promoted. It would be cool to have at least for debian-base, debian-iptables and distroless-iptables, probably it's worth to question sig-release on this :) ( maybe know)
Thanks for the answer! Regarding the - we use it as a compile image for dnsmasq. Will this image still be supported or should I find something other to replace this compile image?
Short answer: I imagine/suppose there is no deprecation plan for this image right now, as it is used as the compilation base for a bunch of other things (and it is base of etcd image) Long answer: In case of distroless-iptables the rationale was: we need to reduce costs (and by reducing costs, reducing image size) we need to use something with a smaller footprint and reduce the amount of packages, and as a result the amount of CVEs in image package. kube-proxy exists in all nodes (unless you use a CNI that implements its features) so it is a good candidate I guess right now debian-base will still be used (I don't remember on the top of my head if other kubernetes components are already running using distroless or compiled based on debian-base, as far as I can see at least etcd image depends on debian-base () so probably any effort on this image should be announced.
Code change:
print "readonly {var}={value}".format(var = k, value = pipes.quote(str(v))) ''' < "${kube_env_yaml}") <del> # Infer master status from presence in node pool if [[ $(hostname) = ${NODE_INSTANCE_PREFIX}* ]]; then KUBERNETES_MASTER="false" else </del> <ins> # Infer master status from hostname if [[ $(hostname) == "${INSTANCE_PREFIX}-master" ]]; then </ins> KUBERNETES_MASTER="true" <ins> else KUBERNETES_MASTER="false" </ins> fi }

Related discussion:
text: Node 2's kubelet logs: controller-manager side: I have a cluster to hand off if someone wants to debug. It's not a transient state: node 2 just failed to ever post a valid status. cc cc (who looked at it briefly with me)
I will take a look soon.
Ok, why kubelet on node 2 talks to ip address of master instead of hostname? root ps -ef | grep kubelet root 1 3 17:35 ? 00:10:45 /usr/local/bin/kubelet --apiservers=https://10.240.49.47 --config=/etc/kubernetes/manifests --allowprivileged=False --v=2 --clusterdns=10.51.240.10 -- It looks like a cluster up issue to me, not a node problem.
It looks like failed to translate the master name into the fqdn (which is done ).
Fascinating. I think we're seeing instance-DNS flakiness, because that name is supposed to get converted ASAP. But what's weirder is ... that's not even the master:
The is definitely right:
Okay, this is messed up. Here's node 1, which is happy: Here's our unhappy node .... that decided to bring itself as a master:
haha... I checked kube-env and did suspect unhappy node treat itself as master node based on those variables. But I didn't check salt state :-)
I ... just can't get there. By the time we get here: , node-2 has set: My best guess is that flaked here: . I think we're seeing a systemic issue where we're racing with GCE network configuration.
I'm going to work up a PR that hopefully improves the reliability of this section to some extent.
<a href=""<img src="" title="made at "/</a
My working theory is that we raced with the configuration done normally by the GCE NetworkManager, leaving us in a bogus state when we made that check.
Why are we relying on inferring the master/node status from the hostname. It should be obvious from the variables set in the kube-env if you are the master or not. Masters don't get passed so why not just check that? Making sure DNS is working is probably also good for reliability...
On GKE, we actually pass the same kube-env on both sides for simplicity.
Code change:
print "readonly {var}={value}".format(var = k, value = pipes.quote(str(v))) ''' < "${kube_env_yaml}") <del> # Infer master status from presence in node pool if [[ $(hostname) = ${NODE_INSTANCE_PREFIX}* ]]; then KUBERNETES_MASTER="false" else </del> <ins> # Infer master status from hostname if [[ $(hostname) == "${INSTANCE_PREFIX}-master" ]]; then </ins> KUBERNETES_MASTER="true" <ins> else KUBERNETES_MASTER="false" </ins> fi }

Related discussion:
text: We can obviously change that, but I was going for a change that didn't involve a deployment change on our side, too.
Code change:
d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err

Related discussion:
text: Hi, Since my upgrade to 1.18 version, I have errors in kube-proxy: I have ipv4/ipv6 dualstack enable. No problem with cluster and IPVS works despite errors. Do you have also this issue ? OS: RHEL7 Kubernetes version (use ):
/sig network
Related to cc
/cc
Taking one of the outputs it gives me the following addresses? can you share more details about your environment, Is it possible that the CoreDNS pod has the address ? what are your cluster-cidr and service-cidr? Indeed that new net.Utils change looks suspicious , a quick grep in the repo gives me some suspects, maybe the service/ipallocator? It's important to mention there were similar issues with IPVS before, interpreting IPv4 addresses as IPv6, has some experience with this kind of issues, maybe he can see something:) /cc
A bit tricky to trace since libnetwork/ipvs is moving around at the moment. Problem seems to be in; The "family" must be wrong. It is called from;
Please do a and attach the output.
Hi, Some informations: cluster-cidr and service-cidr: fd00:10:245:0:0:0:0:0/64,10.245.0.0/16 fd00:10:96:0:0:0:0:0/112,10.96.0.0/12 ipvsadm output:
Ref:
The commit in the ref above should be in v1.18.0 which is the version used in this issue. So the commit does not fix the problem.
Maybe something wrong around here Before , it looks like this: Now it looks like this: However is not initializedit is zero :(.
I do not get these faults. Has the netlink API changed? What kernel version are you using?
d.AddressFamily is supposed to be set from the "NetlinkRouteAttr" slice; It should definitely not be hard-coded to ipv4.
Made a trying to reproduce this in my environment but didn't got this error.
Code change:
d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err

Related discussion:
text: I may try this on some older kernel like from CentOS 7 Added exactly the same IPVS configuration here: And when running this 'gist/program' pointing to the IPVS from the DNS (fd00:10:96::a port 53 UDP) got the expected result: Maybe trying to run this also in some other scenarios could clarify if this is something related to Netlink API changed between Kernels, something with the used library, etc.
After some study, I find that this issue most likely is caused by the low linux kernel version. I have reproduced this problem on . Accroding to the code, netlink will try to get attribute, but I find the kernel just does not support it :( This is the defined in (kernel 3.10) No attribute is defined! But in new kernel version, the is defined like : Obviously, the kernel has some attributes (IPVSDESTATTRADDRFAMILY, IPVSDESTATTRSTATS64...). That is why kube-proxy works well on systems with a higher version of the kernel. So we can address this issue by upgrading our linux kernel. Not sure which kernel version is the minimum requirement, maybe we should document something about this.
Here is the defined in :
Good finding. So, we can conclude that this really is kernel version dependent. It is hard to know what to do about it. We must check the family. I would propose that we make it work for ipv4 with some fallback and put a requirement on kernel version for ipv6-only/dual-stack. But the decision is others to make. /cc
/cc
The symbol in the kernel tree The for service family seems very old. Since the service must have the same family as the real-servers perhaps it can be used for compatibility with old kernels?
The commit that adds the symbol; Seem to go into linux-3.18
I did not know the minimum required for the kernel version, I was using the stable and official version of RHEL7. My kernel version is: Currently the max version in the repository is: According to the documentation (), I should switch to RHEL8 if I want a recent kernel :-
There is no minimum kernel version requirement afaik. So I think this must be fixed for ipv4-only clusters IMO.
Code change:
d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err

Related discussion:
text: However ipv6-only and dual-stack are in "alpha" so a requirement on kernel version might be ok, but I can't say myself. A complication is that the problem is not in k8s itself but in a 3rd party lib.
ipv6-only is beta since 1.18 :smile:
/assign
I opened a PR to fix this in moby/ipvs If someone w/ a cluster on the older kernel version can test it that would be great, otherwise I will setup a cluster to validate this some time this week.
Instead of bootstraping a whole cluster right now (pretty busy here) I can use the Gist I've mentioned to check in a CentOS 7.1 if it faces the problem before and after the patch. At least to have some more test right now
with a small test, it worked fine. Environment: CentOS 7.1 - Kernel 3.10.0-229.el7.x86_64 + ipvsadm 1.2.1 Pre-loaded some IPv6 rules into ipvs: I've used code, searching for the following structure: I then call ` to get the IPVS object, and then try to fetch the destinations. The following are the results: Before patching the ipvs module: After patching the ipvs module (pinning to your repo/branch) Also I've made those tests with the structure but searching for TCP, and for kube-apiserver (that have only one destination) and it worked fine in all those scenarios. If wanted I can improve the gist to pass the parameters as command arguments instead of hardcoded (sorry, did this in a rush to test)
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-triage unresolved
Thanks just waiting on now
same issue,sloved by update kernel to 5.X
I didn't want to wait, I updated to kernel-lt in the version 4.4 . The problem is solved on my side :)
Thanks for confirming. We'll still try to get in since I'm sure other folks with older kernels will run into this issue.
/retitle parseIP error with IPVS proxy
upgrade your system kernel to 4.xx edition
FYI: just merged, need to run some validation & cut a new release (v1.0.1).
Code change:
d.ActiveConnections = int(native.Uint16(attr.Value)) case ipvsDestAttrInactiveConnections: d.InactiveConnections = int(native.Uint16(attr.Value)) <del> case ipvsSvcAttrStats: </del> <ins> case ipvsDestAttrStats: </ins> stats, err := assembleStats(attr.Value) if err != nil { return nil, err

Related discussion:
text: FYI
/retitle DualStack / IPv6-only: parseIP error with IPVS proxy on CentOS 7
v1.18 cherry-pick
Hi, it seems that you're using v1.18.0 and this has been corrected in v1.18.3, can you please update and check? Tks