jiebi/CodeConvo huggingface dataset:
Code change:
package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

Related discussion:
text: Background: kubectl and the Kubernetes reference docs both depend on the spec to operate on Kubernetes resources. The lacks important information required by kubectl and the docs. Usecase: kubectl can identify the patchStrategy and mergeKey for resource fields kubectl can identify the columns to print for on resources docs can parse and display the default value for fields docs can identify top-level objects Solution: Add items to the comment tag: Example: The long term plan is to remove the and struct tags once all code has been updated to use the swagger spec instead. Until then, comments should be for the struct tags and . Some verification process will need to be written to make sure the struct tags and comments remain in sync.
I think this wouldn't fall under sig-apps. cc
I would not do the columns stuff - we're moving get to the server side. Top level objects is already available in some form by the client - specifically the list that we use (should use) to get the swagger doc tells us what the top level type is (or is that for something like explain). Merge key, patch strategy, default values all belong as "real" attributes, although we had not necessarily intended those to be arguments to k8s:openapi-gen, but actual tags in their own right (like "+optional")
Thanks Is there a design proposal / eta for this? This is for generating the docs from swagger. Right now the list of top level resources for the docs to display is hardcoded in a . I would like to move the majority of the metadata out of the config file so that: same tool can generate reference docs for federated apiservers without writing a custom config. don't need to manually maintain and update the config for each release - e.g. adding new types I have no strong opinion about this as long as they appear in the swagger so it is possible right tools the can use this data. Can you comment on why openapi-gen vs specific tags?
Describe in the server makes sense to me. IIUC, just prints out a set of columns. Was the idea that the server would just print out the columns or return a list of column / values or something more complicated? No need to answer if it is in the issue / proposal already.
Code change:
package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

Related discussion:
text: is the bigger issue that touches on a rough overview, I can't find my rambling brain dump of the problem, but get would be rough tabular json, you could also get that as csv, and describe could be a richer but not quite full structured key/value structure.
regarding tags, they do not need to be in the same k8s:openapi-gen tag, it can be repeated: they can also be separate tags but still general like this: The idea is to have a general way to add arbitrary extension to a type so we don't need to change code generator for each specific tag. I have no strong feeling about the format, but I would like it to be general vs tags like +optional. The difference is +optional is a defined tag in OpenAPI () but extensions does not have any meaning for code generator. They better to be general so generate can pass them without change.
No, I mean we should define conventions in our API code that match the intent (optional) rather than have tons and tons of repeated code that is setting x-kubernete-describe-columns (or similar)
I spent some time thinking about your comments last night. I don't think this is mutually exclusive with , and could be complementary. IMHO this is general metadata associated with the type, and including it in the doesn't preclude supporting an api endpoint that pre-parses and formats the data. However, including it in the swagger does allow clients using the swagger to take advantage of it. Following is an example of where the swagger would be useful even if get was in the sever. A generated Java client could use the display columns for the "toString" method of the resource. Another use case that had mentioned to me was to allow some kubectl commands to operate on directly on local files. e.g. (strawman) could print the columns. I don't think we want to do this now, but it is an example of why keeping this object metadata private to the server limits what we can do. My hope is that this swagger tags solution will also oprove portable to easily work with ThirdPartyResources as well by writing the display-columns as an annotation on the TPR and keeping the same format. is a can of worms, and I would agree that the solution is capable of providing a richer experience for aggregating information across multiple resource types - e.g.
Code change:
package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

Related discussion:
text: how describing Nodes fetches Pod information. I still think that the solution is compatible with the richer solution you proposed and there is not much additional cost in doing it. Howv What would you think of supporting a set of optional ALPHA extensions for "display-columns" and "describe-columns" allowing any consumers of the swagger to print the objects. In theory it could also be used by the UI to provide a summary view without having to do separate lookups. As an ALPHA feature we could try it for supporting sig-service-catalog and revisit it in 1.7. If we consider it successful, the solution outlined in could become another consumer of the display-columns tag? I like this approach. My biggest concern here is being able to get this into 1.6. Since kubectl supports +/-1 version skew, we need will need to maintain the old logic for at least 1 full release after we add support for the new swagger based logic. That puts us at 6 months from now to fully migrate if we don't have any swagger extensions in 1.6.
I think that's a nice to have in the long run - but then people will say "why are the columns different when the server prints them from the local print"? And most of the pod columns in no way can represent the full complexity of what they print with swagger, so the best effort local printing is going to be really, really, really best effort. Also, local print is a really specialized case - if we just did the generic metadata print that would probably be ok. So while I agree it's valid, it seems like a lot of work for very little practical value. Again, not opposed, just don't want to do two things halfway and then have nothing. So just biased towards not adding more client complexity in the near term. So you want to hack this in for service catalog in 1.6 on TPR? I feel like that's a lot of hacks - the generic printer stuff at least gives you name and namespace, and I'd rather do nothing in 1.7 for generic client printer rather than jump ahead. I won't stop you if you want to spend time and effort on it, but it's a lot of specialized work that will then be neglected while we do the server side stuff.
Code change:
package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

Related discussion:
text: My intend here was avoiding repeated code for adding each of these tags somewhere (in openapi-gen code or some root level place). Let me distinguish these tags in two categories with respect to OpenAPI code generator: that need some sort of processing that affects OpenAPI spec (e.g. +optional removes the field from OpenAPI field) that need no more processing and will be directly put into OpenAPI spec (e.g. +gvk that we don't have but we can define as a way to define Group/Version/Kind of an object vs register method) My argument is about the second one. I think patch strategy here falls into this category. The question is only about naming it. Naming Option 1: +patch-strategy=... This looks cleaner but we need to define an alias somewhere to tell generator this is the same as x-kubernetes-patch-strategy. The alias definition and the tags are in two places. Putting them in the generator code make the code non-generic and we don't want to do that. Putting these aliases somewhere as an argument to the generator begs for bugs as people that look at the type and add this tag to their types need to know (maybe by reading a doc, but who do that) that it needs to pass the alias to the generator. I, in 6 month, will look at a type and see a tag and copy the tag to my type and expect it to work that will not with this option. Naming Option 2: use something generic for these type of tags. after all generator does not need to do any processing. I suggest +openapi-tag:tag=value that will result in "x-kubernetes-tag: value" in OpenAPI spec. There is no alias defined anywhere and without any code change or alias copying or any knowledge about passing any argument, a tag can be to OpenAPI spec. The value of these tags and their use cases are outside of the scope of what I am arguing about. Anyway a solution is better than no solution, I would suggest we continue with Option 2 as Alpha and change it later if we see fit. Adding an aliasing mechanism to this option later should be easy.
On patch strategy, I had assumed we'd read the value from what's in the struct tag, rather than requiring a net new tag?
Code change:
package foo // Blah is a test. <del> // +k8s:openapi=true </del> <ins> // +k8s:openapi-gen=true // +k8s:openapi-gen=x-kubernetes-type-tag:type_test </ins> type Blah struct { // A simple string String string

Related discussion:
text: , Mehdy Bohlool <:
The get stuff is less urgent for me as long as we aren't doing even more hacky things to work around the current limitations, and can make a commitment to delivering it in a specific release (sounds like 1.7). For service catalog there was a plan to fork kubectl, compile the service catalog types into the binary, and provide the fork as a separate binary. I have some concerns with this approach. I would rather provide a temporary hack in the code that can provide a consistent user experience from day one and re-implement later vs having users get used to downloading separate binaries for each aggregated apiserver. I have no plans to do any work for on this for the core Kubernetes types, and can punt on it until the scope of the work is understood to be relatively small and easily upgraded to the server-side get.
Code change:
} }) <del> // Marking this as LinuxOnly because windows tests run against Azure CNI which doesn't publish // a podCIDR for each family. ginkgo.It("should have ipv4 and ipv6 node podCIDRs [LinuxOnly]", func() { // TODO (aramase) can switch to new function to get all nodes nodeList, err := e2enode.GetReadySchedulableNodes(cs) framework.ExpectNoError(err) for _, node := range nodeList.Items { framework.ExpectEqual(len(node.Spec.PodCIDRs), 2) // assert podCIDR is same as podCIDRs[0] framework.ExpectEqual(node.Spec.PodCIDR, node.Spec.PodCIDRs[0]) // assert one is ipv4 and other is ipv6 framework.ExpectEqual(netutils.IsIPv4CIDRString(node.Spec.PodCIDRs[0]) != netutils.IsIPv4CIDRString(node.Spec.PodCIDRs[1]), true) } }) </del> ginkgo.It("should create pod, add ipv6 and ipv4 ip to pod ips", func() { podName := "pod-dualstack-ips"

Related discussion:
text: The test ""should have ipv4 and ipv6 node podCIDRs" depends on a feature that is optional (one of the ipam controllers in the controller manager) We have discussed this previously, but since this is actually affecting downstream CIs I propose to remove it form the e2e framework. For cloud providers, it can be moved to the own external cloud provider code, since the policy is to not host more tests with dependencies. For the feature itself, it can easily be replaced by a unit or an integration test. ref:
we have discussed it before, but now is affecting downstream CIs and is really a controller-manager test. /cc /cc for sig-testing visibility /sig network
Removing the test sounds reasonable as it's specific to cloud providers. I'll review if we have enough unit tests for this and open a PR to remove it from e2e.
We can add more if they don't exist, as a condition to remove the e2e
/assign
/triage accepted
It's not specific to cloud providers. It's specific to different network plugins; some require it and some don't.
I think that the comment is related to the e2e framework skipper, that only skips by cloud provider, but yeah, it is about CNIs
Code change:
- path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

Related discussion:
text: Istio folks have a iptables image here that is quite small In the past, we did talk about this, for example wrote it well what we could try: We could even do for running the hack script we have for switching the symlinks when vs : kube-proxy image is definitely one of the main causes of the cost we incur in our GCR repository, so we should definitely give this a shot again. /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is a sig/network-only issue. A gives; The looks like this (narroved): If I don't misinterpret this, K8s uses a unified server-image build and it should not be different for . The size of the binary for amd64 in is; It is stripped and can't easily be "made smaller" I guess.
We (Istio) would definitely be interested in sharing ownership (hopefully by upsteaming to distroless mainline) in the image we have if there is interest. Some other projects have expressed similar needs as well
!
kube-proxy make up 9% of current image data traffic
I threw together this little repo (based on our next-gen distroless infrastructure): It pushed which on x64 is roughly another 5x smaller (musl vs. glibc): If folks are interested in this, I'd be happy to create a repo under which is where we've been standing up new images, or you can create your own from pretty easily.
debian based please!! we have KEPs and stuff written that lay down that direction:
For a possible approach + above in that issue and for "distroless" debian based images with arbitrary C/C++ binaries + deps. Note that we need things besides just the distro binaries (see comment linked). and is pretty straightforward for one way to handle the debian binaries part.
I don't even know how we build the kube-proxy image. Where does that happen and form which dockerfile?
Code change:
- path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

Related discussion:
text: Note we need iptables, iptables-save and iptables-restore but also ipvsadm, shell, conntrack, ipset, and maybe other tools
It used to be in kubernetes/kubernetes under build/images, it's now in (Dockerfile etc there)
That's the base - where is the logic to turn that into a kube-proxy image?
Turning that image into a kube-proxy image is just adding the kube-proxy binary with a dockerfile and entirely too much bash which ultimately just does: (except api-server, where we do a setcap in )
/triage accepted Will take a look between this and next week
It doesn't even need bash; the version of the script in intentionally avoids bash-specific features (and gets tested against Alpine where /bin/sh is busybox (which I guess is "ash"?)). And the version in kubernetes/release appears to be just a stripped-down version of that script. (But actually, you should update the script to be based on the latest version anyway, to get the new logic in , in preparation for .)
Probably per one of the things we don't want to do right now is jump into a musl based image like Alpine, right? So I would take the case that we want to use distroless, and copy all the binaries we need to inside that container (conntrack, ipvs, iptables-*, etc etc). per iptables-wrapper, you said it's not bash specific so maybe we can use distroless + the smaller available shell? I will check some approaches here before jumping into changing the release/dockerfiles/whatever scripts, and check if I can run some comformance tests in a distroless k/proxy. Maybe starting with a good binary list is a way to cross check the binaries + libraries required for this distroless container. I did a list with the installed packages per kproxy needs, and some ldd on it: Will start working on something to assemble an image just with the binaries and required libraries, and check size + libraries
I agree no musl/alpine On Sun, Apr 17, 2022, 11:38 AM Ricardo Katz
If you identify the list of binaries, has logic to handle ldd for a binary (e.g. ignore vdso) + keep the copyright and package version info via dpkg.
Code change:
- path: test/conformance/image/Makefile match: BASE_IMAGE_VERSION?= <del> - name: "registry.k8s.io/debian-iptables: dependents" version: bullseye-v1.4.0 </del> <ins> - name: "registry.k8s.io/distroless-iptables: dependents" version: v0.1.1 </ins> refPaths: - path: build/common.sh <del> match: __default_debian_iptables_version= </del> <ins> match: __default_distroless_iptables_version= </ins> - path: test/utils/image/manifest.go <del> match: configs[DebianIptables] = Config{list.BuildImageRegistry, "debian-iptables", "[a-zA-Z]+-v((([0-9]+).([0-9]+).([0-9]+)(?:-([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)(?:+([0-9a-zA-Z-]+(?:.[0-9a-zA-Z-]+)*))?)"} </del> <ins> match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} </ins> - name: "registry.k8s.io/go-runner: dependents" version: v2.3.1-go1.18.3-bullseye.0

Related discussion:
text: We probably need a few more not in the list above for the other proxy modes?
Thanks this is gonna be very helpful! I will work on it using the script you provided during the week!
/cc
From script and some changes: There's a pretty good reduction of size (50%). But this is without kproxy binary, that is almost 40Mb. I will open a Draft PR in k/release so we can discuss over there the approach. BTW, anyone knows why kube-proxy binary is that big? Never digged into that, but curious about it.
https://dr- !
(updated with conntrack-tools) Why? The binary is statically linked so it does not need or use , it only needs some binaries to be executable and it doesn't matter if they are built with musl. They just have to work. I have sucessfully built and executed my own tests, both with proxy-mode and , using an image built with this : The image size on cluster including the binary (~40MB) is; (I use a private local registry) Compare to;
what about the different iptables-nft vs iptables-legacy binaries?
I have assumed that takes care of that (executed by the Dockerfile).
Re size: Are DWARF2 symbols stripped? Ideally k8s should produce artifacts without DWARF2 symbols and distribute them separatelly. Or produce the optional -debug artifacts. This can cut GCR/GCS costs, but introduce some debug/profiling complexity. The Go symbol table should be always included. Related
IIRC wrapper in prod needs both binaries existing in the container, and will use this in a way to check if the host machine is using iptables-legacy or iptables-nft.
Both exist in Alpine; The first example in is Alpine. can you please confirm that it works with Alpine?
I have altered my testenv so I can switch between legacy and nft and verified that the tests work with both and that shows the same in the container and the K8s node, e.g;
Side remark; to my suprise is not needed in the kube-proxy image
Tested the with K8s e2e;
There are / we're legal reasons to avoid Alpine. Have those been resolved? I haven't heard anything.