Change:
EDIT (from - 5/8/20): [x] () New go-runner image for distroless scenarios [x] () Promote the new image - [x] () Bump to [x] () Promote image [x] () Switch core master base images from debian to distroless [x] () Flexvolume master driver is blocking controller-manager from moving to distroless - [x] () Base image (debian-base, debian-iptables) updates & building on K8s Infra [x] () 'pause' base image updates & building on K8s Infra [x] () Remove image [ ] () Remove image Forked from (think of this as V2). is not quite sufficient as a base image for static go binaries. Common dependencies include ca-certificates, , hosts, and nsswtich (see ). See below for a more complete list. We need to decide if we want a scratch+config base image (either found or created by us), or a more complete distro (e.g. debian-base, at the cost of more frequent patching). has also been proposed, or could provide inspiration for a static base. Blocked by: (for the scratch case) , for logs redirection. /kind bug /cc Appendix Here is a list of files referenced from the go source (obviously some of these are not applicable):
/sig cluster-lifecycle
not sure if sig cluster lifecycle is the best owner for this. adding sig release and sig architecture. /sig release /sig architecture
I just hit during a deployment of 1.12, this issue: Please add a sane to all the images or convince the golang folks to fix their defaults.
/assign This is not sig-cluster-lifecycle typically.
is not enough?
before there was , though that didn't make it into 1.12. cc
Yeah, I an as a small stopgap in , but I'd like to see a standard base image as well which should simplify the build and patching a bit. We could backport or to 1.12, I'm not sure what the best choice is. is smaller but is the preferable change going forward.
Note that go's defaults actually match the documentation for not having an : (), it's just that they don't match the default most distros seem to ship of which is probably what most of us want ...
Another option is to just use , although it does include a few c libraries that we don't need.
would distroless/base have what we need for EG the iptables (kubeproxy) variant?
I don't think so - it doesn't have a package manager so iptables would still need to be manually (and managed).
It looks like distroless is planning to put out an image without the c libraries, specifically for this use case. I think we should just use that as soon as it's ready:
Update: is now available. I suggest we work towards this end-state: All official Kubernetes images & addons are based on 1 of 2 images: wherever possible, or a full featured distro for the cases where it's absolutely required. For the latter, our current recommendation is , but that is subject to change. For these cases, the dependencies should be well-documented, and we should eventually work towards building from source, and basing on a distroless image. Action Items: : DONE generate a list of images needing updates for each item below document decision in a more "official" place (community repo) all images on . There are some known-issues with running go static binaries on scratch, and this move should fix those (e.g. ). system images using shell-redirection for logs on distroless/static, by leveraging option to eliminate shell usage. (Requires v1.13+) away at the remaining images with miscellaneous shell dependencies. In some cases, the shell usage should be rewritten as a feature of the binary, in others it's a harder dependency that should be documented and migrated to debian-base (see above). We should have separate tracking issues for fixing these on a case-by-case basis. I propose we try to tackle 1-3 in Kubernetes 1.14. /milestone v1.14 /help
cc (please see 1i from Tim above)
I would like to start digging into this.
SGTM, one thing worth noting is that a few related projects have used (one of mine used to), wish we had a good way to signal intent regarding the future of that image.
it seems like it might be a bit late to tackle this for 1.14 (cc though it might be worth starting on a KEP to outline this in any case?
what would you want to a see in a KEP about this? Just the justification, and an analysis of the risks in moving to the new base image?
/assign
GitHub didn't allow me to assign the following users: yuwenma. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as does not contain any platform specific code and just contains: So we are safe using this instead of scratch wherver needed.
We're well into code freeze for 1.14, is this necessary/release-blocking? Also, this seems to be a major enhancement without a KEP as mentioned above. Based on the work defined above that's appears to be remaining and there is no KEP associated with this do you mind if we push this to 1.15?
Definitely not release-blocking. /milestone v1.15 it sounds like there's demand for a KEP. Would you mind translating your design doc into a KEP? Follow for guidelines.
Will do that. Right now, I'm referring to this issue for the full context but I think KEP is a good idea.
/remove-sig architecture
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
Hi Marko, thanks for the check. Yes! This is planed. We have had the majority of the work done. Have three more PRs need to go in (under review). Please note, this is not a feature request but more of a bug-fix/internal-cleanup due to the security reasons. Does it have to strictly follow the code freeze policy?
Hello, I'm the Bug Triage Lead for 1.15. All issues and PRs are subject to the Code Freeze policy, this is to try and maintain as much stability as possible while preparing a new release. Last release there were comments about creating a KEP for this work, was that ever completed? If so, if your code is complete, and a documentation PR is open then this will likely land in this release. If a KEP has not been created this issue, and associated PRs, may still be incorporated in this release during the Code Thaw phase. For more information please refer to this doc:
Thanks for the reminder. yeah, we are following the policy and here's the We have the PRs ready and they depend on the latest klog (v0.3.2, created yesterday). I'm currently working on importing the new klog version to k/k ( the new klog release may require some test updates) and afterwords we can have the blocking PR merged. Due to our current situation, do you have any suggestions what can be the good release phase for us? should we go with the Code Thaw phase?
If you can get the Klog version updated and the PRs merged by EOD tomorrow (May 30th) you will be fine, otherwise shoot for Code Thaw. I recommend trying to pull in the appropriate Sig leadership to help review/expedite your PRs getting merged
Hello once again! I'd just like to check what's the current status of this issue and is it still planned for v1.15? Thanks!
's the status updates: We have rebased. And here's the .
It looks like Klog is still an issue, can we move this to 1.16?
Since I didn't receive a response I'm bumping this to 1.16 /milestone v1.16
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
Any update on this? Should we move it to the next milestone?
/milestone v1.17 This is an ongoing issue with incremental improvements in each release.
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still on track for 1.17?
/milestone v1.18
Context: sig-architecture subgroup work to search for technical debt Category: Security Reason: Distroless containers decreased attack surface
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/milestone v1.19 /remove-lifecycle stale switching milestone to next cycle. please change back if need.
- have the klog issues that were blocking some of the control plane images been resolved? /cc
we haven't tracked it down yet. last debugging was here -
I've updated the PR description with some of the current work Dims and I are doing. /assign /remove-help /remove-priority important-longterm /priority important-soon
FYI
Hello, here from Bug Triage! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting June 25th and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
-- Yep, it's active for this release. The issue description and tracked items was updated 18 days ago and there are several merged PRs linking back to this.
I still see a couple open PR's. Due to reaching the code freeze deadline, I'm going to bump this to the next release. /milestone v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Heyho I wanted to check how it is going? As the issue is tagged for 1.20, is it still planned for this release?
This is roughly done, but I need to backfill docs. /milestone v1.21
Hi , I'm from the Bug Triage team, is still planned for 1.21? We start code freeze today, so I wonder if we should leave it for this release or move to the next
/milestone v1.22
Is there any update on this? Are the docs backfilled already?
Hey there ( ), Bug-Triage here , looks like this issue hasn't received any update since it has been assigned to . Given the fact that the code freeze for this release already started and the issue has been transitioned a couple of releases yet, I´d like to suggest clearing the milestone label for now. We will remove the label if this issue doesn't receive any significant update in 7d from now.
/milestone clear
I'm prototyping what distroless + additional necessary c binaries (like iptables) would look like in the KIND subproject here: It's not bad actually, especially given that distroless has images intended for C, C++ applications (adds libc6, adds libgcc1). I've been writing a generalizable approach to automatically copy binary + runtime dynamic library deps + their copyrights based on inspecting binaries with in a multi-stage build.
my concern with this approach is that it will essentially hide the packages & versions from vulnerability scanners, which I think mostly just look for package manager manifests. I wonder how hard it would be to keep a package manifest with the relevant packages in the image? Also, how does your approach compare to using bazel to build docker imagse?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
Still needs a review of docs; will work on this during some intercycle downtime /remove-lifecycle rotten /lifecycle frozen /milestone v1.24
hi, I'm with the Bug triage team for 1.24 - we noticed that this issue has not been updated for a while. I wonder if there is any work planned or expected that would go into 1.24? Thanks!
I don't believe there are any further changes in 1.24. Sorry I missed your reply I don't think that will be hard at all. Distroless has pretty much the same proposition when using anything beyond "base", e.g. the "static" image has glibc from debian packaging. If we can settle on what information we wish to preserve (or generate!) we should be able to do this, e.g. currently we preserve package copyright info. And presumably distroless either doesn't handle this and leaves us in the same boat with other images, or we can copy what they're doing. I'd have to dig more to see, I think this is which is in the distroless base image, we should be able to preserve those in the same way we preserve copyright. Well, currently I'm using a multi-stage dockerfile but the overall approach can be implemented many ways. Multi-stage dockerfiles can be pinned to pull source images by digest and are relatively hermetic since they're inherently a sandboxed filesystem. If we have any concerns with this, we could also pull the package contents directly and add them to an image. The bazel rules_docker package largely wraps Bazel is also slightly a moot point for kubernetes following
nudged me about this, digging around shows that distroless images persist package versions by putting the package control file in , which is close to normal dpkg (one big file with a blank line between packages, and some additional info beyond the control file) Source: Scanners already need to implement support for this to handle our other distroless images. (see e.g. ) We can very easily grab the package status () and stage it (into like bazel) along with the copyright and libraries we need for each binary, you can see this to my prototype here: (first commit in )
I think if we wanted to continue with this approach for the iptables image it would look like: 1) rewrite the iptables / iptables-nft select hack script to be a static go binary instead of a shell script 2) convert the image build to look like kubernetes-sigs/kind images/haproxy, but instead of haproxy do the iptables binaries and stage them and their libraries, add the version select tool + symlinks Then we'd have an image with just iptables binaries, their library dependencies, tzdata etc. from distroless base, their copyright files, and their dpkg statuses, along with some symlinks and a go binary for switching between iptables-legacy and iptables-nft. That image should be smaller and have less vulnerability scanner alerts. Step 1) is probably reasonable to consider regardless of step 2) anyhow, if someone wanted to take this up. You'll need to talk to SIG Network about these changes first though. This new image would also be a breaking change for any other projects that base on debian-iptables.
/milestone v1.25 kick this down the road, no activity.
Hi My name Hossein Salahi from Bug-Triage team. I am just checking to ensure if everything is on track for K8s 1.25. Thanks!
the release 1.25 bug triage Lead. I'm reaching out to see the status of this Issue and if we are still targeting the current release.
there was progress (please see ), we will have to take an inventory when 1.26 opens up. /milestone v1.26
Heya and - 1.26 bug triage team checking in to see how we're looking for this release? Thanks!
the last thing we wanted to do here was distroless kubeproxy which got done already. We should close this out! thanks everyone!
Do you mind doing a quick update to and to reflect the fact that we now have kubeproxy with distroless images? thanks!
We should probably still make distroless kubeproxy lighter and make sure our projects pick it up so we can phase out the old full image, do we have follow-up issues for getting the shell etc out of there and making sure we've rolled out the image to subprojects?

Discussion:
# See the License for the specific language governing permissions and # limitations under the License. <del> FROM scratch LABEL maintainer "Shyam JVS <shyamjvs@google.com>" </del> <ins> FROM gcr.io/distroless/static:latest </ins> COPY etcd-version-monitor /etcd-version-monitor