Change:
NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-gci} MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-cos-stable-65-10323-64-0} MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud} <ins> CLEANUP_KUBEMARK_IMAGE=${CLEANUP_KUBEMARK_IMAGE:-true} </ins> # GPUs supported in GCE do not have compatible drivers in Debian 7. if [[ "${NODE_OS_DISTRIBUTION}" == "debian" ]]; then

Discussion:
After looking at 's , I realized that there are some problems with how we're using kubemark images for our CI-testing: run of every kubemark job creates and pushes a kubemark image to the image registry of the project in which the cluster would be created. But the kubemark cluster setup uses the image. This means that a race b/w 2 kubemark jobs while pushing the image is possible, and it may happen that one job is using the image from another one. This can be solved by using a random image tag to uniquify the runs. run of every kubemark job pushes a new image to a docker registry. So we have a huge list of leaked kubemark images in the projects where we're running kubemark jobs - e.g O(10^5) images in used for kubemark pre-submit. Each image is about 72 MB and at the cost of for our GCS buckets, it's ~/month just on stale kubemark images. Few things we should do IMO: delete the kubemark image at the end of the run delete the pre-existing stale kubemark images in general, make janitor garbage collect images also (cc ) changed the kubemark image registry to , instead of the earlier . However, the manifest for the kubemark deployment - so all jobs will continue using old kubemark image (albeit the one). This needs to be fixed asap. cc /assign
PR solves the 1st and 3rd issues above. Wrt 2, the following seem to be the current set of projects running our kubemark jobs (there may be more abandoned ones from the past): We need to clean up all those images. Any ideas on what's the easiest way of doing it? I'm thinking of directly deleting the GCS directories holding those images (if that's possible). cc And for future, I'll send out a PR soon to do automatic GC as part of the run.
I still suggest we push the image as part of the build process :-) for gc, isn't it we won't need anything except for the one? Select all and delete?
I'm not sure I understand. Push where? Do you mean pack it in the release tar (like kube-proxy, etc)?

Yeah, for now we can delete all the images (even the latest one, it'll probably just cause a few one-off failures of the current runs). In general we don't want to gc everything except the latest one as we're going to have multiple unique images in use at a given instant (see my point 1).
I thought we do image pushes already, like in our build jobs, maybe we can fit kubemark build there? cc
I'm not sure - is it? The last I remember, I think we used to pack those docker images (and their tags) into the release tar which was downloaded onto every VM and docker used to locally load those dev image builds (instead of pushing them to somewhere in a registry).
we could run a script with in a cronjob if we had a reliable way to identify which images we want to get rid of.
We still pack the docker images + tags into the release tarballs, since GCE/GKE/kops all still do sideloading, but we also push the images from the to for the kubeadm CI jobs. See for more discussion on this.
maybe combine with an appropriate ?
probably? one issue is that images can be pushed with many tags
you can use to get the list of images, and then will list all tags matching your filter query for a given image.
yeah, we'll need something like: list images we want to keep and their tags list all images and their tags delete set(all tags) - set(tags to keep)
Image retention policy is a feature we (GCR) are investigating. what other tags are you using to push? I see in combination with a , which is going to complicate matters by stripping the build timestamp from the image (thereby enabling reproducible builds). Would the unix timestamp be an acceptable alternative to a completely random tag? If you're continually uploading new images under the exact same tag(s), most of the images will be untagged, and you can list the digests of such images like so: and delete like so:
I'm not sure what uploads are done for this job yet, or what tags would be acceptable but in other places, but in other places we have code that builds images which are doubly tagged ( and ), I think this pattern is fairly common. We have some similar logic for getting rid of stale images on disk (for local caching).

I definitely think we could hack something together, I just want to make sure we'll be able to avoid deleting things we should actually keep... It sounds like for this one we can probably just delete images that aren't latest, but there are lots of other images we should probably do this for.
Ah. 's a bit... interesting, but it shouldn't be too difficult to do the set difference above:
re: bazel-built docker images, I'm hoping we can fix at some point (so we can stamp them with real timestamps).
FYI has been merged, stamping should give you useful timestamp metadata to help reap old images (when a new release is made and adopted).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale On Tue, Jul 10, 2018, 9:08 PM fejta-bot <:
I've sent out as a simple strategy to delete image right after startup.