Change:
I'm pretty sure that building the hyperkube images takes longer than building kubernetes itself now. At least a large part of this seems to be that we download and install ~172 extra debian packages several times (once for each arch). <details</detailsNot sure how to improve this - maybe better caching? Do we actually want to install these extra packages? Should we use a better base image than ? cc
cc (not really sure who "owns" hyperkube, if anyone)
The hyperkube binary is one thing. I think the image ends up being big as we drag in so much stuff. I wasn't involved in creating the docker image so I can't comment. Ideally this thing would be slim and would have minimal external dependencies. Fixing this starts with mapping why we have each of those deps imported.
I actually have written a such patch in my local env, but I never uploaded. It removed the docker and docker-multinode-specific stuff and all the unused deps. Also, we will come one step closer once is in. Should I submit the patch so you can take a look at it?
sure!
This needs to be triaged as a release-blocker or not for 1.5
punting to v1.6.
The main problem here is that we're installing huge packages for supporting some volume types: git nfs-common glusterfs-client cifs-utils With containerized volume mounters we'll get rid of that.
When it comes to local rebuilding, should help here. As I've already noted in the PR, maybe we should make CACHEBUST=0 the default for all hack/dev- scripts. This would at least ease development and testing.
can this be closed? or moved to 1.7?
should this be in v1.7? is there an appropriate sig to own this?
cries nope, not v1.7. I'm still not sure who owns hyperkube.
Everyone and no one I suppose. Yeah, agree it's sad.
/sig release guess we're saying sig-release owns hyperkube under the general guise of "cutting bits" until/unless there is a better home for it
Yeah, hyperkube definitely belongs to sig-release as it hasn't been owned by a sig so far...

Discussion:
<ins> # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. FROM BASEIMAGE # The samba-common, cifs-utils, and nfs-common packages depend on # ucf, which itself depends on /bin/bash existing. # It doesn't seem to actually need bash, however. RUN ln -s /bin/sh /bin/bash RUN echo CACHEBUST>/dev/null && clean-install  iptables  ebtables  ethtool  ca-certificates  conntrack  util-linux  socat  git  jq  nfs-common  glusterfs-client  cifs-utils  ceph-common COPY cni-bin/bin /opt/cni/bin </ins>

<ins> # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Build the hyperkube base image. This image is used to build the hyperkube image. # # Usage: #   [ARCH=amd64] [REGISTRY="gcr.io/google-containers"] make (build|push) REGISTRY?=gcr.io/google-containers IMAGE?=debian-hyperkube-base TAG=0.1 ARCH?=amd64 CACHEBUST?=1 BASEIMAGE=gcr.io/google-containers/debian-base-$(ARCH):0.1 CNI_RELEASE=0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff TEMP_DIR:=$(shell mktemp -d) CNI_TARBALL=cni-$(ARCH)-$(CNI_RELEASE).tar.gz .PHONY: all build push clean all: push cni-tars/$(CNI_TARBALL): mkdir -p cni-tars/ cd cni-tars/ && curl -sSLO --retry 5 https://storage.googleapis.com/kubernetes-release/network-plugins/${CNI_TARBALL} clean: rm -rf cni-tars/ build: cni-tars/$(CNI_TARBALL) cp Dockerfile $(TEMP_DIR) cd $(TEMP_DIR) && sed -i "s|BASEIMAGE|$(BASEIMAGE)|g" Dockerfile ifeq ($(CACHEBUST),1) cd ${TEMP_DIR} && sed -i.back "s|CACHEBUST|$(shell uuidgen)|g" Dockerfile endif mkdir -p ${TEMP_DIR}/cni-bin tar -xz -C ${TEMP_DIR}/cni-bin -f "cni-tars/${CNI_TARBALL}" docker build --pull -t $(REGISTRY)/$(IMAGE)-$(ARCH):$(TAG) $(TEMP_DIR) rm -rf $(TEMP_DIR) push: build gcloud docker -- push $(REGISTRY)/$(IMAGE)-$(ARCH):$(TAG) </ins>

<ins> ### debian-hyperkube-base Serves as the base image for `gcr.io/google-containers/hyperkube-${ARCH}` images. This image is compiled for multiple architectures. #### How to release If you're editing the Dockerfile or some other thing, please bump the `TAG` in the Makefile. ```console # Build for linux/amd64 (default) $ make push ARCH=amd64 # ---> gcr.io/google-containers/debian-hyperkube-base-amd64:TAG $ make push ARCH=arm # ---> gcr.io/google-containers/debian-hyperkube-base-arm:TAG $ make push ARCH=arm64 # ---> gcr.io/google-containers/debian-hyperkube-base-arm64:TAG $ make push ARCH=ppc64le # ---> gcr.io/google-containers/debian-hyperkube-base-ppc64le:TAG $ make push ARCH=s390x # ---> gcr.io/google-containers/debian-hyperkube-base-s390x:TAG ``` If you don't want to push the images, run `make build` instead [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/build/debian-hyperkube-base/README.md?pixel)]() </ins>

FROM BASEIMAGE <del> # If we're building for another architecture than amd64, the CROSS_BUILD_ placeholder is removed so e.g. CROSS_BUILD_COPY turns into COPY # If we're building normally, for amd64, CROSS_BUILD lines are removed CROSS_BUILD_COPY qemu-ARCH-static /usr/bin/ RUN DEBIAN_FRONTEND=noninteractive apt-get update -y  && DEBIAN_FRONTEND=noninteractive apt-get -yy -q install  iptables  ebtables  ethtool  ca-certificates  conntrack  util-linux  socat  git  jq  nfs-common  glusterfs-client  cifs-utils  ceph-common  && DEBIAN_FRONTEND=noninteractive apt-get upgrade -y  && DEBIAN_FRONTEND=noninteractive apt-get autoremove -y  && DEBIAN_FRONTEND=noninteractive apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # CACHEBUST RUN cp /usr/bin/nsenter /nsenter # Copy the the cni-bin folder into /opt/cni/bin COPY cni-bin/bin /opt/cni/bin </del> # Create symlinks for each hyperkube server # Also create symlinks to /usr/local/bin/ where the server image binaries live, so the hyperkube image may be # used instead of gcr.io/google_containers/kube-* without any modifications.

# Build the hyperkube image. # # Usage: <del> #   [ARCH=amd64] [REGISTRY="gcr.io/google_containers"] make (build|push) VERSION={some_released_version_of_kubernetes} </del> <ins> #   [ARCH=amd64] [REGISTRY="gcr.io/google-containers"] make (build|push) VERSION={some_released_version_of_kubernetes} </ins> <del> REGISTRY?=gcr.io/google_containers </del> <ins> REGISTRY?=gcr.io/google-containers </ins> ARCH?=amd64 <del> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) CNI_RELEASE=0799f5732f2a11b329d9e3d51b9c8f2e3759f2ff CACHEBUST?=1 QEMUVERSION=v2.7.0 HYPERKUBE_BIN?=_output/dockerized/bin/linux/${ARCH}/hyperkube HOSTARCH?=amd64 UNAME_S:=$(shell uname -s) ifeq ($(UNAME_S),Darwin) SED_CMD?=sed -i "" endif ifeq ($(UNAME_S),Linux) SED_CMD?=sed -i endif </del> <ins> HYPERKUBE_BIN?=_output/dockerized/bin/linux/$(ARCH)/hyperkube </ins> <del> ifeq ($(ARCH),amd64) BASEIMAGE?=debian:jessie endif ifeq ($(ARCH),arm) BASEIMAGE?=armhf/debian:jessie QEMUARCH=arm endif ifeq ($(ARCH),arm64) BASEIMAGE?=aarch64/debian:jessie QEMUARCH=aarch64 endif ifeq ($(ARCH),ppc64le) BASEIMAGE?=ppc64le/debian:jessie QEMUARCH=ppc64le endif ifeq ($(ARCH),s390x) BASEIMAGE?=s390x/debian:jessie QEMUARCH=s390x endif </del> <ins> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.1 TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) </ins> all: build

$(error VERSION is undefined) endif cp -r ./* ${TEMP_DIR} <del> mkdir -p ${TEMP_DIR}/cni-bin </del> cp ../../../${HYPERKUBE_BIN} ${TEMP_DIR} chmod a+rx ${TEMP_DIR}/hyperkube <del> cd ${TEMP_DIR} && sed -i.back "s|ARCH|${QEMUARCH}|g" Dockerfile </del> cd ${TEMP_DIR} && sed -i.back "s|BASEIMAGE|${BASEIMAGE}|g" Dockerfile <del> ifeq ($(CACHEBUST),1) cd ${TEMP_DIR} && sed -i.back "s|CACHEBUST|$(shell uuidgen)|g" Dockerfile endif ifeq ($(ARCH),$(HOSTARCH)) # When building "normally", remove the whole line, it has no part in the image cd ${TEMP_DIR} && ${SED_CMD} "/CROSS_BUILD_/d" Dockerfile else cd ${TEMP_DIR} && ${SED_CMD} "s/CROSS_BUILD_//g" Dockerfile # When cross-building, only the placeholder "CROSS_BUILD_" should be removed # Register /usr/bin/qemu-ARCH-static as the handler for ARM binaries in the kernel docker run --rm --privileged multiarch/qemu-user-static:register --reset curl -sSL --retry 5 https://github.com/multiarch/qemu-user-static/releases/download/$(QEMUVERSION)/x86_64_qemu-${QEMUARCH}-static.tar.gz | tar -xz -C ${TEMP_DIR} endif # Download CNI curl -sSL --retry 5 https://storage.googleapis.com/kubernetes-release/network-plugins/cni-${ARCH}-${CNI_RELEASE}.tar.gz | tar -xz -C ${TEMP_DIR}/cni-bin </del> docker build --pull -t ${REGISTRY}/hyperkube-${ARCH}:${VERSION} ${TEMP_DIR} rm -rf "${TEMP_DIR}"

gcloud docker -- push ${REGISTRY}/hyperkube:${VERSION} endif <del> .PHONY: all </del> <ins> .PHONY: build push all </ins>