Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-proxy fails to clear IPv6 conntrack entries What you expected to happen: conntrack entries should be cleared How to reproduce it (as minimally and precisely as possible): Deploying a cluster with kind and IPv6 and restarting the coredns pods Error in kube-proxy Running conntrack from the host Running conntrack from the kube-proxy container Anything else we need to know?: Seems to be a bug in the conntrack tools 1.4.4 per snippet pasted previously Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
Seems that is the higher version provided by the debian-iptables base-image
debian buster is the new stable release and have conntrack 1.4.5, what's the best way moving forward? file a bug in debian stretch to fix conntrack tools or ask to upgrade kubernetes base images to ?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/assign
/assign
I don't think is a high priority because the service is working despite the error, but I don't know if this conntrack problem can cause bigger issues
Can confirm the behavior. Tested in a Linux Mint that ships with conntrack-tools 1.4.4 (compiled with libnetfilter-conntrack 1.0.6). After recompiling it to conntrack-tools 1.4.5 and libnefilter-conntrack 1.0.7 it works like a charm :) Debian 9.8 (used in Kubernetes images) and 9.9 ships with conntrack 1.4.4, while Buster (10) ships with the newer version. is this the case we should take this subject to the next sig-network meeting? I think upgrading this touches a lot of other parts from Kubernetes, don't know if this is a subject of a bigger WG instead of a SIG. This doesn't impact in users with people using non-containerized version of kube-proxy (and other components), but kubeadm installs, kind, hyperkube and anything else using the official Kubernetes component images is affected.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry. ‍
OK, I've reached out the package owner in Debian, but the solution is to update the container images to Debian 10. I'll take this to the next sig-network meeting to see how can we deal with this
/priority important-longterm
/remove-triage unresolved
/assign
if I can help with anything else, let me know :)
Do we need to catch this for dual-stack?
This will be fixed by the kube-proxy docker image update

Discussion:
# $1 - server architecture kube::build::get_docker_wrapped_binaries() { local arch=$1 <del> local debian_base_version=v1.0.0 local debian_iptables_version=v11.0.2 </del> <ins> local debian_base_version=v2.0.0 local debian_iptables_version=v12.0.1 </ins> ### If you change any of these lists, please also update DOCKERIZED_BINARIES ### in build/BUILD. And kube::golang::server_image_targets local targets=(

# Several utilities (e.g. ping) were kept for usefulness, but may be removed in later versions. RUN echo "Yes, do as I say!" | apt-get purge  bash  <del> debconf-i18n  e2fslibs  </del> e2fsprogs  <del> init  initscripts  </del> libcap2-bin  <del> libkmod2  </del> libmount1  libsmartcols1  <del> libudev1  </del> libblkid1  <del> libncursesw5  libprocps6  libslang2  </del> libss2  <del> libsystemd0  libtext-charwidth-perl libtext-iconv-perl libtext-wrapi18n-perl  </del> ncurses-base  ncurses-bin  <del> systemd  systemd-sysv  sysv-rc  </del> tzdata # No-op stubs replace some unnecessary binaries that may be depended on in the install process (in

IMAGE ?= $(REGISTRY)/debian-base BUILD_IMAGE ?= debian-build <del> TAG ?= v1.0.0 </del> <ins> TAG ?= v2.0.0 </ins> TAR_FILE ?= rootfs.tar ARCH?=amd64

export DOCKER_CLI_EXPERIMENTAL := enabled ifeq ($(ARCH),amd64) <del> BASEIMAGE?=debian:stretch </del> <ins> BASEIMAGE?=debian:buster-slim </ins> endif ifeq ($(ARCH),arm) <del> BASEIMAGE?=arm32v7/debian:stretch </del> <ins> BASEIMAGE?=arm32v7/debian:buster-slim </ins> QEMUARCH=arm endif ifeq ($(ARCH),arm64) <del> BASEIMAGE?=arm64v8/debian:stretch </del> <ins> BASEIMAGE?=arm64v8/debian:buster-slim </ins> QEMUARCH=aarch64 endif ifeq ($(ARCH),ppc64le) <del> BASEIMAGE?=ppc64le/debian:stretch </del> <ins> BASEIMAGE?=ppc64le/debian:buster-slim </ins> QEMUARCH=ppc64le endif ifeq ($(ARCH),s390x) <del> BASEIMAGE?=s390x/debian:stretch </del> <ins> BASEIMAGE?=s390x/debian:buster-slim </ins> QEMUARCH=s390x endif

FROM BASEIMAGE <ins> # Install latest iptables package from buster-backports RUN echo deb http://deb.debian.org/debian buster-backports main >> /etc/apt/sources.list;  apt-get update;  apt-get -t buster-backports -y --no-install-recommends install iptables # Install other dependencies and then clean up apt caches </ins> RUN clean-install  conntrack  ebtables  ipset  <del> iptables  </del> kmod  netbase <ins> # Install iptables wrapper scripts to detect the correct iptables mode # the first time any of them is run COPY iptables-wrapper /usr/sbin/iptables-wrapper RUN update-alternatives  --install /usr/sbin/iptables iptables /usr/sbin/iptables-wrapper 100  --slave /usr/sbin/iptables-restore iptables-restore /usr/sbin/iptables-wrapper  --slave /usr/sbin/iptables-save iptables-save /usr/sbin/iptables-wrapper RUN update-alternatives  --install /usr/sbin/ip6tables ip6tables /usr/sbin/iptables-wrapper 100  --slave /usr/sbin/ip6tables-restore ip6tables-restore /usr/sbin/iptables-wrapper  --slave /usr/sbin/ip6tables-save ip6tables-save /usr/sbin/iptables-wrapper </ins>

REGISTRY?="staging-k8s.gcr.io" IMAGE=$(REGISTRY)/debian-iptables <del> TAG?=v11.0.2 </del> <ins> TAG?=v12.0.1 </ins> ARCH?=amd64 ALL_ARCH = amd64 arm arm64 ppc64le s390x TEMP_DIR:=$(shell mktemp -d) <del> BASEIMAGE?=k8s.gcr.io/debian-base-$(ARCH):v1.0.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/debian-base-$(ARCH):v2.0.0 </ins> # This option is for running docker manifest command export DOCKER_CLI_EXPERIMENTAL := enabled

<ins> #!/bin/sh # Copyright 2019 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -e # Detect whether the base system is using iptables-legacy or # iptables-nft. This assumes that some non-containerized process (eg # kubelet) has already created some iptables rules. # Bugs in iptables-nft 1.8.3 may cause it to get stuck in a loop in # some circumstances, so we have to run the nft check in a timeout. To # avoid hitting that timeout, we only bother to even check nft if # legacy iptables was empty / mostly empty. num_legacy_lines=$( (iptables-legacy-save || true; ip6tables-legacy-save || true) 2>/dev/null | grep '^-' | wc -l) if [ "${num_legacy_lines}" -ge 10 ]; then mode=legacy else num_nft_lines=$( (timeout 5 sh -c "iptables-nft-save; ip6tables-nft-save" || true) 2>/dev/null | grep '^-' | wc -l) if [ "${num_legacy_lines}" -ge "${num_nft_lines}" ]; then mode=legacy else mode=nft fi fi update-alternatives --set iptables "/usr/sbin/iptables-${mode}" > /dev/null update-alternatives --set ip6tables "/usr/sbin/ip6tables-${mode}" > /dev/null # Now re-exec the original command with the newly-selected alternative exec "$0" "$@" </ins>

# Use go get -u github.com/estesp/manifest-tool to find these values _DEBIAN_BASE_DIGEST = { <del> "manifest": "sha256:6966a0aedd7592c18ff2dd803c08bd85780ee19f5e3a2e7cf908a4cd837afcde", "amd64": "sha256:8ccb65cd2dd7e0c24193d0742a20e4a673dbd11af5a33f16fcd471a31486866c", "arm": "sha256:3432b41de3f6dfffdc1386fce961cfd1f9f8e208b3a35070e10ef3e2a733cb17", "arm64": "sha256:9189251e1d1eb4126d6e6add2e272338f9c8a6a3db38863044625bca4b667f31", "ppc64le": "sha256:50aa659e1e75e4231ee8293c3b4115e5755bb0517142b9b4bddbc134bf4354db", "s390x": "sha256:bbb8ee3a2aaca738c00809f450233d98029fea4e319d8faaa30aa94c8b17a806", </del> <ins> "manifest": "sha256:ebda8587ec0f49eb88ee3a608ef018484908cbc5aa32556a0d78356088c185d4", "amd64": "sha256:d7be39e143d4e6677a28c81c0a84868b40800fc979dea1848bb19d526668a00c", "arm": "sha256:fc731da13b0bc9013b85a86b583fc92e50869b5bc8e7aa6ca730ec0240954c7d", "arm64": "sha256:12502c3eed050fa9b6d5fe353a44bfc5f437dc325c8912b1a48dcc180df36f1e", "ppc64le": "sha256:4277aa59b63c5a1369e6d84a295ecc4ffa08985dcf114de9f7b6de1af4fcbc86", "s390x": "sha256:78ef2a6b017539379c1654b4e52ba8519bfec821c62d0b3a1dbd15104b711e21", </ins> } _DEBIAN_IPTABLES_DIGEST = { <del> "manifest": "sha256:b522b0035dba3ac2d5c0dbaaf8217bd66248e790332ccfdf653e0f943a280dcf", "amd64": "sha256:adc40e9ec817c15d35b26d1d6aa4d0f8096fba4c99e26a026159bb0bc98c6a89", "arm": "sha256:58e8a1d3b187eed2d8d3664cd1c9723e5029698714a24dfca4b6ef42ea27a9d4", "arm64": "sha256:1a63fdd216fe7b84561d40ab1ebaa0daae1fc73e4232a6caffbd8353d9a14cea", "ppc64le": "sha256:9f90adbc7513cc96d92fcec7633c4b29e766dd31cf876af03c0b54374e22fa9c", "s390x": "sha256:4f147708deff2a0163ee49b6980cc95423514bec5f4091612d65773b898fbdae", </del> <ins> "manifest": "sha256:d1cd487e89fb4cba853cd3a948a6e9016faf66f2a7bb53cb1ac6b6c9cb58f5ed", "amd64": "sha256:852d3c569932059bcab3a52cb6105c432d85b4b7bbd5fc93153b78010e34a783", "arm": "sha256:c10f01b414a7cd4b2f3e26e152c90c64a1e781d99f83a6809764cf74ecbc46c3", "arm64": "sha256:5725e6fde13a6405cf800e22846ebd2bde24b0860f1dc3f6f5f256f03cfa85bd", "ppc64le": "sha256:b6d6e56a0c34c0393dcba0d5faaa531b92e5876114c5ab5a90e82e4889724c5a", "s390x": "sha256:39e67e9bf25d67fe35bd9dcb25367277e5967368e02f2741e0efd4ce8874db14", </ins> } _DEBIAN_HYPERKUBE_BASE_DIGEST = {

digest = _digest(_DEBIAN_BASE_DIGEST, arch), registry = "k8s.gcr.io", repository = "debian-base", <del> tag = "0.4.1",  # ignored, but kept here for documentation </del> <ins> tag = "v2.0.0",  # ignored, but kept here for documentation </ins> ) container_pull(

digest = _digest(_DEBIAN_IPTABLES_DIGEST, arch), registry = "k8s.gcr.io", repository = "debian-iptables", <del> tag = "v11.0.2",  # ignored, but kept here for documentation </del> <ins> tag = "v12.0.1",  # ignored, but kept here for documentation </ins> ) container_pull(