Change:
Doesn't seem to haven been caused by my change:
/cc
The test only failed on coreos, and the is full of error messages. Plus the log is almost unreadable because of the extremely high verbosity. I filed for that. shows that the node has been rebooted a couple of times. And the kernel log shows This looks like . The hairpin mode is set to hairpin-veth. /cc
Yeah don't run with hairpin-mode I guess. is coreos using kubenet?
Node problem detector should detect this issue and report KernelDeadLock condition.
Yeah, but this is node e2e test, we don't have node problem detector running on node e2e cluster.
Not sure how to assign coreos issues... Assigning to people to reassign based on thread activity. Please reassign correctly!
ping!
/cc
This happens in node e2e test right? I believe node e2e test is NOT using kubenet.
Running node e2e with kubenet+hairpin-veth could prevent this.
Sorry, this is lost in github notifications.. Feel free to ping me on slack if I am not responding in a day next time :)
To run kubenet on a node, the framework needs to download and unpack the cni plugin in kubelet needs to run with and a fixed pod CIDR (since there is no node controller) I did those in , but it failed on the trusty images because is not installed. We need to either install this in the node e2e framework, or bake it in to the image. what's the best practice for installing packages on the images?
I believe updating is the best way to bake it into the image, and I also think that should be preferred over installing it each time. You could do the same with CNI. Thanks for working on this!
I updated the PR to install nsenter on the trusty images. As for cni, I still keep it in the node e2e framework to install it every time because 1) it only downloads from GCS and untar a file of ~14MB, 2) the tarball is maintained and updated by It'd be an extra burden to update the images just for cni updates.
OK with the plan.
Seeing the error message again after switching to kubenet+hairpin-veth do you want to take a look at the logs?
let me take a look
urrrh... we hit it again. Switching back to
There's a difference in the above kernel log; the device is not a interface but rather an interface (presumably the one within a container's net ns since there is no eth0 on the host). I think all the prior instances of this were veths on the host.
True. This one is slightly different. I think deadlock on interface also showed up before.
Haven't seen any new occurrence since landed. We'll know in a day or so whether this problem has been fixed.
AFAIK, there are no new failures after landed. It should be safe to let this issue stay closed.
unfortunately I hit the bug with hairpinmode sets to promiscuous-bridge kubelet v1.3.0, linux kernel 4.6.3 cbr0 has already enter the promiscuous mode, all veth hairpinmode = 1 start with parameter
can you ? what distro are you running?
also are you using a network-plugin, overlay (like flannel), which cloudprovider?
Are the SELinux warnings normal ?
Maybe a little out of topic. Are you running K8s on GCE? Do you have the node problem detector running in your cluster?
all veths hairpin_mode = 1 my distro is coreos 1068.6.0 I don't use any network-plugins, and no cloudprovider support here, controller-manager start with kubelet setup after address allocated by controller-manager, all pods connected each other with static routes which controlled by ospfd running inside docker. I can confirm the selinux warnings not related to the bug.
not running on GCE. no detector running inside.
Alright, just want to verify whether the NPD works as expected.
here is the logs of kubelet startup.
That's the problem, the reason we ran into this to begin with is because of hairpin mode. Apparently it is still getting set, or isn't getting erased in your environment. I don't think we erase existing values.
that's wired. I checked all my nodes, each veth hairpin_mode is 1. it seems it was set to 1 regardless of --configure-cbr0.
there is no existing veth before kubelet create bridge , docker could not start without cbr0 which created by kubelet.
hairpin_mode is always enabled if the device is a bridge interface,
Trust me it's not on my (and our other e2e test) clusters, or we'd know because tests would fail, and teh default setup is docker running on cbr0 and using veths. In 1.3 we started using kubenet network plugin by default, but hairpin flag wasn't set since 1.2. I can trace the code path down when I have some time, but I suspect it's something to do with the way coreos is setup.
I trust you, I promise. I did some tests. None of results told me the flag is set by kubelet. I could not figure out what was happend to these veths. finally, I duplicate to , the only difference is hairepinEnable default to "0", and invoke the hairpinunset.SetUpContainerPid when false. It works! But, if I create veth by hand, the default hairpin_mode is 0 , oh my god. It will not happen in my test environment, only in production env
will enable hairpin_mode That's my fault, I should not set it to false because kubernetes do not use docker's network model.

Discussion:
var sshOptionsMap map[string]string <del> const archiveName = "e2e_node_test.tar.gz" </del> <ins> const ( archiveName = "e2e_node_test.tar.gz" CNI_RELEASE = "c864f0e1ea73719b8f4582402b0847064f9883b0" ) var CNI_URL = fmt.Sprintf("https://storage.googleapis.com/kubernetes-release/network-plugins/cni-%s.tar.gz", CNI_RELEASE) </ins> var hostnameIpOverrides = struct { sync.RWMutex

} } <ins> // Install the cni plugin. Note that /opt/cni does not get cleaned up after // the test completes. if _, err := RunSshCommand("ssh", GetHostnameOrIp(host), "--", "sh", "-c", getSshCommand(" ; ", "sudo mkdir -p /opt/cni", fmt.Sprintf("sudo wget -O - %s | sudo tar -xz -C /opt/cni", CNI_URL))); err != nil { // Exit failure with the error return "", false, err } </ins> // Create the temp staging directory glog.Infof("Staging test binaries on %s", host) tmp := fmt.Sprintf("/tmp/gcloud-e2e-%d", rand.Int31())

"--config", es.kubeletStaticPodDir, "--file-check-frequency", "10s", // Check file frequently so tests won't wait too long "--v", LOG_VERBOSITY_LEVEL, "--logtostderr", <ins> "--network-plugin=kubenet", "--pod-cidr=10.180.0.0/24", // Assign a fixed CIDR to the node because there is no node controller. "--hairpin-mode=hairpin-veth", </ins> ) cmd := exec.Command("sudo", cmdArgs...) hcc := newHealthCheckCommand(

rm -r etcd-v2.2.5-linux-amd64* fi <ins> # Install nsenter for ubuntu images cat /etc/*-release | grep "ID=ubuntu" if [ $? -eq 0 ]; then if ! which nsenter > /dev/null; then echo "Do not find nsenter. Install it." mkdir -p /tmp/nsenter-install cd /tmp/nsenter-install curl https://www.kernel.org/pub/linux/utils/util-linux/v2.24/util-linux-2.24.tar.gz | tar -zxf- sudo apt-get update sudo apt-get --yes install make sudo apt-get --yes install gcc cd util-linux-2.24 ./configure --without-ncurses make nsenter sudo cp nsenter /usr/local/bin rm -rf /tmp/nsenter-install fi fi </ins> # Install docker hash docker 2>/dev/null if [ $? -ne 0 ]; then

# To copy an image between projects: # `gcloud compute --project <to-project> disks create <image name> --image=https://www.googleapis.com/compute/v1/projects/<from-project>/global/images/<image-name>` # `gcloud compute --project <to-project> images create <image-name> --source-disk=<image-name>` <del> GCE_IMAGES=e2e-node-ubuntu-trusty-docker10-image,e2e-node-ubuntu-trusty-docker9-image,e2e-node-coreos-stable20160622-image,e2e-node-containervm-v20160321-image </del> <ins> GCE_IMAGES=e2e-node-ubuntu-trusty-docker9-v1-image,e2e-node-ubuntu-trusty-docker10-v1-image,e2e-node-coreos-stable20160622-image,e2e-node-containervm-v20160321-image </ins> GCE_ZONE=us-central1-f GCE_PROJECT=kubernetes-jenkins GCE_IMAGE_PROJECT=kubernetes-node-e2e-images

# To copy an image between projects: # `gcloud compute --project <to-project> disks create <image name> --image=https://www.googleapis.com/compute/v1/projects/<from-project>/global/images/<image-name>` # `gcloud compute --project <to-project> images create <image-name> --source-disk=<image-name>` <del> GCE_IMAGES=e2e-node-ubuntu-trusty-docker10-image,e2e-node-ubuntu-trusty-docker9-image,e2e-node-coreos-stable20160622-image,e2e-node-containervm-v20160321-image </del> <ins> GCE_IMAGES=e2e-node-ubuntu-trusty-docker9-v1-image,e2e-node-ubuntu-trusty-docker10-v1-image,e2e-node-coreos-stable20160622-image,e2e-node-containervm-v20160321-image </ins> GCE_ZONE=us-central1-f GCE_PROJECT=kubernetes-jenkins-pull GCE_IMAGE_PROJECT=kubernetes-node-e2e-images