Change:
This is at least partially responsible for - some tests can't run when dns is down. Logs from just before the crash: It got killed because Kubelet decided to kill it (readiness probe failed): It started happening quite recently - I'm trying to locate a build which caused this. Repro: run "Kubelet should not restart containers across restart" test - it caused this behavior for me multiple times today. cc - I applied the P0/v1.3 label/milestone, as I think it should be a blocker. Feel free to decrease the priority if you disagree
gce-serial 1511 seems to be the first one with this problem.
Or 1510 - hard to tell, as it was a broken run. There's no PR around that looks too suspicious though:/
I locally reverted and it seems that DNS is no longer failing. Sadly I have to run to catch a train, but maybe someone can pick it up here ?
Taking a look now. The PR should be unrelated...I am checking to see if the Federation DNS changes went in yet. Add nethealth container to prepull manifest
I search the restart counts for jobs from 1510-latest and gce-serial Job 1540 is the first one to show a non-zero kubedns restart count. If there is a commit responsible, it would be between 1539 and 1540 (June 21 7 AM -11 AM) 1539 end of run 1540 end of run Latest
The kubedns:1.3 container hasn't been updated since (Sat, 04 Jun 2016 03:29:37.847 GMT). GET
cc The kubedns container code is from June 04th. Dawn suggested that its networking issues that are causing kubedns container to fail its readiness check. I will need to add logic to print kubedns container logs during e2e tests, I am trying to search for any logs that might help, otherwise I will have to debug the next run live.
As reported, a simple kubelet restart triggers this issue - it looks like 10.0.0.1 is unreachable from the kubedns container. The kubedns container blocks indefinitely waiting for a connection to the kubernetes service at 10.0.0.1 before setting up the healthz handlers - the readiness probe will fail until kubedns successfully connects to the API server.
- you can try reverting and running it again - it worked for me for some reason (it might be by accident though)
I think the root cause is that cbr0 doesn't have an IP address assigned after kubelet restart - all the pods running behind cbr0 are isolated from the network. Not only kube-dns, but after my kubelet restart, the other addon pods are also stuck in a crash loop.
This sounds like a blocker even more now.
Looks like a regression. I am investigating.
Found the root cause. There is a subnet ip mismatch. vs . Hence kubenet thinks they are different and clears the address. Will submit a fix soon.
i found coredns status is CrashLoopBackOff and i check the logs : i got tips from so i restart my docker , i check the logs , there is no error in logs again

Discussion:
utilerrors "k8s.io/kubernetes/pkg/util/errors" utilexec "k8s.io/kubernetes/pkg/util/exec" utiliptables "k8s.io/kubernetes/pkg/util/iptables" <ins> utilnet "k8s.io/kubernetes/pkg/util/net" </ins> utilsets "k8s.io/kubernetes/pkg/util/sets" utilsysctl "k8s.io/kubernetes/pkg/util/sysctl"

// Ensure cbr0 has no conflicting addresses; CNI's 'bridge' // plugin will bail out if the bridge has an unexpected one <del> plugin.clearBridgeAddressesExcept(cidr.IP.String()) </del> <ins> plugin.clearBridgeAddressesExcept(cidr) </ins> } }

} } <del> func (plugin *kubenetNetworkPlugin) clearBridgeAddressesExcept(keep string) { </del> <ins> func (plugin *kubenetNetworkPlugin) clearBridgeAddressesExcept(keep *net.IPNet) { </ins> bridge, err := netlink.LinkByName(BridgeName) if err != nil { return

} for _, addr := range addrs { <del> if addr.IPNet.String() != keep { glog.V(5).Infof("Removing old address %s from %s", addr.IPNet.String(), BridgeName) </del> <ins> if !utilnet.IPNetEqual(addr.IPNet, keep) { glog.V(2).Infof("Removing old address %s from %s", addr.IPNet.String(), BridgeName) </ins> netlink.AddrDel(bridge, &addr) } }

<ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package net import ( "net" "reflect" ) // IPNetEqual checks if the two input IPNets are representing the same subnet. // For example, //	10.0.0.1/24 and 10.0.0.0/24 are the same subnet. //	10.0.0.1/24 and 10.0.0.0/25 are not the same subnet. func IPNetEqual(ipnet1, ipnet2 *net.IPNet) bool { if ipnet1 == nil || ipnet2 == nil { return false } if reflect.DeepEqual(ipnet1.Mask, ipnet2.Mask) && ipnet1.Contains(ipnet2.IP) && ipnet2.Contains(ipnet1.IP) { return true } return false } </ins>

<ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package net import ( "net" "testing" ) func getIPNet(cidr string) *net.IPNet { _, ipnet, _ := net.ParseCIDR(cidr) return ipnet } func TestIPNetEqual(t *testing.T) { testCases := []struct { ipnet1 *net.IPNet ipnet2 *net.IPNet expect bool }{ //null case { getIPNet("10.0.0.1/24"), getIPNet(""), false, }, { getIPNet("10.0.0.0/24"), getIPNet("10.0.0.0/24"), true, }, { getIPNet("10.0.0.0/24"), getIPNet("10.0.0.1/24"), true, }, { getIPNet("10.0.0.0/25"), getIPNet("10.0.0.0/24"), false, }, { getIPNet("10.0.1.0/24"), getIPNet("10.0.0.0/24"), false, }, } for _, tc := range testCases { if tc.expect != IPNetEqual(tc.ipnet1, tc.ipnet2) { t.Errorf("Expect equality of %s and %s be to %v", tc.ipnet1.String(), tc.ipnet2.String(), tc.expect) } } } </ins>