Change:
We have been running a Kube cluster on 1.2-alpha.7 for quite a while with no problems. Now trying to upgrade to Kube 1.2-beta.1 the pods end up in a restart loop and never come online. The upgrade is done via a complete reinstall of OS and Kubernetes and the issue has been recreated several times by upgrading and downgrading between alpha.7 and beta-1 OS: Centos 7.2 Docker: 1.9.1 Kube version: 1.2-beta.1 CNI Provider: calico cni 1.1.0 / calicoctl: 0.17.0 kubectl --namespace=kube-system get pod -o wide kubectl get events for one pod journalctl -xn 200 -u kubelet (kubelet running with -v=5) I can't see anything weird in the logs for the API Server, Scheduler or Controller Manager
is a process which checks the name to see whether it is resolvable, if it is not, it will kill the container and , maybe you could get some luck by checking the log of
The problem is that it isn't only skydns and kube2sky dying. The registry (which I haven't health checks to) and kube-ui dies as well very quickly
From the log you attached, I saw the error message from Kubelet: Mar 11 06:47:53 srv04 kubelet[]: E0311 06:47:53. ] NetworkPlugin cni failed on the status hook for pod 'kube-ui-v5-whtv9' - invalid CIDR address: Device "eth0" does not Looks like there is a configuration issue of cni on your node, which causes PodInfra container failure. cc/
Hmm, kubernetes will pass eth0 in as CNI_IFNAME and expect the plugin to create such an interface. The error message pointed out by Dawn: makes me think this isn't happening for calico? (you're not running your ui/DNS pods with host networking for some reason, right?).
Looks like it might be an issue with the Calico install - I'm out of the office this week, but should be able to help!
here you go I've also included the config I'm using
Reupload of log with only relevant parts
Thanks, I'm taking a look now.
for your information I ran same version of calico under 1.2-alpha.7 and don't have issues however I do see similar things in the logs.. I can also see that an IP has been assigned to the pod if I do kubectl desribe pod on beta.1
I took a look at the logs (couldn't resist) and it looks like the Calico CNI plugin is doing what I'd expect. From the logs: It reports that it creates in the pod namespace right before kubelet reports that doesn't exist. I'd be interested to know which one is telling the truth... The Calico CNI plugin would notice if the pod was running with host networking, and it wouldn't try to configure networking, so I don't think that is the issue here.
Host networking isn't used unless I've screwed up completely All pods call each other on the 10.240 network and services on 10.241 The hosts are on 172.16
I'm struggling to get to the bottom of this. This piece of code is where the kubelet error is coming from - It would seem that GetContainerIP is returning the string "Device "eth0" does not" and not signalling an error. That is then tripping up net.ParseCIDR. I'll continue working on this tomorrow - still not sure if it's a Calico problem or a more general Kubernetes-with-CNI problem
Hmm this may be a timing issue. I some debug logs around the line above and then the error statement goes away. I'm also pretty sure I've seen this error in alpha7 which means the issue with eth0 may not be the issue causing restarts but still filling the logs with red herrings. Not sure though
Some more information All works well with 1.2-beta.0 The issue with constantly restarting containers still happens with 1.2-beta.1 I can't see the networking errors when running 1.2-beta0 so maybe is related after all
I'm still looking at this but struggling to reproduce it. - have you make any changes in this area? said that you might have.
If it really is a regression from beta0 to beta1, the diff here might help - The one change that jumps out this this one from
the changes that I made were at a higher level, and were around how the kubelet determines the pod IP to inject into: The hosts file that is bind-mounted into the container Environment variables that consume the pod IP via the downward API Relevant PR is
I'm bringing up a new cluster, to be 100%v sure, but given the lack of failure on e2e, it looks like this is CNI specific, yes? I need a signal on whether this is a stop-ship - can you confirm or deny? Marking p0 for triage
Make sure you grab if you're running with kubenet (which btw we can also P0 triage, I thought ship had sailed)
I don't think is a release-blocker, and I am not 100% sure this is either.
xref
It'd be easy for me to try reproducing this with , if that could help?
are you thing whether this may be final release blocker or the the next beta? How beta are cut, by the way? Looks like alphas been getting a few PRs in each, while latest beta only has one PR...
There's 1.2.0-beta.0 and 1.2.0-beta.1. What we thought was the last batch of cherrypicks for 1.2.0 is in and, barring problems, we expect that to merge within the hour. Then, if tests pass consistently, we expect to build 1.2.0 tomorrow morning. So, if we decide this is a release blocker, it is the last remaining blocker.
Any confirmation would be helpful On Mar 15, 2016 5:20 PM, "Brian Grant" wrote:
I'll be able to run my test agains AM GMT, it's pretty late here now...
so come from , which is called with a pipe like this , and we are clearly not handling the return code properly... Although, I do wish this code wasn't shelling out, debugging it as-is should be doable, if I knew how to stop the world in a reliable way, so any hints on that would be hugely appreciated (I heard that gdb doesn't work quite well with Go...).
I'm not able to reproduce this with a simple test that creates a busybox pod using the stock CNI bridge plugin. The "eth0" thing might be a transient race of Status invocation that just fixes itself. To isolate the unknown can either or please try with: and to the kubelet CNI plugins into a netconf into , like: And you're off to the races. Also can you please confirm that this isn't a DNS problem? since by default we proxy all DNS lookups through kube-dns your other containers like kube-ui might be restarting because they can't lookup.
I've done some testing on my side I built 1.2.0-beta0 up until the commit before which works fine and then I rebuilt with and the issues with restarting pods comes back.. looking at the logs of one pod it shows that the termination signal has been sent to it. I also did some logging inside of and can see that the last check comparing hash always returns that it has changed. Not sure why yet
I am testing bridge plugin now, as you suggested, instead of Weave plugin.
I've found the issue for the restarting pods. It is when using CNI and the docker image exports ports as well. This is due to a difference between the creation of the Container object when creating the pod and when checking if it has changed. You should be able to replicate this by running a docker image with exported ports under kubernetes + cni. I should have pull request ready soon
Also, I have discovered , which makes me a little suspicious whether tests agains CNI only ran on internal system of whoever uploaded the build...
One thing that makes this kind of thing very hard to debug (at least for me, not hugely experienced working with k8s) is that after a short while k8s decides things are bad, kills a bunch of things and restarts them to try again. I understand why it does that, but is there a way to stop it while I get a chance to look at what happened?
I am not sure what you are saying exactly, but will wait for your pull-request.
check this if statement vs you can see that the case is slightly different which means the container the hash is calculated against originally will always differ from the one that is checked for changes in the case where the docker image exports ports
that would explain why worked just fine with Weave plugin!
correct
Also, using the CNI binaries from the released tarball with a workaround and config from () I get this:
please see the pull request which is minimal. Personally I'm ok to wait to put it in 1.2.1 as I have a patched build However for someone new to kubernetes that decides to use weave, calico or similar and deploys a docker image with exported ports they will end up with pods restarting and zero logs on why It's quite a bad experiance
cc
Please reference this issue from the PR, not from commits.
sorry about that, it to the pull request.
we're not quite at the stage where we're dog fooding CNI yet (kubenet is off by default). We hope to get there sometime between now and 1.3. Thanks for the quick turn around on this, please keep filing issues where different plugins/environments diverge.
Personal 'm having this error also in version 1.2.3
this error not viewed in version 1.3.0-alpha.2.
Hi I'm having the same error, NetworkPlugin cni failed on the status hook for pod 'nginx' - invalid CIDR address: Device "eth0" does not exist, what was the fix exactly?
Bug for CNI driver tab caracteres not allowed in file, i am submit pull request for correction. Verify tour file contains tab caracters and remove
My and looks like: My has: I've checked and I don't have tabs in either file.
you are running it in a ansible playbook ?, /etc/cni/net.d/10- "etcdauthority": "{{etcdauthority}}" should have ip ETCD server, and the environment must have the tambpém DEFAULTIPV4 settings such as the server ip and ETCDAUTHORITY with the ip of the server ETCD.
you using systemd? /etc/enviroment should be set in kubelet variables section
Yes, on the server I have : and : and the following flags on kubelet:
post your kubelet.service?
you are not setting the / etc / network-environment in your file as variables, it may be the error, add EnvironmentFile=-/etc/network-environment
inform it is work, ok?
Okay, let me try that.
Tried it, but I still get the same error:
I deleted and recreated the nginx pod.
It's an azure issue

Discussion:
glog.V(4).Infof("host: %v, %v", pod.Spec.SecurityContext.HostNetwork, networkMode) return true, nil } <del> } else { </del> <ins> } else if dm.networkPlugin.Name() != "cni" && dm.networkPlugin.Name() != "kubenet" { </ins> // Docker only exports ports from the pod infra container. Let's // collect all of the relevant ports and export them. for _, container := range pod.Spec.Containers {