Change:
Is this a request for help? I think it is an issue either with software or documentation, but I am not quite sure. I have started with a question on stackoverflow: Is this a BUG REPORT or FEATURE REQUEST? (choose one): I think it is a bug or request to improve documentation Kubernetes version (use ): 0 Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: kubeadm init/join Others: What happened: I have got 2 VMs nodes. Both see each other either by hostname (through /etc/hosts) or by ip address. One has been provisioned with kubeadm as a master. Another as a worker node. Following the instructions () I have weave-net. The list of pods looks like the following: CrashLoopBackOff appears for each worker node connected. I have spent several ours playing with network interfaces, but it seems the network is fine. I have found similar question on stackoverflow, where the answer advised to look into the logs and no follow up. So, here are the logs: What you expected to happen: I would expect the weave-net to be in Running state How to reproduce it (as minimally and precisely as possible): I have not done anything special, just followed the documentation on Getting Started. If it is essencial, I can share Vagrant project, which I used to provision everything. Please, let me know if you need one.
Please, let me know if it needs to be reported on weave-net project.
I reported a similar issue here:
I wonder if there is any working step by step instruction on how to get kubernetes cluster up and running on a set of virtual machines? I am happy to downgrade the version of kubernetes if it is an option. I am totally confused with all of these options: kube-deploy, kube-up, and other, I found from 3rd parties..
I have the same issue. I'm using VirtualBox to run 2 VM based on minimal Centos 7 image. All VMs are attached to 2 interfaces, a NAT and an host-only network. The two VMs are able to connect to each other using the host-only network interfaces. I tried also with the instructions about Calico and Canal, and I cannot make them work either.
It seems Calico has got similar issue: What could I try to progress this issue further?
Okay I ran into this exact same issue and here is how I fixed it. This problem seems to be due to kube-proxy looking at the wrong network interface. If you look at the kube-proxy logs on a worker node you will most likely see something like: This is the wrong network interface. The kube-proxy should be looking at the master node's IP address not the NAT IP address. As far as I know the kube-proxy gets this value from the Kube API Server when starting up. If you look at the Kube API Server's documentation it states that if flag isn't set it will default to and if isn't set it will default to host's default interface. Which in my case and yours seems to be the NAT interface, which isn't what we want. So what I did was set the Kube API Server's flag and everything started working. So right after Step 2 and before Step 3 of You will need to update your and add the flag to point to your master node's IP address. So for example: My master node's IP address is , which means right after Step 2 I do: I am not to sure if this is a valid long term solution, because if the default kube- changes then those changes wouldn't get reflected by doing what I am doing. Ideally, I think the user would want some way to set these flags via kubeadm or maybe the user should be responsible for parsing the JSON themselves. Thoughts? However, it still may be a good idea to update Step 2 of to atleast mention to the users that they can update the kube component flags by modifying the their json found at: .
I am looking at this right now. We have a way of reproducing it with , where we have already applied a fix that makes kubelet pickup desride IP for pods in host network namespaces, however somehow service IP still points to 10.0.2.15.
I encountered this issue too. Adding --advertise-address when starting kube-apiserver solved this issue.
I've already started diving into the code, and you saved me time to realise how this part worked exactly, thank you so much! This looks like an easy fix for kubeadm, PR is coming shortly.
Ok, so it turns out that this flag is not enough, we still have an issue reaching service IP. The simplest solution to this is to run with . To enable this, you can use .
I am starting kubeadm with --advertise-address option: kubeadm init --api-advertise-addresses=$masteraddress where my $masteraddress is not NAT interface. It still not enough to remove this issue.
"Ok, so it turns out that this flag is not enough, we still have an issue reaching kubernetes service IP. The simplest solution to this is to run kube-proxy with --proxy-mode=userspace. To enable this, you can use kubectl -n kube-system edit ds kube-proxy-amd64 && kubectl -n kube-system delete pods -l name=kube-proxy-amd64." Could you, please, explain what flag are you setting, to what command in the getting started tutorial? What does your last command do?
see Also, you can do this for now:
We are going to fix this shortly. If you can hop on Slack, I'd be happy to help.
As this thread is getting quite noisy, here is a recap. First, find out what IP address you want to use on the master, it's probably the one on the second network interface. For this example I'll use . Next, run . Now, you want to append to in the static pod manifest, you can do it like this: And finally, you need to update flags in daemonset and append , which can be done like this:
I'm working on docs for this. Also, has provided a Vagrant+Ansible implementation, and latest fixes are in .
can you help me with similar problem, I have multiple eth*. And I am using for private network, but . For now I fixed issue with weave via open port 443 for eth0 and eth1. But found next issue: I could not access to the instances from the cluster(example to redis that hosts on separate instance). I found that weave uses network. And I am looking for solution to change this. Do you know how can I change it? Example to ?
I didn't have to update the for my kube-proxy. All I had to do was make sure the flag for the Kube API Server was set to a value listed in the flag. I'm just curious as to why you needed to set in your setup and I don't? Any ideas? I looked over the and your Vagrant setup looks very similar to mine, except you are using Ubuntu vs my Centos.
No, it uses . So for Weave we have , which expands to - . Kubernetes service range is , which expands to - . And you have , which expands to - . I don't see an overlap here, which is what you seem to be suggesting, but please correct me if I missunderstood. I think your problem is different, and please file another issue. Your previous issue didn't not describe the problem, please try to describe the problem. How many VMs are you running on? If it's a single-node setup, you won't notice anything. If it's not, I'm quite curious and would like to take a closer look, could you share your Vagrantfile?
after applying your changes I was able to get kube-dns and weave-net working! However, kube-scheduler-kubernetes won't run:
sorry, that was a copy/paste bug, I've updated the comments now, it's should had been not .
Yeah I figured that out shortly after my comment. I have everything working as expected now. Thank you to everyone who participated in fixing this! :)
you are welcome! We should eliminate the need for these work-arounds soon, at least as far as week can get without total hacks.
Thank you for the feedback. I'm getting closer to a working solution. With my basic Vagrant 2-node cluster (Centos) I now have weave up and running. The kube-dns service is however still not working. Steps taken: 1) kubeadm init on master with api-advertise-addresses flag set. 2) Update flags in pod manifest and daemonsets per guidelines above. 3) Join node. 4) Install weave on master. kube-dns is still stuck in ContainerCreating state: I experimented with both bridged network settings and private network settings but the problem still persists.
could you provide the output of ?
I thought it should be . After update api server to use specific ip address the wave could not connect to 100.64.0.1:443. Digital Ocean has also Anchor Ip to eth0 with mask: 10.10.0.0/16: Updated: So I have theory that somehow pointed to 159.203.188.178 even when I specify api server address to 10.136.26.23
sure, see below. I assume the last cni-related errors propagate before weave is installed. The local NAT-interface address is still visible in . Is this an issue?
You a right, it's that in the current release, but it changed in master. Yes, there is an issue with that in some DO regions... you need to pass , but not with the build you are using. If you are online now, it'd be easy if you could find me on Slack. Thanks.
just as a sanity check, could you kill the DNS pod, i.e. , may there is some bug on the CNI end and somehow it's not getting picked up for this pod...
Hmm no luck. After deletion:
resolved my issue via adding a routing rule to use eth1 for node machines to kubernetes service range ip. Example:
Despite following the workaround steps ((1) specify flag when running , (2) specify in manifest, (3) append to daemonset), I'm seeing the exact same issue as That is, after installing the weave pod network, is still stuck in : did you ever resolve your problem? Does anyone else have any ideas?
but weave-net was up 15m ago and dns last event was 25m ago. try to restart dns pod.
It would be easier if folks could hop on #kubeadm channel in Slack and we can debug step-by-step. It became really hard to pick individual setup differences in this thread. On Tue, 18 Oct 2016, 14:50 Michael Nikitochkin, wrote:
I did restart the pod with no luck (by the way, when you say restart a pod, I assume you mean deleting the pod (and have the replica set replace it), right)? It is stuck in , although it appears to have reached slightly farther, beeing able to at least create containers. However But I guess I'll follow advice and take it on the slack channel.
So I actually did manage to get it working at last. The last missing piece of the puzzle was to follow the advice of point 5) in If anyone is interested, my ended up looking as follows:
I am very happy to hear you got it working, we planning to add a proper vagrant how-to, once we have tested latest packages. Have you referred to the Ansible code in k8s-playgound repo linked from the docs? On Wed, 19 Oct 2016, 07:54 Peter Gardfjäll, wrote:
yes, I did have a look at that code and just adapted their snippet to my particular setup.
: thanks for the hint. I can also confirm that I have a working system now.
Have you enabled userspace mode in kube-proxy also? On Wed, 19 Oct 2016, 20:49 bach1, wrote:
I run a similar setup based on CoreOS boxes(two nics, eth0 nat and eth1 private) on Vagrant. I had to make two changes to get the networking to work: Set the --node-ip on each kubelet process to the private network ip for each node Enable masquerading on the proxy to respond over the correct interface(--masquerade-all)
Pease checkout , there is code to handle firewalld rules and SELinux stuff (if you are on CentOS), as well as VirtualBox issues (applies to Ubuntu as well as CentOS). We are planing to provide plain text documentation soon too. Find me on Slack if you still have problems. It's always a good idea to create a new issue rather then comment on an existing one that seems like it describes your problem.
I am seeing a similar issue. Here are the details: [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} [root ~]# kubeadm version kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} [root ~]# CentOS 7.2. Kernel Version: [root ~]# uname -a Linux 3.10.0-327.18.2.el7.x8664 SMP Thu May 12 11:03:55 UTC 2016 x8664 x8664 x8664 GNU/Linux These are based on an old version of OpenStack and KVM. if that is relevant, I can find the details
letting you know that workaround command is now being rejected: What could I do to fix it?
i am having the same issue as If you leave validation on, you get this error. i included the input from and both kind and apiVersion are clearly there. { "kind": "List", "apiVersion": "v1", "metadata": {}, "items": [ { "spec": { "template": { "spec": { "containers": [ { "command": [ "--cluster-cidr=10.32.0.0/12" ] } ] } } } } ] } it seems to not get the outermost portion of the input which is where the kind and apiVersion are both at.
Hi, I can confirm the problem still exists on a two nodes kubernetes cluster installed with Kubeadm. The nodes are Ubuntu Xenial 16.04 connected to each other using a tinc VPN. Everything works fine on the tainted master node but on the second (worker only) node every CNI network plugin container keep crashing. (I tried Flannel and Weake) It seems like the network-fabric container is unable to reach the API server on 16.96.0.1 I tried the workaround proposed but couldn't get it through becouse of the issue psoted by and apparentely the first two steps are insufficient.
you need add route to 16.96.0.1 via your you VPN interface.
In my case, my apiserver's advertise-address was correct, but I still had the crashloop issue with the weave pods. Adding "--proxy-mode=userspace" using 's neat jq trick solved the issue for me. Though the issue is solved, I don't have a good understanding of why it solved. I looked at the kube-proxy documentation() and it says: Should we expect any side effects as we are setting it to the "older" option? Thanks.
Hi everyone, I just finished a Vagrant box with centos 7.2 and kubeadm with weave-net and dashboard. I include all solved issues in this post. Thank you to and ! replace all occurences of kube-proxy-amd64 by kube-proxy maybe ... just check it with a get pods in the global namespace
Moved to kubernetes/kubeadm
I'm also experiencing the same issue: Any ideas on how to deal with the issue ?
what do the container's logs say?
Please open a new issue in the kubeadm repo to track that
would you mind sending me the cmd for getting this logs out into a gist for example ? that would help. I will raise a new ticket asap.
either: or should do
Same problem, solved it with

Discussion:
command = append(command, baseFlags[component]...) if component == apiServer { <ins> // Use first address we are given if len(s.API.AdvertiseAddresses) > 0 { command = append(command, fmt.Sprintf("--advertise-address=%s", s.API.AdvertiseAddresses[0])) } </ins> // Check if the user decided to use an external etcd cluster if len(s.Etcd.Endpoints) > 0 { command = append(command, fmt.Sprintf("--etcd-servers=%s", strings.Join(s.Etcd.Endpoints, ",")))