Change:
The kubelet.service EnvironmentFile /etc/sysconfig/kubelet doesn't exist on my minions.
Some extra information from master and minion-1:
I didn't think we used pki with salt. I wonder why salt is complaining about it. , David Reynolds wrote:
Is it possible this is related to salt's tar option having changed? A fix for that went in earlier today, but I don't know that we fixed vagrant yet...
Something else when I try to ./cluster/kube- This happens for each minion.
I think it's looking for eth0 in that ips list but isn't finding it?
since you're in the maintainers file for vagrant: any ideas?
- not yet, but I can take a look
- just to help debug, was this a fresh cluster you created that was working and then stopped after a push, or was this a fresh cluster that never worked?
It was a cluster that was working, I did a push and it broke. So I tore it down completely and rebuilt a fresh cluster that doesn't work either.
I am actually getting an issue where if I run more than one minion, my salt-minion is no longer able to contact the salt-master. So there is a number of things I need to look into.
So eth0 does not exist on a vagrant box for me, we look to need to use eth1 in a vagrant env.
appears to have introduced the regression on the vagrant env. is causing the issue.
Looking to see if I can specify which network interface to use for the function so we can safely use ips[0][0] when in an environment where the interface is not eth0
Just synced my branch, and found the issue is not limited to vagrant. When I ran hack/dev-build-kube-, I have the same kubeletservicecannotrestart issue due to lack of /var/lib/kubelet/kubernetesauth: But kubernetesauth is not installed: root ls /var/lib/kubelet/kubernetesauth ls: cannot access /var/lib/kubelet/kubernetes_auth: No such file or directory cc/
There are a number of issues, the master is not picking up the right network interface to use for the kube-apiserver to serve the readonly content, think we need a entry for address and publicAddressOverride per the arguments to kube-apiserver. I don't have much more time to debug this tonight, but there are a number of issues we need to resolve here.
I ran kube- to recreate the cluster, and now /var/lib/kubelet/kubernetesauth is properly installed on nodes, and kubelet is succussfully started without any issue. Looks like we ran into saltcannotupgradeall_files issue we saw before.
- I do not see same behavior on a fresh cluster.
I am referring to gce, not vagrant. I think there are more issues on vagrant. But on gce, fresh cluster don't have issue.
I think I am close to a fix for vagrant.
Submitted a fix. Testing appreciated.
Bringing up a new cluster on vagrant. Will report back in a bit.
Works for me! Fresh cluster build
Works for me too. Awesome work. Thanks!
Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote:
Brendan, If you are willing to add it to your jenkins box that would be appreciated. -- Derek ----- Original Message ----- From: "Brendan Burns" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Wednesday, November 19, 2014 11:45:46 PM Subject: Re: [kubernetes] Kubelet not starting () Derek, do you folks run vagrant e2e? If not maybe I should add that to our jenkins e2e box... --brendan , David Reynolds wrote: Reply to this email directly or view it on GitHub:
The problem with upgrading the cluster is that it references a new file that doesn't exist if you are doing an upgrade. I'm going to check in a default version of that file to make this work and to smooth things over when the check fails.

Discussion:
# Allow everyone to see cached values of who sits at what IP <ins> {% set networkInterfaceName = "eth0" %} {% if grains.networkInterfaceName is defined %} {% set networkInterfaceName = grains.networkInterfaceName %} {% endif %} </ins> mine_functions: <del> network.ip_addrs: [eth0] </del> <ins> network.ip_addrs: [{{networkInterfaceName}}] </ins> grains.items: []

{% set address = "-address=127.0.0.1" %} <ins> {% set publicAddressOverride = "" %} {% if grains.publicAddressOverride is defined %} {% set publicAddressOverride = "-public_address_override=" + grains.publicAddressOverride %} {% endif %} </ins> {% if grains.etcd_servers is defined %} {% set etcd_servers = "-etcd_servers=http://" + grains.etcd_servers + ":4001" %} {% else %}

{% endif %} {% endif %} <del> DAEMON_ARGS="{{daemon_args}} {{address}} {{etcd_servers}} {{ cloud_provider }} --allow_privileged={{pillar['allow_privileged']}} {{portal_net}} {{cert_file}} {{key_file}} {{secure_port}} {{token_auth_file}}" </del> <ins> DAEMON_ARGS="{{daemon_args}} {{address}} {{etcd_servers}} {{ cloud_provider }} --allow_privileged={{pillar['allow_privileged']}} {{portal_net}} {{cert_file}} {{key_file}} {{secure_port}} {{token_auth_file}} {{publicAddressOverride}}" </ins>

{% endif %} <ins> {% if grains.cloud is defined %} {% if grains.cloud == 'gce' %} </ins> # Kubelet will run without this file but will not be able to send events to the apiserver. /var/lib/kubelet/kubernetes_auth: file.managed:

- user: root - group: root - mode: 400 <ins> {% endif %} {% endif %} </ins> kubelet: group.present:

{% if grains['os_family'] != 'RedHat' %} - file: /etc/init.d/kubelet {% endif %} <ins> {% if grains.cloud is defined %} {% if grains.cloud == 'gce' %} </ins> - file: /var/lib/kubelet/kubernetes_auth <ins> {% endif %} {% endif %} </ins>

grains: node_ip: $MASTER_IP master_ip: $MASTER_IP <ins> publicAddressOverride: $MASTER_IP </ins> network_mode: openvswitch <ins> networkInterfaceName: eth1 </ins> etcd_servers: $MASTER_IP cloud: vagrant cloud_provider: vagrant

network_mode: openvswitch node_ip: $MINION_IP etcd_servers: $MASTER_IP <ins> networkInterfaceName: eth1 apiservers: $MASTER_IP </ins> roles: - kubernetes-pool - kubernetes-pool-vagrant

Key | Value ------------- | ------------- <ins> `apiservers` | (Optional) The IP address / host name where a kubelet can get read-only access to kube-apiserver </ins> `cbr-cidr` | (Optional) The minion IP address range used for the docker container bridge. `cloud` | (Optional) Which IaaS platform is used to host kubernetes, *gce*, *azure*, *aws*, *vagrant* `cloud_provider` | (Optional) The cloud_provider used by apiserver: *gce*, *azure*, *vagrant*

`node_ip` | (Optional) The IP address to use to address this node `minion_ip` | (Optional) Mapped to the kubelet hostname_override, K8S TODO - change this name `network_mode` | (Optional) Networking model to use among nodes: *openvswitch* <ins> `networkInterfaceName` | (Optional) Networking interface to use to bind addresses, default value *etho0* `publicAddressOverride` | (Optional) The IP address the kube-apiserver should use to bind against for external read-only access </ins> `roles` | (Required) 1. `kubernetes-master` means this machine is the master in the kubernetes cluster.  2. `kubernetes-pool` means this machine is a kubernetes-minion.  Depending on the role, the Salt scripts will provision different resources on the machine. These keys may be leveraged by the Salt sls files to branch behavior.