Change:
I saw this error in on the master after a failed cluster start: I'm looking into this, but I want to create a tracking issue as I think some other people may be hitting this.
It's trying to bring up and failing: The weird thing is that neither that file nor the file have been modified that recently:
Yes, to me it looks like salt just isn't finding the plugin. The last change to _states did add , but I've seen this succeed recently as well (http://kubernetes-). I'm going to try this a few times, to try to figure out whether it is failing every time or not. My guess is that it is failing semi-randomly, so it isn't related to the changes since last-fail.
Aha... this is logged at debug level from I'm guessing it's a salt version mismatch... investigating.
I guess AWS needs to lock salt versions like GCE does:
Confirmed that this is because of upstream salt breakage: - deprecated (?) Not sure what we should do here. I can lock to the same version as GCE, but if we find a bug in that version of salt then we can't update without breaking this script.
We don't write bugs. ;)
Can you just put this in an AWS-specific block? It seems like that should work.
(And by "this" I mean "future Salt variable gunk")
I think I'm going to try to lock AWS to the same version as GCE, and then think more about a long-term plan once AWS is working again!
from the looks of things I think vagrant will have the same problem (?) (Note that I am in the process of developing / testing the branch above, so I don't think you can just copy-and-paste the fix - yet!)
As a short term fix, using 2015.05 (ubuntu trusty salt ppa), was to change from: import as ipaddr to: import as ipaddr this actually seems to be working. might be deprecated. but not in 2015.05. why is there a containerbridge custom state at all btw? why not use network.managed and
- I'm not sure we can just change the import, because I think we have to be compatible with GCE's version of salt also. I'm not sure of why we did it this way vs other ways of doing things in salt
Does this mean that the official k8s distribution is broken on AWS, BTW? When is the next point release (and where are these discussed)?
We cherry-pick into point releases as necessary, and it's been pretty informal and usually initiated by us (Google). I'd actually be happy to have a discussion about how to initiate it externally (we're going to need a policy for requesting such things). cc
FWIW, policy aside, I'm happy to cut a point release for this once you think is tested well enough.
That would be great, thanks. I'll be testing it pretty hard today, I think. It has occurred to me that other clouds may also be broken. By grepping, it looks like vagrant and vsphere use I am utterly unfamiliar with the vsphere implementation, and I think vagrant sidesteps the issue because on Redhat is skips the call to It might be good to get confirmation that vagrant hasn't fallen in to a different trap though! Let me test it out today (unless people start filing lots of issues, in which case I think we roll it out more urgently...)
I will test out the Vagrant setup today, but I think we should be good because we do not use the same networking setup.
File a bug for a 0.17.1 to get this cherry-picked and assign it to me, as a reminder?
Do you know the status of 17.1? I'm seeing the same problems trying to AWS working, even tried an older release (e.g. v15), and have the same problem. My script looks 32 times on "[salt-master not working yet]" then gives up.
We're cutting one today.
I'm still seeing the same issue as before with the new release, my master starts then times out waiting for salt-master to start. Do you know some spots I can look at to help troubleshoot? Do you want a new issue created to track against?
Please file a new issue and tag thanks. I'm sorry it didn't fix things for you! On May 22, 2015 4:49 AM, "Steve Sloka" wrote:

Discussion:
md5sum "$file" done } <ins> # Install salt from GCS.  See README.md for instructions on how to update these # debs. install-salt() { local salt_mode="$1" if dpkg -s salt-minion &>/dev/null; then echo "== SaltStack already installed, skipping install step ==" return fi echo "== Refreshing package database ==" until apt-get update; do echo "== apt-get update failed, retrying ==" echo sleep 5 done mkdir -p /var/cache/salt-install cd /var/cache/salt-install DEBS=( libzmq3_3.2.3+dfsg-1~bpo70~dst+1_amd64.deb python-zmq_13.1.0-1~bpo70~dst+1_amd64.deb salt-common_2014.1.13+ds-1~bpo70+1_all.deb ) if [[ "${salt_mode}" == "master" ]]; then DEBS+=( salt-master_2014.1.13+ds-1~bpo70+1_all.deb ) fi DEBS+=( salt-minion_2014.1.13+ds-1~bpo70+1_all.deb ) URL_BASE="https://storage.googleapis.com/kubernetes-release/salt" for deb in "${DEBS[@]}"; do if [ ! -e "${deb}" ]; then download-or-bust "${URL_BASE}/${deb}" fi done for deb in "${DEBS[@]}"; do echo "== Installing ${deb}, ignore dependency complaints (will fix later) ==" dpkg --skip-same-version --force-depends -i "${deb}" done # This will install any of the unmet dependencies from above. echo "== Installing unmet dependencies ==" until apt-get install -f -y; do echo "== apt-get install failed, retrying ==" echo sleep 5 done # Log a timestamp echo "== Finished installing Salt ==" } </ins>

- /srv/reactor/highstate-new.sls EOF <del> # Install Salt # # We specify -X to avoid a race condition that can cause minion failure to # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 # # -M installs the master set +x curl -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s -- -M -X set -x </del> <ins> install-salt master service salt-master start service salt-minion start </ins>

EOF fi <ins> install-salt </ins> <del> # Install Salt # # We specify -X to avoid a race condition that can cause minion failure to # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 curl -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s -- -X </del> <ins> service salt-minion start </ins>

echo "SALT_MASTER='${MASTER_INTERNAL_IP}'" echo "MINION_IP_RANGE='${MINION_IP_RANGES[$i]}'" echo "DOCKER_OPTS='${EXTRA_DOCKER_OPTS:-}'" <ins> grep -v "^#" "${KUBE_ROOT}/cluster/aws/templates/common.sh" </ins> grep -v "^#" "${KUBE_ROOT}/cluster/aws/templates/format-disks.sh" grep -v "^#" "${KUBE_ROOT}/cluster/aws/templates/salt-minion.sh" ) > "${KUBE_TEMP}/minion-start-${i}.sh"