Change:
Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug What happened: Upgrade hangs. API server won't start because it doesn't have valid certs. What you expected to happen: Upgrade succeeds How to reproduce it (as minimally and precisely as possible): Anything else we need to know: https://k8s- is an example. Log from kube- I reproduced this locally and checked the certs: apiserver.{crt,key} are empty.
cc
Looks like this is still failing? https://k8s-
Seems that way.
cc
5, to 1.6
- is this something you are actively looking into? This is blocking all of the GCE upgrade test jobs from running so important to get fixed ASAP. thanks!
As of today I am.
do you have an update on this? Now that the jobs are set up, this is a blocker for 1.6 upgrade testing. Thanks!
I started looking at this yesterday, figured out what was going wrong. The problem is due to enabling RBAC. There are a couple options to fix. I should have a fix prepared and out for review today or tomorrow.
With the I hit another problem. Here is the apiserver error log: Reopen this.
from release-1.5 to master, I am not able to reproduce. verified that this fixed our CI tests.
Sorry, I forgot that my script is upgrade from 1.5.3 to 1.6.0-beta.1, which doesn't include your fix. Will verify again and close this one if it works. :)
Wait, it seems that you only changed , then I'm indeed using the latest to upgrade from v1.5.3 to v1.6.0-beta.1. Is there anything missing in v1.6.0-beta.1?
I think beta.2 rolled up this fix.
master-only upgrades seem to be working correctly now, however full-cluster upgrades aren't getting the cluster into a functioning state: <*errors.errorString | 0xc422c96f10: { I0309 08:45:31.969] s: "failed to get logs from downwardapi-volume--04e7-11e7-986c- for client-container: the server has asked for the client to provide credentials (get pods downwardapi-volume--04e7-11e7-986c-)", See https://k8s-
OK. It works from v1.5.3 to v1.6.0-beta.2. Close this one. :)
looking.... sounds like it might be related :( :(
broke upgrades since it adds a group to certificates that certificates generated with 1.5 don't have. This is causing:
upgrades should still have the existing ABAC policy in place
ahh, ok. I'll look into why that's not helping.
Infinity bugs. PTAL at
can this be closed?

Discussion:
detect-master parse-master-env <ins> backfile-kubeletauth-certs </ins> # Delete the master instance. Note that the master-pd is created # with auto-delete=no, so it should not be deleted.

wait-for-master } <ins> # TODO(mikedanese): delete when we don't support < 1.6 function backfile-kubeletauth-certs() { if [[ ! -z "${KUBEAPISERVER_CERT_BASE64:-}" && ! -z "${KUBEAPISERVER_CERT_BASE64:-}" ]]; then return 0 fi mkdir -p "${KUBE_TEMP}/pki" echo "${CA_KEY_BASE64}" | base64 -d > "${KUBE_TEMP}/pki/ca.key" echo "${CA_CERT_BASE64}" | base64 -d > "${KUBE_TEMP}/pki/ca.crt" (cd "${KUBE_TEMP}/pki" download-cfssl cat <<EOF > ca-config.json { "signing": { "client": { "expiry": "43800h", "usages": [ "signing", "key encipherment", "client auth" ] } } } EOF # the name kube-apiserver is bound to the node proxy # subpaths required for the apiserver to hit proxy # endpoints on the kubelet's handler. cat <<EOF  | "${KUBE_TEMP}/cfssl/cfssl" gencert  -ca=ca.crt  -ca-key=ca.key  -config=ca-config.json  -profile=client  -  | "${KUBE_TEMP}/cfssl/cfssljson" -bare kube-apiserver { "CN": "kube-apiserver" } EOF ) KUBEAPISERVER_CERT_BASE64=$(cat "${KUBE_TEMP}/pki/kube-apiserver.pem" | base64 | tr -d 'rn') KUBEAPISERVER_KEY_BASE64=$(cat "${KUBE_TEMP}/pki/kube-apiserver-key.pem" | base64 | tr -d 'rn') } </ins> function wait-for-master() { echo "== Waiting for new master to respond to API requests =="