Change:
This is for manual upgrade testing: I am running the script on the release-1.5 branch and the script has been running for a long time now (40mins) with no sign of completion. Has anyone experienced a similar issue or has been successfully able to run the upgrade script ? Command I ran: -M v1.5.0-beta.1 $ -M v1.5.0-beta.1 == Pre-Upgrade Node OS and Kubelet Versions == name: "kubernetes-master", osImage: "Google Container-VM Image", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-7593", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-bigx", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-q5uf", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" Your active configuration is: [default] Project: rkouj-test1 Zone: us-central1-b INSTANCEGROUPS=kubernetes-minion-group NODENAMES=kubernetes-minion-group-7593 kubernetes-minion-group-bigx kubernetes-minion-group-q5uf == Upgrading master to ''. Do not interrupt, deleting master instance. == Trying to find master named 'kubernetes-master' Looking for address 'kubernetes-master-ip' Using master: kubernetes-master (external IP: 104.197.149.96) Deleted []. WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: Created []. NAME ZONE MACHINETYPE PREEMPTIBLE INTERNALIP EXTERNAL_IP STATUS kubernetes-master us-central1-b n1-standard-1 10.128.0.2 104.197.149.96 RUNNING == Waiting for new master to respond to API requests == ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
That sounds long to me. Do you see the same behavior when you run it without the flag?
Tried it without the flag. No change. $ v1.5.0-beta.1 == Pre-Upgrade Node OS and Kubelet Versions == name: "kubernetes-master", osImage: "Google Container-VM Image", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-7kpl", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-l2on", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" name: "kubernetes-minion-group-lzgi", osImage: "Debian GNU/Linux 7 (wheezy)", kubeletVersion: "v1.4.7-beta.0.2+-dirty" Your active configuration is: [default] Project: rkouj-test1 Zone: us-central1-b INSTANCEGROUPS=kubernetes-minion-group NODENAMES=kubernetes-minion-group-7kpl kubernetes-minion-group-l2on kubernetes-minion-group-lzgi == Upgrading master to ''. Do not interrupt, deleting master instance. == Trying to find master named 'kubernetes-master' Looking for address 'kubernetes-master-ip' Using master: kubernetes-master (external IP: 23.236.62.71) Deleted []. WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: Created []. NAME ZONE MACHINETYPE PREEMPTIBLE INTERNALIP EXTERNAL_IP STATUS kubernetes-master us-central1-b n1-standard-1 10.128.0.2 23.236.62.71 RUNNING == Waiting for new master to respond to API requests == ..............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
I'm seeing this problem as well. And I saw this on two of the hosts: [ 3870.] Memory cgroup out of memory: Kill process 4697 (google-fluentd) score 1923 or sacrifice child [ 3870.] Killed process 4697 (google-fluentd) total-vm:1224416kB, anon-rss:186420kB, file-rss:8068kB
cc:
After syncing up with This is what we saw in the master configuration. rkouj /var/log $ sudo systemctl status kube-master-configuration kube-master-configuration.service - Configure kubernetes master Loaded: loaded (/etc/systemd/system/kube-master-configuration.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2016-11-22 21:06:55 UTC; 1h 24min ago Process: 1145 ExecStart=/home/kubernetes/bin/configure- (code=exited, status=1/FAILURE) Process: 1140 ExecStartPre=/bin/chmod 544 /home/kubernetes/bin/mounter (code=exited, status=0/SUCCESS) Process: 1136 ExecStartPre=/bin/chmod 544 /home/kubernetes/bin/configure- (code=exited, status=0/SUCCESS) Main PID: 1145 (code=exited, status=1/FAILURE)
cc
This is a cluster lifecycle integration issue. I will leave to drive this. cc/ for the support from the node side. Thanks!
Wrong syntax introduced by
analysis was correct. I was able to ssh into my upgraded Kubernetes master, manually edit to fix the shell, unmount the master pd (), and re-run . Once that finished, my master upgrade succeeded after being in a waiting state for a long time: The bad news is that is baked into the k8s release tar, so we'll need to cut a new beta with a fix before we can resume manual upgrade testing.
Thanks for the investigation, and ! I LGTMd the PR sent with the fix.
FYI this won't actually be fixed until this PR is in the 1.5 release branch and we cut a beta release. At that point we will update the instructions in to point to and should be working properly again.
Agreed; I'll take care of doing that once I see the anago announcement of the new beta.

Discussion:
params+=" --advertise-address=${vm_external_ip}" params+=" --ssh-user=${PROXY_SSH_USER}" params+=" --ssh-keyfile=/etc/srv/sshproxy/.sshkeyfile" <del> else [ -n "${MASTER_ADVERTISE_ADDRESS:-}" ] </del> <ins> elif [ -n "${MASTER_ADVERTISE_ADDRESS:-}" ]; then </ins> params="${params} --advertise-address=${MASTER_ADVERTISE_ADDRESS}" fi