Issue 1:
I got an issue with controller-manager starting from v1.2.0-beta.1 and it still persists in v1.2.0 release.

It keeps restarting pods kube-apiserver, kube-controller-manager, kube-proxy and kube-scheduler every 30s :

NAME READY STATUS RESTARTS AGE
...
kube-apiserver-ip-10-242-128-234.ec2.internal 1/1 Running 0 12s
kube-controller-manager-ip-10-242-128-234.ec2.internal 1/1 Running 0 16s
kube-proxy-ip-10-242-128-234.ec2.internal 1/1 Running 0 16s
kube-scheduler-ip-10-242-128-234.ec2.internal 1/1 Running 0 15s

Age for the listed pods don't go above 30s.

NAME READY STATUS RESTARTS AGE
....
kube-apiserver-ip-10-242-128-234.ec2.internal 0/1 Pending 0 1s
kube-controller-manager-ip-10-242-128-234.ec2.internal 1/1 Running 0 1s
kube-scheduler-ip-10-242-128-234.ec2.internal 0/1 Pending 0 1s

If i switch controller-manager to v1.2.0-beta.0 issue dissapear.

The strange thing is that docker containers are keep running

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
....
c44ff71caf21 gcr.io/google_containers/hyperkube:v1.2.0 "/hyperkube scheduler" 33 minutes ago Up 33 minutes k8s_kube-scheduler.7b10907e_kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_ed62caa9
d8ef6ef809ca gcr.io/google_containers/hyperkube:v1.2.0 "/hyperkube proxy --m" 33 minutes ago Up 33 minutes k8s_kube-proxy.c9b75a4a_kube-proxy-ip-10-242-128-234.ec2.internal_kube-system_e8c6af5a5c65fbcf53b11238b4a403bd_3ee808aa
068a6ab6793a gcr.io/google_containers/hyperkube:v1.2.0 "/hyperkube controlle" 33 minutes ago Up 33 minutes k8s_kube-controller-manager.2e450a4c_kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system_556a90cee78cfb39fd2f1f8d57edb479_8db26042
76e4cda29efb gcr.io/google_containers/hyperkube:v1.2.0 "/hyperkube apiserver" 33 minutes ago Up 33 minutes k8s_kube-apiserver.f4b25f9b_kube-apiserver-ip-10-242-128-234.ec2.internal_kube-system_f3d81ba2e960c16f6848dc5c09eb4eee_471bfae7
11eb022a4939 gcr.io/google_containers/pause:2.0 "/pause" 33 minutes ago Up 33 minutes k8s_POD.6059dfa2_kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system_1adf66daf2f3ee9cce476b5b97a8b58c_a51591f2
a1d9d564218c gcr.io/google_containers/pause:2.0 "/pause" 33 minutes ago Up 33 minutes k8s_POD.6059dfa2_kube-proxy-ip-10-242-128-234.ec2.internal_kube-system_e8c6af5a5c65fbcf53b11238b4a403bd_6accda61
1a1a278f12d5 gcr.io/google_containers/pause:2.0 "/pause" 33 minutes ago Up 33 minutes k8s_POD.6059dfa2_kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system_556a90cee78cfb39fd2f1f8d57edb479_209f305d
48e6e4e4d4bc gcr.io/google_containers/pause:2.0 "/pause" 33 minutes ago Up 33 minutes k8s_POD.6059dfa2_kube-apiserver-ip-10-242-128-234.ec2.internal_kube-system_f3d81ba2e960c16f6848dc5c09eb4eee_3b289a47

Error log from controller-manager:

I0322 15:57:36.787253 1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:36.787288 1 nodecontroller.go:352] <nil>
I0322 15:57:36.817601 1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:36.817644 1 nodecontroller.go:352] <nil>
E0322 15:57:36.819261 1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
E0322 15:57:36.828904 1 nodecontroller.go:352] pods "kube-scheduler-ip-10-242-128-234.ec2.internal" not found
I0322 15:57:37.659569 1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:37.659586 1 nodecontroller.go:352] <nil>
E0322 15:57:38.645497 1 nodecontroller.go:352] pods "kube-scheduler-ip-10-242-128-234.ec2.internal" not found
E0322 15:57:38.678738 1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
I0322 15:57:39.601407 1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:39.601439 1 nodecontroller.go:352] <nil>
I0322 15:57:40.619997 1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:57:40.620013 1 nodecontroller.go:352] <nil>
I0322 15:58:06.283648 1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.295400 1 nodecontroller.go:352] pods "kube-scheduler-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:06.372126 1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.404804 1 nodecontroller.go:352] pods "kube-proxy-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:06.442257 1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.458821 1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:06.527879 1 nodecontroller.go:400] forceful deletion of kube-apiserver-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.528300 1 nodecontroller.go:352] pods "kube-apiserver-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:06.694718 1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.694758 1 nodecontroller.go:352] <nil>
I0322 15:58:06.777777 1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:06.777819 1 nodecontroller.go:352] <nil>
E0322 15:58:06.788130 1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:36.558947 1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.573911 1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:36.614092 1 nodecontroller.go:400] forceful deletion of kube-apiserver-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.615441 1 nodecontroller.go:352] pods "kube-apiserver-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:36.653430 1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
I0322 15:58:36.665033 1 nodecontroller.go:400] forceful deletion of kube-scheduler-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.665048 1 nodecontroller.go:352] <nil>
I0322 15:58:36.736231 1 nodecontroller.go:400] forceful deletion of kube-proxy-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.744630 1 nodecontroller.go:352] pods "kube-proxy-ip-10-242-128-234.ec2.internal" not found
I0322 15:58:36.786456 1 nodecontroller.go:400] forceful deletion of kube-controller-manager-ip-10-242-128-234.ec2.internal succeeded
E0322 15:58:36.786470 1 nodecontroller.go:352] <nil>
E0322 15:58:37.743522 1 nodecontroller.go:352] pods "kube-controller-manager-ip-10-242-128-234.ec2.internal" not found

Error log from kubelet:

Mar 22 16:01:38 master kubelet[1493]: E0322 16:01:38.917605 1493 kubelet.go:1753] Deleting mirror pod "kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(4a513408-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.195729 1493 kubelet.go:1753] Deleting mirror pod "kube-proxy-ip-10-242-128-234.ec2.internal_kube-system(4a5d3ea1-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.196138 1493 kubelet.go:1753] Deleting mirror pod "kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(4a39ac2f-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:39 master kubelet[1493]: E0322 16:01:39.196300 1493 kubelet.go:1753] Deleting mirror pod "kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(4a513408-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197232 1493 kubelet.go:1753] Deleting mirror pod "kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5c68bb91-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197419 1493 kubelet.go:1753] Deleting mirror pod "kube-controller-manager-ip-10-242-128-234.ec2.internal_kube-system(5c78f49f-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:40 master kubelet[1493]: E0322 16:01:40.197556 1493 kubelet.go:1753] Deleting mirror pod "kube-proxy-ip-10-242-128-234.ec2.internal_kube-system(5c4a0855-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:40 master kubelet[1493]: W0322 16:01:40.395965 1493 request.go:627] Throttling request took 110.236442ms, request: POST:http://127.0.0.1:8080/api/v1/namespaces/kube-system/pods
Mar 22 16:01:41 master kubelet[1493]: E0322 16:01:41.199445 1493 kubelet.go:1753] Deleting mirror pod "kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5ca9cde4-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:42 master kubelet[1493]: E0322 16:01:42.201111 1493 kubelet.go:1753] Deleting mirror pod "kube-scheduler-ip-10-242-128-234.ec2.internal_kube-system(5d393e7a-f047-11e5-a5d4-0af71f25e715)" because it is outdated
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.619715 1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.655629 1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.656954 1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.658257 1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.659539 1493 kubelet.go:2741] Error updating node status, will retry: error getting node "ip-10-242-128-234.ec2.internal": nodes "ip-10-242-128-234.ec2.internal" not found
Mar 22 16:01:44 master kubelet[1493]: E0322 16:01:44.659553 1493 kubelet.go:1183] Unable to update node status: update node status exceeds retry count

My understanding is that kubelet thinks that these pods are duplicates and tries to remove them, but at the same time doesn't touch real docker containers.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): vsphere, ResourcePool, ComputeResource

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report

Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.5+coreos.0", GitCommit:"f70c2e5b2944cb5d622621a706bdec3d8a5a9c5e", GitTreeState:"clean", BuildDate:"2016-10-31T19:16:47Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment: CoreOS on vSphere

Cloud provider or hardware configuration: vSphere

OS (e.g. from /etc/os-release): CoreOS 1185.3.0 (MoreOS)

Kernel (e.g. uname -a): Linux k8-m1 4.7.3-coreos-r2 #1 SMP Tue Nov 1 01:38:43 UTC 2016

What happened:

I configured the controller-manager to use vsphere as the cloud provider and supplied it with a provider cloud configuration. When the controller-manager pod started, it panicked with the following stack trace;

2016-11-06T00:34:22.670422172Z panic: reflect.Set: value of type mo.ResourcePool is not assignable to type mo.ComputeResource
2016-11-06T00:34:22.670477454Z
2016-11-06T00:34:22.670494551Z goroutine 68 [running]:
2016-11-06T00:34:22.670592707Z panic(0x38d5200, 0xc820d2b310)
2016-11-06T00:34:22.670754360Z	/usr/local/go/src/runtime/panic.go:481 +0x3e6
2016-11-06T00:34:22.670990355Z reflect.Value.assignTo(0x4a676e0, 0xc8200a5600, 0x99, 0x4df7650, 0xb, 0x4a66e60, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.671019041Z	/usr/local/go/src/reflect/value.go:2164 +0x3be
2016-11-06T00:34:22.671380809Z reflect.Value.Set(0x4a66e60, 0xc82049e780, 0x199, 0x4a676e0, 0xc8200a5600, 0x99)
2016-11-06T00:34:22.671462864Z	/usr/local/go/src/reflect/value.go:1334 +0x95
2016-11-06T00:34:22.671715875Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo.LoadRetrievePropertiesResponse(0xc820d308e0, 0x46ecda0, 0xc82049e780, 0x0, 0x0)
2016-11-06T00:34:22.672055230Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo/retrieve.go:128 +0xe21
2016-11-06T00:34:22.672431372Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).Retrieve(0xc820b54418, 0x7f8a572ab340, 0xc8202d2540, 0xc820d3ca80, 0x1, 0x1, 0xc820cecfd0, 0x1, 0x1, 0x46ecda0, ...)
2016-11-06T00:34:22.672467148Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:167 +0x52f
2016-11-06T00:34:22.672606970Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).RetrieveOne(0xc820b54418, 0x7f8a572ab340, 0xc8202d2540, 0xc820cec9c0, 0xc, 0xc820cec9e0, 0xa, 0xc820cecfd0, 0x1, 0x1, ...)
2016-11-06T00:34:22.672681600Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:173 +0x10e
2016-11-06T00:34:22.672871709Z k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object.Common.Properties(0x0, 0x0, 0xc820b1e500, 0xc8202cedd0, 0xb, 0xc8202cee00, 0xb, 0x7f8a572ab340, 0xc8202d2540, 0xc820cec9c0, ...)
2016-11-06T00:34:22.672904730Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/common.go:97 +0x19f
2016-11-06T00:34:22.673072861Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.readInstance(0xc82025b0e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673104582Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:223 +0xe8f
2016-11-06T00:34:22.673118861Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.newVSphere(0xc820252200, 0x18, 0xc8204c25f0, 0xc, 0xc82011fd60, 0x9, 0xc8204c2048, 0x3, 0x1, 0xc8204c2ae8, ...)
2016-11-06T00:34:22.673128111Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:237 +0x7e
2016-11-06T00:34:22.673204822Z k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.init.1.func1(0x7f8a572c9488, 0xc82014c668, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673274571Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:153 +0xdf
2016-11-06T00:34:22.673365023Z k8s.io/kubernetes/pkg/cloudprovider.GetCloudProvider(0x7ffc9a344abf, 0x7, 0x7f8a572c9488, 0xc82014c668, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673457368Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:62 +0x112
2016-11-06T00:34:22.673526314Z k8s.io/kubernetes/pkg/cloudprovider.InitCloudProvider(0x7ffc9a344abf, 0x7, 0x7ffc9a344ad6, 0x1c, 0x0, 0x0, 0x0, 0x0)
2016-11-06T00:34:22.673582280Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:84 +0x3e2
2016-11-06T00:34:22.673593710Z k8s.io/kubernetes/cmd/kube-controller-manager/app.StartControllers(0xc82062f900, 0xc8203aea80, 0xc8201de340, 0xc8203af860, 0x7f8a572d6790, 0xc820418640, 0x0, 0x0)
2016-11-06T00:34:22.673674949Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:225 +0x719
2016-11-06T00:34:22.673686774Z k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2(0xc8203af860)
2016-11-06T00:34:22.673696396Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:166 +0x6a
2016-11-06T00:34:22.673755265Z created by k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).Run
2016-11-06T00:34:22.673766424Z	/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:177 +0x91

What you expected to happen:

I expected the controller-manager pod to run as normal.

How to reproduce it (as minimally and precisely as possible):

Configure the controller-manager pod to use the vsphere cloud provider and pass the following provider cloud configuration file;

[Global]
server = vcenter
port = 443
user = administrator@vsphere.local
password = removed
insecure-flag = true
datacenter = FC

[Network]
public-network = External

Anything else do we need to know:

This cluster was created by following the CoreOS + Kubernetes Step by Step guide. The only deviation from the guide was to adjust the /etc/kubernetes/manifests/kube-controller-manager.yaml file to include configuration flags for cloud-provider and cloud-config, and adding an additional volume and volume mount for the provider cloud config file.

The CoreOS VMs are part of a single resource group (K8) on the same ESXi host.

My base image for all of the hyperkube containers is quay.io/coreos/hyperkube:v1.4.5_coreos.0

Before adding the additional configuration to use the vsphere cloud provider, this cluster was working as expected. The only change that was made to produce the stack trace above was to enable the vsphere cloud provider and pass the provider cloud config above.

I am still fairly new to Kubernetes so i'm open to the possibility that I might have made a fatal error somewhere, but as far as I can tell this does appear to be a genuine bug. Apologies if this does turn out to be a mistake on my behalf!