Issue 1:
Kubernetes version :

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.8+coreos.0", GitCommit:"9c19ded313d3b3b86eadf179aed553854138abd7", GitTreeState:"clean", BuildDate:"2016-09-19T18:58:14Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

(I'm using v1.4.0-beta.8+coreos.0 version since this fixes "dmidecode" binary dependency, which is not available neither in hyperkube, nor in Core OS #27331)

Environment:

Cloud provider or hardware configuration:

VMWare virtual machine, deployed on as ESXi 6.0.0 Update 2 (Build 4192238) host.

Output from govc datacenter.info

Name: ha-datacenter
Hosts: 1
Clusters: 0
Virtual Machines: 1
Networks: 1
Datastores: 4

Output from govc vm.info -vm.ip=192.168.0.144

Name: ci01
Path: /ha-datacenter/vm/ci01
UUID: 564d8885-7d5b-3c5c-3525-d382ca47ab1c
Guest name: Other 2.6.x Linux (64-bit)
Memory: 1024MB
CPU: 2 vCPU(s)
Power state: poweredOn
Boot time: 2016-09-22 05:59:31.517386 +0000 UTC
IP address: 10.2.24.1
Host: hostname.mydomain.com

where 192.168.0.144 is the public IP of the virtual machine

OS :

NAME=CoreOS
ID=coreos
VERSION=1122.2.0
VERSION_ID=1122.2.0
BUILD_ID=2016-09-06-1449
PRETTY_NAME="CoreOS 1122.2.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

I'm using coreos_production_vmware_ova.ova available in CoreOs stable channel.

Kernel :

Linux ci01 4.7.0-coreos #1 SMP Tue Sep 6 14:39:20 UTC 2016 x86_64 Intel(R) Xeon(R) CPU L5640 @ 2.27GHz GenuineIntel GNU/Linux

Install tools:

VMWare tools version 2147483647

Others:

kubelet service:

[Service]
EnvironmentFile=/etc/environment
Environment=KUBELET_VERSION=v1.4.0-beta.8_coreos.0
TimeoutStartSec=infinity
ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/bin/mkdir -p /srv/kubernetes/manifests
ExecStartPre=/bin/mkdir -p /etc/kubernetes/checkpoint-secrets
ExecStartPre=/bin/rkt fetch quay.io/coreos/hyperkube:v1.4.0-beta.8_coreos.0 --trust-keys-from-https
ExecStart=/usr/lib/coreos/kubelet-wrapper \
--api-servers=https://192.168.0.144:443 \
--kubeconfig=/etc/kubernetes/kubeconfig \
--lock-file=/var/run/lock/kubelet.lock \
--exit-on-lock-contention=true \
--allow-privileged \
--config=/etc/kubernetes/manifests \
--hostname-override=192.168.0.144 \
--node-labels=master=true \
--minimum-container-ttl-duration=3m0s \
--cluster_dns=10.3.0.10 \
--cluster_domain=cluster.local \
--cloud-config=/etc/kubernetes/vsphere.conf \
--cloud-provider=vsphere

Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

vsphere.conf (cloud-config):

[Global]
server=ESXI_host_IP
port=443
user=myuser
password=mypassword
insecure-flag=true
datacenter=ha-datacenter

What happened:

Kubelet fails to start with the following errors:

systemd[1]: Starting kubelet.service...
rkt[1866]: image: using image from local store for image name quay.io/coreos/hyperkube:v1.4.0-beta.8_coreos.0
rkt[1866]: sha512-406472bdf336d7a2a353354ae0597540
systemd[1]: Started kubelet.service.
kubelet-wrapper[1883]: image: using image from file /usr/lib64/rkt/stage1-images/stage1-fly.aci
kubelet-wrapper[1883]: image: using image from local store for image name quay.io/coreos/hyperkube:v1.4.0-beta.8_coreos.0
kubelet-wrapper[1883]: Flag --api-servers has been deprecated, Use --kubeconfig instead. Will be removed in a future version.
kubelet-wrapper[1883]: Flag --config has been deprecated, Use --pod-manifest-path instead. Will be removed in a future version.
kubelet-wrapper[1883]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
kubelet-wrapper[1883]: I0921 13:41:24.809785 1883 server.go:309] acquiring lock on "/var/run/lock/kubelet.lock"
kubelet-wrapper[1883]: I0921 13:41:24.810116 1883 server.go:314] watching for inotify events for: /var/run/lock/kubelet.lock
kubelet-wrapper[1883]: panic: reflect.Set: value of type mo.ComputeResource is not assignable to type mo.ClusterComputeResource
kubelet-wrapper[1883]: goroutine 1 [running]:
kubelet-wrapper[1883]: panic(0x388bb80, 0xc82034aec0)
kubelet-wrapper[1883]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
kubelet-wrapper[1883]: reflect.Value.assignTo(0x4a042a0, 0xc82050e1e0, 0x99, 0x4d90950, 0xb, 0x49ce7e0, 0x0, 0x0, 0x0, 0x0)
kubelet-wrapper[1883]: /usr/local/go/src/reflect/value.go:2164 +0x3be
kubelet-wrapper[1883]: reflect.Value.Set(0x49ce7e0, 0xc8202a4700, 0x199, 0x4a042a0, 0xc82050e1e0, 0x99)
kubelet-wrapper[1883]: /usr/local/go/src/reflect/value.go:1334 +0x95
kubelet-wrapper[1883]: k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo.LoadRetrievePropertiesResponse(0xc820aba760, 0x4690fe0, 0xc8202a4700, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo/retrieve.go:128 +0xe21
kubelet-wrapper[1883]: k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).Retrieve(0xc820962b60, 0x7f3ecae79290, 0xc820256e80, 0xc820581260, 0x1, x1, 0x1, 0x4690fe0, ...)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:167 +0x52f
kubelet-wrapper[1883]: k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).RetrieveOne(0xc820962b60, 0x7f3ecae79290, 0xc820256e80, 0xc82045d180, xe, 0xc82045dd90, 0x1, 0x1, ...)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:173 +0x10e
kubelet-wrapper[1883]: k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object.Common.Properties(0x0, 0x0, 0xc82022a500, 0xc8202cd890, 0xb, 0xc8202cd8f0, 0xe, 20256e80, 0xc82045d180, ...)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/common.go:97 +0x19f
kubelet-wrapper[1883]: k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.readInstance(0xc8201c2820, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:219 +0xe8f
kubelet-wrapper[1883]: k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.newVSphere(0xc820211250, 0x5, 0xc820211490, 0x9, 0xc820210ce0, 0xd, 0xc820210f18, 0x3, ..)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:233 +0x7e
kubelet-wrapper[1883]: k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.init.1.func1(0x7f3ecae69488, 0xc82001c210, 0x0, 0x0, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:149 +0xdf
kubelet-wrapper[1883]: k8s.io/kubernetes/pkg/cloudprovider.GetCloudProvider(0x7ffc4d24cf63, 0x7, 0x7f3ecae69488, 0xc82001c210, 0x0, 0x0, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:62 +0x112
kubelet-wrapper[1883]: k8s.io/kubernetes/pkg/cloudprovider.InitCloudProvider(0x7ffc4d24cf63, 0x7, 0x7ffc4d24cf35, 0x1c, 0x0, 0x0, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:84 +0x3e2
kubelet-wrapper[1883]: k8s.io/kubernetes/cmd/kubelet/app.run(0xc8204fc000, 0x0, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:348 +0x17ca
kubelet-wrapper[1883]: k8s.io/kubernetes/cmd/kubelet/app.Run(0xc8204fc000, 0x0, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:267 +0x4a
kubelet-wrapper[1883]: main.NewKubelet.func1(0xc8201f7400, 0xc82033b6c0, 0x0, 0xd, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/kubelet.go:37 +0x35
kubelet-wrapper[1883]: main.(*HyperKube).Run(0xc82042d560, 0xc82000a0f0, 0xd, 0xd, 0x0, 0x0)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:181 +0x670
kubelet-wrapper[1883]: main.(*HyperKube).RunToExit(0xc82042d560, 0xc82000a0e0, 0xe, 0xe)
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/hyperkube.go:191 +0x5f
kubelet-wrapper[1883]: main.main()
kubelet-wrapper[1883]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/hyperkube/main.go:48 +0x79e
systemd[1]: kubelet.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: kubelet.service: Unit entered failed state.
systemd[1]: kubelet.service: Failed with result 'exit-code'.

What you expected to happen:

It works as expected if I comment out lines related to cloud-config and cloud-provider from kubelet service.

I wolud like to use those flags in orther to be able to use vsphere volumes in kubernetes.

How to reproduce it:

Start kubelet service, it always fails trying to read vm instances on vsphere.go provider plugin

Anything else do we need to know:

It is a sinlge node, self hosted, kubernetes cluster. I generate kubernetes manifests using bootkube binary with its render method, then modify manifest to use v1.4.0-beta.8_coreos.0 version, and finally launch cluster using bootkube start.

As I said before, it works when you don't put cloud-provider and cloud-config in the kubelet servide.

Issue 2:
Recent changes in master #30935 that were pulled into the 1.4 release branch lead to vSphere cloud provider to panic when only a single ESX node is present under a vCenter.

I am in the process of testing and creating a PR for this.

Oct 14 19:11:46 kube kubelet[22061]: panic: reflect.Set: value of type mo.ComputeResource is not assignable to type mo.ClusterComputeResource
Oct 14 19:11:46 kube kubelet[22061]: goroutine 1 [running]:
Oct 14 19:11:46 kube kubelet[22061]: panic(0x287dbe0, 0xc82043b790)
Oct 14 19:11:46 kube kubelet[22061]: /usr/local/go/src/runtime/panic.go:481 +0x3e6
Oct 14 19:11:46 kube kubelet[22061]: reflect.Value.assignTo(0x359f380, 0xc8200f8780, 0x99, 0x382f5a0, 0xb, 0x357c7e0, 0x0, 0x0, 0x0, 0x0)
Oct 14 19:11:46 kube kubelet[22061]: /usr/local/go/src/reflect/value.go:2164 +0x3be
Oct 14 19:11:46 kube kubelet[22061]: reflect.Value.Set(0x357c7e0, 0xc82058c000, 0x199, 0x359f380, 0xc8200f8780, 0x99)
Oct 14 19:11:46 kube kubelet[22061]: /usr/local/go/src/reflect/value.go:1334 +0x95
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo.LoadRetrievePropertiesResponse(0xc820241c20, 0x32db640, 0xc82058c000, 0x0, 0x0)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/vim25/mo/retrieve.go:128 +0xe21
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).Retrieve(0xc8205a3060, 0x7f1cb57faf18, 0xc82038dc00, 0xc8202418a0, 0x1, 0x1, 0xc82043a170, 0x1, 0x1, 0x32db640, ...)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:167 +0x52f
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property.(*Collector).RetrieveOne(0xc8205a3060, 0x7f1cb57faf18, 0xc82038dc00, 0xc820451750, 0xf, 0xc820451770, 0xb, 0xc82043a170, 0x1, 0x1, ...)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/property/collector.go:173 +0x10e
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object.Common.Properties(0x0, 0x0, 0xc820359900, 0xc82032bdd0, 0xb, 0xc82032be00, 0xb, 0x7f1cb57faf18, 0xc82038dc00, 0xc820451750, ...)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/vmware/govmomi/object/common.go:97 +0x19f
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.readInstance(0xc8203cc5a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:256 +0xe8f
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.newVSphere(0xc82042dbe0, 0x1b, 0xc82048a8b0, 0x8, 0xc82048ac70, 0xd, 0xc82048af90, 0x3, 0x1, 0xc82048b550, ...)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:270 +0x7e
Oct 14 19:11:46 kube systemd[1]: kubelet.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 14 19:11:46 kube systemd[1]: Unit kubelet.service entered failed state.
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere.init.1.func1(0x7f1cb57d62c8, 0xc82001e6a8, 0x0, 0x0, 0x0, 0x0)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/vsphere/vsphere.go:186 +0xdf
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/pkg/cloudprovider.GetCloudProvider(0x7ffcb7cebd22, 0x7, 0x7f1cb57d62c8, 0xc82001e6a8, 0x0, 0x0, 0x0, 0x0)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:85 +0x112
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/pkg/cloudprovider.InitCloudProvider(0x7ffcb7cebd22, 0x7, 0x7ffcb7cebd39, 0x19, 0x0, 0x0, 0x0, 0x0)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/plugins.go:107 +0x3e2
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/cmd/kubelet/app.run(0xc820072b00, 0x0, 0x0, 0x0)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:349 +0x18c2
Oct 14 19:11:46 kube kubelet[22061]: k8s.io/kubernetes/cmd/kubelet/app.Run(0xc820072b00, 0x0, 0x0, 0x0)
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:268 +0x4a
Oct 14 19:11:46 kube kubelet[22061]: main.main()
Oct 14 19:11:46 kube kubelet[22061]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:48 +0x94