Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

concurrent map writes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

$ kubectl version

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: Openstack

OS (e.g. from /etc/os-release):

$ cat /etc/os-release

NAME="Ubuntu"

VERSION="16.04.1 LTS (Xenial Xerus)"

ID=ubuntu

ID_LIKE=debian

PRETTY_NAME="Ubuntu 16.04.1 LTS"

VERSION_ID="16.04"

HOME_URL="http://www.ubuntu.com/"

SUPPORT_URL="http://help.ubuntu.com/"

BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

UBUNTU_CODENAME=xenial

Kernel (e.g. uname -a):

$ uname -a

Linux Master1 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Install tools:

Others:

What happened:

I noticed a panic in the kube-controller-manager in a 3 node HA Kubernetes controller deployment. Here are the logs:

Aug 30 15:36:57 ubuntu16 kube-controller-manager[19004]: E0830 15:36:57.457558 19004 leaderelection.go:318] err: Operation cannot be fulfilled on endpoints "kube-controller-manager": the

Aug 30 15:36:57 ubuntu16 kube-controller-manager[19004]: W0830 15:36:57.741242 19004 request.go:347] Field selector: v1 - serviceaccounts - metadata.name - default: need to check if this

Aug 30 15:36:57 ubuntu16 kube-controller-manager[19004]: W0830 15:36:57.762328 19004 reflector.go:334] pkg/controller/serviceaccount/tokens_controller.go:147: watch of _api.ServiceAccount

Aug 30 15:36:57 ubuntu16 kube-controller-manager[19004]: W0830 15:36:57.764460 19004 reflector.go:334] pkg/controller/serviceaccount/serviceaccounts_controller.go:142: watch of *api.Servi

Aug 30 15:36:58 ubuntu16 kube-controller-manager[19004]: W0830 15:36:58.764857 19004 request.go:347] Field selector: v1 - serviceaccounts - metadata.name - default: need to check if this

Aug 30 15:36:58 ubuntu16 kube-controller-manager[19004]: W0830 15:36:58.771127 19004 request.go:347] Field selector: v1 - serviceaccounts - metadata.name - default: need to check if this

Aug 30 15:37:01 ubuntu16 kube-controller-manager[19004]: E0830 15:37:01.594976 19004 leaderelection.go:318] err: Operation cannot be fulfilled on endpoints "kube-controller-manager": the

Aug 30 15:37:03 ubuntu16 kube-controller-manager[19004]: E0830 15:37:03.554584 19004 leaderelection.go:318] err: Operation cannot be fulfilled on endpoints "kube-controller-manager": the

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: fatal error: concurrent map writes

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: goroutine 823 [running]:

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: runtime.throw(0x2c5d4f0, 0x15)

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc8208fcab0 sp=0xc8208fca98

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: runtime.mapassign1(0x1e3e820, 0xc8205518f0, 0xc8208fcbf8, 0xc8208fcc48)

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc8208fcb58 sp=0xc8208fcab0

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: k8s.io/kubernetes/pkg/client/cache.(_DeltaFIFO).queueActionLocked(0xc8200dadc0, 0x2a3fcd0, 0x4, 0x29f02e0, 0xc82106c2c0, 0x0, 0x0)

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:314 +0x5d1 fp=0xc8208fc

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: k8s.io/kubernetes/pkg/client/cache.(_DeltaFIFO).Resync(0xc8200dadc0, 0x0, 0x0)

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:498 +0x4f7 fp=0xc8208fc

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: k8s.io/kubernetes/pkg/client/cache.(_Reflector).ListAndWatch.func1(0xc820107318, 0xc82004ede0, 0xc8202e23c0, 0xc821233aa0, 0xc820107

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:289 +0x252 fp=0xc8208fcf

Aug 30 15:37:04 ubuntu16 kube-controller-manager[19004]: runtime.goexit()

What you expected to happen:

I did not expect to hit a code path where I noticed concurrent map writes.

How to reproduce it (as minimally and precisely as possible):

Used the following steps:

https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-kubernetes-controller.md

The difference is that I used Openstack as the cloud provider

Anything else do we need to know:

I had cloned the 3 HA controller VMs from a single volume, so initially they ended up with the same Linux hostname. That explains the failure in the leader election, but not sure if it's related to the panic. I fixed the hostnames later and the leader election succeeded.

Issue 2:
Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Feature request.

Kubernetes version (use kubectl version):

master

What you expected to happen:

In Mesos 1.x, Mesos container is the default container runtime of tasks; when running Kubernetes on Mesos (kube-mesos), we'd like to also using only one container runtime environment to reduce OP's maintenance effort.

That's one project of kube-mesos ; but I do not want to duplicate Kubelet in kube-mesos project.