Issue 1:
error: google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information. appears in a lot of flakes. Speculation is that the metadata server is slow to respond.

So far, these are the issues where this message has popped up:

#28569

#28565

#28523

#28507

#28493

#28439

#28437

#28429

#28426

#28420

#28293

#27976

#27839

#27715

#27156

#26715

#26490

Sample failed run: https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-e2e-gke/10664

@cjcullen since this is dealing with auth, I'm initially assigning to you. Feel free to reassign to someone more appropriate.

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

no

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

fatal error: concurrent map writes

i found this kubernetes/minikube#368 not the same problem.

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:15:30Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.7", GitCommit:"a2cba278cba1f6881bb0a7704d9cac6fca6ed435", GitTreeState:"clean", BuildDate:"2016-09-12T23:08:43Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: virtual machine ,4 cpu cores,8G RAM

OS (e.g. from /etc/os-release):CentOS Linux release 7.2.1511

Kernel (e.g. uname -a):Linux k8s-master-1 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Install tools:none

Others:two kubernetes master(both with api,controller,scheduler) node use election feature.

What happened:

panic frequently,sometimes 3 times a day.

What you expected to happen:

not panic

How to reproduce it (as minimally and precisely as possible):

not known

Anything else do we need to know:

here is the dump stack:

Nov 23 06:03:13 centos7-image kube-controller-manager: fatal error: concurrent map writes

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 19396 [running]:

Nov 23 06:03:13 centos7-image kube-controller-manager: runtime.throw(0x2c86560, 0x15)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/panic.go:547 +0x90 fp=0xc820cbbab0 sp=0xc820cbba98

Nov 23 06:03:13 centos7-image kube-controller-manager: runtime.mapassign1(0x1e59780, 0xc8201892f0, 0xc820cbbbf8, 0xc820cbbc48)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc820cbbb58 sp=0xc820cbbab0

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).queueActionLocked(0xc8200e0a50, 0x2a67378, 0x4, 0x29fe980, 0xc821752b60, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:314 +0x5d1 fp=0xc820cbbcc0 sp=0xc820cbbb58

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).Resync(0xc8200e0a50, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:498 +0x4f7 fp=0xc820cbbe38 sp=0xc820cbbcc0

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc8201912b0, 0xc82009ad80, 0xc8201503c0, 0xc82124eb40, 0xc8201912d0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:289 +0x252 fp=0xc820cbbf88 sp=0xc820cbbe38

Nov 23 06:03:13 centos7-image kube-controller-manager: runtime.goexit()

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820cbbf90 sp=0xc820cbbf88

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 1 [chan receive]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.WaitFor(0xc8213b7220, 0xc821389780, 0xc821196180, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:209 +0x64

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.pollInternal(0xc8213b7220, 0xc821389780, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:170 +0x8a

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.Poll(0x77359400, 0x2540be400, 0xc821389780, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:164 +0x4f

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).renew.func1()

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:224 +0x7d

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc821389918)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:86 +0x4b

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc821389918, 0x0, 0x0, 0x435701, 0xc820500900)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0xb4

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.Until(0xc821389918, 0x0, 0xc820500900)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).renew(0xc8201b0000)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:235 +0x64

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).Run(0xc8201b0000)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:178 +0x9f

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/leaderelection.RunOrDie(0x2c9c620, 0x17, 0x0, 0x0, 0x2a9cb40, 0xb, 0x0, 0x0, 0x0, 0x0, ...)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:189 +0x98

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.Run(0xc8201758c0, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:194 +0x781

Nov 23 06:03:13 centos7-image kube-controller-manager: main.main()

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/controller-manager.go:53 +0x9b

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 18 [chan receive]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x3f051a0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x67

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x297

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 11 [select, 385 minutes]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc82008a0d8, 0xc82009ad80, 0xc820150d20, 0xc820672540, 0xc82008a0e0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 25 [select]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0x317bb30, 0x12a05f200, 0x0, 0x434301, 0xc82009ad80)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:101 +0x191

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.Until(0x317bb30, 0x12a05f200, 0xc82009ad80)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/util.InitLogs

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/logs.go:50 +0xba

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 26 [IO wait, 275 minutes]:

Nov 23 06:03:13 centos7-image kube-controller-manager: net.runtime_pollWait(0x7fc2c66cc4f0, 0x72, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/netpoll.go:160 +0x60

Nov 23 06:03:13 centos7-image kube-controller-manager: net.(*pollDesc).Wait(0xc8202fc450, 0x72, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a

Nov 23 06:03:13 centos7-image kube-controller-manager: net.(*pollDesc).WaitRead(0xc8202fc450, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36

Nov 23 06:03:13 centos7-image kube-controller-manager: net.(*netFD).accept(0xc8202fc3f0, 0x0, 0x7fc2c64ed078, 0xc820e74700)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/fd_unix.go:426 +0x27c

Nov 23 06:03:13 centos7-image kube-controller-manager: net.(*TCPListener).AcceptTCP(0xc82008a058, 0x453cb0, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d

Nov 23 06:03:13 centos7-image kube-controller-manager: net/http.tcpKeepAliveListener.Accept(0xc82008a058, 0x0, 0x0, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2427 +0x41

Nov 23 06:03:13 centos7-image kube-controller-manager: net/http.(*Server).Serve(0xc820194280, 0x7fc2c64ed040, 0xc82008a058, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2117 +0x129

Nov 23 06:03:13 centos7-image kube-controller-manager: net/http.(*Server).ListenAndServe(0xc820194280, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/net/http/server.go:2098 +0x136

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func1(0xc8201758c0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:153 +0x30b

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/cmd/kube-controller-manager/app.Run

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:154 +0x333

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 27 [chan receive, 385 minutes]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/watch.(*Broadcaster).loop(0xc8202b5240)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/watch/mux.go:201 +0x58

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/watch.NewBroadcaster

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/watch/mux.go:75 +0x129

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 28 [chan receive, 385 minutes]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x7fc2c6697df0, 0xc8201734a0, 0xc8202f3b40)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:227 +0x97

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/record.(*eventBroadcasterImpl).StartEventWatcher

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:239 +0x70

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 29 [chan receive, 385 minutes]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/record.(*eventBroadcasterImpl).StartEventWatcher.func1(0x7fc2c6697df0, 0xc8201735f0, 0xc8201735c0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:227 +0x97

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/record.(*eventBroadcasterImpl).StartEventWatcher

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/record/event.go:239 +0x70

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 40 [semacquire]:

Nov 23 06:03:13 centos7-image kube-controller-manager: sync.runtime_Semacquire(0xc8200e0a58)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/runtime/sema.go:47 +0x26

Nov 23 06:03:13 centos7-image kube-controller-manager: sync.(*RWMutex).Lock(0xc8200e0a50)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/sync/rwmutex.go:88 +0xa1

Nov 23 06:03:13 centos7-image kube-controller-manager: sync.(*Cond).Wait(0xc8200e0a68)

Nov 23 06:03:13 centos7-image kube-controller-manager: /usr/local/go/src/sync/cond.go:64 +0xc1

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*DeltaFIFO).Pop(0xc8200e0a50, 0xc820189350, 0x0, 0x0, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/delta_fifo.go:399 +0xc4

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.(*Controller).processLoop(0xc8202dbc00)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:127 +0x4a

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.(*Controller).(k8s.io/kubernetes/pkg/controller/framework.processLoop)-fm()

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x20

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil.func1(0xc8202cbf88)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:86 +0x4b

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc8202cbf88, 0x3b9aca00, 0x0, 0x1e5f501, 0xc82009ad80)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:87 +0xb4

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.Until(0xc8202cbf88, 0x3b9aca00, 0xc82009ad80)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/util/wait/wait.go:49 +0x43

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/controller/framework.(*Controller).Run(0xc8202dbc00, 0xc82009ad80)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/framework/controller.go:103 +0x1c2

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/controller/endpoint.(*EndpointController).Run

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/controller/endpoint/endpoints_controller.go:157 +0x79

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 65 [select (no cases), 385 minutes]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.StartControllers(0xc8201758c0, 0xc8202b5200, 0xc82006a9c0, 0xc8205008a0, 0x0, 0x0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:484 +0x62e2

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/cmd/kube-controller-manager/app.Run.func2(0xc8205008a0)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-controller-manager/app/controllermanager.go:157 +0x4b

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/leaderelection.(*LeaderElector).Run

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/leaderelection/leaderelection.go:177 +0x91

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 6692 [select, 269 minutes]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc820026ef0, 0xc82009ad80, 0xc82019f680, 0xc820996f00, 0xc820026ef8)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 7159 [select, 259 minutes]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc820812400, 0xc82009ad80, 0xc8202ee640, 0xc821046660, 0xc820812410)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 3403 [select]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch.func1(0xc820812430, 0xc82009ad80, 0xc82019ec80, 0xc820e6ab40, 0xc820812460)

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:283 +0x3c8

Nov 23 06:03:13 centos7-image kube-controller-manager: created by k8s.io/kubernetes/pkg/client/cache.(*Reflector).ListAndWatch

Nov 23 06:03:13 centos7-image kube-controller-manager: /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/client/cache/reflector.go:296 +0xde3

Nov 23 06:03:13 centos7-image kube-controller-manager: goroutine 68 [select]:

Nov 23 06:03:13 centos7-image kube-controller-manager: k8s.io/kubernetes/pkg/util/wait.JitterUntil(0xc820188150, 0x12a05f200, 0x0, 0x7fc2c64eb601, 0xc82009ad80)