Issue 1:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "lock is held by" "has not yet expired"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version):

tested on 1.3.5

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5+coreos.1", GitCommit:"4fc64a21f8a593d97b75296acd1e979391518b28", GitTreeState:"clean", BuildDate:"2016-08-22T17:15:28Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

and 1.3.8 (upgraded today)

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.8+coreos.1", GitCommit:"6b29caf6507ce9989984720212fd3e3341069e06", GitTreeState:"clean", BuildDate:"2016-09-30T16:54:50Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Hardware: vmware guest

OS: CoreOs 1122.2.0

kernel: 4.7.0-coreos

masters: 3

/etc/kubernetes/manifests/kube-scheduler.yaml looks like this:

apiVersion: v1
kind: Pod
metadata:
name: kube-scheduler
namespace: kube-system
labels:
app: kube-scheduler
spec:
hostNetwork: true
containers:
- name: kube-scheduler
image: quay.io/coreos/hyperkube:v1.3.5_coreos.1
command:
- /hyperkube
- scheduler
- --master=http://127.0.0.1:8080
- --leader-elect=true
- --v=2
livenessProbe:
httpGet:
host: 127.0.0.1
path: /healthz
port: 10251
initialDelaySeconds: 15
timeoutSeconds: 1

/etc/kubernetes/manifests/kube-controller-manager.yaml looks like this:

apiVersion: v1
kind: Pod
metadata:
name: kube-controller-manager
namespace: kube-system
labels:
app: kube-controller-manager
spec:
hostNetwork: true
containers:
- name: kube-controller-manager
image: quay.io/coreos/hyperkube:v1.3.5_coreos.1
command:
- /hyperkube
- controller-manager
- --master=http://127.0.0.1:8080
- --leader-elect=true
- --service-account-private-key-file=/etc/kubernetes/ssl/apiserver-key.pem
- --root-ca-file=/etc/kubernetes/ssl/ca.pem
- --v=2
livenessProbe:
httpGet:
host: 127.0.0.1
path: /healthz
port: 10252
initialDelaySeconds: 15
timeoutSeconds: 1
volumeMounts:
- mountPath: /etc/kubernetes/ssl
name: ssl-certs-kubernetes
readOnly: true
- mountPath: /etc/ssl/certs
name: ssl-certs-host
readOnly: true
volumes:
- hostPath:
path: /etc/kubernetes/ssl
name: ssl-certs-kubernetes
- hostPath:
path: /usr/share/ca-certificates
name: ssl-certs-host

What happened:

The logs from kube-scheduler and kube-controller-manager are spamming with:

I1012 07:13:30.700862 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:33.263266 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:36.944429 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:40.486944 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:44.772153 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:47.186971 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:49.914362 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:52.353564 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:56.181010 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:13:59.263994 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:03.553559 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:06.325838 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:10.595729 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:13.347343 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:17.330717 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:21.505816 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:23.950507 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:26.892771 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:30.914393 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:34.408769 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:38.544271 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:40.645166 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:43.919125 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:47.663210 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:51.768576 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:14:55.748494 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:00.116892 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:03.489400 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:06.465890 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:09.703411 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:13.463524 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:15.607621 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:17.896426 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:22.087427 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:26.217274 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:30.269665 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:33.637867 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:36.712575 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:40.783867 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:44.372188 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:47.605168 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:50.434502 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:53.216699 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:57.015595 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:15:59.152636 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired
I1012 07:16:01.301530 1 leaderelection.go:296] lock is held by utv-kube03 and has not yet expired

The node holding the lock only logs this:

I1012 07:27:20.982437 1 leaderelection.go:215] sucessfully acquired lease kube-system/kube-scheduler

What you expected to happen:

If this is expected behaviour i would like to be able to supress those messages with --v=2

How to reproduce it (as minimally and precisely as possible):

I think it would be sufficient to run scheduler or controller-manager with --leader-elect?true.

etcdctl get registry/services/endpoints/kube-system/kube-controller-manager looks like this:

{
"kind": "Endpoints",
"apiVersion": "v1",
"metadata": {
"name": "kube-controller-manager",
"namespace": "kube-system",
"selfLink": "/api/v1/namespaces/kube-system/endpoints/kube-controller-manager",
"uid": "b9ee7a6f-8f82-11e6-b186-0050568802e1",
"creationTimestamp": "2016-10-11T07:17:10Z",
"annotations": {
"control-plane.alpha.kubernetes.io/leader": "{\"holderIdentity\":\"utv-kube03\",\"leaseDurationSeconds\":15,\"acquireTime\":\"2016-10-11T07:17:10Z\",\"renewTime\":\"2016-10-12T07:18:54Z\",\"leaderTransitions\":0}"
}
},
"subsets": []
}

The cluster seems healthy and are scheduling pods correctly.

Issue 2:
What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

kube-dns, kubeadm, headless service, service

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version):

kubectl version
Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"a16c0a7f71a6f93c7e0f222d961f4675cd97a46b", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: hardware x64

OS (e.g. from /etc/os-release):

cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

Kernel (e.g. uname -a):

uname -a
Linux panel4.local.minirz.de 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Install tools:

kubeadm and weave

Others:

What happened:

A headless service doesn't resolve to an A RR

$ kubectl get endpoints
NAME ENDPOINTS AGE
api 2h
registry 10.44.0.2:7866,10.44.0.2:7866,10.44.0.2:7876 2h
webconsole 2h

# Use default DNS Server fails
$ kubectl exec -ti debug -- nslookup registry
Server: 100.64.0.10
Address 1: 100.64.0.10

nslookup: can't resolve 'registry'

kubectl get endpoints --namespace=kube-system
NAME ENDPOINTS AGE
kube-controller-manager <none> 2h
kube-dns 10.36.0.3:53,10.36.0.3:53 4h
kube-scheduler <none> 2h
kubernetes-dashboard 10.44.0.3:9090 4h

# Use the kube-dns Pod IP works
$ kubectl exec -ti debug -- nslookup registry 10.36.0.3
Server: 10.36.0.3
Address 1: 10.36.0.3 kube-dns-2247936740-s5lu1

Name: registry
Address 1: 10.44.0.4 debug

What you expected to happen:

Getting the A RR for all endpoints of this headless service.

How to reproduce it (as minimally and precisely as possible):

Create a cluster with kubeadm + weave then create a headless service:

apiVersion: v1
kind: Service
metadata:
name: registry
spec:
clusterIP: "None" # headless service => kube-dns will return pod IPs
ports:
- name rpc
port: 7866
protocol: TCP
selector:
role: registry

and try to resolve it.

Anything else do we need to know:

On the node running the debug pod:

ifconfig weave
weave: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1410
inet 10.32.0.1 netmask 255.240.0.0 broadcast 0.0.0.0
inet6 fe80::c88:2dff:fe1a:e66c prefixlen 64 scopeid 0x20<link>
ether 0e:88:2d:1a:e6:6c txqueuelen 0 (Ethernet)
RX packets 366 bytes 21068 (20.5 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 303 bytes 23660 (23.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ifconfig | grep 100.64

and the iptables output:

[root@panel4 ~]# iptables-save | grep 100.64
-A KUBE-SERVICES -d 100.64.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -d 100.64.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SERVICES -d 100.64.0.10/32 -p tcp -m comment --comment "kube-system/kube-dns:dns-tcp cluster IP" -m tcp --dport 53 -j KUBE-SVC-ERIFXISQEP7F7OF4
[root@panel4 ~]# iptables-save | grep KUBE-SVC-TCOU7JCQXEZGVUNU
:KUBE-SVC-TCOU7JCQXEZGVUNU - [0:0]
-A KUBE-SERVICES -d 100.64.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-6RTHVOJVKE2IJ2GA
[root@panel4 ~]# iptables-save | grep KUBE-SEP-6RTHVOJVKE2IJ2GA
:KUBE-SEP-6RTHVOJVKE2IJ2GA - [0:0]
-A KUBE-SEP-6RTHVOJVKE2IJ2GA -s 10.36.0.3/32 -m comment --comment "kube-system/kube-dns:dns" -j KUBE-MARK-MASQ
-A KUBE-SEP-6RTHVOJVKE2IJ2GA -p udp -m comment --comment "kube-system/kube-dns:dns" -m udp -j DNAT --to-destination 10.36.0.3:53
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m comment --comment "kube-system/kube-dns:dns" -j KUBE-SEP-6RTHVOJVKE2IJ2GA

Maybe related: #33671 (comment)

FYI: @bboreham