Issue 1:
Do we intend to support prior versions of APIs? E.g. Should we

support v1beta1 inputs once we support v1beta2? To do that properly

we need to version the whole API, not just individual structures, and

we probably want version-specific code that translates old inputs to

"most modern" form well isolated. And Then do we support old versions

on output?

To do this "right" is very intensive.

We also need to write up a policy statement regarding API changes and deprecation policies.

Issue 2:
BUG REPORT

Kubernetes version

v1.4.3+coreos.0

Environment:

Cloud provider or hardware configuration:

Bare metal CoreOS installation. 3 management nodes + 2 worker nodes.

OS (e.g. from /etc/os-release):

CoreOS stable (1185.3.0)

Kernel

4.7.3-coreos-r2

What happened:

DaemonSet pods randomly fail to start on management nodes after nodes reboot.

> kubectl get pods -o wide -a -l app=ds-gluster-slow-01
NAME READY STATUS RESTARTS AGE IP NODE
gluster-slow-01-fgpk3 1/1 Running 0 19h 10.54.147.5 10.54.147.5
gluster-slow-01-payvs 0/1 MatchNodeSelector 0 10h <none> 10.54.147.6
gluster-slow-01-zdr0t 1/1 Running 0 19h 10.54.147.4 10.54.147.4

10.54.147.4, 10.54.147.5, 10.54.147.6 are the management nodes.

The only way to start the pod again is to delete the pod. The MatchNodeSelector happens once every 2-3 reboots. For remaining reboots the pods are started properly.

How to reproduce it:

Create a DaemonSet and assign it to management nodes based on user defined labels.

Here is our configuration:

Hosts labels:

NAME STATUS AGE LABELS
10.54.147.10 Ready 52d beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.54.147.10,type=infra
10.54.147.11 Ready 52d beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.54.147.11,type=infra
10.54.147.4 Ready 53d beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.4,type=management
10.54.147.5 Ready 53d beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.5,type=management
10.54.147.6 Ready,SchedulingDisabled 47d beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,gluster=slow-01,kubernetes.io/hostname=10.54.147.6,type=management

DaemonSet definition:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
labels:
name: gluster-slow-01
name: gluster-slow-01
namespace: default
spec:
selector:
matchLabels:
app: ds-gluster-slow-01
template:
metadata:
labels:
app: ds-gluster-slow-01
spec:
containers:
- image: internal-registry/mic/infra/glusterfs:1
name: gluster-slow-01
livenessProbe:
tcpSocket:
port: 24007
initialDelaySeconds: 30
timeoutSeconds: 1
securityContext:
privileged: true
volumeMounts:
- mountPath: /mnt/localdisk
name: gluster-localdisk
- mountPath: /var/lib/glusterd
name: gluster-varlib
- mountPath: /dev
name: gluster-dev
- mountPath: /sys/fs/cgroup
name: gluster-cgroup
dnsPolicy: ClusterFirst
hostNetwork: true
nodeSelector:
gluster: slow-01
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 1
volumes:
- hostPath:
path: /mnt/localdisk
name: gluster-localdisk
- hostPath:
path: /dev
name: gluster-dev
- hostPath:
path: /sys/fs/cgroup
name: gluster-cgroup
- hostPath:
path: /var/lib/glusterd
name: gluster-varlib

Anything else do we need to know:

The kube-scheduler yaml file for managment nodes is defined as follows:

apiVersion: v1
kind: Pod
metadata:
name: kube-scheduler
namespace: kube-system
spec:
hostNetwork: true
containers:
- name: kube-scheduler
image: quay.io/coreos/hyperkube:v1.4.3_coreos.0
command:
- /hyperkube
- scheduler
- --master=http://127.0.0.1:8080
- --leader-elect=true
livenessProbe:
httpGet:
host: 127.0.0.1
path: /healthz
port: 10251
initialDelaySeconds: 15
timeoutSeconds: 1

The top log lines from the kube-scheduler pod are:

E1109 06:51:42.487688 1 leaderelection.go:252] error retrieving endpoint: Get http://127.0.0.1:8080/api/v1/namespaces/kube-system/endpoints/kube-scheduler: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662848 1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:394: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662918 1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:404: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.662958 1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:391: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%21%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663004 1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:388: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D%2Cstatus.phase%21%3DFailed%2Cstatus.phase%21%3DSucceeded&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663124 1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:414: Failed to list *extensions.ReplicaSet: Get http://127.0.0.1:8080/apis/extensions/v1beta1/replicasets?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663170 1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:409: Failed to list *api.ReplicationController: Get http://127.0.0.1:8080/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663212 1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:399: Failed to list *api.PersistentVolumeClaim: Get http://127.0.0.1:8080/api/v1/persistentvolumeclaims?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E1109 06:51:42.663257 1 reflector.go:214] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:398: Failed to list *api.PersistentVolume: Get http://127.0.0.1:8080/api/v1/persistentvolumes?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
I1109 06:51:45.940628 1 leaderelection.go:295] lock is held by sid-kb-004 and has not yet expired
I1109 06:51:48.664098 1 leaderelection.go:295] lock is held by sid-kb-004 and has not yet expired

I suspect it is a race condition during the management pods initialization. The kube-scheduler is started before the kube-apiserver pod. It may not be handling well the fact that it can't connect to the Kubernetes api service.

I tried to workaround it by adding the init-containers section:

apiVersion: v1
kind: Pod
metadata:
name: kube-scheduler
namespace: kube-system
annotations:
pod.beta.kubernetes.io/init-containers: '[
{
"image": "quay.io/coreos/hyperkube:v1.4.3_coreos.0",
"name": "wait-for-master",
"command": [ "/bin/bash", "-c", "while ! timeout 1 bash -c \"/kubectl --server=http://127.0.0.1:8080/ cluster-info\"; do sleep 1; done" ]
}
]'
spec:
hostNetwork: true
containers:
- name: kube-scheduler
image: quay.io/coreos/hyperkube:v1.4.3_coreos.0
command:
- /hyperkube
- scheduler
- --master=http://127.0.0.1:8080
- --leader-elect=true
livenessProbe:
httpGet:
host: 127.0.0.1
path: /healthz
port: 10251
initialDelaySeconds: 15
timeoutSeconds: 1

but it looks like init-containers only executes during POD creation. When the POD is restarted it does not execute.

My bug is similar to #31123, but provides more data.