Issue 1:
I'm working on standing up a 3 node Zookeeper instance, and instead of having separate RCs and Services for each individual pod, I'd like one RC with three instances that keeps all three pods alive. The image is based off the mesoscloud/zookeeper image, just with the quorum enabled.

zookeeper-rc.json

{
"apiVersion": "v1",
"kind": "ReplicationController",
"metadata": {
"name": "zookeeper-test"
},
"spec": {
"replicas": 1,
"template": {
"metadata": {
"labels": {
"app": "zookeeper"
}
},
"spec": {
"containers": [
{
"env": [
{
"name": "MYID",
"value": "3"
},
{
"name": "SERVERS",
"value": "zk1,zk2,zk3"
}
],
"image": "custom/zookeeper:latest",
"name": "zk3",
"ports": [
{
"containerPort": 2181
},
{
"containerPort": 2888
},
{
"containerPort": 3888
}
]
},
{
"env": [
{
"name": "MYID",
"value": "2"
},
{
"name": "SERVERS",
"value": "zk1,zk2,zk3"
}
],
"image": "custom/zookeeper:latest",
"name": "zk2",
"ports": [
{
"containerPort": 2181
},
{
"containerPort": 2888
},
{
"containerPort": 3888
}
]
},
{
"env": [
{
"name": "MYID",
"value": "1"
},
{
"name": "SERVERS",
"value": "zk1,zk2,zk3"
}
],
"image": "custom/zookeeper:latest",
"name": "zk1",
"ports": [
{
"containerPort": 2181
},
{
"containerPort": 2888
},
{
"containerPort": 3888
}
]
}
]
}
}
}
}

I have them working with one container per RC, but then I have one RC for each pod. Right now this is just testing, but when I scale out to 7+ nodes, and occassionally 13 nodes, I don't want 13 RCs for one ZK deployment.

When I try to deploy this:

mynameismevin$ kubectl create -f zookeeper-rc.json --validate
The ReplicationController "zookeeper-rc" is invalid:
* spec.template.spec.containers[1].name: duplicate value 'server'
* spec.template.spec.containers[2].name: duplicate value 'server'

The EnvVars have to have "name":"server" for each container, so that can be passed to ZK so it knows what servers are in the cluster.

Is this a limitation of k8s or am I using this incorrectly?

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

scheduledjob, cannot determine, too many missed start times

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0+coreos.0", GitCommit:"278a1f7034bdba61cba443722647da1a8204a6fc", GitTreeState:"clean", BuildDate:"2016-09-26T20:48:37Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0+coreos.0", GitCommit:"278a1f7034bdba61cba443722647da1a8204a6fc", GitTreeState:"clean", BuildDate:"2016-09-26T20:48:37Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: bare metal

OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie)

Kernel (e.g. uname -a): 4.4.0-36-generic #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 GNU/Linux

Install tools: quay.io/coreos/hyperkube:v1.4.0_coreos.0, deployed with kargo

Others:

What happened:

I am using a scheduledjob with concurrencyPolicy: Forbid. An instance of my job has taken a long time to finish, thus skipping many start times.

Ever since, the scheduledjob will not schedule anything new.

Controller manager log has:

E1106 13:37:12.457445 1 controller.go:163] Cannot determine if default/my-cron-job needs to be started: Too many missed start times to list

What you expected to happen:

I expect the scheduledjob not to be stuck not scheduling anything anymore. With concurrencyPolicy: Forbid, I expect that it should just re-schedule something as soon as the previously running job has completed.

The "Too many missed start times to list" safeguard appears to apply mostly to cases where job concurrency is allowed to avoid creating a lot of jobs in case of timestamp issues.

kubernetes/pkg/controller/scheduledjob/utils.go

Line 148
in
56bbfd2

// An object might miss several starts. For example, if

:

// An object might miss several starts. For example, if
// controller gets wedged on friday at 5:01pm when everyone has
// gone home, and someone comes in on tuesday AM and discovers
// the problem and restarts the controller, then all the hourly
// jobs, more than 80 of them for one hourly scheduledJob, should
// all start running with no further intervention (if the scheduledJob
// allows concurrency and late starts).
//

// However, if there is a bug somewhere, or incorrect clock
// on controller's server or apiservers (for setting creationTimestamp)
// then there could be so many missed start times (it could be off
// by decades or more), that it would eat up all the CPU and memory
// of this controller. In that case, we want to not try to list
// all the misseded start times.

How to reproduce it (as minimally and precisely as possible):

The issue shows up after about 100 minutes of this running.

apiVersion: batch/v2alpha1
kind: ScheduledJob
metadata:
name: my-cron-job
labels:
app: my-cron-job
spec:
schedule: 0/1 * * * ?
concurrencyPolicy: Forbid
jobTemplate:
spec:
template:
spec:
containers:
- name: my-cron-job
image: busybox:1.25
command: ["sleep", "7000"]
restartPolicy: Never

Anything else do we need to know: