Issue 1:
I enable ResourceQuota in my cluster. I have one namespace ns-xnxin. I have 2 quotas created for that namespace:

(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl describe quota quota --namespace=ns-xnxin
Name: quota-xnxin
Namespace: ns-xnxin
Resource Used Hard
-------- ---- ----
cpu 0 20
memory 0 1Gi
pods 3 3
services 0 5

Name: quota-xnxin-test
Namespace: ns-xnxin
Resource Used Hard
-------- ---- ----
pods 3 1
services 0 5

Before creating quotas, there are already 3 Pods running under ns-xnxin.

Then, I tried to create new Pod two times, but got different response:

(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl create -f busybox-limit.json --namespace=ns-xnxin
Error from server: error when creating "busybox-limit.json": Pod "busybox-limit" is forbidden: limited to 3 pods

(venv)xnxin@LM-SHC-00950210:~/workspace/go/src/k8s.io/kubernetes/yamls$ kubectl create -f busybox-limit.json --namespace=ns-xnxin
Error from server: error when creating "busybox-limit.json": Pod "busybox-limit" is forbidden: limited to 1 pods

It seems that api-server do quota validation in a random behavior by the first get quota record.

I think this is not a consistent behavior and need to be enhanced. Two proposed options:

If k8s accept duplicate resource definition in different quota (for example, in two quotas, both of them contains Pod quota), the quota validation should be done against the sum of them.

Another option is to have resource conflict validation during quota creation. Say, do not allow duplicated resource in different quota.

Issue 2:
One should be able to upgrade a petset through the standard kubectl rolling-update syntax. While it's desirable to have a controlled upgrade strategy (eg: deployments), it may not be necessary for beta.

The tricky part is making sure that pet-v1-0 and pet-v2-0 come up with the same data. For beta I propose we just make kubectl rolling-update on petset update the image and kill pets one by one (as opposed to creating a new petset like a classic rolling-update). If any pet doesn't become ready within a timeout, the rolling-update reverts.

Node upgrade: mark node as unschedulable, delete pets one by one, wait till they show up on new node as ready, upgrade node. Can we get away with just documenting this (upgrading kubelet on a node shouldn't lead to container restarts anyway)?

@smarterclayton @kubernetes/sig-apps Is this enough?

To improve the upgrade UX we could use 2 new events: pre-delete and pre-update. We can use terminationGrace simply to leave the cluster, pre-delete to do something drastic like backup all data, and pre-update to do something like give-up leadership (but retain data). When a member is removed/rejoined the rest of the cluster should receive on-change. #24957 should help with the events.

To improve master/slave upgrade UX, we need to avoid "chasing" the master, i.e by picking the master to upgrade, while the upgrade is happening a new master is elected, the next pod we pick is this new master, and so on.

This might work as follows:

upgrade all slaves

send on-upgrade to master so it gives up mastership, and doesn't compete in re-election

wait for new master

upgrade old master

A leader election service might help here (#28658).

To improve active-active upgrade UX, the on-change sent to all other members needs to exclude the upgrading member by re-writing the config, so writes aren't blocked on its ack. The upgraded member will have to play catchup when it rejoins.

Some databases like Cassandra have special roles, like seed provider. These need to be dealt with in a similar way as leadership transfer (promote new seed, update peers, upgrade existing seed).

To improve node upgrade UX, we could teach kubectl drain to move pets off a node slowly. If people are running 1 pet per node (eg: pod anti-affinity), we have less of a problem.