Prepare for external scheduler

1. Add more Pod statuses. Expectation that PodStatus will get changed by
   future scheduling actuator API, indicating acceptance/rejection of
   pod.
2. Store assigned host in etcd.
3. Change pod key to /registry/pods/<podid>. Container location remains
   the same (/registry/hosts/<machine>/kublet).
Sorry for being even later to the discussion; still trying to grok all these related threads. Here are my thoughts, particularly as related to Mesos (or a Mesos framework) as a consumer of the scheduler API and pod states.

PodStatus correlates closely to Mesos TaskStates. For your reference Mesos has the following TaskStates:
- Staging (Mesos asked to launch task, but not yet downloaded to slave)
- Starting (task+executor downloaded to slave, executor started and ready to run task)
- Running (task is now running)
- Finished (task completed successfully)
- Failed (task failed during execution)
- Killed (killed explicitly by client/user)
- Lost (invalid task info, not enough resources, slave disconnected/removed, task no longer found, authorization or other system failure; each distinguished by an additional message string passed alongside the TaskState).

Mesos has no notion of 'New' or 'Admitted' as those are states internal to the framework-scheduler which will choose to eventually accept one of Mesos' resource offers with a LaunchTask message, at which point the Mesos master considers the task 'Staging'.

PodStatuses: I don't see the value in separating the Accepted state from the Assigned. Since this is an asynchronous, distributed system, a pod that was once Accepted/Feasible may no longer be feasible by the time the scheduler gets ahold of it to schedule/assign it. In fact, even after the scheduler has made an assignment and posted that to the registry, that machine could fill up or go down, meaning that the pod must be rejected/rescheduled anyway.

Brian's simplified NotYetRunning, Running, and Terminated makes sense to me. Mesos clients often write a helper function like isTerminal() that checks for any of Finished/Failed/Killed/Lost, so I can see the benefits of grouping those together alongside an additional reason/message. But I also see from the Scheduler API perspective the benefit of separating New (please schedule me) from Pending/Assigned/Staging (trying to schedule/start). Seems like you can keep Pending, Running, and Stopped, and just add New (or New can be expressed as a message to the scheduler).

Scheduler interface: +1 on not relying on shared state as an implicit API. I want an explicit API where the scheduler is notified of new pods and the scheduler can announce its assignment for each pod.

Handling infeasibility: "is the right behavior to accept the request and keep working on it, or to reject the request after some amount of time" - exactly the reason Mesos uses a resource offer model. If the client asked for something infeasible, Mesos responds with "Here's what I can offer you." and the client can decide whether to accept or decline the offer. In the Kubernetes scenario, I would lean toward a prompt rejection (with helpful error message/code) and let the client retry to its heart's content.

Pod-validity checks: I feel like can-it-fit and quota/fairness checks should be handled by the scheduler. The API server should just be responsible for adding a pod request to the registry, and then the scheduler can figure out whether/where to launch/schedule the pod. In a Mesos scheduler implementation, the pool of available resource offers is constantly changing, so the can-it-fit check would need to be closely tied to the scheduling/assignment.
Maybe apiserver could also have its own preliminary check for user quotas, abusive scheduling, etc. Each step in the process should have a way of erroring-out with a helpful error message/code.

Gang Scheduling: +1. To hoard or wait? Probably deferrable to a separate discussion.

Preemption: +1, but this can probably be deferred to a separate discussion too.