Prepare for external scheduler

1. Add more Pod statuses. Expectation that PodStatus will get changed by
   future scheduling actuator API, indicating acceptance/rejection of
   pod.
2. Store assigned host in etcd.
3. Change pod key to /registry/pods/<podid>. Container location remains
   the same (/registry/hosts/<machine>/kublet).
@lavalamp @smarterclayton @erictune 
Sorry for joining the discussion late. Many deep issues affecting the API layering and system architecture are entangled in this discussion. So, first I'll do a brain dump, then we can discuss how to break up the problem and converge on concrete solutions.

Admission control: Should the request to run this particular Pod be accepted? Clients definitely need to be able to distinguish permanent from temporary rejection, for some definition of "permanent" in a dynamic environment, and users, and in some cases systems, need actionable information about the cause of the rejection. For instance, the rejection could be due to authorization failure, invalid request, exceeding budget or quota, etc. 

However, the same question may also apply to sets of pods. For instance, a batch scheduler may want to make an admission-control decision based on whether a whole set of pods could be gang-scheduled simultaneously. A higher-level service placement system may want to determine whether a whole service could "fit". Update/rollout tools may want to verify that the updated resource requirements will work in aggregate. So, we'll likely need a way to invoke an admission-control decision on an aggregate resource request of some kind, without actually instantiating the underlying resource consumers (i.e., pods in this case).

Additionally, typically these kinds of decisions also want some assurance that the resources will be available beyond the current moment. For deployments of k8s on physical hosts, this may involve ensuring that enough spare capacity has been provisioned, taking into account potential host failures and maintenance. Being able to satisfy the resource requests "now" isn't sufficient. For deployments on virtual hosts, we'll need to figure out what interfaces we need to wrap around the underlying virtual resource provisioning mechanisms. In general, we want to ensure we have the means to communicate resource demand to provisioning mechanisms, including both host provisioning and quota provisioning. 

We'll also need to figure out how we should handle resource (de)fragmentation and workload churn in these decisions. This is related to Clayton's point about retry and back-off.

If we reject pod creation requests due to instantaneous infeasibility/unsatisfiability, then we create a situation where services could be starved indefinitely of resources, regardless of priority, fairness, etc. Mesos addressed this with its resource offer mechanism: http://static.usenix.org/event/nsdi11/tech/full_papers/Hindman_new.pdf. Some sort of resource request queue or scoreboard is needed. Probably we should involve Mesosphere (@adam-mesos) in the discussion of the approach to take here, as well as in the resource demand signaling discussed above.

Speaking of priority, that raises the issue of preemption in order to satisfy higher-priority requests.

Synchronous vs. asynchronous admission control and/or scheduling: Synchronous is easier for users to deal with, but asynchronous provides more flexibility and transparency to higher-level infrastructure layers. A synchronous interface can always be layered upon an asynchronous API, on the server or in libraries, UI, CLI, etc. If asynchronous, that's where it becomes necessary to represent that the request is awaiting an external dependency to be satisfied. Which leads me to...

An explicit enumeration of states and state machine, especially where those states may be persisted and/or interpreted by clients: This approach has a number of problems (e.g., the persisted state value may diverge from the actual current status), but the biggest is that it's not extensible. Components of the system, clients, user applications, etc. will bake in a particular set of states. The bigger the ecosystem becomes, the harder it becomes to add to that set of states, and it quickly becomes impossible to even change the observed runtime behavior of the existing states. Components don't know how to handle new, unrecognized states. Extensions to the system have no way of injecting new states dynamically in a running system, so they instead use hacks rather than states. Instead, I think there are just 3 relevant states: not yet running, running, and terminated. 

There may be many reasons for not yet running. We should have a separate field to indicate the reason, which shouldn't be a strict enumeration, but which should represent what component/decision we're waiting on. It should be possible to dynamically orchestrate a chain of decisions (e.g., resource prediction -> cluster selection -> gang scheduling -> cluster admission control -> instance scheduling) using a continuation-passing approach. (Related to that, some kind of event bus will be needed in order to kick off automation that isn't on the critical path of instantiating the objects.)

Similarly, there are many reasons for termination -- see #137 .

Interface with scheduler: Did this PR imply that the interface should be implicit through the shared state in etcd? Let's not do that. Let's hash out a real API, here or elsewhere.

Master vs. Kubelet APIs: We're going to have to split these, ideally in a way that they cleanly compose.

Desired state vs. current state: I don't like the way these are represented in the current API. We need to split them into separate schemas with distinct (but related) API endpoints. The current approach is a source of user confusion, is hard to even document, and will be problematic for declarative config systems.

Did I miss any issues?