Change:
Currently replication controllers define an exact value. Are there disadvantages to allowing the replication controller to define more of a policy - at least X, at most X, exactly X? Since replication controllers aren't really aware of overlapping sets, I guess you can't really use that to guide allocation between two pools because they would just end up fighting unproductively
What is the use case? This seems sort of like auto-scaling policy. I'd prefer to put those objects in the auto-scaler.
It sounds like you want a canary-controller that scales the old version down while scaling the new one up. Combine with a service that spreads traffic across both versions and this should Just Work (tm).
I'm thinking about this from the mechanics of doing a deployment by composing replication controllers. With controllers as they are today: existing replication controller with old template, name=foo,deployment=1 x5 create new replication controller with new template, name=foo,deployment=2 x0 alter controller 2 to be x1 (or up by the batch size) alter controller 1 to be x4 (or down by the batch size) ... repeat until done delete controller 1 The thought around atLeast was about explicitly representing the invariant you want during an available deployment: existing replication controller with old template, name=foo,deployment=1 x5 create new replication controller with new template, name=foo, atLeast=5 set controller to be atMost=5 delete a pod from the name=foo,deployment=1 query ... continue until 0 pods in that query delete controller 1 update controller 2 to change the label query to name=foo,deployment=2 I'm not sure that's better.... just the for loop for deletes seems more reasonable. Not a huge deal, just trying to work through how we'd model common deploy cases for end users.
Overlapping replicationController sets: I think we should validate that this isn't possible upon creation of a replicationController. We can do this by determining whether the label selector may overlap with any others, though to do this analytically without looking at the combinations of labels attached to pods it requires that all replicationController label selectors have at least one key in common, with a different value than all other replicationControllers. Policy: As mentioned, we intended the policy to go in a separate auto-scaling policy entity, in a higher-level layer / separate component. Deployment: As pointed out, it is possible to create a new replicationController and gradually scale it up while scaling the original down. Or, one could do the Asgard trick of creating a new replicationController and then shift traffic to it. It's possible to add/remove instances from a service by tweaking their labels. For instance one could toggle the value of an in_service=true/false label. The update demo changes the template of a single replication controller and then kills instances one by one until they all get replaced. I don't like that due to the risk of unpredictable replacements and complexity involved with rollbacks. Splitting out the template ( ) would address the rollback issue, but not the unpredictability. It seems to me that the atMost approach suffers from the same risk. That said, I see what you mean that atLeast/atMost are not auto-scaling policies, but are invariants to maintain. I could imagine that flexibility could be useful. For example, a worker pool manager could use that to opportunistically request resources, where some pod requests may fail. Or, perhaps in the case that we need to temporarily create a new pod for some reason (CRIU-based migration?) but we don't want an old one to be killed. So, I like the idea, just not the use case. :-) FWIW, exactly X could be implemented by combining atLeast and atMost.
Agree that without a kill policy on the replication controller the uses cases described don't work. Is a kill policy something you've discussed for the replication controller (oldest, ones that don't match my template first, ones on most overloaded systems)?
I agree that kill policy would be useful and all of the policies you mention are reasonable in particular scenarios. However, I see this as a rich, unbounded policy space, much like scheduling. IIRC, Joe and/or Brendan proposed kill policies early on and I asked for them to be removed. Instead, I propose that we leave the default policy entirely up to the system, much like scheduling, so we can take as many factors into account as we like, and otherwise I'd like to enable the user to implement an arbitrary policy using event hooks . For instance, I could imagine POSTing to a URL whenever a pod needed to be killed. The user's hook would have N seconds to respond (perhaps configurable), after which the replication controller would kill its default choice.
Thanks for the clarification - I think I'll close this for now until the API stabilizes, and then add some of this context to the doc.
I suggest re-opening this issue, but change the title to "add XXX context to doc YYY", so that we can reference in the PR for posterity.

Discussion:
Kubernetes currently supports two objects that use label selectors to keep track of their members, `service`s and `replicationController`s: - `service`: A service is a configuration unit for the proxies that run on every worker node.  It is named and points to one or more pods. <del> - `replicationController`: A replication controller takes a template and ensures that there is a specified number of "replicas" of that template running at any one time.  If there are too many, it'll kill some.  If there are too few, it'll start more. </del> <ins> - `replicationController`: A [replication controller](replication-controller.md) ensures that a specified number of pod "replicas" are running at any one time.  If there are too many, it'll kill some.  If there are too few, it'll start more. </ins> The set of pods that a `service` targets is defined with a label selector. Similarly, the population of pods that a `replicationController` is monitoring is also defined with a label selector.

For management convenience and consistency, `services` and `replicationControllers` may themselves have labels and would generally carry the labels their corresponding pods have in common. <del> Sets identified by labels and label selectors could be overlapping (think Venn diagrams). For instance, a service might point to all pods with `tier in (frontend), environment in (prod)`.  Now say you have 10 replicated pods that make up this tier.  But you want to be able to 'canary' a new version of this component.  You could set up a `replicationController` (with `replicas` set to 9) for the bulk of the replicas with labels `tier=frontend, environment=prod, track=stable` and another `replicationController` (with `replicas` set to 1) for the canary with labels `tier=frontend, environment=prod, track=canary`.  Now the service is covering both the canary and non-canary pods.  But you can mess with the `replicationControllers` separately to test things out, monitor the results, etc. </del> <ins> Sets identified by labels and label selectors could be overlapping (think Venn diagrams). For instance, a service might target all pods with `tier in (frontend), environment in (prod)`.  Now say you have 10 replicated pods that make up this tier.  But you want to be able to 'canary' a new version of this component.  You could set up a `replicationController` (with `replicas` set to 9) for the bulk of the replicas with labels `tier=frontend, environment=prod, track=stable` and another `replicationController` (with `replicas` set to 1) for the canary with labels `tier=frontend, environment=prod, track=canary`.  Now the service is covering both the canary and non-canary pods.  But you can mess with the `replicationControllers` separately to test things out, monitor the results, etc. </ins> Note that the superset described in the previous example is also heterogeneous. In long-lived, highly available, horizontally scaled, distributed, continuously evolving service applications, heterogeneity is inevitable, due to canaries, incremental rollouts, live reconfiguration, simultaneous updates and auto-scaling, hardware upgrades, and so on.

# Pods <del> A _pod_ (as in a pod of whales or pea pod) is a relatively tightly coupled group of containers that are scheduled onto the same host. It models an application-specific "virtual host" in a containerized environment. Pods serve as units of scheduling, deployment, and horizontal scaling/replication, and share fate. </del> <ins> ## What is a _pod_? </ins> <del> Why doesn't Kubernetes just support an affinity mechanism for co-scheduling containers instead? While pods have a number of benefits (e.g., simplifying the scheduler), the primary motivation is resource sharing. </del> <ins> A _pod_ (as in a pod of whales or pea pod) models an application-specific "logical host" in a containerized environment. It may contain one or more containers which are relatively tightly coupled -- in a pre-container world, they would have executed on the same physical or virtual host. </ins> <del> In addition to defining the containers that run in the pod, the pod specifies a set of shared storage volumes. Pods facilitate data sharing and IPC among their constituents. In the future, they may share CPU and/or memory ([LPC2013](http://www.linuxplumbersconf.org/2013/ocw//system/presentations/1239/original/lmctfy%20(1).pdf)). </del> <ins> Like running containers, pods are considered to be relatively ephemeral rather than durable entities. As discussed in [life of a pod](pod-states.md), pods are scheduled to nodes and remain there until termination (according to restart policy) or deletion. When a node dies, the pods scheduled to that node are deleted. Specific pods are never rescheduled to new nodes; instead, they must be replaced (see [replication controller](replication-controller.md) for more details). (In the future, a higher-level API may support pod migration.) </ins> <del> The containers in the pod also all use the same network namespace/IP (and port space). The goal is for each pod to have an IP address in a flat shared networking namespace that has full communication with other physical computers and containers across the network. [More details on networking](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/networking.md). </del> <ins> ## Motivation for pods </ins> <del> While pods can be used to host vertically integrated application stacks, their primary motivation is to support co-located, co-managed helper programs, such as: </del> <ins> ### Resource sharing and communication Pods facilitate data sharing and communication among their constituents. The containers in the pod all use the same network namespace/IP and port space, and can find and communicate with each other using localhost. Each pod has an IP address in a flat shared networking namespace that has full communication with other physical computers and containers across the network. The hostname is set to the pod's Name for the containers within the pod. [More details on networking](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/networking.md). In addition to defining the containers that run in the pod, the pod specifies a set of shared storage volumes. Volumes enable data to survive container restarts and to be shared among the containers within the pod. In the future, pods will share IPC namespaces, CPU, and memory ([LPC2013](http://www.linuxplumbersconf.org/2013/ocw//system/presentations/1239/original/lmctfy%20(1).pdf)). ### Management Pods also simplify application deployment and management by providing a higher-level abstraction than the raw, low-level container interface. Pods serve as units of deployment and horizontal scaling/replication. Co-location, fate sharing, coordinated replication, resource sharing, and dependency management are handled automatically. ## Uses of pods Pods can be used to host vertically integrated application stacks, but their primary motivation is to support co-located, co-managed helper programs, such as: </ins> - content management systems, file and data loaders, local cache managers, etc. - log and checkpoint backup, compression, rotation, snapshotting, etc. - data change watchers, log tailers, logging and monitoring adapters, event publishers, etc.

Individual pods are not intended to run multiple instances of the same application, in general. <ins> ## Alternatives considered </ins> Why not just run multiple programs in a single Docker container? 1. Transparency. Making the containers within the pod visible to the infrastructure enables the infrastructure to provide services to those containers, such as process management and resource monitoring. This facilitates a number of conveniences for users. <del> 2. Decoupling software dependencies. The individual containers may be rebuilt and redeployed independently. Kubernetes may even support live updates of individual containers someday. </del>  No newline at end of file <ins> 2. Decoupling software dependencies. The individual containers may be rebuilt and redeployed independently. Kubernetes may even support live updates of individual containers someday. 3. Ease of use. Users don't need to run their own process managers, worry about signal and exit-code propagation, etc. 4. Efficiency. Because the infrastructure takes on more responsibility, containers can be lighterweight. Why not support affinity-based co-scheduling of containers? That approach would provide co-location, but would not provide most of the benefits of pods, such as resource sharing, IPC, guaranteed fate sharing, and simplified management. </ins>

<ins> # Replication Controller ## What is a _replication controller_? A _replication controller_ ensures that a specified number of pod "replicas" are running at any one time.  If there are too many, it will kill some.  If there are too few, it will start more. As opposed to just creating singleton pods or even creating pods in bulk, a replication controller replaces pods that are deleted or terminated for any reason, such as in the case of node failure. For this reason, we recommend that you use a replication controller even if your application requires only a single pod. As discussed in [life of a pod](pod-states.md), `replicationController` is *only* appropriate for pods with `RestartPolicy = Always`.  `ReplicationController` should refuse to instantiate any pod that has a different restart policy. As discussed in [issue #503](https://github.com/GoogleCloudPlatform/kubernetes/issues/503#issuecomment-50169443), we expect other types of controllers to be added to Kubernetes to handle other types of workloads, such as build/test and batch workloads, in the future. A replication controller will never terminate on its own, but it isn't expected to be as long-lived as services. Services may be comprised of pods controlled by multiple replication controllers, and it is expected that many replication controllers may be created and destroyed over the lifetime of a service. Both services themselves and their clients should remain oblivious to the replication controllers that maintain the pods of the services. ## How does a replication controller work? ### Pod template A replication controller creates new pods from a template, which is currently inline in the `replicationController` object, but which we plan to extract into its own resource [#170](https://github.com/GoogleCloudPlatform/kubernetes/issues/170). Rather than specifying the current desired state of all replicas, pod templates are like cookie cutters. Once a cookie has been cut, the cookie has no relationship to the cutter. There is no quantum entanglement. Subsequent changes to the template or even switching to a new template has no direct effect on the pods already created. Similarly, pods created by a replication controller may subsequently be updated directly. This is in deliberate contrast to pods, which do specify the current desired state of all containers belonging to the pod. This approach radically simplifies system semantics and increases the flexibility of the primitive, as demonstrated by the use cases explained below. Pods created by a replication controller are intended to be fungible and semantically identical, though their configurations may become heterogeneous over time. This is an obvious fit for replicated stateless servers, but replication controllers can also be used to maintain availability of master-elected, sharded, and worker-pool applications. Such applications should use dynamic work assignment mechanisms, such as the [etcd lock module](https://coreos.com/docs/distributed-configuration/etcd-modules/) or [RabbitMQ work queues](https://www.rabbitmq.com/tutorials/tutorial-two-python.html), as opposed to static/one-time customization of the configuration of each pod, which is considerined an anti-pattern. Any pod customization performed, such as vertical auto-sizing of resources (e.g., cpu or memory), should be performed by another online controller process, not unlike the replication controller itself. ### Labels The population of pods that a `replicationController` is monitoring is defined with a [label selector](labels.md), which creates a loosely coupled relationship between the controller and the pods controlled, in contrast to pods, which are more tightly coupled. We deliberately chose not to represent the set of pods controlled using a fixed-length array of pod specifications, because our experience is that that approach increases complexity of management operations, for both clients and the system. The replication controller should verify that the pods created from the specified template have labels that match its label selector. Though it isn't verified yet, you should also ensure that only one replication controller controls any given pod, by ensuring that the label selectors of replication controllers do not target overlapping sets. Note that `replicationControllers` may themselves have labels and would generally carry the labels their corresponding pods have in common, but these labels do not affect the behavior of the replication controllers. Pods may be removed from a replication controller's target set by changing their labels. This technique may be used to remove pods from service for debugging, data recovery, etc. Pods that are removed in this way will be replaced automatically (assuming that the number of replicas is not also changed). Similarly, deleting a replication controller does not affect the pods it created. It's `replicas` field must first be set to 0 in order to delete the pods controlled. In the future, we may provide a feature to do this and the deletion in a single client operation. ## Responsibilities of the replication controller The replication controller simply ensures that the desired number of pods matches its label selector and are operational. Currently, only terminated pods are excluded from its count. In the future, [readiness](https://github.com/GoogleCloudPlatform/kubernetes/issues/620) and other information available from the system may be taken into account, we may add more controls over the replacement policy, and we plan to emit events that could be used by external clients to implement arbitrarily sophisticated replacement and/or scale-down policies. The replication controller is forever constrained to this narrow responsibility. It itself will not perform readiness nor liveness probes. Rather than performing auto-scaling, it is intended to be controlled by an external auto-scaler (as discussed in [#492](https://github.com/GoogleCloudPlatform/kubernetes/issues/492)), which would change its `replicas` field. We will not add scheduling policies (e.g., [spreading](https://github.com/GoogleCloudPlatform/kubernetes/issues/367#issuecomment-48428019)) to replication controller. Nor should it verify that the pods controlled match the currently specified template, as that would obstruct auto-sizing and other automated processes. Similarly, completion deadlines, ordering dependencies, configuration expansion, and other features belong elsehwere. We even plan to factor out the mechanism for bulk pod creation ([#170](https://github.com/GoogleCloudPlatform/kubernetes/issues/170)). The replication controller is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubecfg (run, stop, resize, rollingupdate) are proof-of-concept examples of this. For instance, we could imagine something like [Asgard](http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html) managing replication controllers, auto-scalers, services, scheduling policies, canaries, etc. ## Common usage patterns ### Rescheduling As mentioned above, whether you have 1 pod you want to keep running, or 1000, replication controller will ensure that the specified number of pods exists, even in the event of node failure or pod termination (e.g., due to an action by another control agent). ### Scaling Replication controller makes it easy to scale the number of replicas up or down, either manually or by an auto-scaling control agent, by simply updating the `replicas` field. ### Rolling updates Replication controller is designed to facilitate rolling updates to a service by replacing pods one-by-one. As explained in [#1353](https://github.com/GoogleCloudPlatform/kubernetes/issues/1353), the recommended approach is to create a new replication controller with 1 replica, resize the new (+1) and old (-1) controllers one by one, and then delete the old controller after it reaches 0 replicas. This predictably updates the set of pods regardless of unexpected failures. Ideally, the rolling update controller would take application readiness into account, and would ensure that a sufficient number of pods were productively serving at any given time. The two replication controllers would need to create pods with at least one differentiating label, such as the image tag of the primary container of the pod, since it is typically image updates that motivate rolling updates. ### Multiple release tracks In addition to running multiple releases of an application while a rolling update is in progress, it's common to run multiple releases for an extended period of time, or even continuously, using multiple release tracks. The tracks would be differentiated by labels. For instance, a service might target all pods with `tier in (frontend), environment in (prod)`.  Now say you have 10 replicated pods that make up this tier.  But you want to be able to 'canary' a new version of this component.  You could set up a `replicationController` with `replicas` set to 9 for the bulk of the replicas, with labels `tier=frontend, environment=prod, track=stable`, and another `replicationController` with `replicas` set to 1 for the canary, with labels `tier=frontend, environment=prod, track=canary`.  Now the service is covering both the canary and non-canary pods.  But you can mess with the `replicationControllers` separately to test things out, monitor the results, etc. </ins>