WIP: ip per service

This is filled with cruft, but I wanted to float the idea.  It actually works.

Overview:  When a service object is created, allocate it an IP.  Broadcast that IP to all kube-proxies.  Each one will set up iptables rules to forward traffic for that IP:port to a random local port.  Clients connect to a stable ip:port but no more port collisions!


----- Original Message -----
> > +}
> > +
> > +// NewIPAllocator creates and intializes a new IPAllocator object.
> > +// FIXME: resync from storage at startup.
> > +func NewIPAllocator(subnet string) *IPAllocator {
> > +	return &IPAllocator{
> > +		subnet: ipSubnetFromString(subnet),
> > +	}
> > +}
> > +
> > +// Allocate allocates and returns a new IP.
> > +//FIXME: error if we are "full"?
> > +func (ipa *IPAllocator) Allocate() string {
> > +	ipa.lock.Lock()
> > +	defer ipa.lock.Unlock()
> > +	for i := range ipa.used {
> 
> Turning my attention back to this topic.  I need to store information about
> the IP ranges that have been configured for portal IPs and a mapping of each
> service to an IP.
> 
> Storing the IP along with each service object seems reasonable to me - that's
> the thing that the kube-proxy is already watching.
> 
> I could store some new /registry/portalRange/* objects, each with a bitmap or
> something.

Is portal range globally atomic, or individually atomic?

>  But how do those get created?  Is it kosher for apiserver to
> take a commandline flag and then manifest that into etcd?  Or should it
> assume that configuration already exists in etcd?  Do we have any other
> precedents for configuring the master?

No precedents yet.  I think we probably want to define what default master config is.  We will at least know when a default is not set so we can impose code defaults (vs. requiring all users everywhere to init etcd first).  We probably want to decouple the config for each area, but keep them loosely grouped so you can delete them if necessary.

We didn't do a very good job of prefixing the default keys (`/kubernetes/*`) for simple setups.  I would like to have `/kubernetes/settings|capabilities|config|defaults/<x>` (no pref on name, though).  I would like to separate the configs for unrelated things where possible.

* As an operator, the first time I start etcd, will I know what my allowed port allocation range is? (within error margin maybe, but not exact)
* Once I realize there's a conflict, how do I go clean it up? (delete the conflicting thing, mark the conflicting port as out of bounds, then go?)
* If I need to start from scratch, can I just delete the portal range?
* On APIServer startup, should I attempt to recreate the portal range object if it's not in etcd against my defaults? (makes the previous step easier)

> 
> I don't feel comfortable claiming to know what the plans are around the etcd
> schema, so I am asking :)
> 
> ---
> Reply to this email directly or view it on GitHub:
> https://github.com/GoogleCloudPlatform/kubernetes/pull/1038/files#r17640118