WIP: ip per service

This is filled with cruft, but I wanted to float the idea.  It actually works.

Overview:  When a service object is created, allocate it an IP.  Broadcast that IP to all kube-proxies.  Each one will set up iptables rules to forward traffic for that IP:port to a random local port.  Clients connect to a stable ip:port but no more port collisions!
This response got totally mangled - I had to try to manually extract the
meaning, and I can not find it on GitHub's excellent UI.

I think you said:

> Is portal range globally atomic, or individually atomic?

I'm not sure I get the question.  I think it is globally atomic, since we
can not afford to have dup assignments.  If we had replicated apiservers,
they need to sync between themselves when allocating an IP.

> No precedents yet. I think we probably want to define what default master
> config is. We will at least know when a default is not set so we can
impose
> code defaults (vs. requiring all users everywhere to init etcd first). We

But there will be some things that won't work until configured - Portal IPs
need to be defined - I do not think we can safely auto-assign them.  But
maybe we can find a heuristic around RFC1918 and/or link-local addrs.

> probably want to decouple the config for each area, but keep them loosely
> grouped so you can delete them if necessary. We didn't do a very good job
of
> prefixing the default keys (`/kubernetes/*`) for simple setups. I would
like to
> have `/kubernetes/settings|capabilities|config|defaults/<x>` (no pref on
name,
> though). I would like to separate the configs for unrelated things where
> possible.
>   * As an operator, the first time I start etcd, will I know what my
allowed
>     port allocation range is? (within error margin maybe, but not exact)
>   * Once I realize there's a conflict, how do I go clean it up? (delete
the
>     conflicting thing, mark the conflicting port as out of bounds, then
go?)
>   * If I need to start from scratch, can I just delete the portal range?
>   * On APIServer startup, should I attempt to recreate the portal range
object
>     if it's not in etcd against my defaults? (makes the previous step
easier)

All fair questions.  If we assume that eventually we lose the right to ask
people to reboot their whole cluster, we'll have to make all such actions
work "live".  I figured for things like Portal IPs, you could reconfigure
it and it would start allocating from the new config space, but would leave
teh old portals active.  We'll have to decide and spec the behavior for
every such config option.

But it sounds like we will have to, soon, think about master config (and
maybe minion config) more.

On Thu, Sep 18, 2014 at 4:05 AM, Clayton Coleman <notifications@github.com>
wrote:

> In pkg/registry/service/storage.go:
>
> > +}
> > +
> > +// NewIPAllocator creates and intializes a new IPAllocator object.
> > +// FIXME: resync from storage at startup.
> > +func NewIPAllocator(subnet string) *IPAllocator {
> > +	return &IPAllocator{
> > +		subnet: ipSubnetFromString(subnet),
> > +	}
> > +}
> > +
> > +// Allocate allocates and returns a new IP.
> > +//FIXME: error if we are "full"?
> > +func (ipa *IPAllocator) Allocate() string {
> > +	ipa.lock.Lock()
> > +	defer ipa.lock.Unlock()
> > +	for i := range ipa.used {
>
>  ----- Original Message -----
>  > +} > + > +// NewIPAllocator creates and intializes a new IPAllocator
> object. > +// FIXME: resync from storage at startup. > +func
> NewIPAllocator(subnet string) *IPAllocator { > + return &IPAllocator{ > +
> subnet: ipSubnetFromString(subnet), > + } > +} > + > +// Allocate allocates
> and returns a new IP. > +//FIXME: error if we are "full"? > +func (ipa
> *IPAllocator) Allocate() string { > + ipa.lock.Lock() > + defer
> ipa.lock.Unlock() > + for i := range ipa.used { Turning my attention back
> to this topic. I need to store information about the IP ranges that have
> been configured for portal IPs and a mapping of each service to an IP.
> Storing the IP along with each service object seems reasonable to me -
> that's the thing that the kube-proxy is already watching. I could store
> some new /registry/portalRange/* objects, each with a bitmap or something.
> Is portal range globally atomic, or individually atomic?
>  But how do those get created? Is it kosher for apiserver to take a
> commandline flag and then manifest that into etcd? Or should it assume that
> configuration already exists in etcd? Do we have any other precedents for
> configuring the master?
> No precedents yet. I think we probably want to define what default master
> config is. We will at least know when a default is not set so we can impose
> code defaults (vs. requiring all users everywhere to init etcd first). We
> probably want to decouple the config for each area, but keep them loosely
> grouped so you can delete them if necessary. We didn't do a very good job
> of prefixing the default keys (`/kubernetes/*`) for simple setups. I would
> like to have `/kubernetes/settings|capabilities|config|defaults/<x>` (no
> pref on name, though). I would like to separate the configs for unrelated
> things where possible. * As an operator, the first time I start etcd, will
> I know what my allowed port allocation range is? (within error margin
> maybe, but not exact) * Once I realize there's a conflict, how do I go
> clean it up? (delete the conflicting thing, mark the conflicting port as
> out of bounds, then go?) * If I need to start from scratch, can I just
> delete the portal range? * On APIServer startup, should I attempt to
> recreate the portal range object if it's not in etcd against my defaults?
> (makes the previous step easier)
> ... <#1488870a4cd81947_>
>  I don't feel comfortable claiming to know what the plans are around the
> etcd schema, so I am asking :) --- Reply to this email directly or view it
> on GitHub:
> https://github.com/GoogleCloudPlatform/kubernetes/pull/1038/files#r17640118
>
> --
> Reply to this email directly or view it on GitHub
> <https://github.com/GoogleCloudPlatform/kubernetes/pull/1038/files#r17720779>
> .
>