Change:
Restarting master causes a recreation of fwrule and target pool for type=loadbalancer services. We are saved in the non restart case because we compare a cached verson of the service to decided if we should even call into the cloudprovider and update resources. This is happening because our reconciliation routines are confused about session affintiy and lbip, leading them to recreate, since inplace update isn't allowed. The impact is at best a few seconds of radio silence, at worst who knows? Repro: Create a loadbalanced service, eg: Wait for it to acquire a vip. Try it out Ssh into master and restart kube-controller-manager: Watch the wgets, you'll notice radio silence after a bit. This is because the targetpool and forwarding rule get recreted, for 2 reasons: affinity mismatch So this check fails: Lbip mismatch: So this check fails: (lbip is retrieved here: )
the session affinity fix is trivial. The easiest change for the lb ip fix is to not actually check the ip, but simply backoff as long as the forwarding rule exists. This is "bad" because of the way we allocate a static-ip, assign it to the fwd rule, then de-allocate it. The alternative is to use to reconcile, but that's racy.
cherry pick

Discussion:
if err != nil { return nil, err } <ins> if !fwdRuleExists { glog.Infof("Forwarding rule %v for Service %v/%v doesn't exist", loadBalancerName, apiService.Namespace, apiService.Name) } </ins> // Make sure we know which IP address will be used and have properly reserved // it as static before moving forward with the rest of our operations.

if err != nil { return nil, err } <ins> if !tpExists { glog.Infof("Target pool %v for Service %v/%v doesn't exist", loadBalancerName, apiService.Namespace, apiService.Name) } </ins> // Now we get to some slightly more interesting logic. // First, neither target pools nor forwarding rules can be updated in place -

if err := gce.deleteForwardingRule(loadBalancerName, gce.region); err != nil { return nil, fmt.Errorf("failed to delete existing forwarding rule %s for load balancer update: %v", loadBalancerName, err) } <del> glog.V(4).Infof("EnsureLoadBalancer(%v(%v)): deleted forwarding rule", loadBalancerName, serviceName) </del> <ins> glog.Infof("EnsureLoadBalancer(%v(%v)): deleted forwarding rule", loadBalancerName, serviceName) </ins> } if tpExists && tpNeedsUpdate { if err := gce.deleteTargetPool(loadBalancerName, gce.region); err != nil { return nil, fmt.Errorf("failed to delete existing target pool %s for load balancer update: %v", loadBalancerName, err) } <del> glog.V(4).Infof("EnsureLoadBalancer(%v(%v)): deleted target pool", loadBalancerName, serviceName) </del> <ins> glog.Infof("EnsureLoadBalancer(%v(%v)): deleted target pool", loadBalancerName, serviceName) </ins> } // Once we've deleted the resources (if necessary), build them back up (or for

return nil, fmt.Errorf("failed to create target pool %s: %v", loadBalancerName, err) } if len(hosts) <= maxTargetPoolCreateInstances { <del> glog.V(4).Infof("EnsureLoadBalancer(%v(%v)): created target pool", loadBalancerName, serviceName) </del> <ins> glog.Infof("EnsureLoadBalancer(%v(%v)): created target pool", loadBalancerName, serviceName) </ins> } else { <del> glog.V(4).Infof("EnsureLoadBalancer(%v(%v)): created initial target pool (now updating with %d hosts)", loadBalancerName, serviceName, len(hosts)-maxTargetPoolCreateInstances) </del> <ins> glog.Infof("EnsureLoadBalancer(%v(%v)): created initial target pool (now updating with %d hosts)", loadBalancerName, serviceName, len(hosts)-maxTargetPoolCreateInstances) </ins> created := sets.NewString() for _, host := range createInstances {

if isHTTPErrorCode(err, http.StatusNotFound) { return false, true, "", nil } <del> return false, false, "", fmt.Errorf("error getting load balancer's forwarding rule: %v", err) </del> <ins> // Err on the side of caution in case of errors. Caller should notice the error and retry. // We never want to end up recreating resources because gce api flaked. return true, false, "", fmt.Errorf("error getting load balancer's forwarding rule: %v", err) </ins> } <del> if loadBalancerIP != fwd.IPAddress { </del> <ins> // If the user asks for a specific static ip through the Service spec, // check that we're actually using it. // TODO: we report loadbalancer IP through status, so we want to verify if // that matches the forwarding rule as well. if loadBalancerIP != "" && loadBalancerIP != fwd.IPAddress { glog.Infof("LoadBalancer ip for forwarding rule %v was expected to be %v, but was actually %v", fwd.Name, fwd.IPAddress, loadBalancerIP) </ins> return true, true, fwd.IPAddress, nil } portRange, err := loadBalancerPortRange(ports) if err != nil { <del> return false, false, "", err </del> <ins> // Err on the side of caution in case of errors. Caller should notice the error and retry. // We never want to end up recreating resources because gce api flaked. return true, false, "", err </ins> } if portRange != fwd.PortRange { <ins> glog.Infof("LoadBalancer port range for forwarding rule %v was expected to be %v, but was actually %v", fwd.Name, fwd.PortRange, portRange) </ins> return true, true, fwd.IPAddress, nil } // The service controller verified all the protocols match on the ports, just check the first one if string(ports[0].Protocol) != fwd.IPProtocol { <ins> glog.Infof("LoadBalancer protocol for forwarding rule %v was expected to be %v, but was actually %v", fwd.Name, fwd.IPProtocol, string(ports[0].Protocol)) </ins> return true, true, fwd.IPAddress, nil }

if isHTTPErrorCode(err, http.StatusNotFound) { return false, true, nil } <del> return false, false, fmt.Errorf("error getting load balancer's target pool: %v", err) } if translateAffinityType(affinityType) != tp.SessionAffinity { </del> <ins> // Err on the side of caution in case of errors. Caller should notice the error and retry. // We never want to end up recreating resources because gce api flaked. return true, false, fmt.Errorf("error getting load balancer's target pool: %v", err) } // TODO: If the user modifies their Service's session affinity, it *should* // reflect in the associated target pool. However, currently not setting the // session affinity on a target pool defaults it to the empty string while // not setting in on a Service defaults it to None. There is a lack of // documentation around the default setting for the target pool, so if we // find it's the undocumented empty string, don't blindly recreate the // target pool (which results in downtime). Fix this when we have formally // defined the defaults on either side. if tp.SessionAffinity != "" && translateAffinityType(affinityType) != tp.SessionAffinity { glog.Infof("LoadBalancer target pool %v changed affinity from %v to %v", name, tp.SessionAffinity, affinityType) </ins> return true, true, nil } return true, false, nil