Change:
} } <ins> func updateServiceOrFail(clientset *fedclientset.Clientset, namespace, name string) *v1.Service { service, err := clientset.Services(namespace).Get(name, metav1.GetOptions{}) framework.ExpectNoError(err, "Getting service %q in namespace %q", name, namespace) service.Spec.Selector["name"] = "update-demo" newService, err := clientset.Services(namespace).Update(service) By(fmt.Sprintf("Successfully updated federated service %q in namespace %q", name, namespace)) return newService } func deleteServiceShard(c *fedframework.Cluster, namespace, service string) error { err := c.Clientset.Services(namespace).Delete(service, &metav1.DeleteOptions{}) if err != nil && !errors.IsNotFound(err) { framework.Logf("Failed to delete service %q in namespace %q, in cluster %q", service, namespace, c.Name) return err } By(fmt.Sprintf("Service %q in namespace %q in cluster %q deleted", service, namespace, c.Name)) return nil } </ins> // equivalent returns true if the two services are equivalent.  Fields which are expected to differ between <del> // federated services and the underlying cluster services (e.g. ClusterIP, LoadBalancerIP etc) are ignored. </del> <ins> // federated services and the underlying cluster services (e.g. ClusterIP, NodePort) are ignored. </ins> func equivalent(federationService, clusterService v1.Service) bool { <del> // TODO: I think that we need a DeepCopy here to avoid clobbering our parameters. </del> clusterService.Spec.ClusterIP = federationService.Spec.ClusterIP <del> clusterService.Spec.ExternalIPs = federationService.Spec.ExternalIPs clusterService.Spec.LoadBalancerIP = federationService.Spec.LoadBalancerIP clusterService.Spec.LoadBalancerSourceRanges = federationService.Spec.LoadBalancerSourceRanges // N.B. We cannot iterate over the port objects directly, as their values // only get copied and our updates will get lost. </del> for i := range clusterService.Spec.Ports { clusterService.Spec.Ports[i].NodePort = federationService.Spec.Ports[i].NodePort } <del> return reflect.DeepEqual(clusterService.Spec, federationService.Spec) </del> <ins> if federationService.Name != clusterService.Name || federationService.Namespace != clusterService.Namespace { return false } if !reflect.DeepEqual(federationService.Labels, clusterService.Labels) && (len(federationService.Labels) != 0 || len(clusterService.Labels) != 0) { return false } if !reflect.DeepEqual(federationService.Spec, clusterService.Spec) { return false } return true </ins> }

Discussion:
As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.