Change:
verifyCascadingDeletionForService(f.FederationClientset, clusters, nil, nsName) By(fmt.Sprintf("Verified that services were not deleted from underlying clusters")) }) <ins> It("should recreate service shard in underlying clusters when service shard is deleted", func() { fedframework.SkipUnlessFederated(f.ClientSet) service = createServiceOrFail(f.FederationClientset, nsName, FederatedServiceName) defer func() { // Cleanup By(fmt.Sprintf("Deleting service %q in namespace %q", service.Name, nsName)) err := f.FederationClientset.Services(nsName).Delete(service.Name, &metav1.DeleteOptions{}) framework.ExpectNoError(err, "Error deleting service %q in namespace %q", service.Name, nsName) }() By(fmt.Sprintf("Wait for service shards to be created in all clusters for service "%s/%s"", nsName, service.Name)) waitForServiceShardsOrFail(nsName, service, clusters) framework.Logf("Successfully created and synced service "%s/%s" to all clusters", nsName, service.Name) By(fmt.Sprintf("Deleting a service shard in one underlying cluster")) primaryClusterName := clusters[0].Name err := deleteServiceShard(clusters[0], nsName, FederatedServiceName) framework.ExpectNoError(err, fmt.Sprintf("while deleting service shard %q in cluster %q", FederatedServiceName, primaryClusterName)) waitForServiceShardsOrFail(nsName, service, clusters) framework.Logf("Successfully recreated service shard "%s/%s" in %q cluster", nsName, service.Name, primaryClusterName) }) </ins> }) var _ = Describe("DNS", func() {

Discussion:
As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.