Do not rattempt to remove from load balancer if there are no load balancers