Do not attempt to remove from load balancer if there are no load balancers