Fix possible race condition with regards to abandoning servers