Fix panic when deleting inactive server, fixes #76