server: fix possible join back in TestRemove