Fix potential rance condition in connecting to vanilla servers as well. And move latch into client side only.