Prevent "Already connecting..." from spamming by tracking destination server