Fix a bug that was causing rejected connections to linger too long