Fix a bug that server crushes when connection refused.