[SPARK-3106] Fix the race condition issue about Connection and ConnectionManager

Merge with previous condition? Also, check looks weird (why check for ServerSocketChannel?).