[SPARK-3106] Fix the race condition issue about Connection and ConnectionManager

The reason why Checking ServerSocketChannel is same for above.
About key.synchronized is also.