[SPARK-3106] Fix the race condition issue about Connection and ConnectionManager

Same weird check for ServerSocketChannel.

Also same issue with `key.synchronized` not really protecting against the underlying connection object being closed.