[SPARK-3106] Fix the race condition issue about Connection and ConnectionManager

I separate !key.isValid from the rest intentionally.
When connection is null or connection is closed, !key.isValid is true because connection is closed by another thread calling Connection#close and it's not the case we should log.

If key.isValid is false, even if connection is not closed, it's abnormal case.

Anyway, we (including other committer/contributor) should discuss how we treat this issue.