[SPARK-3106] Fix the race condition issue about Connection and ConnectionManager

It is normal. In a distributed system, hosts will go down and you have to handle that.

Now, about logging that it happened. It's useful to log that a host went down. But is this the place to do it? These exceptions are too generic for that - CancelledKeyExceptions can happen for other reasons too. So the code that is notified that a host went down and didn't expect that to happen should log it instead.