[SPARK-3106] Fix the race condition issue about Connection and ConnectionManager

Sure, but again, all that is doing is avoiding an exception. An exception that is handled properly by the code.

So, what is better:
- synchronize on every key to avoid throwing an exception
- don't synchronize and let exceptions be handled

Given that this is a hot loop, and during normal operation keys are not being cancelled left and right, I think the second approach is better. So I'd just turn the log messages into `logDebug`, since they're expected.