That race condition is quite funny when your running low on memory or CPU.  

sleep 1 gets in; then ssh closes and then  you can log in actually.