Thinking more about it, I think your SSH instance/process is still running even though the daemon is restarted for new connections. If the sshd config is wrong though it can cause SSH to not come back up, but that can be tested with `sshd -t`.

Apparently [restarting the network service can even last](http://unix.stackexchange.com/questions/27636/how-does-ssh-connection-survive-a-network-restart), because the restart takes less time than it takes TCP to timeout.