optimize the sync worker

This change optimize the sync worker when we only have to listen on one
interface. While I'm here, I fixed a long and unnoticed outstanding issue when
we were accepting on multiple interfaces (wonder if someone really use it), at
some point soe interfaces were skipped.