Set events for poll explicitly instead of ORing.

If we are going to use pollfds as a state machine we better not allow
it to get into an inconsistent state.