Fix a possible lockup

Make us not clock to false positive IRQ events, as there is a case we
might get stuck spinning with 0 clocks to add to core, waiting for an
irq that never happens.