Remove assertion again

When doing a double tap, the timeout id might get overridden.