Don't delay after last action
Only block user events when the ghost mouse fires them