Remove untouched timers after reschedule (we're no longer interested in those)