Keep track of time since target loss, drop to zero after several samples