Only start digging if pointer wasn't moved away.