Only stop drifting on anchored lattices

Requested by hg because of the singulo unachoring these things.