I forgot that by squaring the velocity we'd lose the sign of distanceToTravel.  Imagine slowly panning more than half way but then flicking back quickly in the other direction.  In this case we'd like deltaX and distanceToTravel to have opposite signs.

So I'd suggest that when computing distanceToTravel you do 0.5 * Math.abs(velocity) * velocity / swipeFriction.

And then, use Math.abs(deltaX + distanceToTravel)

Does that make sense?