clamp vector dot product to -1..1