DMRECON: clamp dot product to [-1, 1] for acos

Due to floating point precision, the dot product of two unit vectors
can be outside [-1, 1], which causes acos to return NaN. Clamp the
values to the domain of acos to avoid crashes in rare corner cases.