fix edge case of numeric inaccuracy