fix incorrect fractional tolerance definition