account for precision error in floats