calibrate for numerical precision diff between manual calc and clojure calc