adjust precision for double calculation