Weird c++ math problems