Fix rounding errors in rotate tests