Integrate the new rounding modes with BigFloats

This changes the old rounding mode sketch to a more intuitive one,
and using types instead of constants seems to be more logical.