Remove fmt! usage from the compiler

This also prevents future fmt! usage from leaking into the compiler, but it's still turned on by default for everyone else.