Nest default formats and properly merge specified formats

This nests the default formats, and will properly merge the custom `formats` provided via the constructor.