GCC -O3 -flto wasn't doing copy elision

I expected it to get rid of the strings that were dumped directly into
appends.
This makes serializing 15-20x faster.
More code improvements

Dropping the map in favor of the templated if/else seems to have no
effect on performance but drops the binary size by ~10%