Reduce the footprint of swap.c by relying more heavily on SDL_endian.h