Use the 4 byte representation for colors when possible

Much of our data starts in this format, and it doesn't make sense to convert it to 4 floats at load time then back to bytes at run time.