Use byte literals instead of casts

This replaces many instances chars being casted to u8 with byte literals.
This cast should be removable as well.