BPMemory: Cleanup GenMode and fix incorrect bitfield size.

numcolchans only spans 3 bits, not 5. Cf. e.g. http://repo.or.cz/w/libogc.git/blob/HEAD:/libogc/gx.c#l3915.