I think we should hold it until the time when we need it.
3-byte color mode has a bit performance problem at some implementations because it searches closest color entry from palette, but it may useful to solve color scheme compatibility issue between vim and gvim.
I'll research it.