ENH: Ignore labels that are not in the color table