I realized I was using luminance for color segmentation, instead of chrominance.  I switched to chrominance and changed it so that it uses both Cr and Cb.