A method of decoding video data, the method comprising: receiving an encoded block of luma samples of a first block of video data; receiving encoded blocks of luma samples and chroma samples of neighboring blocks of video data; decoding the encoded first block of luma samples to create reconstructed luma samples of the first block; decoding the encoded neighboring blocks of luma samples to create reconstructed luma samples of neighboring blocks and decoding the encoded neighboring blocks of chroma samples to create reconstructed chroma samples of neighboring blocks; selecting a set of reconstructed luma samples of the neighboring blocks to use in predicting the chroma samples of the first block; classifying the reconstructed luma samples of the set that are greater than a first threshold as being in a first sample group of a plurality of sample groups of neighboring samples; classifying the reconstructed luma samples of the set that are less than or equal to the first threshold as being in a second sample group of the plurality of sample groups of neighboring samples; determining a first linear prediction model based on the reconstructed luma samples of the first sample group of neighboring samples and the corresponding reconstructed neighboring chroma samples; determining a second linear prediction model based on the reconstructed luma samples of the second sample group of neighboring samples and the corresponding reconstructed neighboring chroma samples; classifying the reconstructed luma samples of the first block that are greater than the first threshold as being in a first sample group of the first block; classifying the reconstructed luma samples of the first block that are less than or equal to the first threshold as being in a second sample group of the first block; determining whether the number of samples classified into a sample group is less than a predefined minimum number and changing at least one sample between sample groups when the number of samples classified into the group is below the predefined minimum number; applying the first linear prediction model to the reconstructed luma samples in the first sample group of the first block; applying the second linear prediction model to the reconstructed luma samples in the second sample group of the first block; and determining predicted chroma samples in the first block of video data based on the applied first linear prediction model and the applied second linear prediction model. receiving an encoded block of luma samples of a first block of video data; receiving encoded blocks of luma samples and chroma samples of neighboring blocks of video data; decoding the encoded first block of luma samples to create reconstructed luma samples of the first block; decoding the encoded neighboring blocks of luma samples to create reconstructed luma samples of neighboring blocks and decoding the encoded neighboring blocks of chroma samples to create reconstructed chroma samples of neighboring blocks; selecting a set of reconstructed luma samples of the neighboring blocks to use in predicting the chroma samples of the first block; classifying the reconstructed luma samples of the set that are greater than a first threshold as being in a first sample group of a plurality of sample groups of neighboring samples; classifying the reconstructed luma samples of the set that are less than or equal to the first threshold as being in a second sample group of the plurality of sample groups of neighboring samples; determining a first linear prediction model based on the reconstructed luma samples of the first sample group of neighboring samples and the corresponding reconstructed neighboring chroma samples; determining a second linear prediction model based on the reconstructed luma samples of the second sample group of neighboring samples and the corresponding reconstructed neighboring chroma samples; classifying the reconstructed luma samples of the first block that are greater than the first threshold as being in a first sample group of the first block; classifying the reconstructed luma samples of the first block that are less than or equal to the first threshold as being in a second sample group of the first block; determining whether the number of samples classified into a sample group is less than a predefined minimum number and changing at least one sample between sample groups when the number of samples classified into the group is below the predefined minimum number; applying the first linear prediction model to the reconstructed luma samples in the first sample group of the first block; applying the second linear prediction model to the reconstructed luma samples in the second sample group of the first block; and determining predicted chroma samples in the first block of video data based on the applied first linear prediction model and the applied second linear prediction model.