A method of decoding video data, comprising: determining a luma block that corresponds to a chroma block; determining a set of down-sampled samples of reconstructed neighboring luma samples, wherein the reconstructed neighboring luma samples comprise a plurality of reconstructed luma samples that are above the luma block and/or a plurality of reconstructed luma samples that are left to the luma block; calculating a first pair of luma value and chroma value using N down-sampled neighboring luma samples of the set and N reconstructed neighboring chroma samples that correspond to the N down-sampled neighboring luma samples, wherein N is a positive integer larger than 1, and a minimum value of the N down-sampled neighboring luma samples is not less than respective luma values of the remaining down-sampled neighboring luma samples of the set, which are different from said N down-sampled neighboring luma samples; calculating a second pair of luma value and chroma value using M down-sampled neighboring luma samples of the set and M reconstructed neighboring chroma samples that correspond to the M down-sampled neighboring luma samples, wherein M is a positive integer larger than 1, and a maximum value of the M down-sampled neighboring luma samples is not larger than respective luma values of the remaining down-sampled neighboring luma samples of the set, which are different from said M down-sampled neighboring luma samples; determining one or more linear model parameters based on the first and second pairs of luma value and chroma value; determining a predictive block based on the one or more linear model parameters; and reconstructing the chroma block based on the predictive block. determining a luma block that corresponds to a chroma block; determining a set of down-sampled samples of reconstructed neighboring luma samples, wherein the reconstructed neighboring luma samples comprise a plurality of reconstructed luma samples that are above the luma block and/or a plurality of reconstructed luma samples that are left to the luma block; calculating a first pair of luma value and chroma value using N down-sampled neighboring luma samples of the set and N reconstructed neighboring chroma samples that correspond to the N down-sampled neighboring luma samples, wherein N is a positive integer larger than 1, and a minimum value of the N down-sampled neighboring luma samples is not less than respective luma values of the remaining down-sampled neighboring luma samples of the set, which are different from said N down-sampled neighboring luma samples; calculating a second pair of luma value and chroma value using M down-sampled neighboring luma samples of the set and M reconstructed neighboring chroma samples that correspond to the M down-sampled neighboring luma samples, wherein M is a positive integer larger than 1, and a maximum value of the M down-sampled neighboring luma samples is not larger than respective luma values of the remaining down-sampled neighboring luma samples of the set, which are different from said M down-sampled neighboring luma samples; determining one or more linear model parameters based on the first and second pairs of luma value and chroma value; determining a predictive block based on the one or more linear model parameters; and reconstructing the chroma block based on the predictive block.