A method of decoding video data, comprising: performing entropy decoding, inverse quantization process and inverse transform process on a bitstream, to obtain a residual block; determining a luma block that corresponds to a chroma block; determining a set of down-sampled samples of reconstructed neighboring luma samples, wherein the reconstructed neighboring luma samples comprise a plurality of reconstructed luma samples that are above the luma block; calculating a first pair of a luma value and a chroma value using N down-sampled neighboring luma samples of the set and N reconstructed neighboring chroma samples that correspond to the N down-sampled neighboring luma samples, wherein N is a positive integer larger than 1, and a minimum value of the N down-sampled neighboring luma samples is not less than respective luma values of first remaining down-sampled neighboring luma samples of the set, and wherein a number of the first remaining down-sampled neighboring luma samples of the set is equal to a total number of down-sampled neighboring luma samples in the set minus N; calculating a second pair of a luma value and a chroma value using M down-sampled neighboring luma samples of the set and M reconstructed neighboring chroma samples that correspond to the M down-sampled neighboring luma samples, wherein M is a positive integer larger than 1, and a maximum value of the M down-sampled neighboring luma samples is not larger than respective luma values of second remaining down-sampled neighboring luma samples of the set, and wherein a number of the second remaining down-sampled neighboring luma samples of the set is equal to the total number of down-sampled neighboring luma samples in the set minus M; determining one or more linear model parameters based on the first and second pairs of luma value and chroma value; determining a predictive block based on the one or more linear model parameters; and reconstructing the chroma block by adding the predictive block to the residual block. performing entropy decoding, inverse quantization process and inverse transform process on a bitstream, to obtain a residual block; determining a luma block that corresponds to a chroma block; determining a set of down-sampled samples of reconstructed neighboring luma samples, wherein the reconstructed neighboring luma samples comprise a plurality of reconstructed luma samples that are above the luma block; calculating a first pair of a luma value and a chroma value using N down-sampled neighboring luma samples of the set and N reconstructed neighboring chroma samples that correspond to the N down-sampled neighboring luma samples, wherein N is a positive integer larger than 1, and a minimum value of the N down-sampled neighboring luma samples is not less than respective luma values of first remaining down-sampled neighboring luma samples of the set, and wherein a number of the first remaining down-sampled neighboring luma samples of the set is equal to a total number of down-sampled neighboring luma samples in the set minus N; calculating a second pair of a luma value and a chroma value using M down-sampled neighboring luma samples of the set and M reconstructed neighboring chroma samples that correspond to the M down-sampled neighboring luma samples, wherein M is a positive integer larger than 1, and a maximum value of the M down-sampled neighboring luma samples is not larger than respective luma values of second remaining down-sampled neighboring luma samples of the set, and wherein a number of the second remaining down-sampled neighboring luma samples of the set is equal to the total number of down-sampled neighboring luma samples in the set minus M; determining one or more linear model parameters based on the first and second pairs of luma value and chroma value; determining a predictive block based on the one or more linear model parameters; and reconstructing the chroma block by adding the predictive block to the residual block.