A method of decoding a current block of video data, the method comprising: deriving M most probable modes, MPMs, for intra prediction of the current block of video data from among a plurality of intra prediction modes, wherein M is greater than 3, the MPMs including an MPM for a left neighboring column and an MPM for an above neighboring row, and deriving the M most probable modes comprises at least one of: (i) counting, in units of a smallest prediction unit size, intra prediction modes used in a plurality of neighboring blocks in the left neighboring column; defining a representative intra prediction mode for the left neighboring column as a most frequent of the intra prediction modes used in the plurality of neighboring blocks in the left neighboring column; and using the defined representative intra prediction mode for the left neighboring column as the MPM for the left neighboring column, or (ii) counting, in units of the smallest prediction unit size, intra prediction modes used in a plurality of neighboring blocks in the above neighboring row; defining a representative intra prediction mode for the above neighboring row as a most frequent of the intra prediction modes used in the plurality of neighboring blocks in the above neighboring row; and using the defined representative intra prediction mode for the above neighboring row as the MPM for the above neighboring row; decoding a syntax element that indicates whether a MPM index or a non-MPM index is used to indicate a selected intra prediction mode of the plurality of intra prediction modes for intra prediction of the current block of video data; decoding the indicated one of the most probable mode index or the non-most probable mode index, wherein the most probable mode index indicates which of the M MPMs is the selected intra prediction mode, and wherein the non-MPM index indicates which of the plurality of intra prediction modes other than the M MPMs is the selected intra prediction mode; and reconstructing the current block of video data based on the selected intra prediction mode. deriving M most probable modes, MPMs, for intra prediction of the current block of video data from among a plurality of intra prediction modes, wherein M is greater than 3, the MPMs including an MPM for a left neighboring column and an MPM for an above neighboring row, and deriving the M most probable modes comprises at least one of: (i) counting, in units of a smallest prediction unit size, intra prediction modes used in a plurality of neighboring blocks in the left neighboring column; defining a representative intra prediction mode for the left neighboring column as a most frequent of the intra prediction modes used in the plurality of neighboring blocks in the left neighboring column; and using the defined representative intra prediction mode for the left neighboring column as the MPM for the left neighboring column, or (ii) counting, in units of the smallest prediction unit size, intra prediction modes used in a plurality of neighboring blocks in the above neighboring row; defining a representative intra prediction mode for the above neighboring row as a most frequent of the intra prediction modes used in the plurality of neighboring blocks in the above neighboring row; and using the defined representative intra prediction mode for the above neighboring row as the MPM for the above neighboring row; (i) counting, in units of a smallest prediction unit size, intra prediction modes used in a plurality of neighboring blocks in the left neighboring column; defining a representative intra prediction mode for the left neighboring column as a most frequent of the intra prediction modes used in the plurality of neighboring blocks in the left neighboring column; and using the defined representative intra prediction mode for the left neighboring column as the MPM for the left neighboring column, or (ii) counting, in units of the smallest prediction unit size, intra prediction modes used in a plurality of neighboring blocks in the above neighboring row; defining a representative intra prediction mode for the above neighboring row as a most frequent of the intra prediction modes used in the plurality of neighboring blocks in the above neighboring row; and using the defined representative intra prediction mode for the above neighboring row as the MPM for the above neighboring row; decoding a syntax element that indicates whether a MPM index or a non-MPM index is used to indicate a selected intra prediction mode of the plurality of intra prediction modes for intra prediction of the current block of video data; decoding the indicated one of the most probable mode index or the non-most probable mode index, wherein the most probable mode index indicates which of the M MPMs is the selected intra prediction mode, and wherein the non-MPM index indicates which of the plurality of intra prediction modes other than the M MPMs is the selected intra prediction mode; and reconstructing the current block of video data based on the selected intra prediction mode.