A method of processing video data, the method comprising: determining first motion information for a prediction unit of a coding unit of a video block from neighboring blocks of the PU according to a backward view synthesis prediction (BVSP) mode, wherein determining the first motion information for the PU comprises selecting the first motion information as a BVSP candidate from a merge mode candidate list of motion information for predicting the PU, and wherein the first motion information includes at least one disparity motion vector and an associated reference view index that identifies an inter-view reference picture; partitioning the PU into two or more sub-PUs; determining second motion information for each of the sub-PUs, wherein the second motion information includes at least one disparity motion vector associated with a depth block of the inter-view reference picture corresponding to each of the sub-PUs; performing motion compensation to predict each of the sub-PUs from the inter-view reference picture based on the second motion information; storing the second motion information for each of the sub-PUs of the PU in a memory to be used for predicting subsequent PUs; and predicting a subsequent PU based on the stored second motion information for at least one of the sub-PUs of the PU, wherein the subsequent PU is a neighboring block of the PU, and wherein predicting the subsequent PU comprises generating a merge mode candidate list of motion information for predicting the subsequent PU, the merge mode candidate list including the second motion information for at least one of the sub-PUs of the PU stored in the memory as a sub-PU BVSP candidate in the merge mode candidate list. determining first motion information for a prediction unit of a coding unit of a video block from neighboring blocks of the PU according to a backward view synthesis prediction (BVSP) mode, wherein determining the first motion information for the PU comprises selecting the first motion information as a BVSP candidate from a merge mode candidate list of motion information for predicting the PU, and wherein the first motion information includes at least one disparity motion vector and an associated reference view index that identifies an inter-view reference picture; partitioning the PU into two or more sub-PUs; determining second motion information for each of the sub-PUs, wherein the second motion information includes at least one disparity motion vector associated with a depth block of the inter-view reference picture corresponding to each of the sub-PUs; performing motion compensation to predict each of the sub-PUs from the inter-view reference picture based on the second motion information; storing the second motion information for each of the sub-PUs of the PU in a memory to be used for predicting subsequent PUs; and predicting a subsequent PU based on the stored second motion information for at least one of the sub-PUs of the PU, wherein the subsequent PU is a neighboring block of the PU, and wherein predicting the subsequent PU comprises generating a merge mode candidate list of motion information for predicting the subsequent PU, the merge mode candidate list including the second motion information for at least one of the sub-PUs of the PU stored in the memory as a sub-PU BVSP candidate in the merge mode candidate list.