A method of decoding multiview video data, the method comprising: determining, for a first block of video data in a first layer of video data, a temporal motion vector and associated temporal reference block in a temporal reference picture for predicting the first block, wherein the temporal reference picture has a picture order count value, and wherein the first block and the temporal reference block are included in a first view; determining a disparity reference block in a disparity reference picture indicated by a disparity vector associated with the first block, wherein the disparity reference picture is included in an access unit that includes a picture containing the first block, and is included in a second view, the second view being different than the first view; determining whether a decoded picture buffer contains a temporal-disparity reference picture in the second view and having the same picture order count value of the temporal reference picture, wherein the temporal-disparity reference picture is located based on a combination of the temporal motion vector and the disparity vector; based on a determination that the decoded picture buffer does not contain the temporal-disparity reference picture in the second view and having the same picture order count value of the temporal reference picture, modifying an inter-view residual prediction process, which comprises generating a residual predictor for predicting residual data associated with the first block, wherein the residual predictor is generated based on differences between samples of the disparity reference block and corresponding samples of the temporal-disparity reference block, and wherein the residual data associated with the first block represents differences between samples of the first block and corresponding samples of the temporal reference block; and decoding the residual data of the first block of video data with the modified inter-view residual prediction process, wherein modifying the inter-view residual prediction process comprises scaling the temporal motion vector to indicate a temporally nearest picture to the disparity reference picture. determining, for a first block of video data in a first layer of video data, a temporal motion vector and associated temporal reference block in a temporal reference picture for predicting the first block, wherein the temporal reference picture has a picture order count value, and wherein the first block and the temporal reference block are included in a first view; determining a disparity reference block in a disparity reference picture indicated by a disparity vector associated with the first block, wherein the disparity reference picture is included in an access unit that includes a picture containing the first block, and is included in a second view, the second view being different than the first view; determining whether a decoded picture buffer contains a temporal-disparity reference picture in the second view and having the same picture order count value of the temporal reference picture, wherein the temporal-disparity reference picture is located based on a combination of the temporal motion vector and the disparity vector; based on a determination that the decoded picture buffer does not contain the temporal-disparity reference picture in the second view and having the same picture order count value of the temporal reference picture, modifying an inter-view residual prediction process, which comprises generating a residual predictor for predicting residual data associated with the first block, wherein the residual predictor is generated based on differences between samples of the disparity reference block and corresponding samples of the temporal-disparity reference block, and wherein the residual data associated with the first block represents differences between samples of the first block and corresponding samples of the temporal reference block; and decoding the residual data of the first block of video data with the modified inter-view residual prediction process, wherein modifying the inter-view residual prediction process comprises scaling the temporal motion vector to indicate a temporally nearest picture to the disparity reference picture.