Method for encoding video data having a base layer and an enhancement layer, wherein the base layer has lower color resolution than the enhancement layer, the method comprising the steps of - transforming and quantizing base layer data; - inverse transforming (T -1) and inverse quantizing (Q -1) the transformed and quantized base layer data, wherein reconstructed base layer data (BL rec) are obtained; - upsampling (BDUp) the reconstructed base layer data, wherein the upsampling refers at least to bit depth and wherein a predicted version of enhancement layer data (Pre c (BL rec)) is obtained; - generating a residual (EL res) between original enhancement layer data and the predicted version of enhancement layer data; - selecting for the case of an inter coded enhancement layer macroblock between at least two different inter coding modes and for the case of an intra coded enhancement layer macroblock between at least two different intra coding modes, wherein a first inter coding mode comprises using upsampled base layer motion information (MV BLUp) and a second inter coding mode comprises using motion information (MV EL) generated from said enhancement layer residual, and wherein a first intra coding mode comprises coding said residual directly and a second intra coding mode comprises an additional step of spatial prediction of said residual. - encoding (T,Q) the transformed and quantized base layer data; - encoding (T,Q) said enhancement layer residual (EL res) using the selected enhancement layer encoding mode and encoding an indication indicating said encoding mode, wherein the encoding comprises a transform and a quantization step; and - performing entropy coding on the encoded enhancement layer residual (EC EL) and said indication. - transforming and quantizing base layer data; - inverse transforming (T -1) and inverse quantizing (Q -1) the transformed and quantized base layer data, wherein reconstructed base layer data (BL rec) are obtained; - upsampling (BDUp) the reconstructed base layer data, wherein the upsampling refers at least to bit depth and wherein a predicted version of enhancement layer data (Pre c (BL rec)) is obtained; - generating a residual (EL res) between original enhancement layer data and the predicted version of enhancement layer data; - selecting for the case of an inter coded enhancement layer macroblock between at least two different inter coding modes and for the case of an intra coded enhancement layer macroblock between at least two different intra coding modes, wherein a first inter coding mode comprises using upsampled base layer motion information (MV BLUp) and a second inter coding mode comprises using motion information (MV EL) generated from said enhancement layer residual, and wherein a first intra coding mode comprises coding said residual directly and a second intra coding mode comprises an additional step of spatial prediction of said residual. - encoding (T,Q) the transformed and quantized base layer data; - encoding (T,Q) said enhancement layer residual (EL res) using the selected enhancement layer encoding mode and encoding an indication indicating said encoding mode, wherein the encoding comprises a transform and a quantization step; and - performing entropy coding on the encoded enhancement layer residual (EC EL) and said indication.