A method of processing inter-layer residual video representing differences between an input video and a reconstructed version of a base layer video, wherein the inter-layer residual video is encoded as an enhancement layer, the method comprising: encoding inter-layer residual video to produce at least part of a bit stream of encoded data for the inter-layer residual video, wherein the encoding includes processing plural macroblocks of a current coded picture of the inter-layer residual video,; outputting the at least part of the bit stream; characterized in that the processing includes, for a current macroblock of the plural macroblocks that is skipped: selecting a skip mode for the current macroblock that is skipped, wherein if a top neighbor macroblock and a left neighbor of the current macroblock are both motion compensated, the selected skip mode of the current macroblock uses motion compensation, and otherwise, the skip mode of the current macroblock does not use motion compensation; using the skip mode for the current macroblock that is skipped, wherein: if the selected skip mode of the current macroblock does not use motion compensation, using the skip mode for the current macroblock includes setting values of the current macroblock to a defined value; and if the selected skip mode of the current macroblock uses motion compensation, using the skip mode for the current macroblock includes predicting the motion of the current macroblock by computing a component-wise median of motion vectors used by the top neighbor macroblock and the left neighbor macroblock; and wherein the bit stream of encoded data for the inter-layer residual video includes macroblock skip information indicating which of the plural macroblocks are skipped. encoding inter-layer residual video to produce at least part of a bit stream of encoded data for the inter-layer residual video, wherein the encoding includes processing plural macroblocks of a current coded picture of the inter-layer residual video,; outputting the at least part of the bit stream; characterized in that the processing includes, for a current macroblock of the plural macroblocks that is skipped: selecting a skip mode for the current macroblock that is skipped, wherein if a top neighbor macroblock and a left neighbor of the current macroblock are both motion compensated, the selected skip mode of the current macroblock uses motion compensation, and otherwise, the skip mode of the current macroblock does not use motion compensation; using the skip mode for the current macroblock that is skipped, wherein: if the selected skip mode of the current macroblock does not use motion compensation, using the skip mode for the current macroblock includes setting values of the current macroblock to a defined value; and if the selected skip mode of the current macroblock uses motion compensation, using the skip mode for the current macroblock includes predicting the motion of the current macroblock by computing a component-wise median of motion vectors used by the top neighbor macroblock and the left neighbor macroblock; and selecting a skip mode for the current macroblock that is skipped, wherein if a top neighbor macroblock and a left neighbor of the current macroblock are both motion compensated, the selected skip mode of the current macroblock uses motion compensation, and otherwise, the skip mode of the current macroblock does not use motion compensation; using the skip mode for the current macroblock that is skipped, wherein: if the selected skip mode of the current macroblock does not use motion compensation, using the skip mode for the current macroblock includes setting values of the current macroblock to a defined value; and if the selected skip mode of the current macroblock uses motion compensation, using the skip mode for the current macroblock includes predicting the motion of the current macroblock by computing a component-wise median of motion vectors used by the top neighbor macroblock and the left neighbor macroblock; and if the selected skip mode of the current macroblock does not use motion compensation, using the skip mode for the current macroblock includes setting values of the current macroblock to a defined value; and if the selected skip mode of the current macroblock uses motion compensation, using the skip mode for the current macroblock includes predicting the motion of the current macroblock by computing a component-wise median of motion vectors used by the top neighbor macroblock and the left neighbor macroblock; and wherein the bit stream of encoded data for the inter-layer residual video includes macroblock skip information indicating which of the plural macroblocks are skipped.