A method of decoding encoded video information with a video decoder, the method comprising: receiving, at the video decoder, encoded video information from a bitstream; and with the video decoder, decoding an interlaced bi-directionally predicted field using the encoded video information, wherein the interlaced bi-directionaliy predicted field includes plural macroblocks, and wherein the decoding includes, for a current macroblock of the plural macroblocks: calculating a motion vector for each of four blocks in the current macroblock in the interlaced bi-directionally predicted field; and using one or more of the calculated motion vectors in motion compensation for the current macroblock in the interlaced bi-directionally predicted field;wherein motion-compensated prediction modes for the plural macroblocks include forward prediction mode and plural non-forward prediction modes, the plural non-forward prediction modes including backward prediction mode, direct prediction mode and interpolated prediction mode, wherein use of four motion vectors per macroblock is restricted to the forward prediction mode and the backward prediction mode, and wherein the restricted use of four motion vectors per macroblock reduces signaling overhead associated with signaling the backward prediction mode by conditioning use of a non-forward prediction mode selection syntax element in the bitstream on whether motion-compensated prediction mode of the current macroblock is signaled as not the forward prediction mode and on whether the number of motion vectors is signaled as not four for the current macroblock, the non-forward prediction mode selection syntax element, when used, specifying one of the plural non-forward prediction modes for the current macroblock. receiving, at the video decoder, encoded video information from a bitstream; and with the video decoder, decoding an interlaced bi-directionally predicted field using the encoded video information, wherein the interlaced bi-directionaliy predicted field includes plural macroblocks, and wherein the decoding includes, for a current macroblock of the plural macroblocks: calculating a motion vector for each of four blocks in the current macroblock in the interlaced bi-directionally predicted field; and using one or more of the calculated motion vectors in motion compensation for the current macroblock in the interlaced bi-directionally predicted field;wherein motion-compensated prediction modes for the plural macroblocks include forward prediction mode and plural non-forward prediction modes, the plural non-forward prediction modes including backward prediction mode, direct prediction mode and interpolated prediction mode, wherein use of four motion vectors per macroblock is restricted to the forward prediction mode and the backward prediction mode, and wherein the restricted use of four motion vectors per macroblock reduces signaling overhead associated with signaling the backward prediction mode by conditioning use of a non-forward prediction mode selection syntax element in the bitstream on whether motion-compensated prediction mode of the current macroblock is signaled as not the forward prediction mode and on whether the number of motion vectors is signaled as not four for the current macroblock, the non-forward prediction mode selection syntax element, when used, specifying one of the plural non-forward prediction modes for the current macroblock. calculating a motion vector for each of four blocks in the current macroblock in the interlaced bi-directionally predicted field; and using one or more of the calculated motion vectors in motion compensation for the current macroblock in the interlaced bi-directionally predicted field;wherein motion-compensated prediction modes for the plural macroblocks include forward prediction mode and plural non-forward prediction modes, the plural non-forward prediction modes including backward prediction mode, direct prediction mode and interpolated prediction mode, wherein use of four motion vectors per macroblock is restricted to the forward prediction mode and the backward prediction mode, and wherein the restricted use of four motion vectors per macroblock reduces signaling overhead associated with signaling the backward prediction mode by conditioning use of a non-forward prediction mode selection syntax element in the bitstream on whether motion-compensated prediction mode of the current macroblock is signaled as not the forward prediction mode and on whether the number of motion vectors is signaled as not four for the current macroblock, the non-forward prediction mode selection syntax element, when used, specifying one of the plural non-forward prediction modes for the current macroblock.