A system, comprising: a video encoder, which, in operation, encodes and organizes a plurality of picture frames of a video stream at a plurality of temporal layers in a hierarchical P-structure; records one or more encoded reference frames of the video stream in a display picture buffer associated with the video encoder, wherein each of the reference frames has been encoded by the video encoder and can be used by a video decoder as a restarting point for continued decoding of the video stream when one or more of the picture frames is lost; transmits the plurality of picture frames of the video stream over a network; picks one of the recorded reference frames that has been encoded earlier in time than the one or more lost frames when the video decoder provides a negative feedback on the one or more lost frames; and indicates the picked reference frame to the video decoder; and said video decoder, which, in operation, accepts the video stream transmitted over the network, wherein the video stream has said encoded picture frames; provides said negative feedback on the one or more lost frames to the video encoder through a back channel mechanism to trigger selection of the reference frames; and recovers from the one or more lost frames of the plurality of picture frames during decoding of the video stream using a combination of 1) decoding the picture frames of a lower temporal layer in the hierarchical P-structure than a temporal layer of the one or more lost frames and 2) using the picked reference frame as the restarting point for continued decoding of the video stream. a video encoder, which, in operation, encodes and organizes a plurality of picture frames of a video stream at a plurality of temporal layers in a hierarchical P-structure; records one or more encoded reference frames of the video stream in a display picture buffer associated with the video encoder, wherein each of the reference frames has been encoded by the video encoder and can be used by a video decoder as a restarting point for continued decoding of the video stream when one or more of the picture frames is lost; transmits the plurality of picture frames of the video stream over a network; picks one of the recorded reference frames that has been encoded earlier in time than the one or more lost frames when the video decoder provides a negative feedback on the one or more lost frames; and indicates the picked reference frame to the video decoder; and encodes and organizes a plurality of picture frames of a video stream at a plurality of temporal layers in a hierarchical P-structure; records one or more encoded reference frames of the video stream in a display picture buffer associated with the video encoder, wherein each of the reference frames has been encoded by the video encoder and can be used by a video decoder as a restarting point for continued decoding of the video stream when one or more of the picture frames is lost; transmits the plurality of picture frames of the video stream over a network; picks one of the recorded reference frames that has been encoded earlier in time than the one or more lost frames when the video decoder provides a negative feedback on the one or more lost frames; and indicates the picked reference frame to the video decoder; and said video decoder, which, in operation, accepts the video stream transmitted over the network, wherein the video stream has said encoded picture frames; provides said negative feedback on the one or more lost frames to the video encoder through a back channel mechanism to trigger selection of the reference frames; and recovers from the one or more lost frames of the plurality of picture frames during decoding of the video stream using a combination of 1) decoding the picture frames of a lower temporal layer in the hierarchical P-structure than a temporal layer of the one or more lost frames and 2) using the picked reference frame as the restarting point for continued decoding of the video stream. accepts the video stream transmitted over the network, wherein the video stream has said encoded picture frames; provides said negative feedback on the one or more lost frames to the video encoder through a back channel mechanism to trigger selection of the reference frames; and recovers from the one or more lost frames of the plurality of picture frames during decoding of the video stream using a combination of 1) decoding the picture frames of a lower temporal layer in the hierarchical P-structure than a temporal layer of the one or more lost frames and 2) using the picked reference frame as the restarting point for continued decoding of the video stream.