A video apparatus for multi-layered frame-compatible video delivery, comprising at least one processor configured for: a) base layer processing a base layer bitstream signal through a base layer, wherein the base layer bitstream signal encodes images or video frames of multiple views in a frame-compatible representation, the base layer processing comprising: storing past decoded frame-compatible reference pictures for the base layer; and decoding the base layer bitstream signal to create a frame-compatible base layer decode image or video frame by performing disparity compensation at frame-compatible level using the past decoded frame-compatible reference pictures for the base layer; b) enhancement layer processing a plurality of enhancement layer bitstream signals through one or more enhancement layers for multiple views through one or more enhancement layers, wherein each of the multiple views is processed separately in a separate enhancement layer, wherein an enhancement layer bitstream signal encodes images or video frames of a view in an enhancement layer representation having a higher spatial resolution than the same view encoded in the base layer frame-compatible representation, the enhancement layer processing for each of the one or more enhancement layers comprising: providing decoded reference pictures for the respective enhancement layer to retrieve decoded reference pictures for the view; de-multiplexing past decoded frame-compatible reference pictures for the base layer to retrieve decoded reference pictures for the view; storing the decoded reference pictures for the view in an enhancement layer reference picture buffer corresponding to the view; and decoding the enhancement layer bitstream signal corresponding to the respective enhancement layer to create an enhancement layer decode image or video frame for the view by performing motion or disparity compensation using the reference pictures stored in the enhancement layer reference picture buffer corresponding to the view. a) base layer processing a base layer bitstream signal through a base layer, wherein the base layer bitstream signal encodes images or video frames of multiple views in a frame-compatible representation, the base layer processing comprising: storing past decoded frame-compatible reference pictures for the base layer; and decoding the base layer bitstream signal to create a frame-compatible base layer decode image or video frame by performing disparity compensation at frame-compatible level using the past decoded frame-compatible reference pictures for the base layer; storing past decoded frame-compatible reference pictures for the base layer; and decoding the base layer bitstream signal to create a frame-compatible base layer decode image or video frame by performing disparity compensation at frame-compatible level using the past decoded frame-compatible reference pictures for the base layer; b) enhancement layer processing a plurality of enhancement layer bitstream signals through one or more enhancement layers for multiple views through one or more enhancement layers, wherein each of the multiple views is processed separately in a separate enhancement layer, wherein an enhancement layer bitstream signal encodes images or video frames of a view in an enhancement layer representation having a higher spatial resolution than the same view encoded in the base layer frame-compatible representation, the enhancement layer processing for each of the one or more enhancement layers comprising: providing decoded reference pictures for the respective enhancement layer to retrieve decoded reference pictures for the view; de-multiplexing past decoded frame-compatible reference pictures for the base layer to retrieve decoded reference pictures for the view; storing the decoded reference pictures for the view in an enhancement layer reference picture buffer corresponding to the view; and decoding the enhancement layer bitstream signal corresponding to the respective enhancement layer to create an enhancement layer decode image or video frame for the view by performing motion or disparity compensation using the reference pictures stored in the enhancement layer reference picture buffer corresponding to the view. providing decoded reference pictures for the respective enhancement layer to retrieve decoded reference pictures for the view; de-multiplexing past decoded frame-compatible reference pictures for the base layer to retrieve decoded reference pictures for the view; storing the decoded reference pictures for the view in an enhancement layer reference picture buffer corresponding to the view; and decoding the enhancement layer bitstream signal corresponding to the respective enhancement layer to create an enhancement layer decode image or video frame for the view by performing motion or disparity compensation using the reference pictures stored in the enhancement layer reference picture buffer corresponding to the view.