A multi-layer video system, comprising: a first layer encoder that encodes a first layer of video information; at least one second layer encoder that encodes at least one second layer (EL 1...EL n) of video information; and an encoder side reference processing unit that: estimates a filter that applies on a reference picture that is reconstructed from the first video information layer, and processes a current picture of the second video information layer (EL 1...EL n), based on a correlation between the reference picture and the current picture, and sends the filtered reference picture selectively to one of a plurality of different modules (DPB, EL Inter Pred) in the second layer encoder depending on a complexity mode selected from a plurality of complexity modes (BCM, BPM, NCM), each complexity mode using a different configuration for coding the at least one second layer of video information (EL 1...EL n) by the at least one second layer encoder, the plurality of complexity modes (BCM, BPM, NCM) comprising a bypass coding mode, a bypass prediction mode and a normal coding mode; wherein the correlation relates to a complexity characteristic that corresponds to the reference picture and the current picture; and wherein the complexity characteristic comprises a plurality of complexity levels, the plurality of complexity levels including a first complexity level enabling selection from the plurality of complexity modes (BCM, BPM, NCM), and a second complexity level, in which a plurality of complexity profiles are defined, each complexity profile relating to different filters; wherein the encoder side reference processing unit estimates the filter by referring to the one of the plurality of complexity profiles; wherein the first layer of video information comprises a base layer and the at least one second video information layer comprises an enhancement layer; wherein in the normal coding mode, the filtered reconstructed reference picture is used for inter-layer prediction in the enhancement layer; wherein in the bypass coding mode, enhancement layer coding is bypassed; wherein in the bypass prediction mode, inter-prediction and intra-prediction are bypassed in the enhancement layer coding and the filtered reconstructed reference picture functions as the prediction signal; and wherein a scalable video bitstream is outputted. a first layer encoder that encodes a first layer of video information; at least one second layer encoder that encodes at least one second layer (EL 1...EL n) of video information; and an encoder side reference processing unit that: estimates a filter that applies on a reference picture that is reconstructed from the first video information layer, and processes a current picture of the second video information layer (EL 1...EL n), based on a correlation between the reference picture and the current picture, and sends the filtered reference picture selectively to one of a plurality of different modules (DPB, EL Inter Pred) in the second layer encoder depending on a complexity mode selected from a plurality of complexity modes (BCM, BPM, NCM), each complexity mode using a different configuration for coding the at least one second layer of video information (EL 1...EL n) by the at least one second layer encoder, the plurality of complexity modes (BCM, BPM, NCM) comprising a bypass coding mode, a bypass prediction mode and a normal coding mode; estimates a filter that applies on a reference picture that is reconstructed from the first video information layer, and processes a current picture of the second video information layer (EL 1...EL n), based on a correlation between the reference picture and the current picture, and sends the filtered reference picture selectively to one of a plurality of different modules (DPB, EL Inter Pred) in the second layer encoder depending on a complexity mode selected from a plurality of complexity modes (BCM, BPM, NCM), each complexity mode using a different configuration for coding the at least one second layer of video information (EL 1...EL n) by the at least one second layer encoder, the plurality of complexity modes (BCM, BPM, NCM) comprising a bypass coding mode, a bypass prediction mode and a normal coding mode; wherein the correlation relates to a complexity characteristic that corresponds to the reference picture and the current picture; and wherein the complexity characteristic comprises a plurality of complexity levels, the plurality of complexity levels including a first complexity level enabling selection from the plurality of complexity modes (BCM, BPM, NCM), and a second complexity level, in which a plurality of complexity profiles are defined, each complexity profile relating to different filters; wherein the encoder side reference processing unit estimates the filter by referring to the one of the plurality of complexity profiles; wherein the first layer of video information comprises a base layer and the at least one second video information layer comprises an enhancement layer; wherein in the normal coding mode, the filtered reconstructed reference picture is used for inter-layer prediction in the enhancement layer; wherein in the bypass coding mode, enhancement layer coding is bypassed; wherein in the bypass prediction mode, inter-prediction and intra-prediction are bypassed in the enhancement layer coding and the filtered reconstructed reference picture functions as the prediction signal; and wherein a scalable video bitstream is outputted.