Scalable video decoder configured to decode base layer motion parameters from a coded data stream, reconstruct an enhancement layer signal comprising gathering, for a block of a frame of the enhancement layer signal, a set of motion parameter candidates from neighboring already reconstructed blocks (92, 94) of the frame; gathering a set of one or more base layer motion parameters of a block of a base layer signal, collocated to the block of the frame of the enhancement layer signal adding the set of one or more base layer motion parameters or a scaled version thereof to the set of enhancement layer motion parameter candidates so as to obtain an extended motion parameter candidate set for the enhancement layer, selecting one of the motion parameter candidates of the extended motion parameter candidate set, predicting, by motion compensated prediction, the enhancement layer signal using the selected one of the motion parameter candidates of the extended motion parameter candidate set. characterized in that the scalable video decoder is further configured to decode a base layer residual signal of the base layer signal from the coded data stream, in the reconstructing the enhancement layer signal, decode a transform coefficient block of the enhancement layer residual signal using a context model which depends on the base layer residual signal by determining, for a certain transform coefficient at a currently visited transform coefficient position, the context using a template which comprises positions in one or more transform blocks in the base layer residual signal which correspond to the certain transform coefficient position in terms of location and spectral frequency. gathering, for a block of a frame of the enhancement layer signal, a set of motion parameter candidates from neighboring already reconstructed blocks (92, 94) of the frame; gathering a set of one or more base layer motion parameters of a block of a base layer signal, collocated to the block of the frame of the enhancement layer signal adding the set of one or more base layer motion parameters or a scaled version thereof to the set of enhancement layer motion parameter candidates so as to obtain an extended motion parameter candidate set for the enhancement layer, selecting one of the motion parameter candidates of the extended motion parameter candidate set, predicting, by motion compensated prediction, the enhancement layer signal using the selected one of the motion parameter candidates of the extended motion parameter candidate set. characterized in that the scalable video decoder is further configured to decode a base layer residual signal of the base layer signal from the coded data stream, in the reconstructing the enhancement layer signal, decode a transform coefficient block of the enhancement layer residual signal using a context model which depends on the base layer residual signal by determining, for a certain transform coefficient at a currently visited transform coefficient position, the context using a template which comprises positions in one or more transform blocks in the base layer residual signal which correspond to the certain transform coefficient position in terms of location and spectral frequency.