A method of decoding, in a video decoder (150, 1000), a block of enhancement-layer transform domain data for an enhancement-layer encoded video, the block of enhancement-layer transform domain data corresponding to a reconstructed block of reference-layer transform domain data (130, 202, 300) for a reference-layer encoded video, wherein the reference-layer encoded video is a spatially scaled-down version of the enhancement-layer encoded video, wherein the block of enhancement-layer transform domain data is of size W x H, wherein the reconstructed block of reference-layer transform domain data (130, 202, 300) is of size W/R x H/R, and wherein R is a spatial scale factor between the reference-layer encoded video and the enhancement-layer encoded video, the method comprising: for an enhancement-layer element of the block of enhancement-layer transform domain data: if the enhancement-layer element is within the upper left W/R x H/R portion of the block of enhancement-layer transform domain data, determining a context for that enhancement-layer element based upon elements in the reconstructed block of reference-layer transform domain data (130, 202, 300), wherein: the elements are identified by a template (302, 304) including a first reference-layer element and two or more neighboring reference-layer elements (B, C, E, F, I), the first reference-layer element being in the same position in the reconstructed block of reference-layer transform domain data (130, 202, 300) as the position of that enhancement-layer element in the block of enhancement-layer transform domain data and the neighboring reference-layer elements (B, C, E, F, I) neighboring the first reference-layer element; the position of the template (302, 304) in the reconstructed block of reference-layer transform domain data (130, 202, 300) is based upon the position of that enhancement-layer element in the block of enhancement-layer transform domain data; and the context is determined based upon the elements of the reconstructed block of reference-layer transform domain data (130, 202, 300) without upscaling of the reconstructed block of reference-layer transform domain data (130, 202, 300); if the enhancement-layer element is outside the upper left W/R x H/R portion of the block of enhancement-layer transform domain data, upscaling the reconstructed block of reference-layer transform domain data (130, 202, 300) and determining the context based upon the upscaled elements of the reconstructed block of reference-layer transform domain data (130, 202, 300), or determining the context without reference to the reconstructed block of reference-layer transform domain data (130, 202, 300); and entropy decoding that enhancement-layer element using its determined context. if the enhancement-layer element is within the upper left W/R x H/R portion of the block of enhancement-layer transform domain data, determining a context for that enhancement-layer element based upon elements in the reconstructed block of reference-layer transform domain data (130, 202, 300), wherein: the elements are identified by a template (302, 304) including a first reference-layer element and two or more neighboring reference-layer elements (B, C, E, F, I), the first reference-layer element being in the same position in the reconstructed block of reference-layer transform domain data (130, 202, 300) as the position of that enhancement-layer element in the block of enhancement-layer transform domain data and the neighboring reference-layer elements (B, C, E, F, I) neighboring the first reference-layer element; the position of the template (302, 304) in the reconstructed block of reference-layer transform domain data (130, 202, 300) is based upon the position of that enhancement-layer element in the block of enhancement-layer transform domain data; and the context is determined based upon the elements of the reconstructed block of reference-layer transform domain data (130, 202, 300) without upscaling of the reconstructed block of reference-layer transform domain data (130, 202, 300); the elements are identified by a template (302, 304) including a first reference-layer element and two or more neighboring reference-layer elements (B, C, E, F, I), the first reference-layer element being in the same position in the reconstructed block of reference-layer transform domain data (130, 202, 300) as the position of that enhancement-layer element in the block of enhancement-layer transform domain data and the neighboring reference-layer elements (B, C, E, F, I) neighboring the first reference-layer element; the position of the template (302, 304) in the reconstructed block of reference-layer transform domain data (130, 202, 300) is based upon the position of that enhancement-layer element in the block of enhancement-layer transform domain data; and the context is determined based upon the elements of the reconstructed block of reference-layer transform domain data (130, 202, 300) without upscaling of the reconstructed block of reference-layer transform domain data (130, 202, 300); if the enhancement-layer element is outside the upper left W/R x H/R portion of the block of enhancement-layer transform domain data, upscaling the reconstructed block of reference-layer transform domain data (130, 202, 300) and determining the context based upon the upscaled elements of the reconstructed block of reference-layer transform domain data (130, 202, 300), or determining the context without reference to the reconstructed block of reference-layer transform domain data (130, 202, 300); and entropy decoding that enhancement-layer element using its determined context.