A method of encoding an input video into a plurality of encoded streams (310, 316, 328), such that the encoded streams (310, 316, 328) may be combined to reconstruct the input video, the method comprising: receiving an input video comprising respective frames, each frame of the respective frames being divided into a plurality of tiles and each tile of the plurality of tiles being divided into a plurality of blocks; generating a base encoded stream using a base encoder; characterised by determining a temporal mode for one or more further encoded enhancement streams (316, 328) for use in reconstructing the input video together with the base encoded stream, the one or more further encoded enhancement streams (316, 328) being generated using an enhancement encoder, wherein the temporal mode is one of a first temporal mode that does not apply non-zero values from a temporal buffer for generating the one or more further encoded enhancement streams and a second temporal mode that does apply non-zero values from the temporal buffer for generating the one or more further encoded enhancement streams, wherein applying non-zero values from the temporal buffer in the second temporal mode comprises deriving a set of non-zero temporal coefficients from the temporal buffer and using the set of non-zero temporal coefficients to modify a current set of coefficients for generating the one or more further encoded enhancement streams (316, 328) and wherein the temporal buffer stores data associated with a previous frame; and generating the one or more further encoded enhancement streams (316, 328) based on data derived from the base encoded stream and the input video according to the determined temporal mode, wherein generating the one or more further encoded enhancement streams (316, 328) comprises applying a transform to each of a series of blocks of the plurality of blocks, and wherein the temporal mode is determined for one or more of a frame, tile, or block of the input video. receiving an input video comprising respective frames, each frame of the respective frames being divided into a plurality of tiles and each tile of the plurality of tiles being divided into a plurality of blocks; generating a base encoded stream using a base encoder; characterised by determining a temporal mode for one or more further encoded enhancement streams (316, 328) for use in reconstructing the input video together with the base encoded stream, the one or more further encoded enhancement streams (316, 328) being generated using an enhancement encoder, wherein the temporal mode is one of a first temporal mode that does not apply non-zero values from a temporal buffer for generating the one or more further encoded enhancement streams and a second temporal mode that does apply non-zero values from the temporal buffer for generating the one or more further encoded enhancement streams, wherein applying non-zero values from the temporal buffer in the second temporal mode comprises deriving a set of non-zero temporal coefficients from the temporal buffer and using the set of non-zero temporal coefficients to modify a current set of coefficients for generating the one or more further encoded enhancement streams (316, 328) and wherein the temporal buffer stores data associated with a previous frame; and generating the one or more further encoded enhancement streams (316, 328) based on data derived from the base encoded stream and the input video according to the determined temporal mode, wherein generating the one or more further encoded enhancement streams (316, 328) comprises applying a transform to each of a series of blocks of the plurality of blocks, and wherein the temporal mode is determined for one or more of a frame, tile, or block of the input video.