A method comprising: receiving, at data processing hardware, transform coefficients corresponding to a scaled video input signal, the scaled video input signal comprising a plurality of spatial layers L, the plurality of spatial layers L comprising a base layer L 0; determining, by the data processing hardware, a spatial rate factor based on a sample S F of frames from the scaled video input signal, the spatial rate factor comprising an allocation factor for bit rate allocation at each spatial layer L of an encoded bit stream formed from the scaled video input signal, wherein the allocation factor corresponds to a rate of bits per transform coefficient of the base layer L 0, the spatial rate factor is set equal to a difference between the allocation factor and an average rate R L of bits per transform coefficient for the plurality of spatial layers L; wherein the allocation factor comprises a weighted sum of ratios of a product of variances, the ratio comprising a numerator based on estimated variances of the transform coefficients from the base layer L 0 and a denominator based on estimated variances of the transform coefficients from a second spatial layer L j of the plurality of spatial layers, the weighted sum being over the index j; wherein variances of transform coefficients are estimated by averaging across all transform blocks in sample frames S F of the video input signal; and reducing, by the data processing hardware, a distortion for the plurality of spatial layers L of the encoded bit stream by allocating a bit rate B R to each spatial layer L based on the spatial rate factor determined based on the sample S F of frames, the allocating being subject to a constraint on a total bit rate B Rtot available to encode all of the plurality of spatial layers. receiving, at data processing hardware, transform coefficients corresponding to a scaled video input signal, the scaled video input signal comprising a plurality of spatial layers L, the plurality of spatial layers L comprising a base layer L 0; determining, by the data processing hardware, a spatial rate factor based on a sample S F of frames from the scaled video input signal, the spatial rate factor comprising an allocation factor for bit rate allocation at each spatial layer L of an encoded bit stream formed from the scaled video input signal, wherein the allocation factor corresponds to a rate of bits per transform coefficient of the base layer L 0, the spatial rate factor is set equal to a difference between the allocation factor and an average rate R L of bits per transform coefficient for the plurality of spatial layers L; wherein the allocation factor comprises a weighted sum of ratios of a product of variances, the ratio comprising a numerator based on estimated variances of the transform coefficients from the base layer L 0 and a denominator based on estimated variances of the transform coefficients from a second spatial layer L j of the plurality of spatial layers, the weighted sum being over the index j; wherein variances of transform coefficients are estimated by averaging across all transform blocks in sample frames S F of the video input signal; and wherein the allocation factor corresponds to a rate of bits per transform coefficient of the base layer L 0, the spatial rate factor is set equal to a difference between the allocation factor and an average rate R L of bits per transform coefficient for the plurality of spatial layers L; wherein the allocation factor comprises a weighted sum of ratios of a product of variances, the ratio comprising a numerator based on estimated variances of the transform coefficients from the base layer L 0 and a denominator based on estimated variances of the transform coefficients from a second spatial layer L j of the plurality of spatial layers, the weighted sum being over the index j; wherein variances of transform coefficients are estimated by averaging across all transform blocks in sample frames S F of the video input signal; and reducing, by the data processing hardware, a distortion for the plurality of spatial layers L of the encoded bit stream by allocating a bit rate B R to each spatial layer L based on the spatial rate factor determined based on the sample S F of frames, the allocating being subject to a constraint on a total bit rate B Rtot available to encode all of the plurality of spatial layers.