Patent ID: 7944965

Claim:
A method of estimating a distortion level associated with an encoding mode, the method comprising: performing the following steps using an encoder; transforming a motion compensated prediction residual associated with the encoding mode; quantizing the transformed motion compensated prediction residual; inverse quantizing the quantized, transformed motion compensated prediction residual resulting in a reconstructed transform domain residual; estimating a spatial domain residual through a transform domain based analysis of the transformed motion compensated prediction residual and the reconstructed transform domain residual; and calculating a distortion level from the estimated spatial domain residual; wherein the distortion level is defined as the trace ((e−e Q )(e−e Q ) T ), wherein e represents the motion compensated prediction residual, e Q represents the estimated spatial domain residual, and T represents a 4×4 integer transform approximate of a 4×4 cosine transform, and wherein the estimated spatial domain residual is defined as e - e Q = 1 64 ⁢ ( T - 1 ⁢ A ⁡ ( T T ) - 1 ) , where ⁢ ⁢ A = 64 × E - M 1 ⁢ E Q ⁢ M 1 , wherein E represents the transformed residual, E Q represents the reconstructed transform domain residual, and M 1 =diag(4, 5, 4, 5).