Patent ID: 8019804

Claim:
A method for calculating the sum of squared differences between an original block S and a reconstructed block R of image or video data, the method comprising the steps of: (a) computing with a processor a predicted block P corresponding to the original block S, using inter or intra frame prediction, (b) calculating with the processor a residual block D from the original block S and the predicted block P, said residual block D having a plurality of coefficients, (c) applying with the processor an integer image transform to the coefficients of the residual block D so as to obtain a transformed residual block F*, said transformed residual block having a plurality of coefficients f* ij , (d) finding with the processor a plurality of coefficients {circumflex over (f)}* ij , the coefficients {circumflex over (f)}* ij being defined by the equation {circumflex over (f)}* ij =Q −1 (Q(f* ij )), where Q is an operator which performs quantizing and Q −1 is the inverse of the Q operator, wherein finding the coefficients comprises iteratively comparing the absolute value of each coefficient f* ij of the transformed residual block to boundaries quantization sub-zones in a look up table, said boundaries of quantization sub-zones having progressively higher quantization point values, until said absolute value of said coefficient is lower than a quantization sub-zone boundary, and (e) computing with the processor a sum of squared differences between f* ij and {circumflex over (f)}* ij .