Project: ArangoDB (App Definition and Development / Database)
File: crc-doc.1.0.pdf

Everything we know about CRC but afraid to
forget
Andrew Kadatch1and Bob Jenkins2
1Google Inc.
2Microsoft Corporation
September 3, 2010
Abstract
This paper describes a novel interleaved, parallelizeable word-by-word
CRC computation algorithm which computes N-bit CRC ( N64) on
modern Intel and AMD processors in 1.2 CPU cycles per byte, improv-
ing state of the art over word-by-word 32-bit and 64-bit CRCs (2.1 CPU
cycles/byte) and classic byte-by-byte CRC computation (6-7 CPU cy-
cles/byte). It computes 128-bit CRC in 1.7 CPU cycles/byte.
CRC implementations are heavily optimized and hard to understand.
This paper describes CRC algorithms as they evolved over time, splitting
complex optimizations into a sequence of natural improvements.
This paper also presents a collection of CRC \tricks" that we found
handy on many occassions.
Contents
1 Denition of CRC 2
2 Related work 3
3 CRC tricks and tips 4
3.1 Incremental CRC computation . . . . . . . . . . . . . . . . . . . 4
3.2 Changing initial CRC value . . . . . . . . . . . . . . . . . . . . . 4
3.3 Concatenation of CRCs . . . . . . . . . . . . . . . . . . . . . . . 5
3.4 In-place modication of CRC-ed message . . . . . . . . . . . . . 5
3.5 Storing CRC value after the message . . . . . . . . . . . . . . . . 6
4 Ecient software implementation 7
4.1 Mapping bitstreams to hardware registers . . . . . . . . . . . . . 7
4.2 Multiplication of D-normalized polynomials . . . . . . . . . . . . 7
4.3 Multiplication of unnormalized polynomial . . . . . . . . . . . . . 7
14.4 Computing powers of x. . . . . . . . . . . . . . . . . . . . . . . 8
4.5 Simplied CRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.6 Computing a CRC byte by byte . . . . . . . . . . . . . . . . . . . 9
4.7 Rolling CRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.8 Reading multiple bytes at a time . . . . . . . . . . . . . . . . . . 12
4.9 Computing a CRC word by word . . . . . . . . . . . . . . . . . . 12
4.10 Processing non-overlapping blocks in parallel . . . . . . . . . . . 13
4.11 Interleaved word-by-word CRC . . . . . . . . . . . . . . . . . . . 16
4.11.1 Parallelizing CRC computation . . . . . . . . . . . . . . . 16
4.11.2 Combining individual CRCs . . . . . . . . . . . . . . . . . 18
4.11.3 Ecient computation of individual CRCs . . . . . . . . . 19
5 Experimental results 22
5.1 Testing methology . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.2 Compiler comparison . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.3 Choice of interleave level . . . . . . . . . . . . . . . . . . . . . . . 23
5.4 Performance of CRC algorithms . . . . . . . . . . . . . . . . . . . 23
1 Denition of CRC
Cyclic Redundancy Check (CRC) is a well-known technique that allows the
recipient of a message transmitted over a noisy channel to detect whether the
message has been corrupted.
A message M=m0: : : m N1comprised of N=jMjbits ( mk2f0;1g) may
be viewed either as a numeric value
M=N1X
k=0mk2N1k
or as a polynomial of a single variable of degree ( N1)
M(x) =N1X
k=0mkxN1k
where mk2GF(2) =f0;1gand all arithmetic operations on coecients are
performed modulo 2. For example,
Addition: ( x3+x2+x+ 1) + ( x2+x+ 1) = x3+ 2x2+ 2x+ 2 = x3;
Subtraction: ( x3+x+ 1)(x2+x) =x3x2+ 1 = x3+x2+ 1;
Multiplication: ( x+ 1)( x+ 1) = x2+ 2x+ 1 = x2+ 1:
For a given polynomial P(x) of degree D= deg
P(x)
, CRC u
M(x); v(x)
is the reminder from division of
M(x)xD
byP(x). In practice, a more
2complex formula is used:
CRC u
M(x); v(x)
=
v(x)u(x)
xjMj+M(x)xD+u(x)
modP(x);
(1)
where polynomial P(x) of degree Dand polynomial u(x) of degree less than D
are xed.
The use of the non-zero value of u(x) guarantees that the CRC of a sequence
of zeroes is dierent from zero. That allows detection of insertion of zeroes in
the beginning of a message and replacement of both content of the message and
its CRC value with zeroes. Typically,
u(x) =D1X
k=0xk: (2)
The use of auxilary parameter v(x) allows incremental CRC computation as
shown in section 3.1.
2 Related work
Cyclic Redundancy Checks (CRCs) were proposed by Peterson and Brown
[PB61] in 1961. An ecient table-driven software implementation which reads
and processes data byte by byte was described by Hill [Hil79] in 1979, Perez
[Per83] in 1983. The \classic" byte-by-byte CRC algorithm described in section
4.6 was published by Sarwate [Sar88] in 1988.
In 1993, Black [Bla93] published a method that reads data by words (de-
scribed in section 4.8); however, it still computes the CRC byte by byte in strong
sequential order.
In 2001, Braun and Waldvogel [BW01] briey outlined a specialized vari-
ant of a CRC that could read input data by words and process them byte by
byte { but, thanks to the use of multiple tables, dierent bytes from the input
word could be processed in parallel. In 2002, Ji and Killian [JK02] provided de-
tailed description and analysis of a nearly identical scheme. Both solutions were
targeted for hardware implementation. In 2005, Kouvanis and Berry [KB05]
demonstrated clear performance benets of this scheme even when it is im-
plemented in software. A generalized version of this approach is described in
section 4.9.
Surprisingly, until [GGO+10] we have not seen prior art describing or utiliz-
ing a method of computing a CRC by processing in parallel (in an interleaved
manner to utilize multiple ALUs) multiple input streams belonging to non-
overlapping sections of input data, desribed in section 4.10.
A novel method of CRC computation that processes in parallel multiple
words belonging to overlapping sections of input data is described in section
4.11. A special case restricted to the use of 64-bit tables, 64-bit reads, and 32
3or 64-bit generating polynomials was implemented by the authors in February-
March 2007 and was used by a couple of Microsoft products. In 2009, the
algorithm was generalized and these limitations were removed.
The fact that the CRC of a message followed by its CRC is a constant value
which does not depend on the message, described in section 3.5, is well known
and has been widely used in the telecommunication industry for long time.
A method of storing a carefully chosen sequence of bits after a message so
that the CRC of a message and the sequence of bits appended to the mes-
sage produces predened result, described in 3.5, was implemented in 1990 by
Zemtsov [Zem90].
A method for recomputing a known CRC using a new initial CRC value,
described in section 3.2, and the method of computing a CRC of the concatena-
tion of messages having known CRC values without touching the actual data,
described in section 3.3, were implemented by one of the authors in 2005 but
were not published.
3 CRC tricks and tips
3.1 Incremental CRC computation
The use of an arbitrary initial CRC value v(x) allows computation of a CRC
incrementally. If a message M(x) =M1(x)xjM2j+M2(x) is a concatenation
of messages M1andM2, its CRC may be computed piece by piece because
CRC u
M(x); v(x)
= CRC u
M2(x);CRC u
M1(x); v(x)
: (3)
Indeed,
CRC u(M; v) =
(vu)xjMj+MxD+u
modP=
=
(vu)xjM1j+jM2j+ (M1xjM2j+M2)xD+u
modP=
=
(vu)xjM1j+M1xD
xjM2j+M2xD+u
modP=
=
CRC u(M1; v)xjM2j+M2xD+u
modP=
= CRC u
M2;CRC u(M1; v)
3.2 Changing initial CRC value
If CRC u
M(x); v(x)
for some initial value v(x) is known, it is possible to
compute CRC u
M(x); v0(x)
for dierent initial value v0(x) without touching
the value of M(x):
CRC u(M; v0) = CRC u(M; v) +
(v0v)xjMj
modP: (4)
4Proof:
CRC u(M; v0) =
(v0u)xjMj+MxD+u
modP=
=
(v0u) + (vv)
xjMj+MxD+u
modP=
=
(vu) + (v0v)
xjMj+MxD+u
modP=
=
(vu)xjMj+MxD+u
+ (v0v)xjMj
modP=
= CRC u(M; v) +
(v0v)xjMjmodP
:
3.3 Concatenation of CRCs
If a message M(x) =M1(x)xjM2j+M2(x) is a concatenation of messages
M1andM2, and CRCs of M1,M2(computed with some initial values v1(x),
v2(x) respectively) are known, CRC u
M(x); v(x)
may be computed without
touching contents of the message M:
1. Using formula (4), the value of v0
1= CRC u(M1; v) may be computed from
the known CRC u(M1; v1) without touching the contents of M1.
2. Then, v0
2= CRC u(M2; v0
1) may be computed from known CRC u(M2; v2)
without touching the contents of M2.
According to (3), CRC u(M; v) =v0
2.
3.4 In-place modication of CRC-ed message
Sometimes it is necessary to replace a part of message M(x) in-place and re-
compute CRC of modied message M(x) eciently.
If a message M=ABC is a concatenation of messages A,B, and C, and
B0(x) is new message of the same length as B(x), CRC u(M0) of message M0=
AB0Cmay be computed from known CRC u(M). Indeed,
M(x) =A(x)xjBj+jCj+B(x)xjCj+C(x);
M0(x) =A(x)xjBj+jCj+B0(x)xjCj+C(x) =
=M(x) +
B0(x)B(x)
xjCj;
therefore
CRC u
M0(x); v(x)
=
= CRC u
M(x) +
B0(x)B(x)
xjCj
=
=
v(x)u(x)
xjMj+M(x)xD+
B0(x)B(x)
xjCj+D+u(x)
modP(x)
=
CRC u
M(x); v(x)
+
B0(x)B(x)
xjCj+D
modP(x) =
= CRC u
M(x); v(x)
+
B0(x)B(x)
xjCj+DmodP(x)
:
5It is easy to see that
CRC u
B0(x); v(x)
CRC u
B(x); v(x)
=
=
B0(x)B(x)
xDmodP(x);
so
CRC u
M0(x); v(x)
= CRC u
M(x); v(x)
+ 
where
 =
CRC u
B0(x); v(x)
CRC u
B(x); v(x)
xjCjmodP(x):
3.5 Storing CRC value after the message
Often Q(x) = CRC u
M(x); v(x)
is padded with zero bits until the nearest
byte or word boundary and is transmitted as a sequence of Wbits ( WD)
right after the message M(x). This way, the transmitted message T(x) is the
concatenation of M(x) and Q(x) followed by ( WD) zeroes, and is equal to
T(x) =M(x)xW+Q(x)xWD:
According to (1), (3) and taking into account that Q(x) +Q(x) = 0 since
polynomial coecient are from GF(2), CRC u
T(x); v(x)
is a constant value
which does not depend on the contents of the message and is equal to
CRC u
T(x); v(x)
=
= CRC u
Q(x)xWD; CRC
M(x); v(x)
=
= CRC u
Q(x)xWD; Q(x)
=
=
Q(x)u(x)
xW+Q(x)xWDxD+u(x)
modP(x) =
=
u(x)
1xW
modP(x):
A more generic solution is to store a W-bit long value after the message
such that the CRC of the transmitted message is equal to a predened value
R(x) (typically R(x) = 0). The D-bit value followed by ( WD) zero bits that
should be stored after M(x) is
^q
Q(x)
=
R(x)u(x)
xW
Q(x)u(x)
modP(x)
where xWis the multiplicative inverse of xWmodP(x) which exists if P(x)
is not divisble by xand may be found by the extended Euclidean algorithm
6[Has01]:
CRC u
^q
Q(x)
xWD; CRC
M(x); v(x)
=
= CRC u
^q
Q(x)
xWD; Q(x)
=
=
Q(x)u(x)
xW+ ^q
Q(x)
xWDxD+u(x)
modP(x) =
=R(x):
4 Ecient software implementation
4.1 Mapping bitstreams to hardware registers
For little-endian machines (assumed from now on), the result of loading of a
D-bit word from memory into hardware register matches the expectations: the
0-th bit of the 0-th byte becomes the 0-th (least signicant) bit of the word
corresponding to x(D1).
For example, the 32-bit sequence of 4 bytes 0x01, 0x02, 0x03, 0x04 (0x04030201
when loaded into a 32-bit hardware register) corresponds to the polynomial

x31+x22+x15+x14+x5
:
Addition and subtraction of polymonials with coecients from GF(2) is the
bitwise XOR of their coecients. Multiplication of a polynomial by xis achieved
by logical right shift of register contents by 1 bit. If a shift operation causes a
carryover, the resulting polynomial has degree D.
Polynomials of degree less than Dwhose coecients are recorded using ex-
actly Dbits irrespective of actual degree of the polynomial will be called D-
normalized .
Whenever possible { and unless mentioned explicitly { all polynomials will
be represented in D-normalized form.
Since the generating polynomial P(x) is of degree Dand has ( D+ 1) co-
ecients, it does not t into the D-bit register. However, its most signicant
coecient is guaranteed to be 1 and may be implied implicitly.
4.2 Multiplication of D-normalized polynomials
Multiplication of two D-normalized polynomials may be accomplished by tradi-
tional bit-by-bit, shift-and-add multiplication. This is adequate if performance
is not a concern. Sample code is given in listing 1.
4.3 Multiplication of unnormalized polynomial
During initialization of CRC tables it may be necessary to multiply d-normalized
polynomial v(x) of a degree d6=Dby a D-normalized polynomial. It may be
accomplished by representing the operand as a sum of weighted polynomials of
71 // "a" and "b" occupy D l e a s t s i g n i f i c a n t b i t s .
2 Crc Multiply ( Crc a , Crc b) f
3 Crc product = 0 ;
4 Crc bPowX[D] ; // bPowX[ k ] = (b xk ) mod P
5 bPowX [ 0 ] = b ;
6 f o r ( i n t k = 0 ; k <D; ++k )f
7 // I f "a" has non zero c o e f f i c i e n t at x k ,
8 // add ( ( b xk ) mod P) to the r e s u l t .
9 i f ( ( ( a & (1 <<(Dk ) ) != 0) product ^= bPowX[ k ] ;
10
11 // Compute bPowX[ k+1] = (b x( k+1) ) mod P.
12 i f (bPowX[ k ] & 1) f
13 // I f degree of (bPowX[ k ] x ) i s D, then
14 // degree of (bPowX[ k ] xP) i s l e s s than D.
15 bPowX[ k+1] = (bPowX[ k ] >>1) ^ P;
16ge l s ef
17 bPowX[ k+1] = bPowX[ k ] >> 1 ;
18g
19g
20 return product ;
21g
Listing 1: Multiplication of normalized polynomials
degree of no more than ( D1), then calling Multiply () function repeatedly as
shown in listing 2.
4.4 Computing powers of x
Often (see sections 3.2, 3.3, 3.5) it is necessary to compute xNmodP(x) for
very large values of N. This may be accomplished in O
log(N)
time.
Consider the binary representation of N:
N=KX
k=0nk2k
where nk2f0;1g. Then
xN=xPnk2k=KY
k=0xnk2k=Y
nk!=0x2k(5)
and may be computed using no more than ( blog2(N)c+ 1) multiplications of
polynomials of degree less than Dprovided known values of
Pow2k(k) =x2kmodP(x): (6)
Values of Pow2k(k) may be computed iteratively using one multiplication
81 // "v" occupies "d" l e a s t s i g n f i c a n t b i t s .
2 // "m" occupies D l e a s t s i g n i f i c a n t b i t s .
3 Crc MultiplyUnnormalized ( Crc v , i n t d , Crc m) f
4 Crc r e s u l t = 0 ;
5 while (d >D)f
6 Crc temp = v & ((1 <<D)1) ;
7 v > >= D;
8 d= D;
9 // XpowN returns ( x N mod P( x ) ) .
10 r e s u l t ^= Multiply (temp , Multiply (m, XpowN(d) ) ) ;
11g
12 r e s u l t ^= Multiply ( v <<(Dd) , m) ;
13 return r e s u l t ;
14g
Listing 2: Multiplication of unnormalized polynomial
modP(x) per iteration:
Pow2k(0) = 0 ;
Pow2k(k+ 1) = x2k+1modP(x) =
=x22kmodP(x) =
=
x2k2
modP(x) =
=
Pow2k(k1)2
modP(x):
4.5 Simplied CRC
It is sucient to be able to compute
CRC 0
M(x); v(x)
=
v(x)xjMj+M(x)xD
modP(x); (7)
since
CRC u
M(x); v(x)
= CRC 0
M(x); v(x)u(x)
+u(x);
CRC u
M(x); v(x)
of message M=M1: : : M Kmay be computed incrementally
using CRC 0instead of CRC u:
v0(x) =v(x)u(x);
vk(x) = CRC 0
Mk(x); vk1(x)
;
CRC u(M(x); v(x)) =vK+u(x):
4.6 Computing a CRC byte by byte
IfM(x) isW-bit value (typically, W= 8) and deg
v(x)
< D, by denition (7)
CRC 0
M(x); v(x)
=
v(x)xW+M(x)xD
modP(x):
9When DW,
CRC 0
M(x); v(x)
=
v(x)xW+M(x)xD
modP(x) =
=
v(x)xWD+M(x)
xD
modP(x); (8)
which may be obtained via single lookup into precomputed table Tof size 2W
such that T[i] =
i(x)xD)
modP(x) since deg
v(x)xWD+M(x)
< W .
D-normalized representation of v(x) occupies Dleast signicant bits and is
equal to
v(x)xWD
when viewed as W-normalized representation which is
required to form W-bit index into a table of 2Wentries. Therefore, explicit
multiplication of v(x) byxWDin formula (8) is not required.
When DW,v(x) may be represented as
v(x) =vL(x) +vH(x)xDW
where
vH(x) =v(x)
xDW
; deg
vH(x)
< W;
vL(x) =v(x) mod xDW; deg
vL(x)
< DW:
Since deg
vL(x)xW
< D,
vL(x)xW
modP(x) =vL(x)xW. Therefore,
CRC 0
M(x); v(x)
=
=
v(x)xW+M(x)xD
modP(x) =
=
vL(x) +vH(x)xDW
xW+M(x)xD
modP(x) =
=
vL(x)xW+
vH(x) +M(x)
xD
modP(x) =
=
vL(x)xW+
vH(x) +M(x)
xD
modP(x) =
=
vL(x)xW
modP(x) +
vH(x) +M(x)
xD
modP(x) =
=
vL(x)xW
+ MulByXpowD
vH(x) +M(x)
; (9)
where
MulByXpowD
a(x)
=
a(x)xD
modP(x): (10)
The value of
vL(x)xW
may be computed by shifting v(x) byWbits and
discarding Wcarry-over zero bits.
Since deg
vH(x) +M(x)
< W , the value of MulByXpowD
vH(x) +M(x)
may be obtained using precomputed table containing 2Wentries.
The classic table-driven, byte-by-byte CRC computation [Per83, Sar88] im-
plementing formulas (1), (3), (8), (9), and (10) for W= 8 is given in listing
3.
101 Crc CrcByte ( Byte value ) f
2 return MulByXpowD[ value ] ;
3g
4 Crc CrcByteByByte ( Byte data , i n t n , Crc v , Crc u) f
5 Crc crc = v ^ u ;
6 f o r ( i n t i = 0 ; i <n ; ++i )f
7 Crc ByteCrc = CrcByte ( crc ^ data [ i ] ) ;
8 crc > >= 8 ;
9 crc ^= ByteCrc ;
10g
11 return ( crc ^ u) ;
12g
13 void InitByteTable ( ) f
14 f o r ( i n t i = 0 ; i <256; ++i )f
15 MulByXPowD[ i ] = MultiplyUnnormalized ( i , 8 , XpowN(D) ) ;
16g
17g
Listing 3: Computing CRC byte by byte
Experience shows that computing CRC byte by byte is rather slow and,
depending on a compiler and input data size, takes 6 8 CPU cycles per byte
on modern 64-bit CPU for D <= 64. There are two reasons for it:
1. Reading data 8 bits at a time is not the most ecient data access method
on 64-bit CPU.
2. Modern CPUs have multiple ALUs and may execute 3-4 instructions
per CPU cycles provided the instructions handle independent data ows.
However, byte-by-byte CRC contains only one data ow. Futhermore,
most instructions use the result from the previous instruction, leading to
CPU stalls because of result propagation delays.
4.7 Rolling CRC
Given a set of messages Mk=mk: : : m k+N1where mkareW-bit symbols
andNis xed (i.e. each next message is obtained by removing rst symbol
and appending new one), Ck+1= CRC u(Mk+1; v) may be obtained from known
Ck= CRC u(Mk; v) and symbols mkandmk+Nonly, without the need to com-
pute CRC of entire message Mk+1. This property may be utilized to eciently
compute a set of rolling Rabin ngerpints.
Since Mk+1(x) =Mk(x)xWmk(x)xNW+mk+N(x),
Ck+1(x) = CRC u
Mk+1(x); v(x)
=
= 

v(x)u(x)
xNW+u(x) +N1X
n=0mk+1+n(x)xD+W(N1n)!
modP(x) =
=F
Ck(x); mk+N(x)
+G
mk(x)
;
11where
F
Ck(x); mk+N(x)
=
Ck(x)xW+mk+N(x)xD
modP;
G
mk(x)
=
v(x)u(x)
xNW+u
(1xW)mk(x)xD+NW
modP
are polynomials of degree less than D.
G
mk1(x)
may be computed easily via a single lookup in a table of 2W
entries indexed by mk.
Computation of F
Ck(x); mk+N(x)
may be implemented as described in
section 4.6 and requires one bitwise shift, one bitwise XOR, and one lookup into
a precomputed table containing 2Wentries.
4.8 Reading multiple bytes at a time
One straightforward way to speed up byte-by-byte CRC computation is to read
W > 8 bits at once. Unfortunately, this is the path of very rapidly diminish-
ing return as the size of the MulByPowD table increases with Wexponentially.
From practical perspective, it is extremely desirable to ensure that the Mul-
ByPowD table ts into the L1 cache (32-64KB), otherwise table entry access
latency sharply increases from 3-4 CPU cycles (L1 cache) to 15-20 CPU (L2
cache).
The value of MulByXpowD
v(x)
may be computed iteratively using a
smaller table because
MulByXpowD
v(x)
=v(x)xDmodP(x) = CRC 0
v(x);0
(11)
and therefore may be computed using formulas (3) and (9) for smaller values of
W0.
[Bla93] provided the implementation for W= 32 and W0= 8. Our more
general implementation was faster than byte-by-byte CRC but not substentially:
the improvement was in 20-25% range. However, the result is still important {
it demonstrates that reading input data per se is not a bottleneck.
4.9 Computing a CRC word by word
The value of MulByXpowD
v(x)
may be computed using multiple smaller ta-
bles instead of one table. Given that deg
v(x)
< W ,v(x) may be represented
as a weighted sum of polynomials vk(x) such that deg
vk(x)
< B:
v(x) =K1X
k=0vk(x)x(K1k)B;
where K=dW=Beand
vk(x) =v(x)
x(K1k)B
modxB:
12Consequently,
MulByXpowD
v(x)
=v(x)xDmodP(x) =
= K1X
k=0vk(x)x(K1k)B!
xDmodP(x) =
=K1X
k=0
vk(x)x(K1k)B+DmodP(x)
=
=K1X
k=0MulWordByXpowD
k; vk(x)
; (12)
where the values of
MulWordByXpowD
k; vk(x))
=vk(x)x(K1k)B+DmodP(x) (13)
may be obtained using Kprecomputed tables. Given that deg
vk(x)
< B,
each table should contain 2Bentries.
A sample implementation of formulas (1), (3), (12), and (13) is given in
listing 4 using B= 8 and assuming that Wis a multiple of 8.
CrcWordByWord1withW= 64 uses only 2.1-2.2 CPU cycles/byte on mod-
ern 64-bit CPUs (our implementation is somewhat faster than the one described
in [KB05]). It solves the problem with data access and, to lesser degree, allows
instruction level parallelism: in the middle of the unrolled main loop of CrcOf-
Word function the CPU may process multiple bytes in parallel.
However, this solution is still imperfect { the beginning of computation con-
tends for a single source of data (variable value ), and the end of computation
contends for a single destination (variable result ). Further improvement re-
quires processing of multiple independent data streams in interleaved manner
so that when computation of one data ow path is stalled the CPU may proceed
with another one.
4.10 Processing non-overlapping blocks in parallel
Straighforward pipepiling may be achieved by spliting the input message M(x) =
M0(x): : : M N1(x) into Nblocks Mk(x) of approximately the same size and
computing CRC of each block in an interleaved manner, concatenating CRCs
of individual blocks in the end. A sample implementation is given in listing 5.
A tuned implementation of CrcWordByWordBlocks is capable of process-
ing data at 1.3-1.4 CPU cycles/byte on suciently large (64KB and more)
inputs, which is noticeably better that 2.1-2.2 CPU cycles/byte delivered by
word by word CRC computation. It is a good sign that it is a move in right
direction.
1The variant presented in this paper is more general than \slicing" described in [KB05].
Sample implementation given in listing 4 does not include one subtle optimization imple-
mented in [KB05] as it was found to be counter-productive.
131 Crc CrcWord(Word value ) f
2 Crc r e s u l t = 0 ;
3 // Unroll t h i s loop or l e t compiler do i t .
4 f o r ( i n t byte = 0 ; byte <s i z e o f (Word) / 8 ; ++byte ) f
5 r e s u l t ^= MulWordByXpowD [ byte ] [ ( Byte ) value ] ;
6 value > >= 8 ;
7g
8 return r e s u l t ;
9g
10 Crc CrcWordByWord(Word data , i n t n , Crc v , Crc u)
11 Crc crc = v ^ u ;
12 f o r ( i n t i = 0 ; i <n ; ++i )f
13 Crc WordCrc = CrcWord( crc ^ data [ i ] ) ;
14 i f ( s i z e o f ( Crc ) <= s i z e o f (Word) ) f
15 crc = WordCrc ;
16ge l s ef
17 crc > >= 8 ;
18 crc ^= WordCrc ;
19g
20g
21 return ( crc ^ u) ;
22g
23 void InitWordTables ( ) f
24 f o r ( i n t byte = 0 ; byte <s i z e o f (Word) / 8 ; ++byte ) f
25 // (K 1k )B + D = (W /81byte )8 + D = D8 + W8byte .
26 Crc m = XpowN(D 8 + s i z e o f (Word) 88byte ) ;
27 f o r ( i n t i = 0 ; i <256; ++i )f
28 MulWordByXpowD [ byte ] [ i ] =MultiplyUnnormalized ( i , 8 , m) ;
29g
30g
31g
Listing 4: Computing CRC word by word
141 // Processes N s t r i p e s of StripeWidth words each
2 // word by word , in an i n t e r l e a v e d manner .
3 Crc CrcWordByWordBlocks (Word data , Crc v , Crc u) f
4 a s s e r t (n % (N StripeWidth ) == 0) ;
5 // Use N l o c a l v a r i a b l e s instead of the array .
6 Crc crc [N ] ;
7 // I n i t i a l i z e the CRC value f o r each s t r i p e .
8 crc [ 0 ] = v ^ u ;
9 f o r ( i n t s t r i p e = 1 ; s t r i p e <N; ++s t r i p e )
10 crc [ i ] = 0 ^ u ;
11 // Compute each s t r i p e ' s CRC.
12 f o r ( i n t i = 0 ; i <StripeWidth ; ++i ) f
13 // Compute multiple CRCs in i n t e r l e a v e d manner .
14 Word buf [N ] ;
15 f o r ( i n t s t r i p e = 0 ; s t r i p e <N; ++s t r i p e ) f
16 buf [ i ] =
17 crc [ s t r i p e ] ^ data [ i + s t r i p e StripeWidth ] ;
18 i f (D >s i z e o f (Word) 8)f
19 crc [ s t r i p e ] > >= Ds i z e o f (Word) 8 ;
20ge l s ef
21 crc [ s t r i p e ] = 0 ;
22g
23g
24 f o r ( i n t byte = 0 ; byte <s i z e o f (Word) / 8 ; ++byte ) f
25 f o r ( i n t s t r i p e = 0 ; s t r i p e <N; ++s t r i p e ) f
26 crc [ s t r i p e ] ^=
27 MulWordByXpowD [ byte ] [ ( Byte ) buf [ s t r i p e ] ] ;
28 buf [ s t r i p e ] > >= 8 ;
29g
30g
31g
32 // Combine s t r i p e CRCs.
33 f o r ( i n t s t r i p e = 1 ; s t r i p e <N; ++s t r i p e ) f
34 crc [ 0 ] = ChangeStartingValue (
35 crc [ s t r i p e ] , StripeWidth , 0 , crc [ 0 ] ) ;
36g
37 return ( crc [ 0 ] ^ u) ;
38g
Listing 5: Processing non-overlapping blocks in parallel
15The drawbacks of this approach are obvious: it does not work well with
small inputs { the cost of CRC concatentation becomes a bottleneck, { and it
may be susceptible to false cache collisions caused by cache line aliasing.
If the cost of CRC concatenation was not a problem, cache pressure could be
mitigated with the use of very narrow stripes. The code in question, lines 33-37
of listing 5 which combine CRCs of individual stripes, iteratively computes
crc0(x) = crc k(x) +
crc0x8SmodP(x)
fork= 1; : : : ; N1 where NandSare the number and the width of the stripes
respectively. It may be rearranged as
crc0(x) =N1X
k=0
crcK1kx8kSmodP(x)
:
Explicit multiplication by x8kSmay be avoided by moving it into preset
tables
MulWordByXPowDk(n) = MulWordByXPowD( n)xkSmodP(x):
that are used to compute crc0
k(x) = crc k(x)x8kS, so that
crc0(x) =N1X
k=0crc0
k:
Unfortunately, this approach alone does not help because
1. It increases the memory footprint of MulWordByXPowD by factor of N.
Once the cumulative size of MulWordByXPowDktables exceeds the size
of L1 cache (32-64KB), the cost of memory access to multiplication table
data increases from 3-4 CPU cycles to 15-20, eliminating all performance
gains achieved by reducing the number of table operations.
2. It is still necessary to combine all Nvalues of crc kinto crc 0at the end of
the CRC computation.
4.11 Interleaved word-by-word CRC
4.11.1 Parallelizing CRC computation
Assume that input message Mis the concatenation of Kgroups gk, and each
group gkis concatenation of N W -bit long words:
M(x) =K1X
k=0gk(x)x(K1k)NW;
gk(x) =N1X
n=0mk;nx(N1n)W:
16Input message M(x) may be represented as
M(x) =K1X
k=0gk(x)x(K1k)NW=
=K1X
k=0 N1X
n=0mk;nx(N1n)W!
x(K1k)NW=
=N1X
n=0 K1X
k=0mk;nx(K1k)NW!
x(N1n)W=
=N1X
n=0Mn(x)x(N1n)W(14)
where
Mn(x) =K1X
k=0mk;nx(K1k)NW:
In other words, Mnis concatenation of n-thW-bit word from g0followed
by (N1)Wzero bits, then n-th word from g1followed by ( N1)Wzero bits,
etc., ending up with n-th word from last group gK1.
Appending ( N1)Wzero bits to Mnyields M0
n(x) =Mn(x)x(N1)W
which may be viewed as the concatenation of K NW -bit groups fk:
M0
n(x) =Mn(x)x(N1)W=K1X
k=0fk;nx(K1k)NW;
fk;n(x) =mk;n(x)x(N1)W;
so
M(x) =N1X
n=0Mn(x)x(N1n)W
=N1X
n=0M0
n(x)x(N1)Wx(N1n)W
=N1X
n=0M0
n(x)xnW: (15)
According to (3), vK;n(x) = CRC 0
M0
n(x); v0;n(x)
may be computed incre-
17mentally:
vk+1;n(x) = CRC 0
fk;n(x); vk;n(x)
=
= CRC 0
mk;n(x)x(N1)W; vk;n(x)
=
=
vk;n(x)xNW+mk;n(x)x(N1)WxD
modP(x) =
=
vk;n(x)xW+mk;n(x)xD
x(N1)WmodP(x) = (16)
= CrcWordN
mk;n(x); vk;n(x)
: (17)
This approach:
1. Creates Nindependent data ows: computation of vk;0; : : : ; v k;N1may
be performed truly in parallel. There are no contentions on a single data
source or destination like those the word-by-word CRC computation de-
scribed in section 4.9 suered from.
2. Input data is accessed sequentially. Therefore, the load on cache sub-
system and false cache collisions are minimal. Thus, the performance
bottlenecks of approach described in 4.10 are eliminated.
4.11.2 Combining individual CRCs
Once vK;n(x) = CRC 0
M0
n(x); v0;n(x)
are computed starting with
v0;0=v(x);
v0;n= 0; n1;
by denition (7) of CRC 0and relationship (15),
CRC 0
M(x); v(x)
= CRC 0 N1X
n=0M0
n(x)xnW; v(x)!
=
=N1X
n=0CRC 0
M0
n(x)xnW; v0;n(x)
=
=N1X
n=0CRC 0
M0
n(x); v0;n(x)
xnW=
=N1X
n=0vK;n(x)xnW: (18)
Even though this step is performed only once per input message, it still
requires ( N1) non-trivial multiplications modulo P(x) negatively aecting
the performance on small input messages. Also, (18) uses the multiplicative
inverse of xnWmodulo P(x) which does not exists when P(x) mod x= 0.
18There is more ecient and elegant solution. Assume that M(x) is followed
by one more group gK(x). Then
CRC 0
M(x)xNW+gK(x); v(x)
=
= CRC 0
gK(x);CRC 0
M(x); v(x)
=
=
CRC 0
M(x); v(x)
xNW+gK(x)xD
modP(x) =
= 
xNWN1X
n=0vK;n(x)xnW+xDN1X
n=0mK;n(x)x(N1n)W!
modP(x) =
= 
xWN1X
n=0vK;n(x)x(N1n)W+xDN1X
n=0mK;n(x)x(N1n)W!
modP(x)
=N1X
n=0
vK;n(x)xW+mK;n(x)xD
x(N1n)WmodP(x) = (19)
=N1X
n=0CRC 0
mK;n(x); vK;n(x)
x(N1n)WmodP(x): (20)
(20) may be implemented using formula (13) by setting v0
0= 0, and then for
n= 0; : : : ; N1 computing
v0
n+1(x) =
v0
n(x) +vK;n
xW+mK;nxD
modP(x)
= CRC 0
mK;n; v0
n(x) +vK;n
:
Alternatively, this step may be performed using the less ecient technique
described in section 4.8.
4.11.3 Ecient computation of individual CRCs
Given v(x), deg
v(x)
< D andm(x), deg
m(x)
< W ,
CrcWordN
m(x); v(x)
=
v(x)xW+m(x)xD
x(N1)WmodP(x)
may be implemented eciently utilizing the techniques described in sections
4.6, 4.8, and 4.9. When DW,
CrcWordN
m(x); v(x)
=
v(x)xW+m(x)xD
x(N1)WmodP(x) =
=
v(x)xWD+m(x)
x(N1)W+DmodP(x);
and may be implemented using the table-driven multiplication as described in
(13) except that the operand is multiplied by x(N1)W+Dinstead of xD. Like in
(8), explicit multiplication of v(x) byxWDis not required since D-normalized
19representation of v(x), viewed as a W-normalized representation, is equal to
v(x)xWD
.
Using the same technique as in formula (9), for DWlet
vH(x) =v(x)
xDW
; deg
vH(x)
< W;
vL(x) =v(x) mod xDW; deg
vL(x)
< DW;
so that v(x) =vL(x) +vH(x)xDW. Then,
CrcWordN
m(x); v(x)
=
=
v(x)xW+m(x)xD
x(N1)WmodP(x) =
=
vL(x) +vH(x)xDW
xW+m(x)xD
x(N1)WmodP(x) =
=
vL(x)xW+
vH(x) +m(x)
xD
x(N1)WmodP(x) =
=
vH(x) +m(x)
x(N1)W+DmodP(x)
+
+
vL(x)xW
x(N1)WmodP(x)
: (21)
Since deg
vH(x)+m(x)
< W , the rst summand of CrcWordN
m(x); v(x)
,

vH(x) +m(x)
x(N1)W+DmodP(x)
;
may be computed using the table-driven multiplication technique described in
(13) except that the operand is multiplied by xD+(N1)Winstead of xD.
Computation of the second summand of CrcWordN
m(x); v(x)
,

vL(x)xW
x(N1)WmodP(x)
;
is somewhat less intuitive. Since deg
vL(x)
< DW,

vL(x)xW
modP(x) =
vL(x)xW
;
and may be computed by shifting vL(x) byWbits. Additional multiplication
byx(N1)Wis accomplished by adding
vL(x)xW
, produced at step n < N1
of the algorithm described by formula (17), to the value of vk;n+1(x) which will
be additionally multiplied by x(N1)Was shown in formula (16).
Forn=N1, the value of
vL(x)xW
should be added to the value of
vk+1;n0(x) where n0= 0. For k < K , it will be multiplied by x(N1)Wduring
next round of parallel computation as shown in (16). For k=K,vk+1;n0(x)
will be multiplied by x(N1)Wduring CRC concatenation as shown in (19) since
n0= 0.
201 Crc CrcInterleavedWordByWord (
2 Worddata , i n t blocks , Crc v , Crc u) f
3 Crc crc [N+1] = f0g;
4 crc [ 0 ] = v ^ u ;
5 f o r ( i n t i = 0 ; i <N( blocks1) ; i += N)f
6 Word b u f f e r [N ] ;
7 // Load next N words and move overflow
8 // b i t s into " next " word .
9 f o r ( i n t n = 0 ; n <N; ++n)f
10 b u f f e r [N] = crc [ n ] ^ data [ i + n ] ;
11 i f (D >s i z e o f (Word) 8)
12 crc [ n+1] ^= crc [ n ] >>( s i z e o f (Word) 8) ;
13 crc [ n ] = 0 ;
14g
15 // Compute i n t e r l e a v e d word byword CRC.
16 f o r ( i n t byte = 0 ; byte <s i z e o f (Word) ; ++byte ) f
17 f o r ( i n t n = 0 ; n <N; ++n)f
18 crc [ n ] ^=
19 MulInterleavedWordByXpowD [ byte ] [ ( Byte ) b u f f e r [ n ] ] ;
20 b u f f e r [ n ] > >= 8 ;
21g
22g
23 // Combine crc [ 0 ] with delayed overflow b i t s .
24 crc [ 0 ] ^= crc [N ] ;
25 crc [N] = 0 ;
26g
27 // Process the l a s t N bytes and combine CRCs.
28 f o r ( i n t n = 0 ; n <N; ++n)f
29 i f (n != 0) crc [ 0 ] ^= crc [ n ] ;
30 Crc WordCrc = CrcOfWord( crc [ 0 ] ^ data [ i + n ] ) ;
31 i f (D >s i z e o f (Word) 8)f
32 crc [ 0 ] > >= Ds i z e o f (Word) 8 ;
33 crc [ 0 ] ^= WordCrc ;
34ge l s ef
35 crc [ 0 ] = WordCrc ;
36g
37g
38 return ( crc [ 0 ] ^ u) ;
39g
40 void InitInterleavedWordTables ( void ) f
41 f o r ( i n t byte = 0 ; byte <s i z e o f (Word) ; ++byte ) f
42 Crc m = XpowN(D 8 + Ns i z e o f (Word)88byte ) ;
43 f o r ( i n t i = 0 ; i <256; ++i )f
44 MulInterleavedWordByXpowD [ byte ] [ i ] =
45 MultiplyUnnormalized ( i , 8 , m) ;
46g
47g
48g
Listing 6: Interleaved, word by word CRC computation
215 Experimental results
The tests were performed using Intel Q9650 3.0GHz CPU, DDR2-800 memory
with 4-4-4-12 timing, and a motherboard with an Intel P45 chipset.
5.1 Testing methology
All tests were performed using random input data over various block sizes. The
code for all evaluated algorithms was heavily optimized. Tests were performed
on both aligned and non-aligned input data to ensure that misaligned inputs do
not carry performance penalty. CRC tables were aligned on 256-byte boundary.
Tests were performed with warm data and warm CRC tables: as shown in
[KB05], the footprint of CRC tables { as long as they t into L1 cache { is not
a major contributor to the performance.
Performance was measured in number of CPU cycles per byte of input data:
apparently, performance of CRC computation is bounded by performance of
CPU and its L1 cache latency. Spot testing of few other Intel and AMD CPU
models showed little variation in performance measured in CPU cycles per byte
despite substential dierences in CPU clock frequencies.
To minimize performance variations caused by interference with OS and
other applications (context switches, CPU migrations, CPU cache ushes, mem-
ory bus interference from other processes, etc.), the test applications were run
at high priority, each test was executed multiple times, and the minimum time
was measured. That allowed the tests to achieve repeatability within 1%.
5.2 Compiler comparison
Despite CRC code being rather straightforward, there were surprises (see tables
5 and 4).
On 64-bit AMD64 platform, Microsoft CL compiler (version 15.00.30729)
consistently and noticeably generated the fastest code that used general-purpose
integer arithmetics. For instance, CRC-64 and CRC-32 code generated by CL
was 1.24 times faster than the code generated by Intel's ICL 11.10.051, and
1.74 times faster than the code generated by GCC 4.5.0. A tuned, hand-written
inline assembler code for CRC-32 and CRC-64 for GCC was as fast as the code
generated by CL.
When it comes to arithmetics with the use of SSE2 intrinsic functions on
64-bit AMD64 platform, the code generated by GCC 4.5.0 consistenly outper-
formed the code generated by Microsoft and Intel compilers { by factor of 1.15
and 1.30 respectively. However, earlier versions of GCC did not produce ecient
SSE2 code either. For that reason, pre-4.5.0 versions of GCC use hand-written
inline assember code which was as fast as the code generated by GCC 4.5.0.
On 32-bit bit X86 platform, neither compiler was able to generate ecient
code (most likely because because the compilers could not overcome scarsity of
general-purpose registers). Performance of the code that used MMX intrinsic
22functions was better but still not as good as hand-written assember versions,
which were provided for all compilers.
The fastest code for 128-bit CRC on X86 platform was generated by GCC
4.5.0.
5.3 Choice of interleave level
Number of data streams processed by interleaved, word-by-word CRC computa-
tion described in section 4.11 should matter. Too few means underutilization of
available ALUs. Too many will increase the length of the main loop and stress
instruction decoders, and may cause splilling of registers containing hot data
(interleaved processing of Nwords of data uses at least (2 N+ 2) registers).
As table 3 shows, the optimal number of interleaved data streams on modern
Intel and AMD CPUs for integer arithmetics is either 3 or 4 (likely because they
all have exactly 3 ALUs). However, for SSE2 arithmetics on AMD64 platform
the optimal number of streams is 6 (3 on X86), which is quite counter-intuitive
result as it does not correlate with the number of available ALUs. Good old
performance mantra "you need to measure" still applies.
5.4 Performance of CRC algorithms
Average performance of best variants of CRC algorithms for 64-bit AMD64 and
32-bit X86 platforms processing 1KB, 2KB, . . . , 1MB inputs is given in tables
1 and 1 respectively. Proposed interleaved multiword CRC algorithm is 1.7-2.0
times faster that current state of the art \slicing".
As demonstrated in tables 7 and 6, interleaved word-by-word CRC described
in section 4.11, running at 1.2 CPU cycles/byte, is 1.8 times faster than 2.1 CPU
cycles/byte achieved by current state of the art word-by-word CRC algorithm
(\slicing") described in [KB05].
On 64-bit AMD64 platform, the best performance was achieved using 64-bit
reads and 64-bit tables for all variants of N-bit CRC for N64. In particular,
tables 7 and 6 clearly show that performance of 32-bit and 64-bit CRCs is nearly
identical. Consequently, there is no reason to favor CRC-32 over CRC-64 for
performance reasons.
The use of MMX on the 32-bit X86 platform allowed to utilize 64-bit tables
and 64-bit reads achieving 1.3 CPU cyles/byte. Neither compiler generated
ecient code using MMX intrinsic functions, so inline assembler was used.
With the use of SSE2 intrinsics on AMD64 architecture, 128-bit CRC may
be computed takes at 1.7 CPU cycles/byte using the new algorithm (see table
9), compared with 2.9 CPU cycles/byte achieved by word-by-word CRC compu-
tation (see table 8). On the 32-bit X86 architecture, the use of SSE2 intrinsics
and GCC 4.5.0 allowed the computation of 128-bit CRC at 2.1 CPU cycles/byte,
compared with 4.2 CPU cycles/byte delivered by word-by-word algorithm.
Given that MD5 computation takes 6.8-7.1 CPU cycles/byte and SHA-1
takes 7.6-7.9 CPU cycles per byte, CRCs are still the algorithm of choice for
data corruption detection.
23References
[Bla93] Richard Black. Fast CRC32 in software. http://www.cl.cam.ac.
uk/research/srg/bluebook/21/crc/crc.html , 1993.
[BW01] Florian Braun and Marcel Waldvogel. Fast incremental
CRC updates for IP over ATM networks. Technical Re-
port WUCS-01-08, Washington University in St. Louis, April
2001. Available at http://marcel.wanda.ch/Publications/
braun01fast-techreport.pdf .
[GGO+10] Vinodh Gopal, Jim Guilford, Erdinc Ozturk, Gil Wolrich, Wajdi
Feghali, Martin Dixon, and Deniz Karakoyunlu. Fast CRC computa-
tion for iSCSI polynomial using CRC32 instruction. Intel White Pa-
per 323405, February 2010. Available at http://download.intel.
com/design/intarch/papers/323405.pdf .
[Has01] M. A. Hasan. Ecient computation of multiplicative inverses for
cryptographic applications. In ARITH '01: Proceedings of the 15th
IEEE Symposium on Computer Arithmetic , page 66, Washington,
DC, USA, 2001. IEEE Computer Society.
[Hil79] John R. Hill. A table driven approach to cyclic redundancy check
calculations. SIGCOMM Comput. Commun. Rev. , 9(2):40{60, 1979.
[JK02] H. Michael Ji and Eeal Killian. Fast parallel crc algorithm and
implementation on a congurable processor. In ICC, volume 3, pages
1813{1817, April 2002.
[KB05] Michael E. Kounavis and Frank L. Berry. A systematic ap-
proach to building high performance, software-based, CRC gen-
erators. http://www.intel.com/technology/comms/perfnet/
download/CRC_generators.pdf , 2005.
[PB61] W.W. Peterson and D.T. Brown. Cyclic codes for error detection.
InIRE(1) , volume 49, pages 228{235, January 1961.
[Per83] Aram Perez. Byte-wise CRC calculations. IEEE Micro , 3(3):40{50,
1983.
[Sar88] Dilip V. Sarwate. Computation of cyclic redundancy checks via table
look-up. Commun. ACM , 31(8):1008{1013, 1988.
[Zem90] Pavel Zemtsov. Proprietary copy protection system. Personal com-
munication, August 1990.
24Table 1: CRC performance, AMD64 platform
Method Slicing1Multiword2Improvement
CRC-32 2:0831:164;51.79
CRC-64 2:0931:164;51.79
CRC-128 2:9141:684;61.73
Table 2: CRC performance, X86 platform
Method Slicing1Multiword2Improvement
CRC-32 2:5231:293;71.96
CRC-64 3:2831:293;72.55
CRC-128 4:1742:104;81.98
Average number of CPU cycles per byte processing 1KB, 2KB, . . . , 1MB inputs.
Warm data, warm tables.
1\Slicing" implements the algorithm described in section 4.9.
2\Multiword/ N"implements algorithm described in section 4.11 processing N
data streams in parallel in interleaved manner.
3Microsoft CL 15.00.30729 compiler, \-O2" ag.
4GCC 4.5.0 compiler, \-O3" ag.
5Multiword/ N= 4, hand-written inline assembler.
6Multiword/ N= 6, C++.
7Multiword/ N= 4, hand-written MMX inline assember.
8Multiword/ N= 3, C++.
Table 3: Interleaved multiword CRC: choosing the number of stripes N
CRC Platform N=2 N=3 N=4 N=5 N=6 N=7 N=8
CRC-649AMD64 1.42 1.23 1.17 1.46 2.08 2.59 2.73
CRC-12810AMD64 2.07 1.84 1.76 1.70 1.68 1.75 1.79
CRC-12810X86 2.56 2.10 2.46 2.61 2.52 2.62 2.57
Average number of CPU cycles per byte processing 1KB, 2KB, . . . , 1MB inputs.
Interleaved word-by-word CRC computation as described in section 4.11. Warm
data, warm tables.
9Microsoft CL 15.00.30729 compiler, AMD64 platform, C++ code.
10GCC 4.5.0 compiler, AMD64 platform, C++ code.
25Table 4: Compiler comparison: Multiword/4 64-bit CRC
Input size 64 256 1K 4K 16K 64K 256K 1M
GCC/C++ 2.30 2.10 2.04 2.03 2.03 2.05 2.05 2.07
ICL 2.19 1.62 1.49 1.46 1.45 1.45 1.46 1.46
CL 1.75 1.29 1.18 1.15 1.17 1.18 1.18 1.18
GCC/ASM 1.65 1.26 1.17 1.15 1.16 1.17 1.17 1.17
Table 5: Compiler comparison: Multiword/6 128-bit CRC
Input size 64 256 1K 4K 16K 64K 256K 1M
CL 4.08 2.94 2.64 2.62 2.59 2.59 2.59 2.62
ICL 3.48 2.50 2.25 2.08 2.02 2.00 2.00 2.02
GCC 2.90 1.93 1.85 1.72 1.65 1.63 1.63 1.63
Number of CPU cycles per byte. 128-bit CRC (CRC-128/IEEE polynomial) and
64-bit CRC (CRC-64-ECMA-182 polynomial) respectively. 64-bit platform, 64-
bit reads. Warm data, warm tables.
Microsoft CL 15.00.30729 compiler was used with \-O2" ag. Intel ICL
11.10.051 and GCC 4.5.0 were used with \-O3" ag.
01234567
CL-128
ICL-128
GCC-128
GCC-64
ICL-64
00.511.522.533.544.55
CL-128
ICL-128
GCC-128
GCC-64
ICL-64
CL-64
\Multiword/ N"implements algorithm described in section 4.11 processing N
data streams in parallel in interleaved manner.
26Table 6: CRC-32 performance
Input size 64 256 1K 4K 16K 64K 256K 1M
Sarwate 6.61 6.62 6.70 6.68 6.67 6.66 6.67 6.75
Black 5.44 5.46 5.47 5.48 5.47 5.46 5.47 5.53
Slicing 2.15 2.10 2.09 2.09 2.08 2.08 2.08 2.10
Blockword/3 2.27 2.14 2.15 2.13 2.13 1.55 1.39 1.31
Multiword/4 1.75 1.29 1.18 1.16 1.17 1.18 1.18 1.18
Number of CPU cycles per byte. 32-bit CRC (CRC-32C polynomial), 64-bit
platform, 64-bit tables, 64-bit reads (except Sarwate). Microsoft CL 15.00.30729
compiler. Warm data, warm tables.
01234567
Sarwate
Black
Slicing
Blockword/3
Multiword/4
\Sarwate" implements the algorithm described in section 4.6.
\Black" implements the algorithm described in section 4.8.
\Slicing" implements the algorithm described in section 4.9.
\Blockword/3" implements the algorithm described in section 4.10 with 3 stripes
of 15,376 bytes each.
\Multiword/4" implements the algorithm described in section 4.11 processing 4
data streams in parallel in interleaved manner.
27Table 7: CRC-64 performance
Input size 64 256 1K 4K 16K 64K 256K 1M
Sarwate 6.61 6.62 6.70 6.68 6.67 6.65 6.66 6.75
Black 5.44 5.46 5.47 5.47 5.47 5.47 5.47 5.53
Slicing 2.16 2.08 2.09 2.10 2.08 2.08 2.08 2.09
Blockword/3 2.27 2.14 2.15 2.13 2.13 1.59 1.41 1.33
Multiword/4 1.75 1.29 1.18 1.15 1.17 1.18 1.18 1.18
Number of CPU cycles per byte. 64-bit CRC (CRC-64-ECMA-182 polynomial),
64-bit platform, 64-bit tables, 64-bit reads (except Sarwate). Microsoft CL
15.00.30729 compiler. Warm data, warm tables.
00.511.522.53
Slicing
Blockword/3
Multiword/4
\Sarwate" implements the algorithm described in section 4.6.
\Black" implements the algorithm described in section 4.8.
\Slicing" implements the algorithm described in section 4.9.
\Blockword/3" implements the algorithm described in section 4.10 with 3 stripes
of 15,376 bytes each.
\Multiword/4" implements the algorithm described in section 4.11 processing 4
data streams in parallel in interleaved manner.
28Table 8: CRC-128 performance: Slicing CRC
Input size 64 256 1K 4K 16K 64K 256K 1M
CL/SSE2 4.02 3.81 4.01 4.05 4.13 4.18 4.20 4.24
ICL/SSE2 3.40 3.24 3.57 3.59 3.68 3.72 3.75 3.81
GCC/UINT 3.45 3.24 3.36 3.48 3.61 3.64 3.67 3.72
GCC/SSE2 2.67 2.48 2.63 2.79 2.97 2.99 2.99 3.03
Table 9: CRC-128 performance: Multiword CRC
Input size 64 256 1K 4K 16K 64K 256K 1M
GCC/UINT/3 3.83 3.02 3.04 3.01 3.00 2.98 2.98 3.00
CL/SSE2/5 4.08 2.56 2.43 2.25 2.20 2.19 2.18 2.20
ICL/SSE2/5 3.52 2.33 2.23 2.05 2.00 1.99 1.99 2.01
GCC/SSE2/6 2.90 1.93 1.85 1.72 1.65 1.63 1.63 1.63
Number of CPU cycles per byte. 128-bit CRC (CRC-128/IEEE polynomial),
64-bit platform, 128-bit tables, 64-bit reads. Warm data, warm tables.
All compilers were tested using SSE2 intrinsics (/SSE2 variants). GCC was also
tested using 128-bit integers provided by the compiler (GCC/UINT).
012345
CL/Slicing/SSE2
ICL/Slicing/SSE2
GCC/Slicing/UINT
GCC/Slicing/SSE2
GCC/Multiword/UINT/3
CL/Multiword/SSE2/5
ICL/Multiword/SSE2/5
GCC/Multiword/SSE2/6
\Slicing" implements algorithm described in section 4.9.
\Multiword/ N"implements algorithm described in section 4.11 processing N
data streams in parallel in interleaved manner. The optimal (for given compiler)
value of Nwas used.
29