An internal cache memory for a multi-stage motion estimation search in a hardware digital video encoder wherein the motion estimation search has a first predetermined number of search positions within a 32x32 pixel area to search per clock cycle of the encoder, comprising: a plurality of memory blocks, each memory block adapted to store pixel data from a reference search area wherein the plurality of memory blocks comprise memory blocks having a predetermined depth, and the plurality of memory blocks are arranged as four quadrant internal cache modules, wherein each quadrant internal cache module comprises 16 memory blocks in a 4x4 memory grid, wherein each row of the memory grid stores a 16 pixel wide line of pixels, and wherein the memory grid references four rows of the reference search area pixel data per unit of depth up to the predetermined depth; and a memory controller adapted to: store pixel data into the plurality of memory blocks as 16 pixel wide lines of pixels aligned to macroblock boundaries per clock cycle of the encoder; and read pixel data from the plurality of memory blocks as 16 pixel by 16 pixel blocks of pixels for each of the first predetermined number of search positions per clock cycle of the encoder, each quadrant internal cache providing a 16x16 pixel quadrant of the 32x32 pixel area required; wherein the memory controller is adapted to provide search position data from the plurality of memory blocks in 1 pixel or 4 pixel spaced search positions. a plurality of memory blocks, each memory block adapted to store pixel data from a reference search area wherein the plurality of memory blocks comprise memory blocks having a predetermined depth, and the plurality of memory blocks are arranged as four quadrant internal cache modules, wherein each quadrant internal cache module comprises 16 memory blocks in a 4x4 memory grid, wherein each row of the memory grid stores a 16 pixel wide line of pixels, and wherein the memory grid references four rows of the reference search area pixel data per unit of depth up to the predetermined depth; and a memory controller adapted to: store pixel data into the plurality of memory blocks as 16 pixel wide lines of pixels aligned to macroblock boundaries per clock cycle of the encoder; and read pixel data from the plurality of memory blocks as 16 pixel by 16 pixel blocks of pixels for each of the first predetermined number of search positions per clock cycle of the encoder, each quadrant internal cache providing a 16x16 pixel quadrant of the 32x32 pixel area required; wherein the memory controller is adapted to provide search position data from the plurality of memory blocks in 1 pixel or 4 pixel spaced search positions. store pixel data into the plurality of memory blocks as 16 pixel wide lines of pixels aligned to macroblock boundaries per clock cycle of the encoder; and read pixel data from the plurality of memory blocks as 16 pixel by 16 pixel blocks of pixels for each of the first predetermined number of search positions per clock cycle of the encoder, each quadrant internal cache providing a 16x16 pixel quadrant of the 32x32 pixel area required; wherein the memory controller is adapted to provide search position data from the plurality of memory blocks in 1 pixel or 4 pixel spaced search positions.