Integer pixel motion estimation system, motion estimation system for quarter-pixel luminance, motion estimation system for quarter-pixel chrominance, motion estimation system for combined luminance, motion estimation system for combined luminance and chrominance, and motion estimation system for quarter-pixel luminance and chrominance

A RAM_HIME used for integer pixel motion estimation by an IME stores integer pixel luminance data from a SDRAM while satisfying the conditions that improve efficiency in reading an extracted rectangular area. A RAM_HSME used for motion estimation of quarter-pixel accuracy by a SME stores partial quarter-pixel luminance data while satisfying the conditions that improve efficiency in obtaining a rectangular area after calculation by calculation. A RAM_HMEC used for chrominance data generation of quarter-pixel accuracy by a QPG stores integer pixel chrominance data from the SDRAM while satisfying the conditions that improve efficiency in obtaining rectangular areas after calculation by calculation.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to motion estimation systems that estimate motion of a moving image, and more specifically to a method of extracting an image in the course of motion estimation.

2. Description of the Background Art

As the latest international standard for moving image coding, H.264 was developed and has been put to practical use by the VCEG (Video Coding Experts Group) of the ITU-T (International Telecommunication Union Telecommunication Standardization Sector), an international standardization organization for telecommunications, together with the ISO/IEC MPEG (Moving Picture Experts Group).

H.264 achieves compression twice as efficiently as MPEG-2 or MPEG-4 with the same image quality, and is fit for a wide range of uses from a videoconference at low bit rates to HDTV (High Definition TeleVision).

In an H.264 encoder, image data is transferred for motion estimation among a ME (Motion Estimation) system, a MC (Motion Compensation) system, and a frame memory. A memory capable of performing moving image compression efficiently with small frame memory capacitance is disclosed in Japanese Patent Application Laid-Open No. 2004-222262, for example.

In Japanese Patent Application Laid-Open No. 2004-222262, improvements are made mainly to a frame memory, and not to an internal memory used in a motion estimation system. Thus high-speed motion estimation cannot be performed with a motion estimation system having a relatively simple circuit structure.

SUMMARY OF THE INVENTION

The present invention has been made to solve the aforementioned problem, and has an object to provide a motion estimation system capable of performing motion estimation at high speed while simplifying the circuit structure.

According to a first aspect of the invention, a motion estimation system includes: a prescribed number of storage sections for an integer pixel search that divide luminance data of integer-pixel accuracy in a prescribed image area and store the luminance data by prescribed unit as stored luminance data; and an integer pixel motion estimation section that extracts a prescribed reference extracted area based on a combination of the stored luminance data read from the prescribed number of storage sections for an integer pixel search, to perform motion estimation with integer-pixel accuracy, wherein in the prescribed number of storage sections for an integer pixel search, the stored luminance data corresponding to pixels adjacent in a horizontal direction and a vertical direction and four consecutive the stored luminance data corresponding to pixels adjacent in the horizontal direction are stored to vary among the prescribed number of storage sections for an integer pixel search.

The motion estimation system can efficiently read the prescribed reference extracted area, which is extracted based on a combination of the prescribed number of stored luminance data read from the prescribed number of storage sections for an integer pixel search, at an arbitrary position in the prescribed image area.

The improved efficiency of the extraction process of the prescribed reference extracted area allows the integer pixel motion estimation section to perform motion estimation of integer-pixel accuracy at high speed. Further, the improved efficiency reduces the load of doing calculations by the storage sections for an integer pixel search, thereby simplifying the circuit structure.

According to a second aspect of the invention, a motion estimation system for quarter-pixel luminance includes: a temporary storage section for luminance that stores luminance data of integer-pixel accuracy in a prescribed image area; a quarter-pixel generation section for luminance that performs a horizontal-direction interpolation process and a vertical-direction interpolation process based on the luminance data read from the temporary storage section for luminance to generate partial quarter-pixel luminance data, the partially quarter-pixel luminance data being luminance data of quarter-pixel accuracy only in one direction of the horizontal direction and vertical direction; a prescribed number of storage sections for a quarter-pixel search that divide the partial quarter-pixel luminance data and store the partial quarter pixel luminance data by prescribed unit as stored quarter-pixel luminance data; and a quarter-pixel motion estimation section that performs a prescribed calculation process based on a combination of the stored quarter-pixel luminance data read from the prescribed number of storage sections for a quarter-pixel search, to obtain a prescribed reference area after luminance calculation with quarter-pixel accuracy in both of the horizontal direction and vertical direction and perform motion estimation with quarter-pixel accuracy, the prescribed calculation process including an interpolation process in the other direction than the one direction to calculate luminance data of quarter-pixel accuracy in the other direction, wherein in the prescribed number of storage sections for a quarter-pixel search, the stored quarter-pixel luminance data corresponding to pixels adjacent in the other direction and four consecutive the stored quarter-pixel luminance data corresponding to pixels adjacent in units of integer pixels in the one direction are stored to vary among the prescribed number of storage sections for a quarter-pixel search.

The motion estimation system for quarter-pixel luminance can efficiently obtain the prescribed reference area after luminance calculation, which is obtained by a prescribed calculation process based on a combination of the prescribed number of stored quarter-pixel luminance data read from the prescribed number of storage sections for a quarter-pixel search, at an arbitrary position in the prescribed image area.

The improved efficiency in obtaining the prescribed reference area after luminance calculation allows the quarter-pixel motion estimation section to perform motion estimation of quarter-pixel accuracy at high speed. Further, the improved efficiency reduces the load of doing calculations by the quarter-pixel motion estimation section, thereby simplifying the circuit structure.

According to a third aspect of the invention, a motion estimation system for quarter-pixel chrominance includes: a prescribed number of storage sections for a chrominance that divide chrominance data of integer-pixel accuracy in a prescribed image area and store the data in units of stored chrominance data; and a quarter-pixel generation section for chrominance that performs a calculation process based on reference position information of quarter-pixel accuracy in accordance with a combination of the stored chrominance data read from the prescribed number of storage sections for a chrominance, to generate a prescribed reference area after chrominance calculation of quarter-pixel accuracy as estimation result chrominance data, wherein in the prescribed number of storage sections for a chrominance, the stored chrominance data corresponding to pixels adjacent in a horizontal direction are stored by being superimposed in units of a first number of pixels, and a second number of consecutive the stored chrominance data corresponding to pixels adjacent in a vertical direction are stored to vary among the prescribed number of storage sections for a chrominance.

The motion estimation system for quarter-pixel chrominance can efficiently obtain the reference area after chrominance calculation, which is obtained by calculation based on a combination of the stored chrominance data read from the prescribed number of storage sections for a chrominance, at an arbitrary position in the prescribed image area.

The improved efficiency in obtaining the reference area after chrominance calculation allows the quarter-pixel generation section for chrominance to perform motion estimation of quarter-pixel accuracy at high speed with respect to chrominance data. Further, the improved efficiency reduces the load of doing calculations by the quarter-pixel generation section for chrominance, thereby simplifying the circuit structure.

According to a fourth aspect of the invention a motion estimation system for combined luminance includes: the motion estimation system according to the first aspect; and the motion estimation system for quarter-pixel luminance according to the second aspect, wherein the integer pixel motion estimation section in the motion estimation system outputs a reference position signal of integer-pixel accuracy as a motion estimation result, and the quarter-pixel motion estimation section in the motion estimation system for quarter-pixel luminance performs motion estimation of quarter-pixel accuracy with a reference position instructed by the reference position signal as a base point.

Due to the combination of the motion estimation system according to the first aspect and the motion estimation system for quarter-pixel luminance according to the second aspect, the motion estimation system for combined luminance can perform motion estimation based on the luminance data of quarter-pixel accuracy at even higher speed while reducing the circuit structure.

According to a fifth aspect of the invention, a motion estimation system for combined luminance and chrominance includes: the motion estimation system according to the first aspect; the motion estimation system for quarter-pixel luminance according to the second aspect; and the motion estimation system for quarter-pixel chrominance according to the third aspect, wherein the integer pixel motion estimation section in the motion estimation system outputs a reference position signal of integer-pixel accuracy as a motion estimation result, the quarter-pixel motion estimation section in the motion estimation system for quarter-pixel luminance performs motion estimation of quarter-pixel accuracy with a reference position instructed by the reference position signal as a base point, to output estimation result luminance data of quarter-pixel accuracy, and the reference position information used by the motion estimation system for quarter-pixel chrominance includes reference position information on the estimation result luminance data of quarter-pixel accuracy.

Due to the combination of the motion estimation system according to the first aspect, the motion estimation system for quarter-pixel luminance according to the second aspect and the motion estimation system for quarter-pixel chrominance according to the third aspect, the motion estimation system for combined luminance and chrominance can perform motion estimation based on the luminance data and chrominance data of quarter-pixel accuracy at even higher speed while reducing the circuit structure.

According to a sixth aspect of the invention, a motion estimation system for quarter-pixel luminance and chrominance includes: a prescribed number of storage sections for a quarter-pixel search that divide luminance data and chrominance data of integer-pixel accuracy in a prescribed image area and store the luminance data and chrominance data by prescribed unit as stored luminance data and stored chrominance data, respectively; and a quarter-pixel motion estimation section that performs a prescribed calculation process based on a combination of the stored luminance data read from the prescribed number of storage sections for a quarter-pixel search, to obtain a prescribed reference area after luminance calculation with quarter-pixel accuracy in both a horizontal direction and a vertical direction and perform motion estimation thus obtaining estimation result luminance data, and generates a prescribed reference area after chrominance calculation of quarter-pixel accuracy as estimation result chrominance data based on a combination of the stored chrominance data read from the prescribed number of storage sections for a quarter-pixel search, the prescribed reference area after chrominance calculation corresponding to position information of quarter-pixel accuracy of the estimation result luminance data, wherein in the prescribed number of storage sections for a quarter-pixel search, the stored luminance data corresponding to pixels adjacent in the horizontal direction are stored by being superimposed in units of a first number of pixels, a second number of consecutive the stored luminance data corresponding to pixels adjacent in the vertical direction are stored to vary among the prescribed number of storage sections for a quarter-pixel search, and the stored chrominance data corresponding to pixels adjacent in the horizontal direction and a third number of consecutive the stored chrominance data corresponding to pixels adjacent in the vertical direction are stored to vary among the prescribed number of storage sections for a quarter-pixel search.

The motion estimation system for quarter-pixel luminance and chrominance can efficiently obtain the reference area after luminance calculation and the reference area after chrominance calculation, which are obtained by calculations based on combinations of the stored luminance data and stored chrominance data read from the prescribed number of storage sections for a quarter-pixel search, at an arbitrary position in the prescribed image area.

The improved efficiency in obtaining the reference area after luminance calculation and the reference area after chrominance calculation allows the motion estimation system for quarter-pixel luminance and chrominance to perform motion estimation of quarter-pixel accuracy at high speed. Further, the improved efficiency reduces the load of doing calculations by the quarter-pixel motion estimation section, thereby simplifying the circuit structure.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1is a block diagram of a motion estimation system and its peripheral structure according to a first preferred embodiment of the present invention. As shown, a ME2, a motion estimation system for combined luminance and chrominance, receives luminance data D1and chrominance data D2from a SDRAM1, to output luminance and chrominance data D3as a motion estimation result to a MC3. In an H.264 encoder, the SDRAM1corresponds to a frame memory, the ME2to a motion estimation system, and the MC3to a motion compensation system.

The SDRAM1stores the luminance data D1and the chrominance data D2of integer-pixel accuracy. The RAM_HIME21, an integer pixel motion estimation SRAM with respect to luminance data (storage section for an integer pixel search), stores the integer pixel luminance data D1from the SDRAM1while satisfying the conditions described later. The IME11, an integer pixel motion estimation section with respect to luminance data, performs an integer pixel search process to make estimation with integer-pixel accuracy by using the luminance data D1stored in the RAM_HIME21, to output a reference position signal S11indicative of a reference position of estimation result luminance data to the SME13.

The RAM_HMEY22, an integer pixel storage SRAM for a quarter-pixel search with respect to luminance data (temporary storage section for luminance), stores the luminance data D1from the SDRAM1while satisfying the conditions described later. The QPG12, a quarter-pixel generation section for luminance, performs a quarter-pixel generation process for luminance described later by using the luminance data D1stored in the RAM_HMEY22, to output partial quarter-pixel luminance data S12in which integer pixel data is added with (horizontal direction, vertical direction, oblique direction) half-pixel data and (vertical direction) quarter-pixel data to the RAM_HSME23.

The RAM_HSME23, a partial quarter-pixel storage SRAM (storage section for a quarter-pixel search), stores the partial quarter-pixel luminance data S12while satisfying the conditions described later. The SME13, a quarter-pixel motion estimation section with respect to luminance data, conducts a pixel search of quarter-pixel accuracy in a range of {−0.75˜+0.75} in a horizontal direction and a vertical direction from the reference position instructed by the reference position signal S11. Namely, the SME13performs a quarter-pixel search process described later by using the partial quarter-pixel luminance data S12stored in the RAM_HSME23to output luminance data S13indicative of a quarter-pixel search result (estimation result luminance data), and outputs a reference position signal S13pinstructing a quarter-pixel reference position of the luminance data S13(reference position information) to the QPG14.

The RAM_HMEC24, a motion estimation SRAM with respect to chrominance data (storage section for a chrominance), stores the integer pixel chrominance data D2from the SDRAM1while satisfying the conditions described later. The QPG14, a quarter-pixel generation section for chrominance, performs a quarter-pixel generation process for chrominance described later based on the reference position signal S13pby using the chrominance data D2stored in the RAM_HMEC24, to generate chrominance data S14(estimation result chrominance data) of quarter-pixel accuracy corresponding to the luminance data S13. The luminance data S13and the chrominance data S14form the luminance and chrominance data D3.

(Integer Pixel Motion Estimation System with Respect to Luminance Data)

The RAM_HIME21and the IME11shown inFIG. 1form an integer pixel motion estimation system with respect to luminance data.

FIG. 2is a schematic diagram of an original image area30stored in the RAM_HIME21. In the diagram, the n (pixel)×m (line) original image area30(prescribed image area) stores an arbitrary 8 (pixel)×4 (line) extracted rectangular area42(prescribed reference extracted area) in an extractable manner.FIG. 2is an illustration where n=40 and m=40.

FIGS. 3A to 3Dillustrate examples of an obtained rectangular area of integer-pixel (position) accuracy of luminance data used for motion estimation. An 8×8 rectangular area31can be obtained from two 8×4 extracted rectangular areas42as shown inFIG. 3A, or an 8×16 rectangular area32can be obtained from four extracted rectangular areas42as shown inFIG. 3B, or a 16×8 rectangular area33can be obtained from four extracted rectangular areas42as shown inFIG. 3C, or a 16×16 rectangular area34can be obtained from eight extracted rectangular areas42as shown inFIG. 3D.

The RAM_HIME21includes four RAM_HIMEs21ato21d, and stores the luminance data D1while satisfying the conditions indicated below. The premise is that the luminance data D1is read from the SDRAM1in units of 8 (pixel)×2 (line). For the purpose of illustration, one of the RAM_HIMEs21ato21dwill hereafter be indicated as a RAM_HIME21x.

(1) A storage unit of stored luminance data per address of the RAM_HIME21xis 8×2.

(2) The 8×2 stored luminance data corresponding to pixels adjacent in vertical and horizontal directions are stored to vary among the RAM_HIMEs21ato21d.

(3) Four consecutive 8×2 stored luminance data corresponding to pixels adjacent in a horizontal direction are stored to vary among the RAM_HIMEs21ato21d.

(4) Reading is performed after all data of the original image area30are stored in the RAM_HIMEs21ato21dfrom the SDRAM1.

(5) Reading is performed simultaneously from the four RAM_HIMEs21ato21d, to obtain a 16×4 or 32×2 read rectangular area in one cycle (by one reading).

(6) For the 16×4 read rectangular area, an 8×4 rectangular area is extracted as an extracted rectangular area.

(7) For the 32×2 read rectangular area, a 16×2 rectangular area is extracted as an extracted rectangular area.

FIGS. 4A to 4Cillustrate a process of extracting an 8×4 extracted rectangular area. As shown inFIG. 4A, read luminance data D21to D24are obtained from the RAM_HIMEs21ato21d. Then as shown inFIG. 4B, a 16×4 read rectangular area41can be obtained from the read luminance data D21to D24to satisfy the above condition (2). Then as shown inFIG. 4C, an arbitrary 8×4 rectangular area in the read rectangular area41can be extracted as the extracted rectangular area42.

FIGS. 5A and 5Bare schematic diagrams of a specific storage example in the RAM_HIME21x. In the diagrams, “i” of luminance data “Yij_k” indicates a horizontal position in units of 8 pixels, “j” indicates a vertical position in units of 8 lines, and “k” indicates a line position in units of 2 lines. As shown inFIG. 5A, the original image area30has a 40×40 integer pixel area. Meanwhile, as shown inFIG. 5B, the luminance data D1is stored to be distributed among the RAM_HIMEs21ato21dso that the RAM_HIMEs21ato21dstore the luminance data in units of 8×2 per address, and a 16×4 or 32×2 rectangular area can be extracted in units of 8×2 by one reading operation from the original image area30by satisfying the above conditions (2) and (3). Since 8 bits are allocated per pixel, 128-bit (=8×8×2) capacitance is allocated per address as shown inFIG. 5B.

By way of example, assume that the read rectangular area41(Y10—3, Y20—3, Y11—0, Y21—0) is selected to obtain the extracted rectangular area42as shown inFIG. 5A. As the RAM_HIME21astores “Y20—3” at an address number11, the RAM_HIME21bstores “Y11—0” at an address number2, the RAM_HIME21cstores “Y21—0” at an address number12, and the RAM_HIME21dstores “Y10—3” at an address number1, 8×2 luminance data at these addresses are read from the RAM_HIMEs21ato21d, to obtain the 16×4 read rectangular area41. After that, the arbitrary 8×4 extracted rectangular area42can be extracted from the read rectangular area41.

FIGS. 6A to 6Cillustrate a process of extracting a 16×2 extracted rectangular area. As shown inFIG. 6A, the read luminance data D21to D24are obtained from the RAM_HIMEs21ato21d. Then as shown inFIG. 6B, a 32×2 read rectangular area43can be obtained from the read luminance data D21to D24to satisfy the above condition (3). Then as shown inFIG. 6C, an arbitrary 16×2 rectangular area in the read rectangular area43can be extracted as an extracted rectangular area44.

FIG. 7is a schematic diagram of a (first) example of extracting a 16×16 macro block. The premise is that the extraction process of the extracted rectangular area42shown inFIG. 4Cis referred to as a main extraction process, and the extraction process of the extracted rectangular area44shown inFIG. 6Cas a sub-extraction process.

When a beginning line position pp of a rectangular area34amatches with a first line-1position p1of two lines of luminance data stored at one address in the RAM_HIME21xas shown inFIG. 7, the 16×16 rectangular area34acan be obtained from eight extracted rectangular areas42without fail, and thus can be read in eight cycles by eight main extraction processes.

FIG. 8is a schematic diagram of a (second) example of extracting a 16×16 macro block. When the beginning line position pp of a rectangular area34bmatches with a second line-1position p2of two lines of luminance data stored at one address in the RAM_HIME21xas shown inFIG. 8, the 16×16 rectangular area34bneeds to be read in nine cycles by eight main extraction processes and one sub-extraction process.

In this manner an arbitrary 8×4 extracted rectangular area can be read in one to two cycles and a 16×16 macro block can be read in eight to nine cycles from the RAM_HIME21x. The result is that the IME11in the subsequent stage reads the 16×16 macro block at high speed by changing the search pixel position in units of integers, thus performing motion estimation at high speed. The IME11performs the motion estimation itself by an existing method, and outputs the reference position signal S11of integer-pixel accuracy which is the estimation result to the SME13. The actual motion estimation itself, which is performed in units of 8×8 integer pixels in the 16×16 macro block, is not substantially relevant to the characteristics of the present invention and is thus not described in detail.

As described above, in the integer pixel motion estimation system with respect to luminance data including the RAM_HIME21and the IME11, the four RAM_HIMEs21ato21dstore the stored luminance data D1while satisfying the above conditions (1) to (7).

Accordingly, the arbitrary extracted rectangular area42(44) in the original image area30can be extracted by the combination of the four read luminance data D21to D24read from the RAM_HIMEs21ato21dby one or two reading operations, with improved efficiency in reading the extracted rectangular area42.

The improved efficiency of the extraction process of the extracted rectangular area42allows the IME11to perform motion estimation of integer-pixel accuracy at high speed. Further, the improvement in reading efficiency reduces the load of doing calculations by the IME11, thereby simplifying the circuit structure.

The RAM_HMEY22, the QPG12, the RAM_HSME23, and the SME13shown inFIG. 1form a motion estimation system for quarter-pixel luminance with respect to luminance data.

FIG. 9is a schematic diagram of an original image area35stored in the RAM_HMEY22. In the diagram, the n (pixel) ×m (line) original image area35stores an arbitrary 16 (pixel)×1 (line) extracted rectangular area46in an extractable manner.FIG. 9is an illustration where n=40 and m=40.

The RAM_HMEY22serving as a temporary storage section for luminance includes two RAM_HMEYs22aand22b, and stores the luminance data D1while satisfying the conditions indicated below. The premise is that the luminance data D1is read from the SDRAM1in units of 8 (pixel)×2 (line). For the purpose of illustration, one of the RAM_HMEYs22aand22bwill hereafter be indicated as a RAM_HMEY22x.

(1) A storage unit of stored luminance data per address of the RAM_HMEY22xis 16×1.

(2) The 8×2 luminance data read from the SDRAM1is divided into two 8×1 data and written simultaneously to the RAM_HMEYs22aand22b, respectively.

(3) The 16×1 stored luminance data corresponding to pixels adjacent in a horizontal direction are stored to vary between the RAM_HMEYs22aand22b.

(4) Reading is performed after all data of the original image area35are stored in the RAM_HMEYs22aand22bfrom the SDRAM1.

(5) Reading is performed simultaneously from the two RAM_HMEYs22aand22b, to obtain a 32×1 read rectangular area in one cycle.

(6) For the 32×1 read rectangular area, a 16×1 rectangular area is extracted as an extracted rectangular area.

FIGS. 10A to 10Cillustrate a process of extracting a 16×1 extracted rectangular area. As shown inFIG. 10A, read luminance data D25and D26are obtained from the RAM_HMEYs22aand22b. Then as shown inFIG. 10B, a 32×1 read rectangular area45can be obtained from the read luminance data D25and D26by one reading operation to satisfy the above condition (3). Then as shown inFIG. 10C, an arbitrary 16×1 rectangular area in the read rectangular area45can be extracted as the extracted rectangular area46.

(Quarter-Pixel Generation Process for Luminance (QPG12))

FIGS. 11A and 11Bare schematic diagrams of integer pixel reading by the QPG12in the course of the quarter-pixel generation process for luminance. The QPG12reads integer pixels sequentially from 0 through 39 lines from the 40×40 original image area35stored in the RAM_HMEY22shown inFIG. 11Aby shifting the horizontal position of the extracted rectangular area46in units of 8 or 9 pixels as shown inFIGS. 11A and 11B. The QPG12then generates the partial quarter-pixel luminance data S12in which an integer pixel in a quarter-pixel generation area37is added with a half-interpolation pixel62and a quarter-interpolation pixel64in a vertical direction from a quarter-pixel calculation area36.

FIGS. 12A to 12Dare schematic diagrams of processing contents of the QPG12. The luminance data of the 16×1 extracted rectangular area46shown inFIG. 12Aextracted from the RAM_HMEY22is stored in a 15×1 lead register shown inFIG. 12B. Since the partial quarter-pixel luminance data S12can be generated with 14 to 15 integer pixels in a horizontal direction, 15×1 pixel data in the 16×1 extracted rectangular area46is stored selectively in the lead register50.

As shown inFIG. 12B, the data stored in the lead register50is shifted to a 15×1 line buffer51upon reading of luminance data of the newly extracted rectangular area46, and subsequently shifted to 15×1 line buffers52to56in this order upon reading of luminance data of the newly extracted rectangular area46. Six integer pixels adjacent in a vertical direction can thus be stored in the line buffers51to56(buf0to buf5).

FIG. 13illustrates an example of generating half-interpolation pixels by the “6 tap filter”. As shown, one half-interpolation pixel62a(H) can be obtained from 6 integer pixels61ato61f(I) adjacent in a horizontal direction, and one half-interpolation pixel62b(V) can be obtained from 6 integer pixels61gto611(I) adjacent in a vertical direction. Likewise, one half-interpolation pixel62c(S) can be obtained from 6 half-interpolation pixels62a(H) adjacent in a vertical direction.

Referring back toFIG. 12C, a quarter-interpolation pixel64a(iv) or a quarter-interpolation pixel64c(vi) is obtained from the integer pixel61(I) and the half-interpolation pixel62b(V) adjacent in a vertical direction (2 tap filter), and a quarter-interpolation pixel64b(hs) or a quarter-interpolation pixel64d(sh) is obtained from the half-interpolation pixel62a(H) and the half-interpolation pixel62c(S) adjacent in a vertical direction. The QPG12does not generate quarter-interpolation pixels69ato69hin consideration of memory efficiency of the RAM_HSME23in the subsequent stage.

In each of the 0/4 to 3/4 lines, 27 pixels of information is obtained by the 6 tap filter processes and 2 tap filter processes based on the stored data in the line buffers51to56. However, since the half-interpolation pixel62a, the quarter-interpolation pixel64b, the half-interpolation pixel62c, and the quarter-interpolation pixel64dare obtained from a sixth interpolation, only 7 to 8 pixels will be obtained effectively.

The 27 or 29 pixels of luminance data thus obtained are divided into write luminance data D11and write luminance data D12, and written in two different RAM_HSMEs23, respectively.

The RAM_HSME23includes eight RAM_HSMEs23ato23h, and stores the partial quarter-pixel luminance data S12generated by the QPG12in units of quarter-pixels so that 4 lines in a vertical direction and 16 pixels in a horizontal direction vary among the RAM_HSMEs23ato23h.

FIG. 14is a schematic diagram of an original pixel area (original image area)70stored in the RAM_HSME23. In the diagram, the n*2 (pixel)×m*4 (line) original pixel area70stores a 16 (pixel)×4 (line) extracted rectangular area47for calculation (extracted area for luminance data calculation) in an extractable manner. After a horizontal-direction quarter-pixel generation process described later, an 8×4 rectangular area48after calculation (reference area after luminance calculation) can be obtained with a quarter-pixel accuracy position from the extracted rectangular area47for calculation.

FIGS. 15A to 15Dillustrate examples of an obtained rectangular area of luminance data used for motion estimation of quarter-pixel accuracy. An 8×8 rectangular area71can be obtained from two rectangular areas48after calculation as shown inFIG. 15A, or an 8×16 rectangular area72can be obtained from four rectangular areas48after calculation as shown inFIG. 15B, or a 16×8 rectangular area73can be obtained from four rectangular areas48after calculation as shown inFIG. 15C, or a 16×16 rectangular area74can be obtained from eight rectangular areas48after calculation as shown inFIG. 15D, all with quarter-pixel position accuracy.

The RAM_HSME23includes the eight RAM_HSMEs23ato23h, and stores the partially quarter-pixel luminance data S12while satisfying the conditions indicated below. For the purpose of illustration, one of the RAM_HSMEs23ato23hwill hereafter be indicated as a RAM_HSME23x.

(1) A storage unit of stored partial quarter-pixel luminance data per address of the RAM_HSME23xis 16×1. The four kinds of lines (seeFIG. 12C) indicated below store the data collectively:

(2) Writing is performed simultaneously in two or three RAM_HSMEs23xby two-way or three-way split in units of 27×1 or 29×1 (two RAM_HSMEs23xcan store a maximum of 32×1).

(3) The 16×1 stored partially quarter-pixel luminance data corresponding to pixels adjacent in a horizontal direction are stored to vary among the RAM_HSMEs23ato23h.

(4) Four consecutive 16×1 stored partially quarter-pixel luminance data corresponding to pixels adjacent in a vertical direction in units of integers are stored to vary among the RAM_HSMEs23ato23h.

(5) Reading is performed after all data of the original pixel area70are stored in the RAM—HSMEs 23ato23h.

(6) Reading is performed simultaneously from the eight RAM_HSMEs23ato23h, to obtain a 32×4 read rectangular area in one cycle with quarter-pixel position accuracy.

(7) A 16×4 rectangular area obtained by using the 32×4 read rectangular area is extracted as the extracted rectangular area47for calculation in units of quarter-pixels.

(8) The 16×4 extracted rectangular area47for calculation is subjected to a horizontal-direction half-interpolation calculation (2 tap filter) which is a prescribed calculation process, to further subject the partial quarter-pixel luminance data S12to quarter-pixel interpolation in a horizontal direction, thus obtaining an 8×4 rectangular area with quarter-pixel position accuracy in horizontal and vertical directions as the rectangular area48after calculation.

FIGS. 16A to 16Dillustrate a process of extracting an 8×4 extracted rectangular area. As shown inFIG. 16A, read luminance data D31to D38are obtained from the RAM_HSMEs23ato23h. Then as shown inFIG. 16B, a 32×4 read rectangular area58can be obtained from the read luminance data D31to D38to satisfy the above conditions (1), (3) and (4). Then as shown inFIG. 16C, an arbitrary 16×4 rectangular area in the read rectangular area58can be extracted as the extracted rectangular area47for calculation. Then as shown inFIG. 16D, the 8×4 rectangular area48after calculation can be obtained by performing a horizontal-direction 2 tap filter process using the luminance data of the extracted rectangular area47for calculation.

FIG. 17illustrates a method of calculating the rectangular area48after calculation by the SME13. It is assumed in the diagram that the extracted rectangular area47for calculation in the 0/4 line was obtained. The extracted rectangular area47for calculation in this case has a 16×4 structure including only the integer pixels61(I) and the half-interpolation pixels62a(H). Namely, 4 lines of the total of 16 pixels including 8 integer pixels61and 8 half-interpolation pixels62aform the extracted rectangular area47for calculation.

It is now assumed that the rectangular area48after calculation including the quarter-interpolation pixels69a(see “ih” inFIG. 12C) in the horizontal-direction 1/4 line is obtained from the extracted rectangular area47for calculation. In this process, in each of the 4 lines, each of (8) pairs of the integer pixel61and the half-interpolation pixel62ato the right of the integer pixel61is subjected to half-interpolation to obtain 8 quarter-interpolation pixels69a. Ultimately, the rectangular area48after calculation including the 8×4 quarter-interpolation pixels69acan be obtained.

The SME13performs motion estimation of quarter-pixel accuracy based on the rectangular area48after calculation to output the luminance data S13that defines the rectangular area48after calculation obtained as a final estimation result to the MC3, and outputs the reference position signal S13pindicative of a reference position of quarter-pixel accuracy of the rectangular area48after calculation to the QPG14.

In the motion estimation system for quarter-pixel luminance with respect to luminance data including the RAM_HMEY22, the QPG12, the RAM_HSME23and the SME13, the eight RAM_HSMEs23ato23hstore the stored quarter-pixel luminance data while satisfying the above conditions (1) to (8).

Accordingly, the extracted rectangular area47for calculation capable of calculating the arbitrary rectangular area48after calculation in the original pixel area70can be extracted by the combination of the eight read luminance data D31to D38read from the RAM_HSMEs23ato23hby one or reading operation, with improved efficiency in obtaining the rectangular area48after calculation.

The improved efficiency in obtaining the rectangular area48after calculation allows the SME13to perform motion estimation of quarter-pixel accuracy at high speed. Further, the improved efficiency reduces the load of doing calculations by the SME13, thereby simplifying the circuit structure.

As has been described, the ME2includes a motion estimation system for combined luminance. The system includes the motion estimation system having the RAM_HIME21and the IME11, and the motion estimation system for quarter-pixel luminance having the RAM_HMEY22, the QPG12, the RAM_HSME23and the SME13.

Thus the SME13can perform motion estimation of quarter-pixel accuracy with an integer pixel reference position instructed by the reference position signal S11as a base point, thereby performing motion estimation based on the luminance data of quarter-pixel accuracy at even higher speed while simplifying the circuit structure.

The RAM_HMEC24and the QPG14form a motion estimation system for quarter-pixel chrominance with respect to chrominance data.

FIGS. 18A and 18Bare schematic diagrams of an original pixel area (original image area)38U and an original pixel area (original image area)38V stored in the RAM_HMEC24. InFIG. 18A, the n (pixel)×m (line) original pixel area38U stores a 5 (pixel)×5 (line) rectangular area39U for calculation (extracted area for chrominance data calculation) in an extractable manner. InFIG. 18B, likewise, the n (pixel)×m (line) original pixel area38V stores a 5 (pixel)×5 (line) rectangular area39V for calculation (extracted area for chrominance data calculation) in an extractable manner.FIGS. 18A and 18Bare illustrations where n=40 and m=40.

FIGS. 19A to 19Dare schematic diagrams of storage areas of the RAM_HMEC24. As shown, chrominance pixels (U, V) in a horizontal direction are shifted in units of 4 pixels, to store 8 chrominance pixels U and 8 chrominance pixels V at one address.

As shown inFIG. 19A, a storage area (0) stores chrominance pixels (U, V) of 0 to 7 pixels in a horizontal position, with a U pixel and a V pixel being paired, at one address in each of a 0 line to a (m−1) line. As shown inFIG. 19B, a storage area (1) stores chrominance pixels (U, V) of 4 to 11 pixels in a horizontal position, with a U pixel and a V pixel being paired, at one address in each of a 0 line to a (m−1) line. As shown inFIG. 19C, a storage area (2) stores chrominance pixels (U, V) of 8 to 15 pixels in a horizontal position, with a U pixel and a V pixel being paired, at one address in each of a 0 line to a (m−1) line. Subsequently, as shown inFIG. 19D, a storage area (k) similarly stores chrominance pixels (U, V) of (k×4) to {(k×4)+7} pixels in a horizontal position, with a U pixel and a V pixel being paired, at one address in each of a 0 line to a (m−1) line. When the original pixel areas38U and38V shown inFIGS. 18A and 18Bhave a 20×20 structure (n=20, m=20), k is equal to 4 inFIG. 19D.

The RAM_HMEC24includes five RAM_HMECs24ato24e, and stores the chrominance data D2so that the chrominance pixels of 5 consecutive lines in a vertical direction in the storage area (i(i=one of 1 to k)) certainly vary among the RAM_HMECs24ato24e.

The five RAM_HMECs24ato24estore the chrominance data D2while satisfying the conditions indicated below. For the purpose of illustration, one of the RAM_HMECs24ato24ewill hereafter be indicated as a RAM_HMEC24x.

(1) A storage unit per address of the RAM_HMEC24xis 16×1(2×(8×1)). Eight chrominance pixels U and8chrominance pixels V are stored. Namely, a storage unit of stored chrominance data is 8×1 for each of the U and V pixels.

(2) Eight chrominance pixels U and 8 chrominance pixels V are stored at one address while superimposing four pixels of the stored chrominance data in each of the U and V pixels in a horizontal direction.

(3) Five consecutive 16×1(2×(8×1)) chrominance data corresponding to pixels adjacent in a vertical direction (8×1 stored chrominance data in each of the U and V pixels) are stored to vary among the RAM_HMECs24ato24e.

(4) Reading is performed after all data of the original pixel areas38U and38V are stored in the RAM_HMECs24ato24efrom the SDRAM1.

(5) Reading is performed simultaneously from the five RAM_HMECs24ato24e, to obtain a 16(8+8)×5 read rectangular area in one cycle.

(6) The (5+5)×5 rectangular area39U for calculation and rectangular area39V for calculation are obtained from the (8+8)×5 read rectangular area.

(7) The (5+5)×5 rectangular area39U for calculation and rectangular area39V for calculation are subjected to a two-dimensional linear interpolation process (bi-linear filter), to extract (4+4)×4 rectangular area40U after calculation and rectangular area40V after calculation (reference areas after chrominance calculation) of quarter-pixel accuracy.

A chrominance pixel stores a half amount of information compared with a luminance pixel. Thus, with sufficient processing time, the RAM_HMECs24ato24emay be replaced by one RAM_HMEC24t. This reduces the number of SRAMs of the RAM_HMEC24.

When the RAM_HMEC24consists of the one RAM_HMEC24t, the “RAM_HMEC24x” mentioned in the above condition (1) for the RAM_HMECs24ato24eis replaced by “one RAM_HMEC24t”, the condition (3) becomes unnecessary, and the “RAM_HMECs24ato24e” mentioned in the condition (4) are replaced by “one RAM_HMEC24t”.

Also, the condition (5) is changed to read “Reading is performed from the one RAM_HMEC24t, to obtain a 16 (8+8)×5 read rectangular area in five cycles”. The other conditions are identical to those mentioned above.

FIGS. 20A to 20Cillustrate a process of extracting a 5×5 extracted rectangular area. As shown inFIG. 20A, read chrominance data D27is obtained from the RAM_HMEC24x. Then as shown inFIG. 20B, an arbitrary (8+8)×5 read rectangular area (in only one line is shown) can be obtained from the read chrominance data D27of each of the RAM_HMECs24ato24eto satisfy the above conditions (2) and (3). When reading is performed from the one RAM_HMEC24t, an arbitrary (8+8)×5 read rectangular area can be obtained by reading the read chrominance data D27in five cycles.

Then as shown inFIG. 20C, an extracted rectangular area49(in only one line is shown) can be extracted which will become the arbitrary (5+5)×5 rectangular area39U for calculation and rectangular area39V for calculation in the read rectangular area shown inFIG. 20B. The extracted rectangular area49includes an extracted rectangular area49uand an extracted rectangular area49v. Five lines of the extracted rectangular area49uform the rectangular area39U for calculation, and five lines of the extracted rectangular area49vform the rectangular area39V for calculation.

FIGS. 21A and 21Bare schematic diagrams of a two-dimensional linear interpolation process (bi-linear filter), a calculation process by the QPG14. The QPG14receives the reference position signal S13pof quarter-pixel accuracy from the SME13, and extracts the rectangular areas39U and39V for calculation capable of calculating the 4×4 rectangular areas40U and40V after calculation.

As shown inFIG. 21A, in 5×5 U integer pixels65in the rectangular area39U for calculation, one U interpolation pixel67is calculated by interpolation with four U integer pixels65, thereby obtaining the rectangular area40U after calculation that includes 4×4 U interpolation pixels67. Weights assigned to the interpolation calculation of the four U integer pixels65are changed depending on the reference position signal S13p.

As shown inFIG. 21B, in 5×5 V integer pixels66in the rectangular area39V for calculation, one V interpolation pixel68is calculated by interpolation with four V integer pixels66, thereby obtaining the rectangular area40V after calculation that includes 4×4 V interpolation pixels68. Weights assigned to the interpolation calculation of the four V integer pixels66are changed depending on the reference position signal S13p.

The U interpolation pixels67and V interpolation pixels68of the rectangular area40U after calculation and rectangular area40V after calculation thus obtained are defined, to be output as the chrominance data S14.

As described above, in the motion estimation system for quarter-pixel chrominance with respect to chrominance data including the RAM_HMEC24and the QPG14, the five RAM_HMECs24ato24estore the stored chrominance data while satisfying the above conditions (1) to (7).

Accordingly, the arbitrary rectangular areas40U and40V after calculation in the original pixel areas38U and38V can be obtained by the combination of the five read chrominance data D27read from the RAM_HMECs24ato24eby one reading operation, with improved efficiency in obtaining the rectangular areas40U and40V after calculation.

The improved efficiency in obtaining the rectangular areas40U and40V after calculation allows the QPG14to perform motion estimation of quarter-pixel accuracy at high speed. Further, the improved efficiency reduces the load of doing calculations by the QPG14, thereby simplifying the circuit structure.

Moreover, when the RAM_HMEC24consists of the one RAM_HMEC24tin an environment with sufficient processing time for chrominance pixels, the number of RAMs can be reduced and efficient motion estimation of quarter-pixel accuracy can be performed.

As has been described, the ME2according to the first preferred embodiment includes the motion estimation system having the RAM_HIME21and the IME11, the motion estimation system for quarter-pixel luminance having the RAM_HMEY22, the QPG12, the RAM_HSME23and the SME13, and the motion estimation system for quarter-pixel chrominance having the RAM_HMEC24and the QPG14. Thus the ME2can perform motion estimation based on the luminance data and chrominance data of quarter-pixel accuracy at even higher speed, and output the luminance and chrominance data D3which is the estimation result including the luminance data S13and the chrominance data S14with quarter-pixel position accuracy at high speed, while simplifying the circuit structure.

A minimum number of SRAMs is required by storing the luminance data D1or the chrominance data D2in the RAM_HIME21, the RAM_HMEY22, the RAM_HSME23and the RAM_HMEC24serving as storage sections, respectively, while satisfying the storage conditions mentioned above. Further, a calculation section requires a simplest circuit structure by dispersing the calculation functions of the IME11, the QPG12, the SME13and the QPG14serving as the calculation sections in accordance with purposes (a luminance data search (integer-pixel accuracy, quarter-pixel accuracy), a chrominance data search). Therefore, the ME2can be attained with a relatively inexpensive structure.

In the first preferred embodiment, the ME2serving as a motion estimation system for combined luminance and chrominance includes the motion estimation system for integer pixel luminance having the RAM_HIME21and the IME11, the motion estimation system for quarter-pixel luminance having the RAM_HMEY22, the QPG12, the RAM_HSME23and the SME13, and the motion estimation system for quarter-pixel chrominance having the RAM_HMEC24and the QPG14. Alternatively, the ME2may employ motion estimation systems indicated below:

While the QPG14uses the reference position signal S13pas reference position information for determining the chrominance data S14in the first preferred embodiment, the QPG14may obtain estimation result chrominance data of integer-pixel accuracy by capturing information instructing the reference position of the luminance data of integer-pixel accuracy which is the estimation result from the IME11. Namely, the above motion estimation system for integer pixel luminance and the above motion estimation system for quarter-pixel chrominance may form a motion estimation system for combined luminance and chrominance of integer-pixel accuracy.

While the SME13performs motion estimation of quarter-pixel accuracy based on the reference position signal S11from the IME11, the SME13may perform motion estimation of quarter-pixel accuracy in a searchable entire image area without having to use the reference position signal S1. Namely, the above motion estimation system for quarter-pixel luminance and the above motion estimation system for quarter-pixel chrominance may form a motion estimation system for combined luminance and chrominance of quarter-pixel accuracy while omitting the above motion estimation system for integer pixel luminance.

While the QPG12obtains the partial quarter-pixel luminance data S12including luminance data of quarter-pixel accuracy only in a vertical direction and the SME13obtains luminance data of quarter-pixel accuracy in a horizontal direction by a prescribed calculation process, the horizontal and vertical directions can be reversed. Namely, the QPG12may obtain partial quarter-pixel luminance data including luminance data of quarter-pixel accuracy only in a horizontal direction and the SME13may obtain luminance data of quarter-pixel accuracy in a vertical direction by a prescribed calculation process.

Moreover, the number of RAMs can be reduced by sharing RAMs between the RAM_HMEY22and the RAM_HMEC24.

FIG. 22is a block diagram of a motion estimation system and its peripheral structure according to a second preferred embodiment of the present invention. As shown, a RAM_HME (H.264 motion estimation)26, a storage section for a quarter-pixel search, receives luminance data D4and chrominance data D5from a SDRAM4, and stores the data while satisfying the conditions described later.

A ME5, a quarter-pixel motion estimation section capable of performing motion estimation with quarter-pixel accuracy that includes integer-pixel accuracy, performs a comprehensive search process described later by using luminance data D6and chrominance data D7provided from the RAM_HME26, to output luminance and chrominance data D8(estimation result luminance data and estimation result chrominance data) as an estimation result of quarter-pixel accuracy to a MC6, a motion compensation system. The RAM_HME26and the ME5form a quarter-pixel motion estimation system.

FIG. 23is a schematic diagram of an original pixel area80stored in the RAM_HME26. In the diagram, the nL (pixel)×mL (line) original pixel area80stores a 14 (pixel)×10 (line) read rectangular area91in an extractable manner. After vertical-direction and horizontal-direction quarter-pixel generation processes described later, an 8×4 rectangular area60after calculation (reference area after luminance calculation) is obtained with a quarter-pixel accuracy position from the read rectangular area91.FIG. 23is an illustration where nL=32 and mL=40.

FIGS. 24A to 24Dillustrate examples of an obtained rectangular area of luminance data used for motion estimation. An 8×8 rectangular area81can be obtained from two rectangular areas60after calculation as shown inFIG. 24A, or an 8×16 rectangular area82can be obtained from four rectangular areas60after calculation as shown inFIG. 24B, or a 16×8 rectangular area83can be obtained from four rectangular areas60after calculation as shown inFIG. 24C, or a 16×16 rectangular area84can be obtained from eight rectangular areas60after calculation as shown inFIG. 24D.

FIGS. 25A to 25Dare schematic diagrams of luminance data storage areas of the RAM_HME26. As shown, luminance pixels in a horizontal direction are shifted in units of 4 pixels, to store 16 pixels at one address.

As shown inFIG. 25A, a storage area L (0) stores luminance pixels of 0 to 15 pixels in a horizontal position at one address in each of a 0 line to a (mL−1) line. As shown inFIG. 25B, a storage area L (1) stores luminance pixels of 4 to 19 pixels in a horizontal position at one address in each of a 0 line to a (mL−1) line. As shown inFIG. 25C, a storage area L (2) stores luminance pixels of 8 to 23 pixels in a horizontal position at one address in each of a 0 line to a (mL−1) line. Subsequently, as shown inFIG. 25D, a storage area L (k) similarly stores luminance pixels of (k×4) to {(k×4)+15} pixels in a horizontal position at one address in each of a 0 line to a (mL−1) line. When the original pixel area80shown inFIG. 23has a 32×32 structure (nL=32, mL=32), k is equal to 4 inFIG. 25D.

The RAM_HME26includes ten RAM_HMEs26ato26j, and stores the luminance data D4and the chrominance data D5so that the luminance pixels of 10 (at least 9) consecutive lines in a vertical direction in the storage area L (i (i=one of 1 to k)) certainly vary among the RAM_HMEs26ato26j.

The RAM_HME26includes the ten RAM_HMEs26ato26jas mentioned above, and stores the luminance data D4and the chrominance data D5while satisfying the conditions indicated below. For the purpose of illustration, one of the RAM_HMEs26ato26jwill hereafter be indicated as a RAM_HME26x.

(1) A storage unit of stored luminance data per address of the RAM_HME26xis 16×1. Sixteen luminance pixels are stored.

(2) Sixteen pixels of stored luminance data are stored at one address while being superimposed in units of four pixels in a horizontal direction.

(3) Ten (at least nine) consecutive 16×1 stored luminance data corresponding to pixels adjacent in a vertical direction are stored to vary among the RAM_HMEs26ato26j.

(4) Reading is performed after all data of the original pixel area80are stored in the RAM_HMEs26ato26jfrom the SDRAM4.

(5) Reading is performed simultaneously from the ten RAM_HMEs26ato26j, to obtain a 16×10 read rectangular area in one cycle.

(6) A 13×9 rectangular area59for calculation is obtained from the 16×10 read rectangular area.

(7) The 13×9 rectangular area59for calculation is subjected to parallel calculations by a 2 tap filter and a 6 tap filter, to extract the 8×4 rectangular area60after calculation of quarter-pixel accuracy in both horizontal and vertical directions.

FIGS. 26A to 26Dillustrate a process of extracting the 8×4 rectangular area60after calculation. As shown inFIG. 26A, read luminance data D41to D50are obtained from the RAM_HMEs26ato26j. Then as shown inFIG. 26B, an arbitrary 16×10 read rectangular area57can be obtained from the read luminance data D41to D50of each of the RAM_HMEs26ato26jto satisfy the above conditions (2) and (3).

Then as shown inFIG. 26C, the arbitrary 13×9 rectangular area59for calculation (extracted area for luminance data calculation) in the read rectangular area68shown inFIG. 26Bcan be extracted. A quarter-pixel parallel calculation process (prescribed calculation process) is performed based on integer pixel data in the rectangular area59for calculation, thereby obtaining the rectangular area60after calculation of quarter-pixel accuracy as shown inFIG. 26D.

FIG. 27illustrates the calculation process for obtaining the 8×4 rectangular area60after calculation of quarter-pixel accuracy. As shown, 14 pixels of data of each of the read luminance data D41to D50form the 14×10 read rectangular area91made of integer pixels61. To obtain the 8×4 rectangular area60after calculation, the 13×9 rectangular area59for calculation made of integer pixels61is sufficient enough. Yet for the purpose of illustration, the read rectangular area91will be used as an example.

The arbitrary 8×4 rectangular area60after calculation of quarter-pixel accuracy can be obtained from the read rectangular area91. Namely, the following interpolation processes (quarter-pixel parallel calculation process) are executable:

(1) A 6 tap filter process is performed with six integer pixels61adjacent in a horizontal direction, to sequentially obtain (horizontal direction) half-interpolation pixels62a(H) (6 tap filter).

(2) A 6 tap filter process is performed with six integer pixels61adjacent in a vertical direction, to sequentially obtain (vertical direction) half-interpolation pixels62b(V) (6 tap filter).

(3) A 6 tap filter process is performed with six half-interpolation pixels62aadjacent in a vertical direction, to obtain an oblique half-interpolation pixel62c(S) (6 tap filter).

(4) A 2 tap filter process is performed with the integer pixels61and the half-interpolation pixels62badjacent in a vertical direction, to obtain (vertical direction) quarter-interpolation pixels64a(iv) and64c(vi).

(5) A 2 tap filter process is performed with the half-interpolation pixels62aand62cadjacent in a vertical direction, to obtain (vertical direction) quarter-interpolation pixels64b(hs) and64d(sh).

(6) A 2 tap filter process is performed with the integer pixels61and the half-interpolation pixels62aadjacent in a horizontal direction, to obtain (horizontal direction) quarter-interpolation pixels69a(ih) and69b(hi) (seeFIGS. 12Cfor “ih” and “hi”).

(7) A 2 tap filter process is performed with the quarter-interpolation pixels64aand64badjacent in a horizontal direction, to obtain (horizontal direction) quarter-interpolation pixels69c(vh) and69d(hv) (seeFIGS. 12Cfor “vh” and “hv”).

(8) A 2 tap filter process is performed with the half-interpolation pixels62band62cadjacent in a horizontal direction, to obtain (horizontal direction) quarter-interpolation pixels69e(vs) and69f(sv) (seeFIGS. 12Cfor “vs” and “sv”).

(9) A 2 tap filter process is performed by the quarter-interpolation pixels64cand64dadjacent in a horizontal direction, to obtain (horizontal direction) quarter-interpolation pixels69g(vh′) and69h(hv′) (seeFIGS. 12Cfor “vh” and “hv”).

The quarter-pixel parallel calculation process including the above processes (1) to (9) can be performed upon obtainment of the base data for each of the processes (1) to (9). Thus quite a large amount of processes can be performed in parallel (for example, the processes (1) and (2) can be performed in parallel). This allows the rectangular area60after calculation to be obtained faster compared with the first preferred embodiment.

To obtain an 8×4 rectangular area60aafter calculation made of the integer pixels61(I), for example, the existence of the integer pixels61in the rectangular area60aafter calculation is sufficient enough. The rectangular area60aafter calculation can therefore be extracted from the read rectangular area91without having to performing the processes (1) to (9).

Also, an 8×4 rectangular area60bafter calculation made of the quarter-interpolation pixels64b(hs) can be obtained by performing the processes (1), (3) and (5). Likewise, an 8×4 rectangular area60cafter calculation made of the quarter-interpolation pixels64c(vi) can be obtained by performing the processes (2) and (4).

In this manner, the ME5obtains the rectangular area60after calculation of quarter-pixel accuracy that includes integer-pixel accuracy at high speed based on the luminance data D6of integer-pixel accuracy stored in the RAM_HME26to perform motion estimation with integer-pixel accuracy or quarter-pixel accuracy, thereby obtaining the estimation result luminance data of integer-pixel accuracy or quarter-pixel accuracy.

Thus the ME5can obtain the estimation result luminance data of quarter-pixel accuracy forming the luminance and chrominance data D8faster than when the ME2obtains the luminance data S13after performing motion estimation with integer-pixel accuracy to detect a reference position of the estimation result luminance data of integer-pixel accuracy, and then performing motion estimation with quarter-pixel accuracy in a range of {−0.75˜+0.75} from the reference position in the first preferred embodiment.

The ME5is capable of performing motion estimation of quarter-pixel accuracy directly based on the luminance data D6made of integer pixels. Therefore, estimation result luminance data of quarter-pixel accuracy can be obtained at the beginning of estimation by performing motion estimation of quarter-pixel accuracy without performing motion estimation of integer-pixel accuracy.

As described above, in the ME5according to the second preferred embodiment, the ten RAM_HMEs26ato26jstore the stored luminance data while satisfying the above conditions (1) to (9).

Accordingly, the arbitrary rectangular area60after calculation in the original pixel area80can be obtained by the combination of the read luminance data D41to D50read from the RAM_HMEs26ato26jby one reading operation, with improved efficiency in obtaining the rectangular area60after calculation.

The improved efficiency in obtaining the rectangular area60after calculation allows the ME5to perform motion estimation of quarter-pixel accuracy at high speed. Further, the improved reading efficiency reduces the load of doing calculations by the ME5, thereby simplifying the circuit structure.

FIGS. 28A and 28Bare schematic diagrams of an original pixel area85for U pixels and an original pixel area86for V pixels stored in the RAM_HME26. In the diagrams, the nC (pixel)×mC (line) original pixel areas85and86store a 5 (pixel)×5 (line) rectangular area87for U calculation and a 5 (pixel)×5 (line) rectangular area88for V calculation (extracted areas for chrominance data calculation) in an extractable manner, respectively. A rectangular area89after U calculation and a rectangular area90after V calculation (reference areas after chrominance data calculation) corresponding to the 8×4 rectangular area60after calculation based on the estimation result luminance data mentioned above are obtained from the rectangular area87for U calculation and the rectangular area88for V calculation, respectively.

FIGS. 29A and 29Bare schematic diagrams of chrominance data storage areas of the RAM_HME26. As shown, chrominance pixels (U, V) in a horizontal direction are shifted in units of 16 pixels without being superimposed, to store 8 chrominance pixels U and 8 chrominance pixels V at one address. An area C (0) to an area C (k) are set inFIGS. 29A and 29Bin a corresponding manner to the area L (0) to the area L (k) shown inFIGS. 25A to 25D, so the amount of storage per address of the area C (i) is apparently 32 pixels. However, 8 U pixels and 8 V pixels are actually stored at one address. For example, at the address0of the area C (0), the RAM_HME26astores high-order 8 U pixels and high-order 8 V pixels (8 to 15) of the line0, and the RAM_HME26bstores low-order 8 U and V pixels (0 to 7) of the line0.

As shown inFIG. 29A, the storage area C (0) stores chrominance pixels (U, V) of 0 to 15 pixels in a horizontal position, with a U pixel and a V pixel being paired, at one address in each of a 0 line to a (m−1) line by allocating high-order 8 pixels and low-order 8 pixels between the two RAM_HMEs26as described above. Subsequently, as shown inFIG. 29B, a storage area C (k) similarly stores chrominance pixels (U, V) of (k×16) to {(k×16)+15} pixels in a horizontal position, with a U pixel and a V pixel being paired, at one address in each of a 0 line to a (m−1) line by allocating high-order 8 pixels and low-order 8 pixels between the two RAM_HMEs26. When the original pixel areas85and86shown inFIGS. 28A and 28Bhave a 16×16 structure (nC=16, mC=16), k is equal to 0 inFIG. 29B.

The RAM_HME26xstores the chrominance data while satisfying the conditions indicated below:

(1) A storage unit per address of the RAM_HME26xis 16×1(2×(8×1)). Eight chrominance pixels U and 8 chrominance pixels V are stored. Namely, a storage unit of stored chrominance data is 8×1 for each of the U and V pixels.

(2) Eight chrominance pixels U and 8 chrominance pixels V are stored at one address without being superimposed in a horizontal direction.

(3) 16×1(2×(8×1)) chrominance data corresponding to pixels adjacent in a horizontal direction (8×1 stored chrominance data for each of the U and V pixels) are stored to vary among the RAM_HMEs26ato26j.

(4) Five consecutive 16×1(2×(8×1)) chrominance data corresponding to pixels adjacent in a vertical direction (8×1 stored chrominance data for each of the U and V pixels) are stored to vary among the RAM_HMEs26ato26j.

(5) Reading is performed after all data of the original pixel areas85and86are stored in the RAM_HMEs26ato26jfrom the SDRAM4.

(6) Reading is performed simultaneously from the ten RAM_HMEs26ato26j, to obtain a 16 (8+8)×5 read rectangular area in one cycle.

(7) The (5+5)×5 rectangular area87for U calculation and rectangular area88for V calculation are obtained from the (8+8)×5 read rectangular area.

(8) The (5+5)×5 rectangular area87for U calculation and rectangular area88for V calculation are subjected to a bi-linear filter, to extract the (4+4)×4 rectangular area89after U calculation and rectangular area90after V calculation of quarter-pixel accuracy.

FIGS. 30A to 30Cillustrate a process of extracting 5×5 extracted rectangular areas. As shown inFIG. 30A, read chrominance data D51to D55are obtained from the RAM_HMEs26ato26j. Then as shown inFIG. 30B, an arbitrary (16+16)×5 read rectangular area can be obtained from the read chrominance data D51to D55from the RAM_HMEs26ato26jto satisfy the above conditions (3) and (4). Namely, a 16×5 read rectangular area for U pixels is obtained from read chrominance data D51uto D55ufor U pixels in the read chrominance data D51to D55, and a 16×5 read rectangular area for V pixels is obtained from read chrominance data D51vto D55vfor V pixels in the read chrominance data D51to D55.

Then as shown inFIG. 30C, the arbitrary 5×5 rectangular area87for U calculation and 5×5 rectangular area88for V calculation in the read rectangular area shown inFIG. 30Bcan be extracted.

FIGS. 31A and 31Bare schematic diagrams of a two-dimensional linear interpolation process (bi-linear filter) by the ME5. As shown inFIG. 31A, in 5×5 U integer pixels65in the rectangular area87for U calculation, one U interpolation pixel67is calculated by interpolation with four U integer pixels65, thereby obtaining the rectangular area89after U calculation that includes 4×4 U interpolation pixels67. Weights assigned to the interpolation calculation of the four U integer pixels65are changed depending on the reference position of the rectangular area60after calculation in the estimation result luminance data of quarter-pixel accuracy.

As shown inFIG. 31B, in 5×5 V integer pixels66in the rectangular area88for V calculation, one V interpolation pixel68is calculated by interpolation with four V integer pixels66, thereby obtaining the rectangular area90after V calculation that includes 4×4 V interpolation pixels68. Weights assigned to the interpolation calculation of the four V integer pixels66are changed depending on the reference position of the rectangular area60after calculation in the luminance data.

The U interpolation pixels67and V interpolation pixels68of the rectangular area89after U calculation and rectangular area90after V calculation are defined as estimation result chrominance data. The estimation result luminance data mentioned above and this estimation result chrominance data are output to the MC6as the luminance and chrominance data D8.

In this manner, the rectangular area87for U calculation and the rectangular area88for V calculation which are necessary for calculating the rectangular area89after U calculation and the rectangular area90after V calculation can be obtained from the RAM_HME26in one cycle. The ME5can therefore calculate chrominance data of the rectangular area89after U calculation and the rectangular area90after V calculation at high speed based on the chrominance data D7stored in the RAM_HME26.

As described above, in the ME5according to the second preferred embodiment, the ten RAM_HMEs26ato26jstore the stored chrominance data while satisfying the above conditions (1) to (8).

Accordingly, the arbitrary rectangular area89after U calculation and rectangular area90after V calculation in the original pixel areas85and86can be obtained by the combination of the read chrominance data D51to D55read from the RAM_HMEs26ato26jby one reading operation, with improved efficiency in obtaining the areas89and90.

The improved efficiency in obtaining the rectangular area89after U calculation and the rectangular area90after V calculation allows the ME5to perform motion estimation of quarter-pixel accuracy with respect to chrominance data. Further, the improved efficiency reduces the load of doing calculations by the ME5, thereby simplifying the circuit structure.

As described above, the ME5according to the second preferred embodiment can obtain the luminance and chrominance data D8faster than when the ME2according to the first preferred embodiment obtains the luminance and chrominance data D3. Further, the first preferred embodiment needs four RAM_HIMEs21, two RAM_HMEYs22, eight RAM_HSMEs23and five (one) RAM_HMECs24, totaling nineteen (fifteen) (S)RAMs, while the second preferred embodiment only requires ten RAM_HMEs26. This reduces the number of (S)RAMs compared with the first preferred embodiment.

Moreover, the RAM_HME26according to the second preferred embodiment is only required to store integer pixels and does not need to store quarter-pixels like the RAM_HSME23according to the first preferred embodiment. This reduces (S)RAM capacitance. Note however that the ME5, which needs to perform quite a large amount of parallel calculation processes in order to obtain luminance data of quarter-pixel accuracy based on the luminance data D6of integer-pixel accuracy, has a more complicated circuit structure than the IME11, the QPG12, the SME13and the QPG14forming the ME2.