Image processing apparatus and control method thereof

According to an image processing apparatus and a control method thereof of the present invention, a graphics drawing result is obtained in accordance with one or more graphics drawing commands included in drawing commands, a clipping command is generated from one or more moving image drawing commands included in the drawing commands, and clipped graphics is obtained by clipping the graphics drawing result using the clipping command. Further, moving image data processed in accordance with the one or more moving image drawing commands included in the drawing commands is generated, and the generated moving image data and the clipped graphics are composed and output.

TECHNICAL FIELD

The present invention relates to an image processing apparatus for composing graphics and moving image data input thereto, and to a control method thereof.

BACKGROUND ART

Conventionally, an image that includes a moving image and graphics was constituted by capturing a frame image of the moving image and combining the graphics with the frame image, which is treated as a still image. However, the use application of images, the data format and the type of drawing process required differ between moving images and graphics. For example, in the case of moving images, a technique that prevents a reduction in moving image quality due to dropped frames or the like is sought. The systems for processing moving images and graphics are therefore separated, and the output results of the respective processing systems are composed and output.

Japanese Patent Laid-Open No. 07-72850, which relates to an image forming apparatus that composes and outputs a moving image and graphics, proposes a method for composing graphics data and moving image data stored in memory while switching the readout addresses in memory. Japanese Patent Laid-Open No. 2005-321481 proposes achieving high quality video by keeping a video signal processing function independent from a graphics controller, and inputting images created by the graphics controller as video signals.

However, while the respective quality of moving images and graphics can be enhanced according to the above-mentioned conventional methods since the systems for processing a moving image and graphics are separated, in the case where a moving image and graphics are displayed so as to overlap, a plurality of buffers are needed for storing the graphics drawing results in order for display to be performed correctly. The resultant increase in device cost was problematic.

DISCLOSURE OF INVENTION

An aspect of the present invention is to eliminate the above-mentioned problems with the conventional technology.

An aspect of the present invention is to provide a technique being capable of composing a moving image and a graphic by reducing a capacity of a memory that stores graphics drawing results.

According to an aspect of the present invention, there is provided an image processing apparatus for receiving drawing commands that include one or more moving image drawing commands and one or more graphics drawing commands, and performing a drawing process, comprising:

graphics drawing means for obtaining a graphics drawing result in accordance with the one or more graphics drawing commands included in the drawing commands;

clipping means for generating a clipping command based on the one or more moving image drawing commands included in the drawing commands;

storage means for storing clipped graphics obtained by clipping the graphics drawing result using the clipping command;

moving image drawing means for generating moving image data processed in accordance with the one or more moving image drawing commands included in the drawing commands; and

composition means for composing the clipped graphics stored in the storage means and the moving image data generated by the moving image drawing means.

According to another aspect of the present invention, there is provided a control method of an image processing apparatus for receiving drawing commands that include one or more moving image drawing commands and one or more graphics drawing commands, and performing a drawing process, the method comprising:

a graphics drawing step of obtaining a graphics drawing result in accordance with the one or more graphics drawing commands included in the drawing commands;

a clipping step of generating a clipping command from the one or more moving image drawing commands included in the drawing commands;

a storage step of storing clipped graphics obtained by clipping the graphics drawing result using the clipping command;

a moving image drawing step of generating moving image data processed in accordance with the one or more moving image drawing commands included in the drawing commands; and

a composition step of composing the clipped graphics stored in the storage step and the moving image data generated in the moving image drawing step.

BEST MODE FOR CARRYING OUT THE INVENTION

FIG. 1is a block diagram describing a schematic configuration of an image processing apparatus according to exemplary embodiments of the present invention.

InFIG. 1, a CPU102controls operation of the image processing apparatus in accordance with programs loaded in a ROM103or a RAM104. The ROM103stores programs and various parameters that do not require modification. The RAM104is constituted by a SDRAM, a DRAM or the like, and provides a work area for use by the CPU102, as well as being used for temporarily storing programs supplied from an external device or the like and various data such as image data. A display unit105displays images drawn by programs. A system bus101connects the CPU102, the ROM103, the RAM104and the display unit105to one another.

FIG. 2is a functional block diagram illustrating the functions of an image processing apparatus201according to the exemplary embodiments.

InFIG. 2, a drawing controller202in the image processing apparatus201according to the present embodiment inputs drawing commands including one or more moving image drawing commands and one or more graphics drawing commands, and creates graphics. A graphics storage unit203stores the created graphics. A moving image drawing unit204creates moving image frames in accordance with input moving image data and instructions from a drawing command separator206, and outputs created moving image frames to a composition unit205. The composition unit205composes graphics and moving images.

The drawing controller202includes the drawing command separator206, a graphics drawing unit207, and a clipping unit208. The drawing command separator206receives drawing commands and separates received drawing commands into graphics drawing commands and moving image drawing commands. The graphics drawing unit207creates graphics by executing graphics drawing commands separated by the drawing command separator206, and stores created graphics in the graphics storage unit203. The clipping unit208acquires parameters representing the geometric information of a moving image from moving image drawing commands separated by the drawing command separator206. The clipping unit208then generates a clipping command for graphics drawn by the graphics drawing unit207, and outputs the generated clipping command to the graphics storage unit203. The graphics storage unit203thereby obtains clipped graphics by performing a clipping process in accordance with the clipping command on graphics stored therein. The moving image drawing unit204receives moving image data, acquires the parameters of a two-dimensional affine transformation, for example, for a moving image from moving image drawing commands separated by the drawing command separator206, and performs a transformation process on the moving image data. The composition unit205composes graphics stored in the graphics storage unit203and moving image data transformed by the moving image drawing unit204, and outputs the resultant composition image to the display unit105for display.

First Embodiment

Next, the flow of processing in the image processing apparatus according to a first embodiment of the present invention will be described. The geometric information of a moving image according to the first embodiment is represented by the parameters of a two-dimensional affine transformation.

FIG. 3Adepicts a view describing exemplary drawing commands input to the image processing apparatus according to the present embodiment.

These drawing commands include a rectangle drawing command301, a moving image drawing command302, a rectangle drawing command303and a moving image drawing command304, and are called in the order shown inFIG. 3A. The rectangle drawing command301is for drawing a rectangle with lower-left corner at (x1, y1), width w1in the X-axis direction and height h1in the Y-axis direction with a color c1. This rectangle is shown by a rectangle305inFIG. 3B. The rectangle drawing command303is similarly for drawing a rectangle with lower-left corner at (x3, y3), width w3in the X-axis direction and height h3in the Y-axis direction with a color c3. This rectangle is shown by a rectangle306inFIG. 3D.

The moving image drawing command302is for scaling the moving image by sx2in the X-axis direction and sy2in the Y-axis direction, and translating the moving image by x2in the X-axis direction and y2in the Y-axis direction. This rectangle is shown by a rectangle307inFIG. 3C. The moving image drawing command304is similar for scaling the moving image by sx4in the X-axis direction and sy4in the Y-axis direction, and translating the moving image by x4in the X-axis direction and y4in the Y-axis direction. This rectangle is shown by a rectangle308inFIG. 3E.

The drawing content in the case where the respective drawing processes are independently executed are shown inFIGS. 3B to 3E. InFIG. 3C, the width vw2of the image results from scaling the width of the moving image by sx2, and the height vh2results from scaling the height of the moving image by sy2. Similarly, the width vw4of the image inFIG. 3Eresults from scaling the width of the moving image by sx4, and the height vh4results from scaling the height of the moving image by sy4. Note that while the present embodiment is described using scaling and translation as a two-dimensional affine transformation, rotation, skewing or the like may be used, or an affine transformation that combines these transformations may be used.

The drawing command separator206receives drawing commands one at a time, and separates these drawing commands into moving image drawing commands and graphics drawing commands. If the input drawing command is a graphics drawing command, the graphics drawing unit207receives the graphics drawing command and creates graphics by executing the graphics drawing command. The graphics thus created are stored in the graphics storage unit203. On the other hand, if the input drawing command is a moving image drawing command, firstly a clipping command is generated by the clipping unit208and stored in the graphics storage unit203. The graphics storage unit203thereby performs a clipping process on graphics stored therein. Here, the clipping command is generated so as to calculate an area that will be occupied by the moving image from the parameters of the moving image drawing command and to eliminate the graphics in the area. The drawing command separator206sends the moving image drawing command to the moving image drawing unit204, after the instruction for the clipping process has been given by the clipping unit208. The moving image drawing unit204receives the moving image drawing command and performs a transformation process on the moving image data. The composition unit205composes a plurality of transformed moving images input from the moving image drawing unit204starting with the moving image whose drawing command was input first, and then composes graphics input from the graphics storage unit203over the composed moving images. The processing after input of drawing commands has been received will be described using the figures, with the drawing commands shown inFIG. 3Aas examples. Firstly, the graphics drawing process will be described.

FIGS. 4A to 4Dare schematic diagrams describing the content of graphics drawing results stored by the graphics storage unit203when the drawing commands shown inFIG. 3Aare executed sequentially.

Firstly, the content of graphics drawing results stored by the graphics storage unit203is cleared and made transparent. When the rectangle drawing command301shown inFIG. 3Ais input, this drawing command is determined by the drawing command separator206to be a graphics drawing command, and processing is executed by the graphics drawing unit207. A rectangle is thereby drawn with the color c1, as shown inFIG. 4A. Reference numeral411inFIG. 4Adenotes this drawing result.

Next, when the moving image drawing command302shown inFIG. 3Ais input, this drawing command is determined by the drawing command separator206to be a moving image drawing command, and a clipping command421is generated by the clipping unit208. The processing in this case is shown inFIG. 4B. This clipping command421changes the color values of pixels in the area of the rectangle with lower-left corner at (x2, y2), width vw2in the X-axis direction and height vh2in the Y-axis direction to make the pixels transparent. The drawing result when this clipping command421is processed will be as shown by reference numeral412. InFIG. 4B, the rectangular area inFIG. 4A(equivalent to305inFIG. 3B) has been clipped with the area shown by307inFIG. 3C.

Next, when the rectangle drawing command303shown inFIG. 3Ais input, processing is executed by the graphics drawing unit207, and a rectangle is drawn with the color c3(FIG. 4C). This rectangle is equivalent to306inFIG. 3D. A drawing result413shown inFIG. 4Cis obtained by this processing.

Next, when the moving image drawing command304shown inFIG. 3Ais input, this drawing command is determined by the drawing command separator206to be a moving image drawing command, and a clipping command422(FIG. 4D) is generated by the clipping unit208. This is for clipping the rectangle306inFIG. 3Dwith the rectangle308inFIG. 3E. The clipping command422clips the area of the rectangle with lower-left corner (x4, y4), width vw4in the X-axis direction and height vh4in the Y-axis direction, and a drawing result414shown inFIG. 4Dis obtained by this clipping process.

Next, the moving image drawing process will be described using the schematic diagram ofFIG. 5.

FIG. 5is a schematic diagram describing the moving image drawing process according to the first embodiment of the present invention.

The moving image drawing unit204has a plurality of moving image processing units (503,504), and these moving image processing units correspond one-to-one with the moving image drawing commands. Here, moving image data drawn using the moving image drawing command302is given as moving image data501, and moving image data drawn using the moving image drawing command304is given as moving image data502. A moving image processing unit503processes the moving image drawing command302, and a moving image processing unit504processes the moving image drawing command304. The moving image processing units503and504respectively receive the moving image data501and502. Note that here the format of moving image data may be any of MPEG-1, MPEG-2, AVI or the like, and is not specified here. Further, the moving image processing units503and504respectively perform a transformation process on this moving image data in accordance with the moving image drawing commands supplied from the drawing command separator206. For example, the moving image processing unit503scales the moving image data501by sx2in the X-axis direction and sy2in the Y-axis direction, and translates the moving image data501by x2in the X-axis direction and y2in the Y-axis direction, based on the parameters in the moving image drawing command302, as a result of which transformed moving image data505is obtained. The moving image processing unit504similarly scales the moving image data502by sx4in the X-axis direction and sy4in the Y-axis direction, and translates the moving image data502by x4in the X-axis direction and y4in the Y-axis direction, based on the parameters in the moving image drawing command304, as a result of which transformed moving image data506is obtained. These processes are the same as those illustrated with the foregoingFIGS. 3C and 3E.

FIG. 6is a schematic diagram describing the composition process in the composition unit205according to the present embodiment.

The drawing result414from the graphics storage unit203and the transformed moving image data505and506are input to the composition unit205. The composition unit205firstly inputs the transformed moving image data505and506, and composes these moving image data. The transformed moving image data505and506are respectively generated using the moving image drawing command302and the moving image drawing command304. Because the moving image drawing command302is input to the drawing controller202before the moving image drawing command304, the composition process is performed by layering the transformed moving image data505and506such that the moving image data505is positioned under the moving image data506. A composition result with the moving image data506placed over the moving image data505is obtained as a result, as shown by611inFIG. 6. Next, the drawing result414stored in the graphics storage unit203and the moving image composition result611are composed. This time the composition process is performed by layering the moving image composition result611and the drawing result414such that the moving image composition result611is under the drawing result414. A composition result612from composing moving images and graphics is thus ultimately obtained, and this composition result is displayed on the display unit105.

Note that while the first embodiment was described in terms of two moving image drawing commands being included in the drawing commands, the present invention is realized even in the case of there being one moving image drawing command or three or more moving image drawing commands. Also, while the composition process is here performed starting with the moving image, the composition process may be performed in any order provided that the hierarchical relation of the layers is maintained.

Second Embodiment

Hereinafter, a second embodiment of the present invention will be described with reference to the figures. Note that because the hardware configuration and the functional configuration of the image processing apparatus according to the second embodiment are the same as that described for the foregoing first embodiment, description thereof will be omitted.

Next, the flow of processing in the image processing apparatus according to the second embodiment of the present invention will be described. Here, the geometric information of a moving image used in the second embodiment is represented by a mask image (mask information) as well as by specification of the parameters of a two-dimensional affine transformation.

FIG. 7Ashows an exemplary sequence of drawing commands input to the image processing apparatus according to the second embodiment.

These drawing commands include a polygonal drawing command701, a moving image drawing command702, an elliptic drawing command703and a moving image drawing command704, and are called in the respective order in which they are input. Next, the drawing results from the respective drawing commands will be described using the schematic diagram shown inFIG. 7B.

The polygonal drawing command701is a graphics drawing command in which six X coordinates and Y coordinates are here transferred as an array px[ ], py[ ]. For example, in the case of a hexagon, line segments are drawn in the order (px[0], py[0]), (px[1], PY[1]), (Px[2], PY[2]), (Px[3], PY[3]), (Px[4], PY[4]), (px[5], py[5]) and (px[0], py[0]). The area enclosed by these line segments is then filled in with a color cp. Reference numeral710inFIG. 7Bshows an exemplary polygon drawn using this polygonal drawing command701.

The elliptic drawing command703is a graphics drawing command for filling in an ellipsoid with central coordinates (cx, cy), width2rxand height2ry, that is, the pixels of coordinates (x, y) satisfying an equation represented by the following expression (1) with a color ce. Reference numeral711inFIG. 7Bshows an exemplary ellipsoid drawn using this elliptic drawing command703.
(x−cx)2/rx2+(y−cy)2/ry2≦1   (1)

The moving image drawing command702performs a masking process on moving image data using mask image data m5, scales the masked moving image by sx5in the X-axis direction and sy5in the Y-axis direction, and translates the masked moving image by x5in the X-axis direction and y5in the Y-axis direction. Mask image data m5is provided with a “0” or “1” mask value for each pixel of input moving image data712, and the moving image data712is transformed such that pixels whose mask value is “0” will be transparent and pixels whose mask value is “1” will be the color of the pixels in the moving image data.

A mask image722illustrates the mask image data m5. The mask values of the black area are “0” and the mask values of the white area are “1”. Masked moving image data732is generated by processing the moving image data712using this mask image722. Moving image data such as shown by713inFIG. 7Bis obtained using the moving image drawing command702. Here, the width vw5of the moving image results from scaling the width of the moving image712by sx5, and the height vh5results from scaling the height of the moving image712by sy5.

The moving image drawing command704similarly performs a masking process on moving image data using mask image data m6, scales the masked moving image by sx6in the X-axis direction and sy6in the Y-axis direction, and translates the masked moving image by x6in the X-axis direction and y6in the Y-axis direction. A mask image724illustrates the mask image data m6, and masked moving image data734is generated by performing the masking process on moving image data714. Moving image data such as shown by715inFIG. 7Bis obtained using the moving image drawing command704. Here, the width vw6of the moving image results from scaling the width of the moving image data714by sx6, and the height vh6results from scaling the height of the moving image data714by sy6.

Next, the processing after drawing commands such as shown inFIG. 7Aare received in the second embodiment will be described using the examples shown inFIGS. 7A and 7B. Firstly, the graphics drawing process will be described.

FIGS. 8A to 8Dare schematic diagrams describing the exemplary content of graphics drawing results stored by the graphics storage unit203when the drawing commands shown inFIG. 7Aare executed sequentially.

Firstly, the content of graphics drawing results stored by the graphics storage unit203is cleared and all of the pixels are made transparent.

When the polygonal drawing command701inFIG. 7Ais input, this drawing command is determined by the drawing command separator206to be a graphics drawing command, and a polygon is drawn with a color cp as a result of a drawing process being executed by the graphics drawing unit207, as shown inFIG. 8A. The drawing result stored by the graphics storage unit203will be as shown by a drawing result911.

Next, when the moving image drawing command702inFIG. 7Ais input, this drawing command is determined by the drawing command separator206to be a moving image drawing command. New mask image data m15obtained by scaling the mask image data m5by sx5in the X-axis direction and sy5in the Y-axis direction and translating the mask image data m5by x5in the X-axis direction and y5in the Y-axis direction is then generated by the clipping unit208. As shown inFIG. 8B, a clipping command921is then generated with this generated mask image data m15as a parameter. This clipping command921is for converting the graphics drawing result911to make pixels whose mask value in the mask image data m15is “1” transparent. A drawing result912shown inFIG. 8Bis then obtained by executing this clipping command921.

When the ellipsoid drawing command703inFIG. 7Ais input, an ellipsoid is drawn with a color ce as a result of a drawing process being executed by the graphics drawing unit207to obtain a drawing result913, as shown inFIG. 8C.

Next, when the moving image drawing command704inFIG. 7Ais input, this drawing command is determined by the drawing command separator206to be a moving image drawing command. New mask image data m16obtained by scaling the mask image data m6by sx6in the X-axis direction and sy6in the Y-axis direction and translating the mask image data m6by x6in the X-axis direction and y6in the Y-axis direction is then generated by the clipping unit208. As shown inFIG. 8D, a clipping command922is then generated with this generated mask image data m16as a parameter and processed. This clipping command922is for converting the graphics drawing result to make pixels whose mask value in the mask image data m16is “1” transparent. A drawing result914is then obtained by executing this clipping command922.

FIG. 9illustrates the moving image drawing process in the moving image drawing unit204according to the second embodiment.

The moving image drawing unit204has a plurality of moving image processing units (1003,1004), and these moving image processing units correspond one-to-one with the moving image drawing commands. Moving image data processed by a moving image processing unit1003in accordance with the moving image drawing command702shown in the foregoingFIG. 7Ais given as moving image data712, and moving image data processed by a moving image processing unit1004in accordance with the moving image drawing command704is given as moving image data714. The moving image processing units1003and1004respectively receive the moving image data712and714. The format of the moving image data at this time may be any of MPEG-1, MPEG-2, AVI or the like, and is not specified here. Further, the drawing command separator206receives moving image drawing commands, and the moving image processing units1003and1004perform a masking process and a transformation process on the respective moving image data in accordance with the moving image drawing commands. The moving image processing unit1003firstly transforms the moving image data712using the mask image data m5, based on the parameters in the moving image drawing command702. Further, the moving image processing unit1003then scales the moving image data by sx5in the X-axis direction and sy5in the Y-axis direction and translates the moving image data by x5in the X-axis direction and y5in the Y-axis direction, and outputs processed moving image data1005. Similarly, the moving image processing unit1004transforms the moving image data using the mask image data m6, based on the parameters in the moving image drawing command704. Further, the moving image processing unit1004scales the moving image data by sx6in the X-axis direction and sy6in the Y-axis direction and translates the moving image data by x6in the X-axis direction and y6in the Y-axis direction, and outputs processed moving image data1006.

FIG. 10is a schematic diagram describing the composition process in the composition unit205according to the second embodiment.

Here, the drawing result914shown inFIG. 8Dand the processed moving image data1005and1006shown inFIG. 9are input to the composition unit205. The composition unit205firstly composes the processed moving image data1005and the processed moving image data1006. As illustrated withFIG. 9, the processed moving image data1005is generated using the moving image drawing command702, and the processed moving image data1006is generated using the moving image drawing command704. Here, because the moving image drawing command702is input to the drawing controller202before the moving image drawing command704, the composition process is performed by layering the processed moving image data1005and1006such that the moving image data1005is under the moving image data1006. A moving image composition result1111is obtained as a result.

Next, the drawing result914stored in the graphics storage unit203and the moving image composition result1111are composed. This time the composition process is performed by layering the moving image composition result1111and the drawing result914such that the moving image composition result1111is under the drawing result914. A composition result1112from composing moving images and graphics is thus ultimately obtained, and this composition result is displayed on the display unit105.

Other Embodiments

While the embodiments of the present invention have been detailed above, the present invention may be applied to a system constituted by a plurality of devices or to an apparatus composed of a single device.

Note that the present invention can be achieved by a software program for realizing the functions of the foregoing embodiments being directly or remotely supplied to a system or an apparatus, and a computer in the system or apparatus reading out and executing the supplied program. In this case, as long as the functions of the program are provided, this embodiment need not be in the form of a program.

Consequently, the present invention is also realized by the actual program code installed on a computer, in order to realize the functions and processes of the present invention by computer. In other words, the claims of the present invention also encompass the actual computer program for realizing the functions and processes of the present invention. In this case, as long as the functions of the program are provided, the program may take any form, such as an object code, a program executed by an interpreter, script data supplied to an operating system, or the like.

The functions of the foregoing embodiments can be realized in forms other than by a computer executing the read program. For example, an operating system or the like running on a computer can perform part or all of the actual processing based on instructions in the program, and the functions of the foregoing embodiments can be realized by this processing.

This application claims the benefit of Japanese Patent Application No. 2008-154589, filed Jun. 12, 2008, which is hereby incorporated by reference herein in its entirety.