Apparatus and method for video signal processing

Abstract: Image processing method and apparatus for producing an image from image data provides an image memory (10) including a drawing area to store image data of an object to be drawn and a texture area to store texture data to decorate the object, a supply circuit (5) for supplying image data to the image memory, a selector for selectively supplying the image data from the supply circuit to the drawing area and the texture area, an image synthesizing circuit (6) for synthesizing an image from the image data stored in the image memory and an output circuit for converting an output signal from the image synthesizing circuit to a video signal for outputting the video signal. The method provides the steps of storing image data of an object to be drawn in a drawing area of an image memory, storing moving image data in a texture area of the image memory, and mapping a texture of the moving image stored in the texture area onto the surface of the object stored in the drawing area.

Description: This invention relates generally to an image processing apparatus and method for generating an image from data stored in a memory and, more specifically, to generating an image that has texture or ornamentation on its surfaces, such as an image for use in a computer game. In order to enjoy a computer game, high-speed processing and an excellent graphics display ability are required. Among video games, it is required to display a 3-dimensional object at high speed, to realize visual effects such as textures or the like, or to display other information with reality. The texture is intended to cause a visual effect that is peculiar to the particular 3-dimensional object by seemingly adhering a pattern, which was defined in a different place, onto the surface of the 3-dimensional object. As examples of such patterns which are adhered to the surface of the object, there is a geometric pattern or a still pattern, such as a photograph, a hand written drawing, or the like. It is also desirable that a moving image be used as a texture pattern in order to provide distinctive computer graphics. Hitherto, however, a texture pattern has been predetermined and the texture pattern cannot dynamically be rewritten onto selected images. Therefore, the texture pattern of a moving image cannot be adhered to the surface of an object using computer graphics. Further, since the texture pattern cannot dynamically be rewritten as mentioned above, a texture having a large change cannot be adhered to the surface of an object. Moreover, hitherto, the number of colors of the texture pattern is limited due to a limitation in the number of bits and a number of colors that can be expressed. US 4,935,879 describes a texture mapping apparatus for the real-time display of a figure in which an animated image has been mapped. The apparatus includes a device which generates 2D (2 dimensional) source coordinates data which provide a source for texture mapping data corresponding to 3D (3 dimensional) destination coordinates data which in turn provide destinations for texture mapping data. SMPTE JOURNAL, Vol 100, No. 3, March 1991, WHITE PLAINS, NY, US pp162-166 VIGNEAUX ET AL "A RealTime Video Mapping and Manipulation System" describes a system that offers real-time input video mapping to internally create 3-d objects of revolution. These objects may be purely curvilinear or possess multiple individually crafted local deformations. Full-motion video may then be mapped to the object in real time (real time is defined as video rate, 30 frames/sec). EP-A-0,447,227 describes methods and apparatus for texture mapping graphics primitives in a graphics pipeline architecture system. The methods and apparatus utilize rectangular box filters to down-sample original texture maps thereby optimizing aliasing and blurring when graphics primitives have a two-dimensional texture mapped to a three-dimensional object. The methods comprise the steps of determining an original texture map of two dimensions for a surface, storing the original texture map in the frame buffer, sampling the original texture map independently using an asymmetrical filter to construct multiple versions of a texture and to address textured pixels on a display in the frame buffer graphics systems, mapping the textured pixels to areas on the frame buffer, and displaying the textured graphics primitives on the display. According to an aspect of the invention, there is provided a real time texture mapping method for producing a two-dimensional image of an object having a surface that has dynamically changing texture by using three-dimensional image data and two-dimensional moving image data, comprising the steps of: (a) receiving the three-dimensional image data and the two-dimensional moving image data; (b) transforming the three-dimensional image data to respective two-dimensional image data; (c) storing the two-dimensional image data in a drawing area of a frame memory; (d) storing the two-dimensional moving image data frame-by-frame as texture pattern data in a texture area of the frame memory; and (e) mapping a frame of the moving image data stored in the texture area per vertical display period onto the surface of the object presented by the two-dimensional image data stored in the drawing area of the frame memory. According to another aspect of the present invention, there is provided a video game apparatus for producing a video signal corresponding to an image of an object having a dynamically changing texture by using three-dimensional image data and moving image texture pattern data, comprising: a main memory for receiving and storing said three-dimensional image data, said three-dimensional image data being indicative of shape, coordinate position and orientation of said object in three-dimensional space; means for performing three-dimensional coordinate transformation on said three-dimensional image data stored in said main memory, for converting said transformed three-dimensional image data to respective two-dimensional image data for display on a display, and for sending said two-dimensional image data back to said main memory, said three-dimensional coordinate transformation including a geometric transformation of said object in said three-dimensional space; receiving means for receiving said moving image texture pattern data, and for supplying said moving image texture pattern data frame-by-frame; a frame memory including a texture area for storing said moving image texture pattern data supplied frame-by-frame by said receiving means, a display area for storing image data of a current frame being displayed and a drawing area providing a work area for composing image data of a next frame to be displayed, said two-dimensional image data of said object to be displayed in a next frame being stored in said display area; image synthesizing means for synthesizing a dynamically changing textured image of said object by using said two-dimensional image data and said moving image texture pattern data stored in said frame memory; and output means for converting said image data of a current frame which is stored in said display area to supply said video signal. The present invention will be more easily understood from the following detailed description given by way of example only, with reference to the accompanying drawings in which: Fig. 1 is a block diagram of an embodiment of the present invention; Fig. 2 is a flowchart useful in explaining an embodiment of the present invention; Fig. 3 is a diagram useful in explaining a memory construction in an embodiment of the present invention; Figs. 4A-4C are diagrams useful in explaining a texture operation in an embodiment of the present invention; Figs. 5A-5C are diagrams useful in explaining a texture operation in an embodiment of the present invention; Fig. 6 is a diagram useful in explaining a texture operation in an embodiment of the present invention; Fig. 7 is a diagram useful in explaining a texture operation in an embodiment of the present invention; and Fig. 8 is a diagram useful in explaining a color look-up table in an embodiment of the present invention. Fig. 1 shows one embodiment of the present invention, in which a CPU 2, a main memory 3, an image expansion unit 4, a CD-ROM decoder 5, an image synthesis unit 6, a coordinate transformation unit 7, and a DMA controller 8 are connected to a system bus 1. A CPU of 32 bits is advantageously used as the CPU 2, and this CPU 2 manages the whole system. For example, a sorting process to decide the order within which to draw a polygon onto a screen is executed by the CPU 2. The image expansion unit 4 executes an expanding process of an image that was compressed by a discrete cosine transform (DCT), for example. The CD-ROM decoder 5 is connected to a CD-ROM driver 9 and decodes an application program or data from a CD-ROM loaded in the CD-ROM driver 9. Moving image data, which can be image data compressed by a DCT, can be recorded in the CD-ROM. The image synthesis unit 6 is connected to a frame memory 10 through a local bus 11. The frame memory 10 is constructed by two frame memories, and the two frame memories are switched every vertical period. The frame memory 10 includes a texture area to store a texture image and a drawing area to store an image to be drawn. Further, a color look-up table (CLUT) is provided in the frame memory 10. An output for display of the image generated by the image synthesis unit 6 is generated through a D/A converter 12. As an example of the computer generated image operation the image synthesis unit 6 executes a slope calculation and a polygon drawing. The polygon data in the main memory 3, which was sorted by the CPU 2, is sent to the image synthesis unit 6 in the sorted order and a slope is calculated by a slope calculating unit (not shown) that is part of the image synthesis unit 6. The slope calculation is a calculation to obtain a slope of a plane of the modified mapping data when the inside of a polygon is filled with mapping data in the drawing of a polygon. In the case of texture, the polygon is filled with texture data and in the case of Gouraud shading, the polygon is filled with luminance values. The coordinate transformation unit 7 executes a 3-dimensional coordinate transformation and a conversion from 3-dimensional data to 2-dimensional data on the screen. The coordinate transformation unit 7 and the main memory 3 can perform direct memory access (DMA) transfer of the data by the DMA controller 8. The coordinate transformation unit apparatus 7 obtains the data from the main memory 3 through the system bus 1 and executes a coordinate transformation of the data and then again transfers the converted data back into the main memory 3 through the system bus 1. Data of an object represented in the 3-dimensional space comprises modelling data indicative of a shape of the object and geometry data indicative of the coordinate position in which the object is displayed. The DMA controller 8 executes controls relevant to image synthesizing processes that use memories, such as the writing of data to memories and the reading of data from memories. Fig. 2 shows a flowchart of the overall steps involved in providing the polygon drawing. When a command is generated from the CPU 2 in step 101, the data from the main memory 3 is transferred to the coordinate transforming apparatus 7 through the system bus 1. The 3-dimensional coordinate transformation is then executed in the coordinate transforming apparatus 7 in step 102, so that the data is converted from 3-dimensional data to 2-dimensional data in step 103. The converted data is transferred to the main memory 3 through the system bus 1 and is sorted by the CPU 2 in step 104. The data sorted by the CPU 2 is sent to the image synthesis unit 6 through the system bus 1. The slope is calculated by the image synthesizing apparatus 6 in step 105 and a polygon is drawn in step 106. The resultant image is generated and displayed in step 107. The frame memory 10 has a texture area to store the texture image and a drawing area to store the image to be drawn, and Fig. 3 shows a memory space of the frame memory 10. The frame memory 10 has been addressed by 2-dimensional addresses of columns and rows. In the 2-dimensional address space, areas AT1, AT2, AT3, and so on are set as the texture areas and any number of kinds of texture patterns can be arranged into the texture areas AT1, AT2, AT3, and so on. AD1 denotes a drawing area and an image plane to be drawn is developed in the drawing area AD1. The areas AC1, AC2, AC3, and so on denote color look-up table (CLUT) areas. In the case of adhering the texture to the surface of the object that has.been generated, the texture data in the texture areas AT1, AT2, AT3 is 2-dimensional mapping transformed. For example, the texture pattern TP1 as shown in Fig. 4A is coordinate transformed onto a 2-dimensional screen as shown in Fig. 4B. The mapping transformed texture pattern TP1 is sent to the drawing area AD1 as shown in Fig. 4C and synthesized onto the surface of the object on the drawing area AD1. In the case of mapping the texture to the image of a polygon, texture patterns T1, T2, and T3 in the texture areas AT1, AT2, and AT3 as shown in Fig. 5A are read out and are 2-dimensional mapping transformed and adhered onto the surfaces of a solid object OB1 shown in Fig. 5B. Subsequently, as shown in Fig. 5C, the textures T1, T2, and T3 are adhered onto the surfaces of the object OB1 and are arranged in the drawing area AD1. The image plane of the drawing area AD1 is then displayed on the screen as the computer generated graphic. In the case of still image texture, the texture patterns in the main memory 3 are transferred to the texture areas AT1, AT2, AT3, and so on in the frame memory 10 through the image synthesis unit 6, and the image synthesis unit 6 adheres the texture patterns to the surfaces of the polygon that appear on the screen. Due to this operation, the texture of the still image is realized on the object. Further, a texture can also be applied to a moving image. In the case of a moving image texture, the compressed moving image data from the CD-ROM is once read into the main memory 3 and the compressed image data is then sent to the image expander 4. The image data is expanded by the image expander 4 and the moving image data is sent to the texture areas AT1, AT2, AT3, and so on in the frame memory 10. Because the texture areas AT1, AT2, AT3 are provided in the frame memory 10, the texture pattern itself can be also rewritten every frame. When a moving image is sent to the texture areas AT1, AT2, and AT3 the texture is dynamically rewritten and changes in the texture of the moving image are realized. The compressed moving image data from the CD-ROM is read into the main memory 3, the compressed image data is expanded by the image expander 4, and the expanded data is sent to the drawing area AD1 in the frame memory 10, so that the moving image can be directly drawn on the screen. Further, the object in the drawing area AD1 in the frame memory 10 can be moved to the texture areas AT1, AT2, AT3. Due to this, the object with the texture to the texture thereto can be further set to the next texture pattern. Due to this, a texture pattern having a very large change can be formed. For instance, as shown in Fig. 7, texture patterns T11, T12, and T13 existing in the texture areas AT1, AT2, and AT3, respectively, are adhered to the object and developed into the drawing area AD1. The image plane in the drawing area AD1 is subsequently moved to the texture area AT4 and a texture pattern T14 in the texture area AT4 is further adhered to an object OB22. As mentioned above, the object to which the texture was adhered can itself become a texture. The color look-up table (CLUT) areas AC1, AC2, and AC3 are provided in the frame memory 10. For instance, as shown in Fig. 8, CLUT tables C1, C2, C3 are provided in the areas AC1, AC2, AC3 for texture patterns T21, T22,.T23, respectively. The colors of the texture patterns T21, T22, T23 are designated by the CLUT tables C1, C2, and C3, respectively. In many cases, such as where an image is used as a texture pattern, a special color is used, like a surface pattern of brick. In such an image, by previously registering into the CLUT the colors that are used, the number of bits per pixel can be reduced. Nevertheless, in general, the kinds of colors that are used have a tendency such that they are different for every texture, for example, the surface pattern of brick and the surface pattern of stone. When such different surfaces must be expressed by a single CLUT, the image quality is deteriorated due to the decrease in number of colors. According to an embodiment of the invention, therefore, a plurality of CLUT tables C1, C2, and C3 are prepared and the CLUT can be selected for each specific texture. There is no need to have a one-to-one corresponding relation between the CLUT table and the texture pattern. One CLUT table can also be commonly used by a plurality of texture patterns. For example, assuming that the number of bits of the texture is equal to four, when a single CLUT table is used, the number of colors which can be expressed by the texture mapping is equal to only sixteen, however, if the CLUT table can be selected for every texture pattern, the number of colors that can be expressed is equal to 48 colors, even when the number of bits is only equal to four. Having described a specific preferred embodiment of the present invention with reference to the accompanying drawings, it is to be understood that the invention is not limited to that precise embodiment, and that various changes and modifications may be effected therein by one skilled in the art without departing from the scope or the spirit of the invention as defined in the appended claims. For example, according to the embodiment described above, the image data has been compressed and recorded in the CD-ROM, however, the image data can be also compressed and recorded on other recording medium, such as a magnetic disc or a semiconductor memory, like a memory card. The image compressing method is not limited to the DCT but other compressing methods can be also advantageously used.

Claims: A real time texture mapping method for producing a two-dimensional image of an object having a surface that has dynamically changing texture by using three-dimensional image data and two-dimensional moving image data, comprising the steps of: (a) receiving the three-dimensional image data and the two-dimensional moving image data; (b) transforming the three-dimensional image data to respective two-dimensional image data; (c) storing the two-dimensional image data in a drawing area (AD1) of a frame memory (10); (d) storing the two-dimensional moving image data frame-by-frame as texture pattern data in a texture area (AT1,AT2,AT3,AT4) of the frame memory (10); and (e) mapping a frame of the moving image data stored in the texture area (AT1,AT2,AT3,AT4) per vertical display period onto the surface of the object presented by the two-dimensional image data stored in the drawing area (AD1) of the frame memory (10). A real time texture mapping method according to claim 1 comprising the further step of reproducing the moving image data from a CD-ROM, the reproducing step preceding the receiving step. A real time texture mapping method according to claim 1 or 2 wherein step (e) stores a plurality of frames of moving image data in a respective plurality of subareas of the texture area (AT1,AT2,AT3,AT4). A real time texture mapping method according to claim 1 or 2 wherein step (e) stores a plurality of frames of moving image data in a single texture area (AT1,AT2,AT3,AT4) one frame per vertical display period. A real time texture mapping method according to claim 1 further comprising the steps of: storing in the texture area (AT1,AT2,AT3,AT4) of the frame memory (10) the new two-dimensional image data having the texture pattern data mapped on the surface of the object therein as new texture pattern data; using the new two-dimensional image data having the texture pattern data mapped on the surface of the object therein as a current image frame for display; and using the new texture pattern data of the object having mapped thereon the texture pattern data as a new texture pattern to map onto a second object stored in the drawing area (AD1) for a next image frame to be displayed. A real time texture mapping method according to claim 5 comprising the further step of reproducing the texture pattern data from a CD-ROM, the texture pattern data from the CD-ROM comprising frames of a moving image, the reproducing step preceding the receiving step. A video game apparatus for producing a video signal corresponding to an image of an object having a dynamically changing texture by using three-dimensional image data and moving image texture pattern data, comprising: a main memory (3) for receiving and storing said three-dimensional image data, said three-dimensional image data being indicative of shape, coordinate position and orientation of said object in three-dimensional space; means (7) for performing three-dimensional coordinate transformation on said three-dimensional image data stored in said main memory (3), for converting said transformed three-dimensional image data to respective two-dimensional image data for display on a display, and for sending said two-dimensional image data back to said main memory (3), said three-dimensional coordinate transformation including a geometric transformation of said object in said three-dimensional space; receiving means for receiving said moving image texture pattern data, and for supplying said moving image texture pattern data frame-by-frame; a frame memory (10) including a texture area (AT1,AT2,AT3,AT4) for storing said moving image texture pattern data supplied frame-by-frame by said receiving means, a display area for storing image data of a current frame being displayed and a drawing area (AD1) providing a work area for composing image data of a next frame to be displayed, said two-dimensional image data of said object to be displayed in a next frame being stored in said display area; image synthesizing means (6) for synthesizing a dynamically changing textured image of said object by using said two-dimensional image data and said moving image texture pattern data stored in said frame memory (10); and output means (12) for converting said image data of a current frame which is stored in said display area to supply said video signal. The video game apparatus according to claim 7 wherein said display and drawing areas (AD1) of said frame memory (10) being functionally interchangeable during a first vertical display period one of said display and drawing areas (AD1) performing a first function of storing image data of a current frame being displayed and other of said display and drawing areas (AD1) performing a second function as a work area for composing image data of a next frame to be displayed and during a second vertical display period the one of said display and drawing areas (AD1) performing the second function and the other of said display and drawing areas (AD1) performing the first function. The video game apparatus according to claim 7 further comprising: reproducing means (5,9) for reproducing compressed moving image data from a recording medium and for supplying said reproduced compressed moving image data to said receiving means, said receiving means expanding said compressed moving image data, and supplying said expanded data frame-by-frame as said moving image texture pattern data to said frame memory (10). The video game apparatus according to claim 9 wherein said reproducing means (5,9) optically reproduces. said compressed moving image data from said recording medium. The video game apparatus according to claim 9, said recording medium being a CD-ROM, and said reproducing means (5,9) comprising: a CD-ROM driver (9) for reproducing data recorded on said CD-ROM; and a CD-ROM decoder (5) connected to said system bus for decoding said compressed moving image data from said data reproduced by said CD-ROM driver (9) and supplying said decoded compressed moving image data said receiving means. The video game apparatus according to any one of claims 7 to 11 wherein said output means (12) comprises a digital to analog converter (12). The video game apparatus according to any one of claims 7 to 12 further comprising a system bus, wherein said main memory (3) and said image synthesizing means (6) are connected to said system bus, and said frame memory (10) is connected to said system bus through said image synthesizing means (6).

Inventors: FURUHASHI MAKOTO, SUZUOKI MASAKAZU, FURUHASHI, MAKOTO, SUZUOKI, MASAKAZU
Applicants: SONY COMP ENTERTAINMENT INC, SONY COMPUTER ENTERTAINMENT INC.
Application year: 1993
Main classification: G06T17
IPC codes: G06T15, A63F13