Moving image conversion device, method and program, moving image distribution device, and e-mail relay device

A matrix of processing object frame data and a matrix of temporally consecutive next frame data of the processing object frame data, are prepared as matrices for error diffusion processing. An error generated at a pixel position in a frame, which is represented by the processing object frame data, is diffused to a corresponding pixel in a frame represented by the temporally consecutive next frame data of the processing object frame data.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a moving image conversion device and its method for converting a moving image into a color reduction moving image, a moving image distribution device for distributing a moving image to a mobile terminal device or the like, an e-mail relay device for relaying an e-mail, to which a moving image is attached, and sending the e-mail to a transmission destination, and a program for causing a computer to execute the moving image conversion method.

2. Description of the Related Art

Images obtained with digital cameras, scanners or the like can be represented in 8-bit colors for each of the RGB colors, which are 24-bit colors in total. In other words, the images can be represented in approximately 16.77 million colors. However, some image reproduction devices can reproduce a smaller number of colors (for example, 256 colors). Further, image formats such as gif (Graphic Interchange Format) and png (Portable Network Graphic), which are frequently used in websites, can process only 256 colors. Therefore, when the images, which were obtained with the digital cameras or the like, are displayed at the reproduction devices, which can reproduce a smaller number of colors, or the images are converted to image formats such as gif and png, color reduction processing is required to reduce the number of colors from 16.77 million to 256.

Conventionally, the color reduction processing as described above is performed by generating palette data representing palette colors of 256 colors, which are optimal colors for representing a color reduction processing object image in 256 colors. A color, into which the color of each pixel should be converted, is selected from the palette colors represented by the palette data, and color reduction processing is performed on each pixel.

Further, a method is proposed, wherein when moving images are obtained with digital video cameras, color reduction processing is performed on the moving images in real time, and images with movement (hereinafter called color reduction moving images) are produced by displaying still images consecutively in switching them (Japanese Unexamined Patent Publication No. 11(1999)-259640). In this method, the color reduction processing is performed by generating palette data for every few frames instead of each frame. A single set of palette data is used to perform color reduction processing on every few frames so that the color reduction processing is performed in real time without dropping any frames included in the moving image, which change every 1/60 second.

Here, in some cases, gradations in images are lost when color reduction processing is performed on the images. For example, if the palette data does not include orange, but includes only red and yellow, an orange area in an original image, which is represented by original image data before color reduction processing, can be represented only in red or yellow. Therefore, the orange area in the original image cannot be reproduced as an orange area in a color reduction moving image.

Therefore, color reduction processing is performed on the orange area using error diffusion processing. In the error diffusion processing, red pixels and yellow pixels are produced in the area in appropriate ratios to each other. Accordingly, although when each pixel is seen, the color of the pixel is red or yellow, when the area is seen as a whole, the color of the area is pseudo-orange.

The color reduction processing using error diffusion processing will be described below.FIGS. 19A and 19Bare diagrams for explaining the color reduction processing using error diffusion processing.FIG. 19Aillustrates the arrangement of pixels in an original image, which is represented by original data.FIG. 19Billustrates the arrangement of pixels in a processed image, which is represented by processed data, on which error diffusion processing has been performed. Here, to simplify the explanation, it is assumed that the original image includes 5×3 pixels, and the position of each pixel is represented by using (x, y) coordinates. It is also assumed that the pixel (1,1) at the upper left corner is a start point inFIG. 19A, and the color reduction processing is performed by performing error diffusion processing in the x direction and moving a pixel string, on which error diffusion processing is performed, in the y direction at the same time. It is also assumed that the palette data includes only four values of 10, 20, 30 and 40.

FIG. 20illustrates an example of a matrix for error diffusion processing. As illustrated inFIG. 20, a matrix T0for error diffusion processing is used to diffuse an error generated at a pixel (namely, a pixel corresponding to the center of the matrix, called P0), which is a color reduction processing object, to four neighboring pixels at the right side, the lower right side, the lower side and the lower left side of the pixel P0in the ratios of 7/16, 3/16, 5/16, and 1/16, respectively. Since error diffusion processing has been already performed on neighboring pixels at the upper left side, the upper side, the upper right side and the left side of the pixel P0, the elements at the upper left side, the upper side, the upper right side and the left side of the element at the center of the matrix T0do not have any values.

In the original data as illustrated inFIG. 19A, the pixel value of the pixel (1,1) is 24. Therefore, the pixel value of the pixel (1,1) is changed to 20, which is the closest value to 24 among the values included in the palette data. Accordingly, an error of 4 is generated at the pixel (1,1), and the error of 4 is diffused to pixels (2,1), (2,2), and (1,2) in the ratios of 7/16, 3/16, and 5/16, respectively. Next, color reduction processing is performed on the pixel (2,1). In the original data, the pixel value of the pixel (2,1) was 24. However, since a value of 7×4/16 has been added to 24 in the error diffusion processing performed on the pixel (1,1), the pixel value of the pixel (2,1) is 25.75. Therefore, the pixel value of the pixel (2,1) is changed to 30, which is the closest value to 25.75 among the values included in the palette data. Accordingly, an error of −4.25 is generated at the pixel (2,1), and the error of −4.25 is diffused to pixels (3,1), (3,2), (2,2), and (1,2) in the ratios of 7/16, 3/16, 5/16, and 1/16, respectively. Then, color reduction processing using error diffusion processing is sequentially performed on all of the pixels, and processed data can be obtained.

Here, each of the pixels in the original image has a pixel value of 23 to 27, but each of the pixels in the processed image has a pixel value of 20 or 30. However, the pixels having the pixel value of 20 and the pixels having the pixel value of 30 are produced in appropriate ratios to each other. Therefore, although there is no pixel, which has a pixel value between 20 and 30, in the processed image, pseudo gradations in the range of pixel values between 23 and 27 in the original data can be reproduced.

Meanwhile, functions of mobile terminal devices such as cellular camera phones are rapidly improving, and communication speeds thereof are improving. Therefore, mobile terminal devices have been proposed, which can capture moving images and send the captured moving images by attaching them to e-mails, or access moving image distribution servers, which distribute moving image contents, and download and reproduce desired moving image contents.

Further, although functions are limited to still images, an e-mail relay device has been proposed, which can perform appropriate image processing on images attached to e-mails, which were sent from a mobile terminal device with a camera, based on the model type of a terminal device at a destination, (U.S. Patent Laid-Open No. 20040105119). By using the e-mail relay device as described above, image processing may be performed on image data so that an image is produced, which is appropriate for an image size, the brightness of a display, a color tone, or the like of the mobile terminal device at the destination, and an e-mail, to which the image is attached, can be sent.

The color reduction moving images may be produced from the moving images by the method disclosed in Japanese Unexamined Patent Publication No. 11(1999)-259640. However, there is demand for production of high quality color reduction moving images by using the characteristics of the moving images.

SUMMARY OF THE INVENTION

In view of the foregoing circumstances, it is an object of the present invention to make it possible to produce higher quality color reduction moving images from moving images.

A moving image conversion device according to the present invention is a moving image conversion device for converting a moving image having a plurality of frames into a color reduction moving image including a predetermined number of colors, the device comprising:

a frame sampling means for sampling a plurality of frames for color reduction moving image production, which is used to produce the color reduction moving image, from the plurality of frames in the moving image;

a palette data generation means for generating palette data to produce the color reduction moving image;

a color reduction processing means for converting each of the plurality of frames for color reduction moving image production into a color reduction moving image frame by color reduction processing using error diffusion processing, wherein the error diffusion processing is performed with reference to the palette data and an error between the palette data and actual data, which is generated at each pixel position in each of the plurality of frames for color reduction moving image production, is diffused to a corresponding pixel position or a neighboring pixel position of the corresponding pixel position, which includes the corresponding pixel position, in the temporally consecutive next frame for color reduction moving image production, of each of the plurality of frames for color reduction moving image production; and

a frame combination means for combining the color reduction moving image frame to produce the color reduction moving image.

In the moving image conversion device according to the present invention, the color reduction processing means may increase the degree of diffusing the error to the corresponding pixel position in the temporally consecutive next frame for color reduction moving image production, when the number of the plurality of frames for color reduction moving image production per unit time is larger.

A moving image distribution device according to the present invention is a moving image distribution device for sending a requested moving image to a transmission request terminal device, which has requested transmission of the moving image, the device comprising:

a model type distinguishing means for distinguishing the model type of the transmission request terminal device; and

the moving image conversion device according to the present invention, wherein when the model type of the transmission request terminal device is a model type, which cannot display the moving image but can display a color reduction moving image, the requested moving image is converted to the color reduction moving image, and the color reduction moving image is sent to the transmission request terminal device instead of the moving image.

A e-mail relay device according to the present invention is a e-mail relay device for sending an e-mail, to which a moving image is attached, to a transmission destination terminal device, which is a transmission destination of the e-mail, the device comprising:

a model type distinguishing means for distinguishing the model type of the transmission destination terminal device; and

the moving image conversion device according to the present invention, wherein when the model type of the transmission destination terminal device is a model type, which cannot display the moving image but can display a color reduction moving image, the moving image, which is attached to the e-mail, is converted to the color reduction moving image, and the color reduction moving image is sent to the transmission destination terminal device instead of the moving image.

A moving image conversion method according to the present invention is a moving image conversion method for converting a moving image having a plurality of frames into a color reduction moving image including a predetermined number of colors, the method comprising the steps of:

sampling a plurality of frames for color reduction moving image production, which is used to produce the color reduction moving image, from the plurality of frames in the moving image;

generating palette data to produce the color reduction moving image;

converting each of the plurality of frames for color reduction moving image production into a color reduction moving image frame by color reduction processing using error diffusion processing, wherein the error diffusion processing is performed with reference to the palette data and an error between the palette data and actual data, which is generated at each pixel position in each of the plurality of frames for color reduction moving image production, is diffused to a corresponding pixel position or a neighboring pixel position of the corresponding pixel position, which includes the corresponding pixel position, in the temporally consecutive next frame for color reduction moving image production, of each of the plurality of frames for color reduction moving image production; and

combining the color reduction moving image frame to produce the color reduction moving image.

Further, a program for causing a computer execute the moving image conversion method according to the present invention may be provided.

According to the moving image conversion device and method of the present invention, a plurality of frames for color reduction moving image production is sampled from a plurality of frames of moving images to produce color reduction moving images, and palette data for producing the color reduction moving images are generated. Then, each of the plurality of frames for color reduction moving image production is converted to a color reduction moving image frame by performing color reduction processing using error diffusion processing. The error diffusion processing is performed with reference to the palette data. In the error diffusion processing, an error between actual data and the palette data, which is generated at each pixel position in each of the plurality of frames for color reduction moving image production, is diffused to a corresponding pixel position in the temporally consecutive frame for color reduction moving image production, of each of the plurality of frames for color reduction moving image production. Alternatively, the error is diffused to the neighboring pixel position of the corresponding pixel position, which includes the corresponding pixel position, in the temporally consecutive frame for color reduction moving image production, of each of the plurality of frames for color reduction moving image production. Further, the color reduction moving image frames are combined, and a color reduction moving image is produced.

Here, the color reduction moving image has a plurality of color reduction moving image frames, and the color reduction moving image has a characteristic similar to moving images, that a corresponding pixel in each of the frames of color reduction moving images closely correlates with each other. Therefore, when error diffusion processing is performed, an error, which is generated at each pixel in a frame for color reduction moving image production, is diffused to a corresponding pixel position or a neighboring pixel position of the corresponding pixel position, which includes the corresponding pixel position, in the temporally consecutive frame for color reduction moving image production. Accordingly, color reduction processing can be performed so that when a color reduction moving image is reproduced, the discontinuity in colors between frames is reduced. Therefore, the continuity in colors between frames is improved, and a higher quality color reduction moving image can be produced in comparison with the case of performing error diffusion processing only within a frame.

If a time interval between the frames for color reduction moving image production to produce the color reduction moving image is shorter, corresponding pixels in the frames correlate with each other more closely. Therefore, if the number of frames for color reduction moving image production per unit time is larger, the degree of diffusing an error to the corresponding pixel position in the temporally consecutive frame for color reduction moving image production is increased. Accordingly, the continuity in colors between the frames can be improved, and an even higher quality color reduction moving image can be produced.

According to a moving image distribution device of the present invention, if the model type of a transmission request terminal device, which has requested transmission of a moving image, is distinguished, and the transmission request terminal device cannot display a moving image, but can display a color reduction moving image, the moving image is converted to the color reduction moving image in the same manner as the processing by the moving image conversion device according to the present invention. Then, the color reduction moving image is sent to the transmission request terminal device instead of the moving image. Therefore, even if the terminal device, which has requested transmission of the moving image, cannot display the moving image, a high quality pseudo moving image can be displayed at the terminal device by using the color reduction moving image.

According to an e-mail relay device of the present invention, if the model type of a destination terminal device of an e-mail, to which a moving image is attached, is distinguished, and the destination terminal device cannot display a moving image, but can display a color reduction moving image, the moving image is converted to the color reduction moving image in the same manner as the processing by the moving image conversion device according to the present invention. Then, the color reduction moving image is sent to the destination terminal device instead of the moving image. Therefore, even if the destination terminal device cannot display the moving image, a high quality pseudo moving image can be displayed at the destination terminal device by using the color reduction moving image attached to the e-mail.

Note that the program of the present invention may be provided being recorded on a computer readable medium. Those who are skilled in the art would know that computer readable media are not limited to any specific type of device, and include, but are not limited to: floppy disks, CD's RAM'S, ROM's, hard disks, magnetic tapes, and internet downloads, in which computer instructions can be stored and/or transmitted. Transmission of the computer instructions through a network or through wireless transmission means is also within the scope of this invention. Additionally, computer instructions include, but are not limited to: source, object and executable code, and can be in any language including higher level languages, assembly language, and machine language.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention will be described with reference to the attached drawings.FIG. 1is a schematic block diagram illustrating the configuration of a moving image distribution system, to which a moving image conversion device and a moving image distribution device according to a first embodiment of the present invention are applied. As illustrated inFIG. 1, the moving image distribution system according to the first embodiment of the present invention includes a moving image distribution server1. Moving image data M0, which are generated at a personal computer2or the like, which is a moving image content producing device, are stored in the moving image distribution server1. Then, a mobile terminal device3such as a cellular phone, PHS (Personal Handy-Phone System), and PDA (Personal Digital Assistant) sends a moving image distribution request to the moving image distribution server1via a cellular phone communication network. Accordingly, the moving image distribution server1distributes the moving image data M0to the mobile terminal device3.

As illustrated inFIG. 1, the moving image distribution server1includes a moving image storage unit11, a request receiving unit12, a model type distinguishing unit13, a moving image format conversion unit14, a color reduction processing unit15, and an image output unit16.

The moving image storage unit11stores a plurality of sets of moving image data M0, which are generated by the personal computer2. The moving image data M0are encoded in a predetermined compression format and stored in the moving image storage unit11.

The request receiving unit12receives a moving image distribution request, which is sent from the mobile terminal device3. The moving image distribution request, which is sent from the mobile terminal device3, includes a file name of the requested moving image data M0and model type information indicating the model type of the mobile terminal device3.

The model type distinguishing unit13distinguishes the model type of the mobile terminal device3based on the model type information included in the request.

The moving image format conversion unit14reads out the moving image data M0, of which the file name is included in the request, from the moving image storage unit11. Further, the moving image format conversion unit14decodes the moving image data M0, and converts the format of the read-out moving image data M0based on the model type of the mobile terminal device3, which was distinguished by the model type distinguishing unit13. At this time, the moving image format conversion unit14converts the format of the moving image data M0with reference to a table showing the relationship between model types and formats.

FIG. 2is a table illustrating the relationship between model types and formats. As illustrated inFIG. 2, if the mobile terminal device3is a cellular phone, a table L1shows whether the cellular phone can display moving images, a display size of the cellular phone, the number of colors, which can be displayed at the cellular phone, and the number of color reduction moving image frames, which can be displayed at the cellular phone, for each model type of cellular phone. Then, the moving image format conversion unit14refers to the table L1and converts the format of the moving image data M0into a format, which is appropriate for the display size of the mobile terminal device3and the number of colors, which can be displayed at the mobile terminal device3based on the model type of the mobile terminal device3, which has sent the request. Accordingly, moving image data M1, of which format has been converted, is generated. Further, the moving image data M1is encoded.

If the mobile terminal device3, which has sent the request, cannot display moving images, the color reduction processing unit15converts the moving image data M1, of which format has been converted, into color reduction moving image data R1.FIG. 3is a schematic block diagram illustrating the configuration of the color reduction processing unit15. As illustrated inFIG. 3, the color reduction processing unit15includes a moving image input unit51, a decoding unit52, a frame division unit53, a palette data generation unit54, a frame sampling unit55, a processing unit56, a frame combination unit57, an encoding unit58, and a color reduction moving image output unit59.

The moving image input unit51receives an input of the moving image data M1, of which format has been converted, to the color reduction processing unit15.

The decoding unit52decodes the encoded moving image data M1, and generates raw moving image data M1′.

The frame division unit53divides the raw moving image data M1′ into image data for each frame (hereinafter called frame data Fr0). In the present embodiment, a single frame represents a moving image of 1/60 second, for example.

The palette data generation unit54generates standard frame data B0, which is used for generating palette data, from the frame data Fr0. Here, it is assumed that a first frame data Fr0in frames, which are arranged in time series, is used as the standard frame data B0. However, an arbitrary frame data may be used as the standard frame data B0. Alternatively, all of the sets of frame data Fr0or a plurality of sets of frame data Fr0′, which is selected from all of the sets of frame data Fr0, may be combined as a single set of image data, and the combined frame data may be generated as the standard frame data B0. The standard frame data B0may also be generated by averaging all of the sets of frame data Fr or a plurality of sets of frame data Fr, which is selected from all of the sets of frame data Fr. As the plurality of sets of frame data Fr0′, frame data sampled by the frame sampling unit55, which will be described later, may be used.

Further, the palette data generation unit54generates palette data P0including a required number of colors for color reduction processing, based on the standard frame data B0. Specifically, the palette data P0may be generated by using a median cut method or the like. However, the generation method of the palette data P0is not limited to the median cut method.

Here, the median cut method is a method for obtaining palette colors to be included in the palette data. In the median cut method, each of the RGB color data included in the standard frame data B0is plotted in three-dimensional RGB color space, a median value in the existing range of the color data in the three-dimensional space is obtained, and the obtained median value is used as one of the palette colors included in the palette data. Further, the space is divided into two parts by using the median value, and a median value in each of the divided parts of the space is obtained. These processes are repeated until the number of obtained colors reaches the number of colors in the palette data (256 colors, for example).

The frame sampling unit55samples frame data Fr1, which should be included in the color reduction image, from the frame data Fr0. Specifically, the frame sampling unit55refers to the table L1illustrated inFIG. 2and obtains information regarding the number of frames of the color reduction moving image, which can be displayed at the mobile terminal device3, which has requested the transmission. The frame sampling unit55samples the frame data Fr1, which corresponds to the number of frames. The frame sampling unit55samples the frame data Fr1in the period of time, which is obtained by dividing reproduction time of the moving image data M0by the number of frames. Besides sampling the frame data, new frame data may be generated by interpolating between frames.

The processing unit56performs color reduction processing on the frame data Fr1, which was sampled by the frame sampling unit55, by using the palette data P0. Specifically, the processing unit56performs color reduction processing using error diffusion processing. The color reduction processing using error diffusion processing will be described below.

FIG. 4is an example of a matrix, which is used for the error diffusion processing.FIGS. 5A and 5Bare diagrams for explaining the error diffusion processing. Here, to simplify explanations, it is assumed that a frame represented by the frame data Fr1includes 5×3 pixels, and the position of each pixel is represented by using (x, y) coordinates. It is assumed that the pixel (1,1) is a start point. The color reduction processing is performed by performing the error diffusion processing sequentially in the x direction and moving a pixel string, on which error diffusion processing is performed, in the y direction. It is also assumed that the palette data includes only four values of 10, 20, 30 and 40.

As illustrated inFIG. 4, matrices for error diffusion processing include a matrix T1for the processing object frame data Fr1and a matrix T2for the temporally consecutive next frame data Fr1′ of the processing object frame data Fr1. Each of the matrix T1and the matrix T2includes 3×2 elements. The matrix T1is used to diffuse an error generated at a color reduction processing object pixel (namely, a pixel P0, which corresponds to the center of elements in the matrix). The error is diffused to four neighboring pixels at the right side, the lower right side, the lower side and the lower left side of the pixel P0in the ratios of 7/24, 3/24, 5/24, and 1/24, respectively. The matrix T2is used to diffuse the error generated at the pixel P0to a pixel (P0′), which corresponds to the pixel P0, and four neighboring pixels of the pixel P0′ at the upper side, the lower side, the left side, and the right side in the temporally consecutive next frame data Fr1′, in the ratios of 4/24, 1/24, 1/24, 1/24 and 1/24, respectively. The ratio of the error, which is diffused to the pixel P0′, is larger than the ratio of the error, which is diffused to the neighboring pixels of the pixel P0′. Since the error diffusion processing has been performed on neighboring pixels of the pixel P0at the upper left side, the upper side, the upper right side and the left side of the pixel P0, the elements at the upper left side, the upper side, the upper right side and the left side of the element at the center of the matrix T1do not have any values.

If the error diffusion processing object is the pixel (2,2), since the pixel value of the pixel (2,2) in the frame data Fr1is 24 as illustrated inFIG. 5A, the pixel value of the pixel (2,2) is changed to 20, which is the closest value to 24 among the values included in the palette data. Consequently, an error of 4 is generated at the pixel (2,2). Therefore, the error of 4 is diffused to the pixels (3,2), (3,3), (2,3), and (1,3) in the ratios of 7/24, 3/24, 5/24, and 1/24, respectively, by using the matrix T1. Then, the error of 4 is diffused to the pixels (2,2), (2,1), (3,2), (2,3), and (1,2) in the next frame data Fr1′ in the ratios of 4/24, 1/24, 1/24, 1/24, and 1/24, respectively, by using the matrix T2.

Next, error diffusion processing is performed on the pixel (3,2) in the frame data Fr1. In the frame data Fr1, the original pixel value of the pixel (3,2) was 26, However, since when error diffusion processing was performed on the pixel (2,2), a value of 7×4/16 was added to 26, the pixel value of the pixel (3,2) is about 27.7. Therefore, the pixel value of the pixel (3,2) is changed to 30, which is the closest value to 27.7 among the values included in the palette data. Consequently, an error of −2.3 is generated at the pixel (3,2). The error of −2.3 is diffused to pixels (4,2), (4,3), (3,3), and (2,3), which are neighboring pixels of the pixel (3,2), in the frame data Fr1and a corresponding pixel (3,2) and neighboring pixels (3,1), (4,2), (3,3), and (2,2) of the corresponding pixel (3,2) in the next frame data Fr1′ by using the matrix T1and the matrix T2. Then, the error diffusion processing is performed on all of the pixels sequentially to diffuse an error generated at each pixel, which is a processing object, to neighboring pixels of the processing object pixel and to a corresponding pixel in the next frame data Fr1′. Accordingly, frame data Fr2, of which color is reduced, is obtained.

Meanwhile, as illustrated inFIG. 5A, the pixel value of the pixel (2,2) in the frame, which is represented by the next frame data Fr1′ of the frame data Fr1, is 24. However, since the error of 4 generated at the pixel (2,2) in the previous frame data Fr1is added to the pixel value of the pixel (2,2) in the frame data Fr1′ in the ratio of 4/24, the pixel value of the pixel (2,2) in the frame data Fr1′ is changed to about 24.2. Then, the pixel value of the pixel (2,2) is changed to 20, which is the closest value to 24.2 among the values included in the palette data. Then, error diffusion processing is performed and an error generated at each pixel is diffused to neighboring pixels of the processing object pixel and a corresponding pixel in the next frame data in the same manner as the case of the frame data Fr1. Accordingly, frame data Fr2′, of which color is reduced, is obtained.

The frame sampling unit55performs the processing as described above on all of the sets of frame data Fr1, which have been sampled, and the frame data Fr2, of which color is reduced, is generated.

The frame combination unit57arranges the sets of frame data Fr2, which are generated by the processing unit56, in time series and combines the sets of frame data Fr2into a single file. Accordingly, the frame combination unit57generates color reduction raw moving image data R0to reproduce the image of each frame consecutively in switching frames.

The encoding unit58encodes the color reduction raw moving image data R0, and generates color reduction moving image data R1. Here, animation gif (Graphic Interchange Format) and mng (Multiple-Image Network Graphics) for png (Portable Network Graphic Network), which corresponds to the animation gif, may be used as the file format of the color reduction moving image data R1. However, the file format of the color reduction moving image data R1is not limited to the gif or the mng.

The color reduction moving image output unit59outputs the color reduction moving image data R1, which is generated by the encoding unit58, to the image output unit16.

Next, processing in the first embodiment will be described.FIG. 6is a flow chart illustrating the processing in the first embodiment. The request receiving unit12receives a moving image distribution request, which is sent from the mobile terminal device3, and processing starts. The model type distinguishing unit13distinguishes the model type of the mobile terminal device3based on the model type information included in the request (step S1). Then, the moving image format conversion unit14reads out the moving image data M0, which includes a file name included in the request, from the moving image storage unit11(step S2). The moving image format conversion unit14refers to the table L1and converts the format of the moving image data M0into a format, which is appropriate for the model type of the mobile terminal device3, which has requested transmission. Accordingly, the moving image format conversion unit14obtains the moving image data M1, of which format has been converted (step S3).

Next, the color reduction processing unit15judges whether the model type of the mobile terminal device3, which has requested transmission, is a model type, which can display moving images (step S4). If step S4is NO, the color reduction processing unit15performs color reduction processing on the moving image data M1(step S5).

FIG. 7is a flow chart illustrating the color reduction processing. First, the moving image input unit51receives an input of the moving image data M1(step S11). The decoding unit52decodes the moving image data M1, and obtains the raw moving image data M1′ (step S12). Then, the frame division unit53divides the raw moving image data M1′ into frame data Fr0, which is image data for each frame (step S13). The palette data generation unit54generates the standard frame data B0based on the frame data Fr0(step S14). Further, the palette data generation unit54generates the palette data P0based on the standard frame data B0(step S15).

Meanwhile, the frame sampling unit55samples frame data Fr1, which should be included in the color reduction moving image, from the frame data Fr0(step S16). The processing in step S16may be performed before the processing in steps S14and S15. Alternatively, the processing in steps S14, S15and S16may be performed in parallel.

Then, the processing unit56generates the frame data Fr2, of which color is reduced, by performing color reduction processing on the frame data Fr1, which has been sampled by the frame sampling unit55, using error diffusion processing (step S17). In the error diffusion processing, an error is diffused to the temporally consecutive frame data by using the palette data as described above. Next, the frame combination unit57arranges the frame data Fr2, which was generated by the processing unit56, in time series and combines the frame data Fr2into a single file. Accordingly, the frame combination unit57generates the color reduction raw moving image data R0(step S18). The encoding unit58encodes the color reduction raw moving image data R0, and generates the color reduction moving image data R1(step S19). Further, the color reduction moving image output unit59outputs the color reduction moving image data R1(step S20), and the color reduction processing ends.

Processing returns to the processing illustrated inFIG. 6, and if step S4is YES and the color reduction processing in step S5ends, the image output unit16sends the color reduction moving image data R1or the moving image data M1to the mobile terminal device3, which has requested the transmission (step S6), and processing ends.

The mobile terminal device3receives the color reduction moving image data R1or the moving image data M1. Here, if the mobile terminal device3can reproduce moving images, the moving image data M1is sent to the mobile terminal device3, and the moving images can be displayed. In contrast, if the mobile terminal device3cannot reproduce the moving images, the color reduction moving image data R1is sent to the mobile terminal device3, and frames, which have been sampled from the moving image data M1, are displayed consecutively based on the color reduction moving image data R1in switching the frames. Accordingly, images with movement are displayed.

Here, the color reduction image includes a plurality of frames, and has a characteristic, which is similar to the characteristic of an moving image, that a corresponding pixel in each frame closely correlates with each other. Therefore, when the error diffusion processing is performed, an error generated at each pixel in a frame represented by the frame data Fr1is diffused to a corresponding pixel position and the neighboring pixel positions of the corresponding pixel position in the frame, which is represented by the temporally consecutive next frame data Fr1′. Accordingly, when the color reduction moving image data R1is reproduced, the color reduction processing can be performed so that the discontinuity in colors between the frames is reduced. Therefore, in comparison with the case of performing error diffusion processing only within a frame, the continuity in colors between frames can be improved, and color reduction moving image data R1for producing higher quality color reduction moving images can be generated.

Next, a second embodiment of the present invention will be described.FIG. 8is a schematic block diagram illustrating the configuration of an e-mail system, to which a moving image conversion device and a e-mail relay device according to the second embodiment of the present invention are applied. As illustrated inFIG. 8, the e-mail system according to the present embodiment includes an e-mail relay server101. In the e-mail system according to the present embodiment, an e-mail, to which a moving image is attached, is sent from the mobile terminal device3such as a cellular camera phone, a PHS with a camera and a PDA with a camera, which can capture moving images, and the e-mail is relayed by the e-mail relay server101and sent to a destination of the e-mail.

As illustrated inFIG. 8, the e-mail relay server101includes an e-mail receiving unit111, a moving image separation unit112, a model type distinguishing unit113, an image storage unit114, a color reduction processing unit115, an image attaching unit116, a URL (Uniform Resource Locator) attaching unit117, an e-mail transmission unit118, a request receiving unit119, an image read-out unit120, and an image transmission unit121.

The e-mail receiving unit111receives an e-mail E0, to which moving image data M0representing a moving image is attached. The e-mail E0was sent from a mobile terminal device3.

The moving image separation unit112separates the moving image data M0from the e-mail E0, and generates a mail body E1, which includes all the information in the e-mail E1except the moving image data M0. The mail body E1includes a mail text and a header including various kinds of information such as a sender's mail address and a destination mail address. The moving image separation unit112inputs the mail body E1and the moving image data M0to the model type distinguishing unit113.

The model type distinguishing unit113refers to a database113A, which shows the correspondence between mail addresses and model types, and judges whether the mobile terminal device3at the destination can display moving images based on the mail address of a mobile terminal device3at the destination, which is included in the header of the mail body E1. The model type distinguishing unit133also judges whether the mobile terminal device3at the destination can receive the e-mail, to which the moving image is attached, without reducing the number of colors.

Here, judgment results by the model type distinguishing unit113are described below:

(1) The mobile terminal device3at the destination can display moving images and receive an e-mail, to which an image is attached, without reducing the number of colors;

(2) The mobile terminal device3at the destination cannot display moving image, but the mobile terminal device3can receive an e-mail, to which an image is attached, without reducing the number of colors;

(3) The mobile terminal device3at the destination can display the moving image, but the mobile terminal device3cannot receive an e-mail, to which an image is attached, without reducing the number of colors; and

(4) The mobile terminal device3at the destination cannot display the moving image nor receive an e-mail, to which an image is attached, without reducing the number of colors.

If the judgment is as described in the above item (1), the model type distinguishing unit113inputs the moving image data M0to the image attaching unit116without causing the color reduction processing unit115to perform color reduction processing. Further, the moving image separation unit112inputs the mail body E1to the image attaching unit116.

If the judgment is as described in the above item (2), the model type distinguishing unit113inputs the moving image data M0to the color reduction processing unit115to cause the color reduction processing unit115to perform color reduction processing. Further, the moving image separation unit112inputs the mail body E1to the image attaching unit116.

If the judgment is as described in the above item (3), the model type distinguishing unit113inputs the moving image data M0to the image storage unit114without causing the color reduction processing unit115to perform color reduction processing. Further, the moving image separation unit112inputs the mail body E1to the URL attaching unit117.

If the judgment is as described in the above item (4), the model type distinguishing unit113inputs the moving image data M0to the color reduction processing unit115and causes the color reduction processing unit115to perform color reduction processing. Further, the moving image separation unit112inputs the mail body E1to the URL attaching unit117.

The image storage unit114stores the moving image data M0and the color reduction moving image data R1, which is generated by the color reduction processing unit115.

If the model type of the mobile terminal device3at the destination is a model type, which cannot display the moving image, the color reduction processing unit115converts the moving image data M0into the color reduction moving image data R1. The configuration of the color reduction processing unit115is the same as that of the color reduction processing unit15in the moving image distribution server1in the first embodiment as described above. Further, the processing in the color reduction processing unit115is also the same as the processing in the color reduction processing unit15. Therefore, detailed explanations on the color reduction unit115will be omitted.

If the mobile terminal device3at the destination can receive the e-mail, to which an image is attached, without reducing the number of colors, the image attaching unit116attaches the moving image data M0or the color reduction moving image data R1to the mail body E1, and generates an e-mail E2, to which an image is attached.

If the mobile terminal device3at the destination cannot receive the e-mail, to which the image is attached, without reducing the number of colors, the URL attaching unit117attaches a URL of a storage location of the moving image data M0or the color reduction moving image data R1in the image storage unit114to the mail body E1, and generates an e-mail E3, to which the URL is attached.

The e-mail transmission unit118sends the e-mail E2or the e-mail E3to the mobile terminal device3at the destination via a cellular phone communication network.

The request receiving unit119receives an image download request from the mobile terminal device3, which has received the e-mail E3, to which the URL is attached. The image distribution request from the mobile terminal device3includes the URL of the storage location of the moving image data M0or the color reduction moving image data R1.

The image read-out unit120refers to the URL included in the request, which has been received by the request receiving unit119, and reads out the moving image data M0or the color reduction moving image data R1from the image storage unit114.

The image transmission unit121sends the moving image data M0or the color reduction moving image data R1, which has been read out by the image read-out unit120, to the mobile terminal device3, which sent the request, via the cellular phone communication network.

Next, processing in the second embodiment will be described.FIG. 9is a flow chart illustrating the processing performed in the second embodiment. When the e-mail receiving unit111receives an e-mail E0, to which moving image data M0is attached, from the mobile terminal device3, processing starts. Then, the moving image separation unit112separates the moving image data M0from the e-mail E0(step S101). Next, the model type distinguishing unit113refers to the database113A, and judges which item the mobile terminal device3at the destination satisfies among the above items (1)-(4), based on the mail address of the mobile terminal device3, which is included in the header of the mail body E1(step S102).

If the mobile terminal device3satisfies the item (1), processing goes to the steps illustrated in the flow chart ofFIG. 10. Then, the model type distinguishing unit113inputs the moving image data M0to the image attaching unit116, and the moving image separation unit112inputs the mail body E1to the image attaching unit116, respectively (step S103). The image attaching unit116attaches the moving image data M0to the mail body E1, and generates the e-mail E2, to which the image is attached (step S104). The e-mail transmission unit118sends the e-mail E2to the mobile terminal device3at the destination (step S105), and processing ends.

Since the mobile terminal device3, which has received the e-mail E2, can display the moving image, the moving image data M0, which is attached to the e-mail E2, is displayed.

Meanwhile, if the mobile terminal device3satisfies the item (2), processing goes to the steps illustrated in the flow chart ofFIG. 11. Then, the model type distinguishing unit113inputs the moving image data M0to the color reduction processing unit115, and the moving image separation unit112inputs the mail body E1to the image attaching unit116(step S106). The color reduction processing unit115performs color reduction processing on the moving image data M0(step S107). The color reduction processing is performed using the error diffusion processing in the similar manner to the color reduction processing in the first embodiment. In the error diffusion processing, an error generated at each pixel in the frame, which is represented by the frame data Fr1, is also diffused to a corresponding pixel position in a frame, which is represented by the temporally consecutive next frame data Fr1′.

After step S107, the image attaching unit116attaches the color reduction moving image data R1to the mail body E1, and generates the e-mail E2, to which the image is attached (step S108). Then, the e-mail transmission unit118sends the e-mail E2to the mobile terminal device3at the destination (step S109), and processing ends.

The mobile terminal device3, which has received the e-mail E2, cannot display the moving image. Therefore, the mobile terminal device3displays frames, which have been sampled from the moving image data M0, consecutively in switching frames based on the color reduction moving image data R1attached to the received e-mail E2. Accordingly, the mobile terminal device3can display images with movement.

If the mobile terminal device3satisfies the item (3), processing goes to the steps illustrated in the flow chart ofFIG. 12. Then, the model type distinguishing unit113inputs the moving image data M0to the image storage unit114, and the moving image separation unit112inputs the mail body E1to the URL attaching unit117(step S110). The image storage unit114stores the moving image data M0(step S111). The URL attaching unit117attaches the URL of the storage location of the moving image data M0to the mail body E1, and generates the e-mail E3, to which the URL is attached (step S112). Then, the e-mail transmission unit118sends the e-mail E3to the mobile terminal device3at the destination (step S113), and processing ends.

If the mobile terminal device3satisfies the item (4), processing goes to the steps illustrated in the flow chart ofFIG. 13. Then, the model type distinguishing unit113inputs the moving image data M0to the color reduction processing unit115, and the moving image separation unit112inputs the mail body E1to the URL attaching unit117(step S114). The color reduction processing unit115performs color reduction processing on the moving image data M0(step S115). The image storage unit114stores the color reduction moving image data R1(step S116). The URL attaching unit117attaches the URL of the storage location to the mail body E1, and generates the e-mail E3, to which the URL is attached (step S117). Then, the e-mail transmission unit118sends the e-mail E3to the mobile terminal device3at the destination (step S118), and processing ends.

If the mobile terminal device3satisfies the item (3) or (4) as described above, the mobile terminal device3, which has received the e-mail E3, sends a download request of the moving image data M0or the color reduction moving image data R1to the e-mail relay server101.

FIG. 14is a flow chart illustrating the processing performed by the e-mail relay server101when the request is sent to the e-mail relay server101. When the request receiving unit119receives a download request, the e-mail relay server101starts processing. The image read-out unit120refers to a URL included in the request, and reads out the moving image data M0or the color reduction moving image data R1from the image storage unit114(step S141). Then, the image transmission unit121sends the moving image data M0or the color reduction moving data R1to the mobile terminal device3, which has sent the request (step S142), and processing ends.

The mobile terminal device3receives the color reduction moving image data R1or the moving image data M0. Here, if the mobile terminal device3can reproduce the moving image, the moving image data M0is sent. In contrast, if the mobile terminal device3cannot reproduce the moving image, the color reduction moving image data R1is sent, and frames, which have been sampled from the moving image data M0, are displayed consecutively based on the color reduction moving image data R1in switching frames, and images with movement are displayed.

As described above, in the second embodiment, the color reduction processing using the error reduction processing is performed in the similar manner to the first embodiment. In the color reduction processing using the error reduction processing, an error generated at each pixel in a frame represented by the frame data Fr1is also diffused to a corresponding pixel position in a frame represented by the temporally consecutive next frame data Fr1′. Therefore, when the color reduction moving image data R1is reproduced, the color reduction processing can be performed so that the discontinuity in colors between frames is reduced. Hence, in comparison with the case of performing the error diffusion processing only within a frame, the continuity in colors between frames can be improved, and the color reduction moving image data R1, from which a higher quality color reduction moving image can be produced, can be generated.

In the second embodiment as described above, the moving image separation unit112temporarily stores the e-mail E0, to which the moving image data M0is attached. If the model type distinguishing unit133judges that the mobile terminal device3satisfies the item (1) as described above, the transmission unit118may send the e-mail E0, to which the moving image data M0is attached, to the mobile terminal device3at the destination without reducing the number of colors.

In the second embodiment as described above, the e-mail relay server101relays the e-mail regardless of whether the mobile terminal device3at the destination can receive the e-mail, to which the image is attached, without reducing the number of colors. However, the e-mail relay server101may also be configured as a specialized relay server101, which is used when the mobile terminal device3at the destination can receive the e-mail, to which the image is attached, without reducing the number of colors. The e-mail relay server (called101A) which is configured in this manner as a third embodiment is illustrated inFIG. 15. When the e-mail relay server101A inFIG. 15and the e-mail relay server101inFIG. 8are compared, the image storage unit114, the URL attaching unit117, the request receiving unit119, the image read-out unit120, and the image transmission unit121in the e-mail relay server101are not in the e-mail relay server101A.

Alternatively, the e-mail relay server101may also be configured as a specialized e-mail relay server, which is used when the mobile terminal device3at the destination cannot receive the e-mail, to which the image is attached, without reducing the number of colors. The e-mail relay server (called101B), which is configured in this manner, is illustrated inFIG. 16as a fourth embodiment. When the e-mail relay server101B inFIG. 16and the e-mail relay server101inFIG. 8are compared, the image attaching unit116in the e-mail relay server101is not in the e-mail relay server101B.

In the first to fourth embodiments as described above, a file size of image data, which can be displayed at the mobile terminal device3, is limited. Therefore, if the color reduction moving image data R1is sent to the mobile terminal device3without considering the limitation in the file size, even if the color reduction moving image data R1is sent, the mobile terminal device3may not display the color reduction image in some cases.

Therefore, the color reduction moving image data R1is generated so that the file size is appropriate for the model type of the mobile terminal device3, which has sent the request, or the model type of the mobile terminal device3at the destination. Accordingly, the problems that although the color reduction moving image data R1is sent, the mobile terminal device3cannot display the color reduction moving image can be prevented.

Further, the color reduction processing unit15in the first embodiment and the color reduction processing unit115in the second to fourth embodiments may be used separately as a moving image conversion device for converting a moving image into a color reduction moving image. In this case, the moving image data M0may be converted into the color reduction moving image data R1regardless of the model type of the mobile terminal device3.

Further, in the first to fourth embodiments as described above, the error generated at each pixel, which is an error diffusion processing object, in the frame, which is represented by the frame data Fr1, is diffused to a corresponding pixel position and neighboring pixel positions of the corresponding pixel position in the frame represented by the temporally consecutive next frame data Fr1′ by using the matrix T2, which is illustrated inFIGS. 5A and 5B. However, when the error is diffused to the temporally consecutive next frame data Fr1′, the error may be diffused only to the corresponding pixel position by using the matrix T2′, which is illustrated inFIG. 17.

When the time interval between sets of frame data Fr1for generating the color reduction moving image data R1is shorter, the correlation between corresponding pixels in the frames represented by the sets of frame data Fr1becomes higher. Therefore, it is preferable to increase the degree of diffusing the error to the corresponding pixel in the temporally consecutive frame if the number of frames per unit time in the frame data Fr1is larger. For example, regarding the elements in the matrix T1and T2, which are illustrated inFIGS. 5A and 5B, if the elements in the matrix T1are decreased (7/24 is changed to 6/24, and 5/24 is changed to 4/24), and the elements in the matrix T2are increased (4/24 is changed to 6/24) as illustrated inFIG. 18A, the degree of diffusing the error to the corresponding pixel position in the temporally consecutive next frame data may be increased. In contrast, regarding the elements in the matrix T1and the matrix T2, which are illustrated inFIGS. 5A and 5B, if the elements in the matrix T1are increased (7/24 is changed to 8/24, and 5/24 is changed to 6/24), and the elements in the matrix T2are decreased (4/24 is changed to 2/24) as illustrated inFIG. 18B, the degree of diffusing the error to the corresponding pixel position in the temporally consecutive next frame data may be decreased.

The continuity of the colors between frames can be improved by increasing the degree of diffusing the error to the corresponding pixel in the temporally consecutive next frame, if the number of frames per unit time in the frame data Fr1is larger. Accordingly, the color reduction moving image data R1, which can represent even higher quality color reduction moving image data, can be generated.

Further, in the first to fourth embodiments as described above, a single set of palette data is generated, and color reduction processing is performed. However, a single set of palette data may be generated for every few frames by using the method disclosed in Japanese Unexamined Patent Publication No. 11(1999)-259640, and color reduction processing may be performed by using the single set of palette data for every few frames.