Abstract:
This invention provides a still image data generating apparatus for generating still image data from a plurality of frame mage data. The still image data generating apparatus comprises a positional relationship data generator configured to generate positional relationship data representing a relative positional relationship of images represented by the plurality of frame image data and an image data arrangement unit configured to generate still image data for which the plurality of frame image data are arranged in accordance with the positional relationship data.

Description:
BACKGROUND OF THE INVENTION  
       [0001]     1. Field of the Invention  
         [0002]     The present invention relates to an art for generating still image data from a plurality of image data.  
         [0003]     2. Description of the Related Art  
         [0004]     In recent years, art for generating still image data from moving image data recorded using a digital video camera or other moving image imaging device has been becoming more widespread. For the process of generating still image data from moving image data, generally, it is possible to generate a plurality of still image data from moving image data, so the task of organizing the generated plurality of still image data arises. Furthermore, cases are also assumed of it being difficult to grasp where a subject, for an image for which part of the subject has been imaged, has moved to.  
         [0005]     In such a case, if the positional relationship of the images represented by a plurality of still image data is displayed, this lightens the burden of the organizing task and of understanding the still images. Furthermore, the type of problems described above are not limited to generation of still image data from moving image data, but the same problems may also occur for a plurality of successively imaged still image data, for example.  
         [0006]     However, conventionally, use of information representing the positional relationship of a plurality of still image data for this kind of application had not been thought of.  
       SUMMARY OF THE INVENTION  
       [0007]     The present invention was created to solve the problems described above for the prior art, and its purpose is to provide an art for acquiring information representing the positional relationship of a plurality of still image data for the process of generating still image data from a plurality of still image data.  
         [0008]     To solve at least part of the problems described above, the present invention provides a still image data generating method of generating still image data from a plurality of frame image data. The still image data generating method comprises: (a) a positional relationship data generating step of generating positional relationship data representing a relative positional relationship of images represented by the plurality of frame image data; and (b) an image data arrangement step of generating still image data for which the plurality of frame image data are arranged in accordance with the positional relationship data.  
         [0009]     With the still image data generating device of the present invention, it is possible to arrange these frame image data according to the positional relationship data representing the relative positional relationship of images represented by the plurality of frame image data acquired from the plurality of frame image data, allowing for reduction of the burden on the user for understanding and organizing still images.  
         [0010]     Here, “frame image data” includes not only still image data for forming non-interlace format moving image data, but also still image data for forming interlace format moving image data. In the case of the interlace format, each of the frame image data correlates to still image data generated from, for example, odd numbered field still image data formed from image data of odd numbered scan lines, and even numbered field still image data formed from image data of even numbered scan lines. Furthermore, “frame image data” is a broad concept including not only still image data that forms moving image data, but also a plurality of still image data imaged successively, for example.  
         [0011]     Note that the present invention may be realized in various formats such as a still image data output device, an image file generating device, a computer program for realizing on a computer the functions of a still image data generating method or device, a recording medium on which that computer program is recorded, data signals realized within carrier signals including that computer program, computer program products, and the like. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0012]      FIG. 1  is an explanatory drawing showing an image processing system  100  as an embodiment of the present invention.  
         [0013]      FIG. 2  is an explanatory drawing showing an operating display screen  200  displayed on a display  18   a.    
         [0014]      FIG. 3  is a flow chart showing the contents of the still image data generating process of a first embodiment of the present invention.  
         [0015]      FIG. 4  is an explanatory drawing showing a frame image selected by a user.  
         [0016]      FIG. 5  is an explanatory drawing showing the status of still image data being generated with an embodiment of the present invention.  
         [0017]      FIG. 6  is an explanatory drawing showing the operating display screen  200  when a fourth frame image Iframe [486] is selected.  
         [0018]      FIG. 7  is an explanatory drawing showing the typical estimate results of optical flow estimation when there are nine sample points.  
         [0019]     FIGS.  8 ( a ),  8  ( b ),  8  ( c ), and  8  ( d ) are flow charts showing the contents of the still image data generating process of a second embodiment of the present invention.  
         [0020]      FIG. 9  is an explanatory diagram showing a frame image selected by a user for the second embodiment of the present invention.  
         [0021]      FIG. 10  is an explanatory diagram showing the operating display screen  200  when a fourth frame image Iframe [415] is selected. 
     
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0022]     Following, aspects for embodiment of the present invention are described based on embodiments in the following sequence. 
    A. Image Processing System Constitution:     B. Still Image Data Generating Process for a First Embodiment of the Present Invention:     C. Still Image Data Generating Process for a Second Embodiment of the Present Invention:     D. Variation Examples:    
 
         [0027]     A. Image Processing System Constitution:  
         [0028]      FIG. 1  is an explanatory drawing showing an image processing system  100  as an embodiment of the present invention. The image processing system  100  comprises an moving image database unit  30  for supplying moving image data, a user interface unit  18  for specifying a plurality of still image data acquired from moving image data, a personal computer PC as an image processing device for performing image processing on a plurality of still image data acquired from the moving image database unit  300  [sic;  30 ], and a color printer  20  as an output device for outputting processed still image data.  
         [0029]     The personal computer PC comprises a CPU  11  for executing arithmetic processing for image processing, a ROM  12 , a hard disk  14  for storing application software for image processing, a RAM  13  for temporarily storing image data or application programs, and an interface circuit  15  for performing the interfacing with an external device such as a moving image database unit  30 , a user interface unit  18 , or a color printer  20 .  
         [0030]     The moving image database unit  30  comprises a digital video camera  30   a , a DVD  30   b , and a hard disk  30   c  as moving image data supply sources. The moving image data is supplied to the personal computer PC as a collection of frame image data successively in time sequence. With this embodiment, frame image data is still image data for each frame. Note that with this embodiment, the time for one frame is 33 ms, and frame numbers are given in order of time sequence to each frame image data.  
         [0031]     The user interface unit  18  is a user interface for the user to specify frame image data acquired from moving image data. The user interface unit  18  comprises a display  18   a  for displaying moving images supplied from the moving image database unit  20 , still images being generated, and an operating display screen to be described later; and a keyboard  18   b  and a mouse  18   c  for receiving input from a user.  
         [0032]      FIG. 2  is an explanatory drawing showing an operating display screen  200  displayed on the display  18   a . The operating display screen  200  comprises a moving image area  210  for displaying moving images, a still image area  220  for displaying still images being generated, and an operating area  230  for receiving operation input by the user. Note that a preview screen for which the resolution has been dropped is displayed in the moving image area  210  and the still image area  220 .  
         [0033]     The operating area  230  comprises a plurality of buttons including various buttons for controlling moving images displayed in the moving image area  210 , a frame image acquisition button  236 , and a still image generating button  237 . Various buttons for controlling moving images include a play button  231 , a stop button  232 , a pause button  233 , a rewind button  234 , and a fast forward button  235 .  
         [0034]     With the hardware constitution described above, still image data including a plurality of frame images arranged so as to represent the positional relationship during image acquisition is generated as shown below. Note that the personal computer PC functions as the “frame image data acquisition unit,” the “positional relationship data generating unit,” and the “image data arrangement unit” noted in the claims.  
         [0035]     B. Still Image Data Generating Process for a First Embodiment of the Present Invention:  
         [0036]      FIG. 3  is a flow chart showing the contents of the still image data generating process of the first embodiment of the present invention. At step S 100 , the personal computer PC receives supply of moving image data from the moving image database unit  30 . The supplied moving image data is buffered in the RAM  13  and also displayed in the moving image area  210 .  
         [0037]     At step S 200 , the personal computer PC acquires frame image data according to input from the user. Input from the user is performed by placing a cursor (not illustrated) over the frame image acquisition button  236  and clicking the mouse  18   c.    
         [0038]      FIG. 4  is an explanatory drawing showing a frame image selected by the user. With this example, it is assumed that the digital video camera  30   a  zooms in on a scene ( FIG. 2 ) with the setting sun as a background and also pans the screen ( FIG. 2 ) from the upper left toward the lower right.  
         [0039]     The four images Iframe [238], Iframe [317], Iframe [415], and Iframe [486] indicate each part of the scene ( FIG. 2 ). Here, the image Iframe [238] means the frame image of the frame number 238.  
         [0040]     When the second image Iframe [317] is selected, the personal computer PC automatically generates positional relationship data (step S 300 ). The positional relationship data is data representing the positional relationship between the first image Iframe [238] and the second image Iframe [317]. With this embodiment, it is assumed that the digital video camera  30  will do panning, so the positional relationship data is a translational movement vector. The translational movement vector is represented by the horizontal direction and the vertical direction pixel count.  
         [0041]     For the translational movement vector, for example, when the selected successive frame image data time interval is relatively short, the overlapping image area is wide, so it is possible to generate this using publicly known image processing art such as pattern matching or special point tracking of these images. Meanwhile, when the successive frame image data time interval is relatively long, it is desirable to form the translational movement vector by generating and synthesizing it for each 10 frames (0.33 seconds), for example. In this case, the 10 frames correlates to the “specified frame interval” noted in the claims.  
         [0042]     At step S 400 , the personal computer PC arranges the two images Iframe [238] and Iframe [317] based on the positional relationship data. At step S 500 , overlapping processing is implemented on the arranged two images Iframe [238] and Iframe [317]. The overlapping process is a process of using as the pixel value of the overlapping area the pixel data of the image frame that is later in terms of time for the area for which the two images Iframe [238] and Iframe [317] overlap.  
         [0043]     This kind of process is processed in real time for each image selection by the user and the kind of still image being generated shown in  FIG. 5  is displayed in the still image area  220 . The same process is performed for selection of the third image Iframe [415] and the fourth image Iframe [486].  
         [0044]      FIG. 6  is an explanatory drawing showing the operating display screen  200  when the fourth frame image Iframe [486] is selected. The selected frame image Iframe [486] is displayed in the moving image area  210 . In this state, when the still image generating button  237  is clicked, still image data for representing the image displayed in the still image area  220  is generated (step S 600 ).  
         [0045]     In this way, with the first embodiment, it is possible to generate still image data for which frame image data is arranged according to the relative positional relationship of the images, so it is possible to generate still image data for which it is easy to understand the positional relationship of each frame image.  
         [0046]     C. Still Image Data Generating Process for a Second Embodiment of the Present Invention:  
         [0047]      FIG. 7  is a flow chart showing the contents of the still image data generating process of a second embodiment of the present invention. The still image data generating process of the first embodiment has a prerequisite of panning between successive images, but the still image data generating process of the second embodiment is different in that it assumes not only panning but also performance of other camera operations such as the zoom operation and rotation operation.  
         [0048]     This kind of feature is realized by changing the contents of the positional relationship data generating process (step S 300 ), the overlapping process (step S 500 ), and the still image data generating process (step S 600 ).  
         [0049]     With the positional relationship data generating process (step S 300   a ) of the second embodiment, the positional relationship data includes not only the translational movement vector but also a rotation angle and an image magnification. The rotation angle and the image magnification may be realized by combining an optical flow estimation and pattern matching, for example. There are various methods for optical flow estimation, and with the gradient-based method, for example, under the presumption that the brightness of the subject between the observed frame images does not change, the relationship between the space gradient and the time gradient of the concentration distribution of the subject in the image is used and the movement of the subject is estimated with the camera as a reference.  
         [0050]     Generation of relative position data for this embodiment is performed by roughly estimating the image translational movement and rotation as well as the zoom magnification change based on the results of the optical flow estimation, and also performing pattern matching between processed frame images based on these estimation results.  
         [0051]     FIGS.  8 ( a ),  8 ( b ),  8 ( c ), and  8 ( d ) are explanatory drawings showing the typical estimate results of optical flow estimation when there are nine sample points.  FIG. 8 ( a ) to  FIG. 8 ( d ) show the estimation results in a case when the subject has stopped, when the digital video camera  30   a  pans side to side, rotates counterclockwise, zooms in, and zooms out, respectively. In this way, with the optical flow estimation, it is possible to estimate the contents of camera operations not only of panning, but also when performing other camera operations such as the zoom operation and the rotation operation, so it is possible to generate positional relationship data including information such as the rotation angle and the image magnification as well.  
         [0052]     With the overlapping process (step S 500   a ) of the second embodiment, processing is performed for which overlapping of the plurality of frame image data in the overlapping area uses the magnification as a reference. In specific terms, image processing is performed so that the larger the magnification of the frame image data, the greater the priority for selection. The reason that selection priority is given the bigger the magnification of the frame image data is that it is possible to reproduce images with higher resolution the bigger the magnification of the frame image data.  
         [0053]      FIG. 9  is an explanatory diagram showing a frame image selected by the user for the second embodiment of the present invention. With this example, the same as with the first embodiment, the digital video camera  30   a  zooms in on a scene for which the setting sun is the background ( FIG. 2 ), and also assumes there will be panning from the upper left toward the lower right of the screen ( FIG. 9 ).  
         [0054]     A first image Iframe [086] is an image for representing the overall scene ( FIG. 2 ). Three images from the second and thereafter Iframe [238], Iframe [317], and Iframe [415] are zoomed-in images with relatively high magnification, indicating each part of the background ( FIG. 2 ). As a result, the three images from the second and thereafter Iframe [238], Iframe [317], and Iframe [415] are selected with priority in relation to the first image Iframe [086]. Furthermore, for the three images from the second and thereafter Iframe [238], Iframe [317], and Iframe [415], the same as with the first embodiment, images that come later in terms of time have priority for selection.  
         [0055]      FIG. 10  is an explanatory diagram showing the operating display screen  200  when the fourth frame image Iframe [415] is selected. Displayed in the still image area  220  are still image data for which the four frame images Iframe [086], Iframe [238], Iframe [317], and Iframe [415] are synthesized (step S 500 ). In this state, when the still image generating button  237  is clicked, still image data representing the image displayed in the still image area  220  is generated (step S 500   a ).  
         [0056]     With the still image data of the second embodiment (step S 500   a ), in contrast to the first embodiment, when generating the still image data, a resolution adjustment process is performed. The reason that the resolution adjustment process is performed is because the magnification of each frame image data is different. In specific terms, by performing an interpolation process of frame interpolation or median interpolation on frame image data with a relatively low magnification (with the example noted above, the initial image Iframe [086]), the resolution is made to match the other three frame image data.  
         [0057]     In this way, the second embodiment has the advantage of being able to generate positional relationship data not only with panning but also in cases when doing other camera operations such as zoom operations and rotation operations.  
         [0058]     D. Variation Examples:  
         [0059]     Note that the present invention is not limited to the embodiments and working examples noted above, but can be implemented in various aspects without straying from the main spirit of the claims, and variations such as the following are possible, for example.  
         [0060]     D-1. With the embodiments described above, because the subject is a scene with little movement, with the optical flow estimation, the camera operation is estimated, but, for example, it is also possible to apply the present invention when the subject is moving. With the optical flow estimation, it is possible to estimate the relative positional relationship of the subject and the moving image imaging device, so it is also possible to use the present invention for application such as synthesizing parts of images of a subject moving at high speed, for example.  
         [0061]     D-2. With the embodiments described above, positional relationship data is generated according to frame image data, but, for example, when the zoom magnification or other attribute information during other moving image imaging is included in the moving image data, the constitution may be such that the positional relationship data is generated using this attribute information.  
         [0062]     D-3. With the embodiments described above, the image data was formed with non-interlace format frame image data, but the present invention may also be used for interlace format moving image data. In this case, each frame image data for the embodiments described above would correlate to still image data generated from odd numbered field still image data formed from image data of odd numbered scan lines, and even numbered field still image data formed from image data of even numbered scan lines.  
         [0063]     D-4. With the embodiments described above, still image data is generated from moving image data, but this is not limited to moving image data, and, for example, it is also possible to generate still image data from a plurality of successively imaged still image data. Note that the “frame image data” noted in the claims is a broad concept including this kind of still image data.  
         [0064]     With the embodiments noted above, part of a constitution realized using hardware may be replaced by software, and conversely, part of a constitution realized using software may also be replaced by hardware.  
         [0065]     When part or all of the functions of the present invention are realized using software, that software (computer program) may be provided in a format stored on a computer readable recording medium. For this invention, a “computer readable recording medium” is not limited to a portable recording medium such as a flexible disk or a CD-ROM, but also includes internal storage devices within the computer such as various types of RAM or ROM or the like, or external recording devices attached to a computer such as a hard disk or the like.  
         [0066]     Finally, the Japanese patent application used as a basis for the priority claim of this application (Patent Application 2003-353176 (application date: Oct. 14, 2003)) is disclosed herein by way of reference.