Method and medium for recording video information

A method and a medium for recording video information make it possible to retrieve particular scenes quickly based on representative images alone, which are chosen by taking the contents of images and sounds into account. The method for recording video information comprises a first step of storing video, a second step of dividing the video into segments, and extracting the top frame number of each segment, a third step of extracting the frame number of the frame having the maximum feature amount in each segment, and a fourth step of storing the frame numbers extracted by the second and third step in the first step.

BACKGROUND OF THE INVENTION

The present invention relates to the retrieval of video information and provides a method and a medium for recording video information that make it possible to retrieve video information quickly.

With the ever-increasing processing speed and capacity of computers, database systems and presentation tools capable of handling video information such as is recorded on movie films, videotapes, etc. have lately attracted considerable attention. However, it requires a lot of time and labor on the part of a user to retrieve desired pieces of information individually from a huge amount of video information and process the retrieved pieces of information. Accordingly, techniques to aid users in such retrieval and processing by using computers have been made available. For example, a technique, “Method of automatically compiling indexes and method of searching objects in color video images,” has been reported in the Transactions of Information Processing Society of Japan, Vol. 33, No. 4; and, another technique, “Method of detecting change points in video,” has been disclosed in Japanese Patent Laid-Open No. 4-111181. In accordance with these techniques, video is automatically divided into scenes, and a table of the first images of the scenes is prepared and stored. By using the table as if it were the index of a book, the retrieval and the editing of the video can be facilitated. According to these techniques, because the user can grasp the contents of video in a table at a glance, any scenes searched for can easily be found. In addition, because video is handled scene by scene, the video can easily be edited roughly, or scene-wise.

SUMMARY OF THE INVENTION

In accordance with the above techniques, however, the first, or the last, or the middle image is merely chosen from each scene on a fixed-timing basis to have the chosen image represent the scene. In other words, the importance of each image in a scene is not taken into account in choosing a representative image. In the video of a piece of news, for example, images captured by using a flashbulb or images on which subtitles are superimposed would be important. In the video of a sportscast, images captured when a great cheer arose would be important. Furthermore, images containing a close-up of a particular person or images wherein a large number of people appear can be important. In conventional methods of choosing representative images, however, the contents of images or sounds are not taken into account in choosing representative images. Therefore, when the user searches for a particular scene in a table of representative images, the scene often cannot be found because the information provided by the representative images alone is not sufficient. The user usually has to replay scenes as motion picture images to find the particular scene, which consumes a considerable time.

The object of the present invention is to provide a method and a medium for recording video information that make it possible to retrieve particular scenes quickly based on representative images alone, which are chosen by taking the contents of images and sounds into account.

To solve the above problem, the method of recording video information in accordance with the present invention comprises a first step of storing video on a medium, a second step of dividing the video into segments and extracting the top frame number of each segment, a third step of extracting the frame number of the frame image having the maximum feature amount in each video segment, and a step of storing the frame numbers extracted in the second and third steps on the medium.

In the second step, video is divided into segments each containing one and the same number of frames, or the division of video takes place each time the scene changes in the video.

The feature amount of each frame image in the third step is calculated from the video data or the audio data of the frame image. The feature amount calculated from the video data of each frame image is the sum of brightness of the pixels of the frame image, or the sum of the pixels having a specific color in the frame image, or the sum of differences in brightness between adjoining pixels in the frame image. The feature amount calculated from the audio data of each frame image is the sum of absolute values of the audio data corresponding to each frame.

The medium in the first step is a random-access medium, such as a magnetic disk, a compact disk, a digital video disc, or a flash memory.

These and other objects, features and advantages of the present invention will become more apparent in view of the following detailed description of the preferred embodiments in conjunction with accompanying drawings.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

An embodiment of the present invention will be described in detail below.

FIG. 1is a schematic block diagram of a system in which the present invention is embodied. The reference numeral1denotes a display unit such as a CRT, which displays the output images generated by a computer4. Instructions can be given to the computer4by using an input device5, such as a keyboard or a pointing device. A video playback device10is an optical-disk drive, a VCR (video cassette recorder), or the like. The video signals output from the video playback device10are sequentially converted by an A/D converter3into digital video data and sent to the computer. In the computer, the digital video data enters through an interface8and is stored into a memory9to be processed by a CPU7in accordance with a video indexing program9-1stored in the memory9. In the case when each of the frames of video handled by the video playback device10is assigned a number (a frame number) in the order of the top frame down to the bottom one, the video of a scene can be replayed by sending the corresponding frame numbers through a control line2to the video playback device. Various information necessary for processing can be stored in an external storage6, including video data6-1, index data6-2such as the frame numbers of representative images, and a video browsing program6-3.

According to the present embodiment, in the system including the video-storing device described above, video from the video playback device10is input frame by frame into the memory9in accordance with the video indexing program9-1. The input video is displayed on the display1.

Besides, the feature amounts of the frame images in the video are calculated and displayed, together with the video, on the display1. As the video advances from scene to scene, the frame of the maximum feature amount in each scene is treated as the representative image of the scene. Icons of representative images are created, and a table of icons is displayed on the display1. Information on the change of scenes and information on the representative images obtained during the above processing are, together with the video, stored in the external storage6.

Moreover, the video browsing program6-3is stored in the external storage6. Accordingly, even if the external storage6containing the video and the index information is disconnected from the computer4and connected to another computer, the video can be browsed without installing a browsing program in the computer. An editing program as well as the video browsing program may be stored in the external storage6as the occasion demands, which increases the utility of the system.

Although video is input from a video playback device such as a VTR, a DVD drive, or the like in the present embodiment, video of a telecast from a tuner, video from a video camera, or video stored in an external storage such as a HDD may be input.

In the case in which video data is of ten carried from place to place, an external storage capable of reading and writing a removable medium, such as an optical disk (compact disk), a digital video disc, or a flash memory, may be used.

FIG. 2shows a picture produced on the display unit of FIG.1. The window20includes a number of video windows displayed by the video indexing program9-1, including a video window21, a feature amount window22; a representative image window23. Check buttons24-1,24-2,24-3, and24-4are used to choose one among the various kinds of features, of which the amounts are extracted from frame images. If the check button24-1is selected, the video indexing program9-1extracts the brightness of each frame image as its feature amount.

In the same way, if the check button24-2,24-3, or24-4is selected, the color, the edge, or the sound, as the case may be, is extracted as the feature amount. Two or more buttons can be chosen simultaneously. Extracted feature amounts are displayed frame by frame as a graph22-1in the window22. The feature amount of each frame image of each video segment is calculated, and the frame image of the maximum feature amount is treated as the representative image of said video segment.

Each video segment is indicated in image window23by a rectangular parallelepiped23-1and the first frame number23-2of said video segment. The depth of the rectangular parallelepiped23-1represents the number of frames of said video segment, and the extracted representative image23-3is reduced in size and disposed at the depth corresponding to its frame number in the rectangular parallelepiped. Video is divided into segments by detecting a change of the scene or is so divided into segments that every segment contains the same number of frames.

The table of representative images so extracted is suitable for a user to browse the video from the viewpoint of his or her interest. Images of newscasts captured by using flashbulbs would be entered in the table of representative images of maximum brightness, on which the user can easily find important scenes. Close-ups of persons would be entered in the table of representative images of maximum flesh-color amounts. Images with subtitles would be entered in the table of representative images of maximum edge amounts, and the user, by using the table, can quickly browse programs wherein subtitles are used. By using the table of representative images of maximum sound amounts, the user can easily find highlight scenes of sport programs in which great cheers arose.

FIG. 3shows a data structure of video data6-1and the index data6-2stored in the external storage6. The video data6-1is stored in the form of a number of frame images6-1-3and the same number of frame numbers6-1-2following a header6-1-1wherein the video size, the compression format, etc. are described. The index data6-2comprise the total number of scenes6-2-1representing the number of video segments and scene structures6-2-2describing the contents of each scene. Each scene structure6-2-2comprises the top frame number6-2-2-1of the scene, and the number of frames6-2-2-2of the scene, the frame numbers of the representative images6-2-2-3corresponding to a plurality of features. According to the present invention, because the video data6-1and the index data6-2are linked to each other by the frame numbers, the user can replay pieces of video at random, referring to the index data. The time code, instead of the frame number, may be used to attain the same effect.

FIG. 4shows a flowchart of the video indexing program9-1that runs on the system of FIG.1. In accordance with the video indexing program9-1which is stored in the memory9, the CPU7executes a process operation40, in the first place, to initialize the windows21,22, and23ofFIG. 2, and, at the same time, resets the frame number “f” and the scene number S to “O's”. Then, in process operation41, the CPU7resets to “O” the status variable, “Status,” which controls the status of indexing. And, the CPU7executes the following process repeatedly until Status becomes “2”.

First, in process operation42, it is checked whether the button25has been pushed or not. In the case when the but ton25has been pushed in process operation43, it is checked whether Status is “O” or not. If Status is “O”, the status is advanced to “1” in process operation44to render the system ready for indexing. If the Status is “1”, it is advanced to “2” in process operation45to end the indexing.

Then, in process operation46, it is checked whether Status is “1” or not. In the case when Status is “1”, the following indexing process is executed.

First, in process operation47, a frame image constituting video is input from the video playback device10and stored into the memory9. The audio data attached to the frame, if necessary, is also stored in the memory9. Then, in process operation48, the frame number “f” is renewed by doing one to it. If the frame number can be read out from the video playback device10, the frame number so read out may be substituted for the variable “f” Then, in process operation49, the input frame image is displayed in the window21and, at the same time, is stored as video data6-1in the external storage6.

Then, in process operation50, the feature amounts are calculated from frame images or audio data to choose a representative image. The details of this process will be described later with reference to FIG.5. In process operation51, the graph of the feature amounts calculated in the process operation50is displayed in the window22.

Lastly, the process of dividing video into video segments is executed. First, in process operation52, the change of a scene is detected. The correlation between the current frame image and previous ones is determined from the differences in their color histogram, and the point in time when the correlation has been interrupted is regarded as the change of the scene. The technique disclosed in Japanese Patent Laid-Open No. 8-227462, “Method and device for detecting change points of video,” of the inventors, for example, can be used to execute the process operation52. Then, in process operation53, it is determined whether the scene has changed or not. In the case when the scene has changed, process operations54and55are executed. In the process operation54the scene number S is renewed by adding one to it. Then, in the process operation55, index data is prepared and displayed in the window23, and, at the same time, it is stored as index data6-2in the external storage6. This process will be described in detail later with reference to FIG.6. In the case when the scene has not changed, the representative image of the scene number S is extracted in process operation56. The details of this process will be described later with reference to FIG.7.

After the above process of indexing, it is checked in process operation60to determine whether Status is “2” or not. If it is “2”, the indexing is ended.

FIG. 5shows the details of the process of calculating feature amounts according to the present invention. First, in process operation100, it is checked to determine whether the brightness check button24-1has been selected or not. In the case when this button is chosen, process operation01is executed. In the process operation101, the sum of the brightness of the individual pixels of each frame image of each scene is calculated and stored in a variable, “Feature I”.

Then, in process operation110, it is checked to determine whether the color check button24-2has been selected or not. In the case where it is selected, process operation111is executed. In the process operation111, the number of pixels of the color designated in advance of each input frame image is calculated and stored in a variable, “Feature2”.

Similarly, in process operation120, it is checked to determine whether the edge check button24-3has been selected or not. In the case where it is selected, process operation121is executed. In the process operation121, the sum of absolute values of differences in brightness between adjacent pixels of each input frame image is calculated and stored in a variable, “Feature3”.

Lastly, in process operation130, it is checked to determine whether the sound check button244has been selected or not. In the case where it is selected, process operation131is executed. In the process operation131, the sum of sound intensities of the audio data corresponding to the frame span of each input frame image is calculated and stored in a variable, “Feature4”.

Although the above feature amounts are the number of specific pixels or the simple sum of a variable for each input frame image or for each frame span, they may be normalized by the number of pixels in the frame or by the frame span as the case may be.

FIG. 6shows the details of the process of preparing index data.

In process operation200, the current frame number “f” is treated as the top frame number Ts of a new scene S. In process operation210, the top frame number Ts of the scene S is displayed in the window22.

Then, in process operation220, the number of frames of the preceding scene S-1is calculated from the difference between the top frame number of the preceding scene S-I and that of the scene S. In process operation230, the representative image, or the frame image of the maximum feature amount, of the preceding scene S-1is disposed at an appropriate depth in the rectangular parallelepiped and displayed in the window23. Then, in process operation230, the index data of the previous scene S-1is stored in the external storage6.

Lastly, the process of initialization is executed to find the representative image of the new scene S.

First, in process operation250, it is checked to determine whether the brightness check button has been selected or not. In the case where it is selected, process operation251is executed. In the process operation251, the variable, “Feature1,” is substituted for the variable, “Max Feature1,” and the current frame number “f” is substituted for the variable, “Maxfno1.” Max Feature1is a variable to hold the maximum value of the feature amounts, and Maxfno1is a variable to hold the number of the frame of the maximum feature amount.

Next, in process operation252, it is checked to determine whether the color check button24-2has been selected or not. In the case where it is selected, process operation253is executed in the process operation253, the variable, “Feature2” is substituted for the variable, “Max Feature2”, and the current frame number “f” is substituted for the variable, “Maxfno2.

Similarly, in process operation254, it is checked to determine whether the edge check button24-3has been selected or not. In the case where it is selected, process operation255is executed. In the process operation255, the variable, “Feature3” is substituted for the variable, “Max Feature3”, and the current frame number “f” is substituted for the variable, “Maxfno3”.

Lastly, in process operation256, it is checked to determine whether the sound check button244has been selected or not. In the case where it is selected, process operation257is executed. In the process operation257, the variable, “Feature4” is substituted for the variable, “Max Feature4”, and the current frame number “f” is substituted for the variable, “Maxfno4”.

Thus, the process of initialization to find the representative image is ended. A process operation to be performed after this serves to update the maximum value of the feature amount on every entry of a frame image. This process will be described with reference to FIG.7.

FIG. 7shows the details of the process of extracting representative images according to the present invention.

First, in process operation300, it is checked to determine whether the brightness check button24-1has been selected or not. In the case where it is selected, process operation301is executed.

In the process operation301, it is determined whether the variable, “Feature I” is greater than the variable, “Max Feature1” or not. In the case where it is greater, the maximum value is updated in process operation302. Namely, the variable, “Feature1” is substituted for “Max Feature1”. Then, in process operation303, the frame number of the representative image is also updated. Namely, the frame number “f” is substituted for the variable, “Maxfno1”.

Next, in process operation310, it is checked to determine whether the color check button24-2has been selected or not. In the case when it is selected, process operation311is executed. In process operation311, it is checked to determine whether the variable, “Feature2” is greater than the variable, “Max Feature2”. In the case where it is greater, the maximum value is updated in process operation312. Namely, the variable, “Feature2” is substituted for “Max Feature2”. Then, in process operation313, the frame number of the representative image is also updated. Namely, the frame number “f” is substituted for the variable, “Maxfno2”.

Similarly, in process operation320, it is checked to determine whether the edge check button24-3has been selected or not. In the case where it is selected, process operation321is executed. In the process operation321, it is determined whether the variable, “Feature3” is greater than the variable, “Max Feature3” or not. In the case where it is greater, the maximum value is updated in process operation322. Namely, the variable, “Feature3” is substituted for “Max Feature3”. Then, in process operation323, the frame number of the representative image is also updated. Namely, the frame number “f” is substituted for the variable, “Maxfno3”.

Lastly, in process operation330, it is checked to determine whether the sound check button244has been selected or not. In the case where it is selected, process operation331is executed. In the process operation331, it is determined whether the variable, “Feature4” is greater than the variable, “Max Feature4”. In the case where it is greater, the maximum value is updated in process operation332. Namely, the variable, “Feature4” is substituted for “Max Feature4”. Then, in process operation333, the frame number of the representative image is also updated. Namely, the frame number “f” is substituted for the variable, “Maxfno4”.

According to the present invention, the entire video is divided into video segments, and the feature amounts are determined by contents of images and audio data of the video segments. Since index data is created using one or a plurality of representative images for each feature amount, it is possible to browse the video quickly in accordance with the viewpoint of a user.

Further, according to the present invention, since video data and index data are stored in the same medium, operability is not disturbed even when such data is carried and browsed by some other devices. Still further, a video browsing program as well as the video data and the index data may be stored in the same medium, which eliminates the inconvenience of installing the video browsing program on another device by which the former data can be browsed.

Also, according to the present invention, frame numbers are extracted and stored instead of frame images. Therefore, information corresponding to representative images can be stored using a memory with very little capacity.

While the present invention has been described above in conjunction with the preferred embodiments, one of ordinary skill in the art would be enabled by this disclosure to make various modifications to this embodiment and still be within the scope and spirit of the invention as defined in the appended claims.