Moving image processing apparatus, moving image processing method, and program

There is provided a moving image processing apparatus, including a similarity determination unit configured to determine a degree of similarity between a subsequent cut and first and second cut groups based on feature amounts generated from extraction images of the first cut group included in a moving image and feature amounts generated from extraction images of the second cut group included in the moving image, a cut grouping unit configured to group the subsequent cut into a similar cut group similar to the subsequent cut, which is one of the first and second cut groups, when the subsequent cut is similar to the first or second cut group, and group the subsequent cut into a third cut group when the subsequent cut is not similar to either of the first and second cut groups, a feature amount generation unit configured to compare extraction images extracted from the third cut group with the extraction images extracted from the first and second cut groups when the subsequent cut is not similar to either of the first and second cut groups, and generate feature amounts of the third cut group, and an image extraction unit configured to preferentially extract an image with a later time code of the moving image from images included in each cut group, thereby obtaining extraction images of each cut group.

BACKGROUND

The present disclosure relates to a moving image processing apparatus, a moving image processing method, and a program.

In general, a video (or a moving image) includes a plurality of cuts. In video expression, there is a case in which a cut structure is devised such as an arrangement of cuts with the same content at temporal intervals according to the intention of a maker.

SUMMARY

Further, there is a case in which it is necessary to understand a cut structure of a video in order to view or use the video. In such a case, the cut structure of the video is understood by dividing the video into a plurality of cuts and grouping similar cuts into a common cut group. However, when the cuts are not appropriately grouped, it may not be possible to appropriately understand the cut structure.

In light of the foregoing, it is desirable to provide a moving image processing apparatus, a moving image processing method, and a program, which can appropriately group cuts of a video.

According to an embodiment of the present disclosure, there is provided a moving image processing apparatus, including a similarity determination unit configured to determine a degree of similarity between a subsequent cut and first and second cut groups based on feature amounts generated from extraction images of the first cut group included in a moving image and feature amounts generated from extraction images of the second cut group included in the moving image, a cut grouping unit configured to group the subsequent cut into a similar cut group similar to the subsequent cut, which is one of the first and second cut groups, when the subsequent cut is similar to the first or second cut group, and group the subsequent cut into a third cut group when the subsequent cut is not similar to either of the first and second cut groups, a feature amount generation unit configured to compare extraction images extracted from the third cut group with the extraction images extracted from the first and second cut groups when the subsequent cut is not similar to either of the first and second cut groups, and generate feature amounts of the third cut group, and an image extraction unit configured to preferentially extract an image with a later time code of the moving image from images included in each cut group, thereby obtaining extraction images of each cut group.

When the subsequent cut is similar to the first or second cut group, the feature amount generation unit may compare extraction images extracted from the similar cut group including the subsequent cut with extraction images extracted from a cut group not similar to the subsequent cut, and may generate feature amounts of the similar cut group.

When the subsequent cut is similar to the first or second cut group, the image extraction unit may extract the extraction images from the subsequent cut and the cut group similar to the subsequent cut.

When the subsequent cut is similar to the first or second cut group, the image extraction unit may extract the extraction images from the subsequent cut.

The feature amount generation unit may compare extraction images extracted from the first cut group with extraction images extracted from the second cut group, and may generate feature, amounts of the first cut group and feature amounts of the second cut group.

The image extraction unit may extract images positioned at a regular interval in order from the latest time code from the images included in each cut group.

The image extraction, unit may extract images positioned at an interval expressed by geometric progression in order from the latest time code from the images included in each cut group.

The image extraction unit may extract images positioned at an interval expressed by progression of differences in order from the latest time code from the images included in each cut group.

According to another embodiment of the present disclosure, there is provided a moving image processing method, including, determining a degree of similarity between a subsequent cut and first and second cut groups based on feature amounts generated from extraction images of the first cut group included in a moving image and feature amounts generated from extraction images of the second cut group included in the moving image, when the subsequent cut is similar to the first or second cut group, grouping the subsequent cut into a similar cut group similar to the subsequent cut, which is one of the first and second cut groups, and when the subsequent cut is not similar to either of the first and second cut groups, grouping the subsequent cut into a third cut group, and comparing extraction images extracted froth the third cut group with extraction images extracted from the first and second cut groups to generate feature amounts of the third cut group. Extraction images of each cut group are obtained by preferentially extracting an image with a later time code of the moving image from images included in each cut group.

According to yet another embodiment of the present disclosure, there is provided a program causing a computer to execute the moving image processing method. The program may be provided in a computer-readable medium or via means of communication and the like.

According to the embodiment of the present disclosure described above, it is possible to provide a moving image processing apparatus, a moving image processing method, and a program, which can appropriately group cuts of a video.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

1. OVERVIEW OF MOVING IMAGE PROCESSING METHOD

First, the overview of the moving image processing method according to an embodiment of the present disclosure will be described with reference toFIG. 1.FIG. 1illustrates the overview of the moving image processing method according to the embodiment of the present disclosure.

As illustrated inFIG. 1, in the moving image processing method, among images of a first cut group included in the moving image MP, images with a later time code of the moving image MP are preferentially extracted (step S1). In the same manner, among images of a second cut group included in the moving image MP, images with a later time code of the moving image MP are preferentially extracted (step S2). Next, the degree of similarity between subsequent cuts and the first and second cut groups is determined based on feature amounts S1generated from extraction images of the first cut group and feature amounts S2generated from extraction images of the second cut group (step S3).

Then, it is determined whether the subsequent cuts are similar to the first or second cut group (step S4). When the subsequent cuts are similar to the first or second cut group (“Yes” in step S4), the subsequent cuts are grouped into a similar cut group (e.g., the first cut group) similar to the subsequent cut, which is one of the first and second cut groups (step S5).

Meanwhile, when the subsequent cuts are not similar to either of the first and second cut groups (“No” in step S4), the subsequent cuts are grouped into a third cut group (step S6). In addition, among images of the third cut group including the subsequent cut, images with a later time code of the moving image MP are preferentially extracted (step S7). Then, extraction images of the third cut group are compared with the extraction images of the first and second cut groups, and feature amounts S3of the third cut group are generated (step S8).

Here, in relation to the extraction images of each cut group, images with a later time code of the moving image MP are preferentially extracted from among the images included in each cut group. Furthermore, feature amounts S (the general term for feature amounts of cut groups) of each cut group are generated by emphasizing feature amounts of images near the subsequent cuts, which are to be subject to determination of similarity, among the images included in each cut group based on the time codes of the moving image MP. Consequently, it is possible to appropriately group the subsequent cuts by sufficiently considering the temporal context (a story and the like of the moving image MP) between cuts.

2. CONFIGURATION OF MOVING IMAGE PROCESSING APPARATUS1

Next, the moving image processing apparatus1according to the embodiment of the present disclosure will be described with reference toFIG. 2.FIG. 2illustrates the main functional configuration of the moving image processing apparatus1. As illustrated inFIG. 2, the moving image processing apparatus1includes a data acquisition unit11, a cut transition detection unit13, a cut grouping processing unit15, a cut structure image generation unit17, a cut structure image output unit19, a cut structure information output unit21, and a data storage unit23.

The data acquisition unit11acquires moving image data MP including a plurality of cuts, and supplies the moving image data MP to the cut transition detection unit13, the cut grouping processing unit15, and the cut structure image generation unit17. In general, the moving image data MP is data in a frame format, and may be only image data or a combination of image data and voice data. The moving image data MP may be acquired from the data storage unit23or an external apparatus (not shown).

The cut transition detection unit13detects cut transition in the moving image data MP, and supplies a detection result to the cut grouping processing unit15. The cut transition represents a change in cuts in the moving image MP. The feature amounts of images and/or voice in frames in succession is obtained and the cut transition is detected based on the degree of similarity of the feature amounts. As the feature amounts of the images and/or the voice, a color histogram and facial image detection, and/or the volume, tone/rhythm and the like of voice can be used. In addition, the detection result of the cut transition may be supplied to the cut structure information output unit21, the data storage unit23, an external apparatus and the like, in addition to the cut grouping processing unit15. Furthermore, feature amounts obtained when detecting the cut transition may be stored in the data storage unit23and the like and used for other processes.

The cut grouping processing unit15groups the plurality of cuts into cut groups based on the degree of inter-cut similarity, which will be described, in detail later. The cut grouping result is supplied to the cut structure image generation unit17together with the detection result of the cut transition. However, the cut grouping result may be supplied to the cut structure information output unit21, the data storage unit23, the external apparatus and the like. The cut grouping processing unit15assigns group IDs representing cut groups to the cuts based on the cut grouping result. The cut grouping processing unit15serves as an image extraction unit, a similarity determination unit, a cut grouping unit, and a feature amount generation unit

The degree of inter-cut similarity is generated from a color histogram of images and/or voice included in the cuts and a detection result of a facial image, and/or feature amounts of cuts including the volume, tone/rhythm and the like of the voice. The cut group represents a combination of cuts with similar feature amounts.

In order to generate a cut structure image CI, the cut structure image generation unit17extracts a representative image I (the general term for representative images) from images included in each cut based on the detection result of the cut transition according to predetermined standards. The representative image I is an image representing each cut, and for example, is extracted as an image corresponding to the center frame of the cut. Then, the cut structure image generation unit17generates the cut structure image CI indicating the cut structure of the moving image MP based on the moving image data MP and the cut grouping result. For example, the cut structure image CI is an image including a representative image I of each cut arranged in the order of the cut transition and indicating the cut groups.

The cut structure image output unit19outputs the cut structure image CI supplied from the cut structure image generation unit17such that a user can easily understand the cut structure of the moving image MP. The cut structure image CI may be output to a display apparatus, a printing apparatus, a storage apparatus, or an external apparatus (not shown) connected to the moving image processing apparatus1.

The cut structure information output unit21outputs the cut grouping result and the like as cut structure information such that a user can use the cut structure information to understand the cut structure. The cut structure information may be output to the display apparatus, the printing apparatus, the storage apparatus, or the external apparatus (not shown) connected to the moving image processing apparatus1. The cut structure information can be used as data for performing a moving image search considering the cut structure.

The data storage unit23stores the moving image data MP and data belonging to the moving image data MP. The data storage unit23may store the cut structure information, such as the detection result of the cut transition and the cut grouping result, and the cut structure image CI, which are associated with the moving image data MP. Furthermore, inFIG. 2, a part of a connection relationship between the data storage unit23and other elements is omitted.

Here, in the functional configuration, the data acquisition unit11, the cut transition detection unit13, the cut grouping processing unit15, the cut, structure image generation unit17, the cut structure image output unit19, and the cut structure information output unit21include an arithmetic processing unit such as a CPU or a digital signal processing apparatus (DSP). The data storage unit23includes an internal storage device such as a flash memory, or an external storage device such as a hard disk drive or a Blu-ray disc drive. The CPU develops and executes a program read from a ROM and the like on a RAM, thereby performing the moving image processing method. In addition, at least a part of the functional configuration may be realized by hardware such as a dedicated logic.

3. OPERATION OF MOVING IMAGE PROCESSING APPARATUS1

Next, the operation of the moving image processing apparatus1according to the embodiment of the present disclosure will be described with reference toFIGS. 3 to 11.FIG. 3illustrates the operation procedure of the moving image processing apparatus1.

As illustrated inFIG. 3, the data acquisition unit11firsts acquires moving image data MP (step S11) and supplies the moving image data MP to the cut transition detection unit13and the like. The cut transition detection unit13detects cut transition in a moving image MP based on the moving image data MP (step S13), and supplies the detection result to the cut grouping processing unit15and the like. The cut transition is detected based on the feature amounts of images and/or voice in frames in succession.

FIG. 4illustrates an example of a cut structure obtained from the detection result of the cut transition. In order to facilitate understanding,FIG. 4illustrates a cut structure using representative images11to17of respective cuts. In addition, the representative images I are extracted by the cut structure image generation unit17from the moving image data MP as will be described later. As illustrated inFIG. 4, cuts1,3and6are similar to one another, cuts2,4and7are similar to one another, and a cut5is not similar to any of the cuts1to4,6and7.

Returning to the description ofFIG. 3, next, the cut grouping processing unit15performs a cut grouping process of grouping the cuts into cut groups (step S15).FIG. 5illustrates the procedure of the cut grouping process. As illustrated inFIG. 5, in the cut grouping process, an initialization process is first performed (step S31). In the initialization process, the number m of groups is initialized. Furthermore, a group ID of 1 is assigned to the cut1and a group ID of 2 is assigned to the cut2. Here, the number m of the groups indicates the number (cut groups1and2are specified in the initialization process) of the cut groups specified from the moving image data MP. The group ID is assigned to each cut in order to indicate a cut group to which each cut belongs.

Then, feature amounts of the cut1are generated and stored in the data storage unit23and the like as the feature amounts S1of the cut group1(step S33). In the same manner, feature amounts of the cut2are generated and stored in the data storage unit23and the like as the feature amounts S2of the cut group2(step S35).FIG. 6illustrates the procedure of a feature amount generation process.

As illustrated inFIG. 6, a predetermined number of images are first extracted from images of the cut1as first extraction images, and a predetermined number of images are extracted from images of the cut2as second extraction images (steps S71and S73). Usually, each cut includes images of several tens to several thousands of frames, and for example, images of about 10 frames are extracted as the first and second extraction images, respectively. The first and second extraction images, for example, are extracted according to the following extraction rules.

FIG. 7illustrates the extraction rules of images.FIG. 7illustrates the case in which the cut grouping process is performed in the order of the time codes of the moving image data MP, that is, according to the reproduction direction of the moving image data MP. The time code is time information assigned to each frame (image) of the moving image MP, and is assigned by employing a start frame as the starting point.

InFIG. 7, among the images of the cut1, the first extraction images are preferentially extracted at the cut transition time point from the cut1to the cut2, that is, from images near the final frame n of the cut1. In addition, inFIG. 7, extracted images (frames) are hatched by dots.

According to the rule1ofFIG. 7, when setting the final frame of the cut1as a frame n, a predetermined number of images (e.g. frames n−1, n−5, n−9, n−13, n−17, . . . ) are extracted at a regular interval (arithmetic progression with a common difference of 4) of four frames. According to the rule2ofFIG. 7, in the same manner, a predetermined number of images (e.g. frames n−1, n−3, n−7, n−15, n−31, . . . ) are extracted at an interval (geometric progression with a common ratio of 2) of two frames, four frames, eight frames, 16 frames, . . . . According to the rule3ofFIG. 7, in the same manner, a predetermined number of images (e.g. frames n−1, n−3, n−6, n−10, n−15, . . . ) are extracted at an interval (progression of differences with a common difference of 1) of two frames, three frames, four frames, five frames, . . . .

In addition, in any one of the rules1to3, the common difference and the common ratio are set as arbitrary values, respectively. In the same manner, the extraction images may be extracted including the final frame or extracted by excluding several frames around the final frame. Furthermore, although not shown inFIG. 7, a predetermined number of frames with a large pixel difference relative to an adjacent frame may be extracted. Moreover, the extraction images, for example, may be extracted at an interval by arithmetic progression or geometric progression by combining the rule1with the rule2.

In the processes of steps71and73, in relation to the first extraction images, images near the final frame of the cut1are preferentially extracted from the images of the cut1according to the extraction rules. In relation to the second extraction images, images near the final frame of the cut2are preferentially extracted from the images of the cut2.

Next, various feature amounts of the images extracted as the first and second extraction images are calculated. Various feature amounts, for example, are calculated as a color histogram and facial image detection, and/or the volume, tone/rhythm and the like of voice, or a combination thereof. The calculated feature amounts of the images are stored in the data storage unit23and the like in order to be used for the processes which will be described later (steps S75and77).

Then, a “correct” tag is assigned to the first extraction images and an “incorrect” tag is assigned to the second extraction images (steps S79and81). Here, the “correct” tag is assigned to the first extraction images extracted from the cut1because it is highly probable that the first extraction images include the feature amounts S1of the cut group1as compared with the second extraction images, and the “incorrect” tag is assigned to the second extraction images because it is less probable that the second extraction images include the feature amounts S1of the cut group1as compared with the first extraction images.

Then, feature amounts corresponding to the first extraction, images having the “correct” tag assigned thereto and not corresponding to the second extraction images having the “incorrect” tag assigned thereto are generated as the feature amounts S1of the cut group1(step S83). That is, the feature amounts S1of the cut group1are generated as feature amounts for distinguishing the first extraction images from the second extraction images.

For example, it is assumed that arbitrary feature amounts A have a predetermined quantitative or qualitative value in the first extraction images, but do not have the predetermined quantitative or qualitative value in the second extraction images. In this case, feature amounts (features) S1, in which the feature amounts A have the predetermined quantitative or qualitative value, are generated. Of course, for a combination of arbitrary feature amounts A, B, . . . , feature amounts (features) are generated in the same manner.

Next, a “correct” tag is assigned to the second extraction images and an “incorrect” tag is assigned to the first extraction images (steps S85and87). Then, feature amounts corresponding to the second extraction images having the “correct” tag assigned thereto and not corresponding to the first extraction images having the “incorrect” tag assigned thereto are generated as the feature amounts S2of the cut group2(step S89). That is, the feature amounts S2of the cut group2are generated as feature amounts for distinguishing the second extraction images from the first extraction images.

Returning to the description ofFIG. 5, the presence or absence of a subsequent cut to be processed is checked (step S37). The subsequent cut is checked in the order of the cut3, the cut4, whenever performing the process of step S37. When the subsequent cut is present (“Yes” in step S37), the feature amounts S′ of the subsequent cut are generated (step S39), and, the similarity of the feature amounts is determined between the subsequent cut and cut groups1to m (step S41).

In the determination of similarity, a predetermined number of images are first extracted from images of the subsequent cut as comparison images. The comparison images are extracted as a predetermined number of images representing the images of the subsequent cut. The comparison images may be extracted according to the extraction rules illustrated inFIG. 7or other extraction rules.

Next, it is determined whether feature amount are similar between the comparison images and the images of the cut groups1to m, that is, whether the subsequent cut is similar to the cut groups1to m.

Here, the feature amounts S1to Sm of the cut groups1to m are generated by emphasizing feature amounts of images near the subsequent cuts, which are to be subject to the determination of similarity, among the images of each cut group based on the time codes of the moving image MP. For example, the feature amounts S1of the cut group1are generated by emphasizing the feature amounts of the images near the final frame of the cut1among the images of the cut1. Therefore, the similarity of feature amounts is determined between the comparison images and images near the subsequent cut among the images of the cut groups1to m based on the time codes of the moving image MP.

In addition, in the determination of similarity, the degree of similarity of the feature amounts may be preferentially determined between the subsequent cut and a cut group with a group ID larger than that of an immediately previous cut. This is because the group ID of a cut group including the subsequent cut is larger than that of the immediately previous cut when the cuts are repeated.

When the degree of similarity of the feature amounts is equal to or more than a predetermined threshold value between the subsequent cut and any one of the cut groups1to m (“Yes” in step S41), the group ID of a cut group with the highest similarity is assigned to the subsequent cut (step S43).

For example, when the degree of similarity between the subsequent cut and the cut group1satisfies the predetermined threshold value, the subsequent cut is grouped into the cut group1. In such a case, the feature amounts S1of the cut group1are regenerated and stored according to the following procedure (step S45).

First, a predetermined number of images are extracted from the images included in the cut group1including the subsequent cut as first extraction images. The first extraction images are also extracted according to the extraction rules illustrated inFIG. 7.FIG. 8illustrates rules of a feature amount regeneration process.FIG. 8illustrates a cut structure including a cut1, a cut2, and a cut3similar to the cut1. In addition, inFIG. 8, the positions of images preferentially extracted as the extraction images of a cut group are densely hatched.

According to the rule a ofFIG. 8, among images included in the cut3grouped into the cut group1, images near the final frame of the cut3are preferentially extracted as first extraction images. In such a case, images included in the cut1are not extracted. Meanwhile, according to the rule b ofFIG. 8, among images included in the cut1and the cut3grouped into the cut group1, images near the final frame of the cut3are preferentially extracted as the first extraction images. In such a case, the images included in the cut1are also extracted.

In addition, the number of the first extraction images may be equal to or more than the number of extraction images at the time of the initial generation of feature amounts. Furthermore, the first extraction images may be extracted under the same conditions as those of the extraction images at the time of the process of step S33, or under different conditions. For example, at the time of the initial generation of feature amounts, different common differences and common ratios may be used, or different may be used (e.g., the rule1is used when feature amounts are initially generated, the rule2is used when the feature amounts are regenerated, and the like).

Furthermore, the first extraction images may be extracted by combining a plurality of different conditions with one another. For example, the rule a may be combined with the rule1and the rule b may be combined with the rule2. In such a case, while images are extracted at an interval by arithmetic progression and at an interval expressed by geometric progression from the subsequent cut, images are extracted at an interval expressed by progression of differences from cuts other than the subsequent cut.

Next, various feature amounts of the images extracted as the first extraction images of the cut group1are calculated. In addition, as the feature amounts of second to mthextraction images of the cut groups2to m, feature amounts generated at the time of the generation of each cut group are used.

Then, a “correct” tag is assigned to the first extraction images and an “incorrect” tag is assigned to the second to mthextraction images. Here, the “correct” tag is assigned to the first extraction images because it is highly probable that the first extraction images extracted from the cut group1include the feature amounts of the cut group1as compared with the second to mthextraction images, and the “incorrect” tag is assigned to the second to mthextraction images because it is less probable that the second to mthextraction images include the feature amounts of the cut group1as compared with the first extraction images.

Then, feature amounts corresponding to the first extraction images having the “correct” tag assigned thereto and not corresponding to the second to mth extraction images having the “incorrect” tag assigned thereto are generated as the feature amounts S1of the cut group1. That is, the feature amounts S1of the cut group1are generated as feature amounts for distinguishing the first extraction images from the second to mthextraction images. In this way, the feature amounts S1of the cut group1are generated as the feature amounts of the first extraction images obtained by preferentially extracting the images near the final frame of the subsequent cut from the images of the cut group1.

In the same manner, when the degree of similarity between the comparison images and a cut group x (1<x<m) satisfies a predetermined threshold value, the subsequent cut is grouped into the cut group x. In such a case, in the same manner as the cut group1, feature amounts Sx of the cut group x are regenerated. In this way, the feature amounts Sx of the cut group x including the subsequent cut are calculated as feature amounts of xthextraction images obtained by preferentially extracting the images near the final frame of the subsequent cut from images of the cut group x.

Meanwhile, when the degree of similarity of the feature amounts is smaller than the predetermined threshold value between the subsequent cut and all the cut groups1to m (“No” in step S41), the number m of the groups is incremented by 1 in order to generate a new cut group (step S47). A group ID corresponding to the number m of the groups is assigned to the subsequent cut as the new cut group (step S49). In such a case, the feature amounts Sm of the cut group m are generated and stored according to the following procedure (step S51).

First, a predetermined number of images are extracted from the images included in the subsequent cut as mthextraction images. The mthextraction images are also extracted according to the extraction rules illustrated inFIG. 7. Here, the mthextraction images are obtained by preferentially extracting images near the final frame of the subsequent cut from the images of the cut frame m.

Next, various feature amounts of the images extracted as the mthextraction images of the cut frame m are calculated. In addition, as the feature amounts of the first to (m−1)thextraction images of the cut groups1to m−1, feature amounts generated at the time of generation of each cut group are used.

Then, a “correct” tag is assigned to the mthextraction images and an “incorrect” tag is assigned to the first to (m−1)thextraction images. Here, the “correct” tag is assigned to the mthextraction images extracted from the cut group in because it is highly probable that the mthextraction images include the feature amounts Sm of the cut group m as compared with the first to (m−1)thextraction images, and the “incorrect” tag is assigned to the first to (m−1)thextraction images because it is less probable that the first to (m−1)thextraction images include the feature amounts Sm of the cut group m as compared with the mthextraction images.

Then, feature amounts corresponding to the mthextraction images having the “correct” tag assigned thereto and not corresponding to the first to (m−1)thextraction images having the “incorrect” tag assigned thereto are generated as the feature amounts Sm of the cut group m. That is, the feature amounts Sm of the cut group m are generated as feature amounts for distinguishing the mthextraction images from the first to (m−1)thextraction images. In this way, the feature amounts Sm of the cut group m are generated as the feature amounts of the mthextraction images obtained by preferentially extracting the images near the final frame of the subsequent cut from the images of the cut group m.

Returning to the description ofFIG. 5, the processes of steps S37, S39, S41, S43, S45, S47, S49, and S51are repeated until there is no subsequent cut. When there is no subsequent cut (“No” in step S37), the cut grouping process ends, and the process of the next step S17is performed as illustrated inFIG. 3. The cut grouping processing unit15supplies the detection result of the cut transition and the cut grouping result to the cut structure image generation unit17.

FIG. 9illustrates an example of the result of the cut grouping process.FIG. 10illustrates an example of the course of the feature amount generation process.FIGS. 9 and 10illustrate the case in which the cut grouping process is performed with respect to the cut structure illustrated inFIG. 4. In addition, inFIG. 10, the positions of images preferentially extracted as the extraction images of a cut group are densely hatched.

First, as illustrated inFIG. 9, group IDs of 1 and 2 are assigned to the cuts1and2, respectively. As illustrated in the state1ofFIG. 10, the feature amounts S1of the cut group1are generated as feature amounts of images obtained by preferentially extracting images near the final frame of the cut1from the images of the cut1, and the feature amounts S2of the cut group2are generated as feature amounts of images obtained by preferentially extracting images near the final frame of the cut2from the images of the cut2.

Next, the similarity between the cut3and the cut groups1and2is determined. The similarity of the cut3is determined using feature amounts of images near the cut3based on the time codes of the moving image MP among the images of the cut groups1and2, as illustrated in the state1ofFIG. 10. In the example ofFIG. 9, the cut3has the degrees 0.9 and 0.2 of similarity with respect to the cut groups1and2, respectively. Thus, it is determined that the cut3is similar to the cut group1and is not similar to the cut group2(a threshold value for the determination of similarity is set to be equal to or more than similarity 0.5). For this reason, a group ID of 1 is assigned to the cut3. As illustrated in the state2ofFIG. 10, the feature amounts S1of the cut group1are regenerated as feature amounts of images obtained by preferentially extracting images near the final frame of the cut3from the images of the cuts1and3.

Then, the similarity between the cut4and the cut groups1and2is determined. The similarity of the cut4is determined using feature amounts of images near the cut4based on the time codes of the moving image MP among the images of the cut groups1and2, as illustrated in the state2ofFIG. 10. In the example ofFIG. 9, the cut4has the degrees 0.3 and 0.8 of similarity with respect to the cut groups1and2, respectively. Thus, it is determined that the cut4is similar to the cut group2and is not similar to the cut group1. For this reason, a group ID of 2 is assigned to the cut4. As illustrated in the state3ofFIG. 10, the feature amounts S2of the cut group2are regenerated as feature amounts of images obtained by preferentially extracting images near the final frame of the cut4from the images of the cuts2and4.

Thereafter, the similarity between the cut5and the cut groups1and2is determined. The similarity of the cut5is determined using feature amounts of images near the cut5based on the time codes of the moving image MP among the images of the cut groups1and2, as illustrated in the state3ofFIG. 10. In the example ofFIG. 9, the cut5has the degrees 0.1 and 0.2 of similarity with respect to the cut groups1and2, respectively. Thus, it is determined that the cut5is not similar to either of the cut groups1and2. For this reason, a cut group3is generated and a group ID of 3 is assigned to the cut5. As illustrated in the state4ofFIG. 10, the feature amounts S3of the cut group3are generated as feature amounts of images obtained by preferentially extracting images near the final frame of the cut5from the images of the cut5.

Next, the similarity between the cut6and the cut groups1to3is determined. The similarity of the cut6is determined using feature amounts of images near the cut6based on the time codes of the moving image MP among the images of the cut groups1to3, as illustrated in the state4ofFIG. 10. In the example ofFIG. 9, the cut6has the degrees 0.8, 0.3 and 0.2 of similarity with respect to the cut groups1to3, respectively. Thus, it is determined that cut6is similar to the cut, group1and is not similar to the cut groups2and3. For this reason, a group ID of 1 is assigned to the cut6. As illustrated in the state5ofFIG. 10, the feature amounts S1of the cut group1are regenerated as feature amounts of images obtained by preferentially extracting images near, the final frame of the cut6from the images of the cuts1,3and6.

Then, the similarity between the cut7and the cut groups1to3is determined. The similarity of the cut7is determined using feature amounts of images near the cut7based on the time codes of the moving image MP among the images of the cut, groups1and2, as illustrated in the state5ofFIG. 10. In the example ofFIG. 9, the cut7has the degrees 0.1, 0.9 and 0.1 of similarity with respect to the cut groups1to3, respectively. Thus, it is determined that the cut7is similar to the cut group2and is not similar to the cut groups1and3. For this reason, a group ID of 2 is assigned to the cut7. As illustrated in the state6ofFIG. 10, the feature amounts S2of the cut group2are regenerated as feature amounts of images obtained by preferentially extracting images near the final frame of the cut7from the images of the cuts2,4and7.

As a consequence, as illustrated inFIG. 9, for the cut structure illustrated inFIG. 4, the cuts1,3and6are grouped into the cut group1, the cuts2,4and7are grouped into the cut group2, and the cut5is grouped into the cut group3.

Returning to the description ofFIG. 3, when the cut grouping process ends, the cut structure image generation unit17first extracts the representative images1from a series of images included in each cut based on the moving image data MP and the detection result of the cut transition according to predetermined standards (step S17). In addition, the representative image I of each cut may be extracted in advance when the cut transition is detected. Next, the cut structure image CI is generated based on the specifying result of the cut groups (step S19). The cut structure image CI is generated as an image including the representative image I of each cut arranged in the order of the cut transition and indicating the cut groups.

FIG. 11illustrates an example of the cut structure image CT generated from the specifying result of the cut groups illustrated inFIG. 9. In the cut structure image CI illustrated inFIG. 11, the representative images11and12of the cuts1and2are transversely arranged, the representative images13to15of the cuts3to5are transversely arranged below the representative images11and12of the cuts1and2, and the representative images16and17of the cuts6and7are transversely arranged below the representative images13and14of the cuts3and4. Using the cut structure image CI as described above, the cut structure is easily understood. In addition, the structure of the cut structure image CI is not limited to the structure illustrated inFIG. 11.

As described above, in accordance with the moving image processing apparatus1and the moving image processing method according to the embodiment of the present disclosure, the feature amounts S of each cut group are generated by emphasizing the feature amounts of images near the subsequent cuts, which are to be subject to determination of similarity, among the images included in each cut group based on the time codes of the moving image MP. Consequently, it is possible to appropriately group the subsequent cuts by sufficiently considering the temporal context (a story and the like of the moving image MP) between cuts.

The present disclosure contains subject matter related to that disclosed in Japanese Priority Patent Application JP 2010-179696 filed in the Japan Patent Office on Aug. 10, 2010, the entire content of which is hereby incorporated by reference.