Image generation apparatus and control method

An image generation apparatus stores a plurality of selection condition rows including a plurality of selection conditions used for selecting an image from a plurality of material images, selects at least one material image from the plurality of material images as a first material image, obtains a feature quantity of the first material image, selects a selection condition row which is stored in the storage unit and includes a selection condition including the obtained feature quantity, selects a second material image from the plurality of material images based on a selection condition row which has been selected, and generates an image based on the first and the second material images which have been selected and the selection condition row which has been selected.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present disclosure relates to an image generation apparatus configured to generate an image and a control method thereof.

2. Description of the Related Art

In recent years, there is a growing need for generating an attractive output image such as a moving image or a slideshow by performing image editing on material images such as still images and moving images. Image editing includes, for example, selection, composing, and deletion of scenes using the material images, and applying effect and transition to the scenes. However, for users not used to generating output images, processing like selection, composing, and deletion of scenes, and designation of effect and transition is very challenging.

Thus, a tool that generates an output image in a full-automatic or a semiautomatic manner has been developed. A tool that generates an output image full-automatically needs only a material image to be processed and a template designated by a user in generating an output image. When receiving the material image and the template, the tool automatically generates an output image.

For example, a template discussed in Japanese Patent No. 4424389 includes a condition and an order of material images to be used and information of the effect/transition to be applied. Feature quantities of the material images are compared with the condition of the material image to be used described in the template. Then, the material images that match the condition are arranged in order and the designated effect/transition is applied to the material images. In this manner, an output image is generated. Japanese Patent No. 4343027 discusses a method that selects a template with a theme that matches a theme set in advance to a material image to be processed and generates an output image using the template.

Regarding a tool for semi-automatically generating an output image, there is a tool that presents candidate material images to a user. Japanese Patent Application Laid-Open No. 2011-30159 discusses a method for calculating a recommendation degree of each material image concerning whether the image can be arranged at the (N+1)-th image from a relationship between a feature quantity of an N-th material image which is selected by a user and a feature quantity of a different material image. Then, based on the calculated recommendation degree, the user is provided with candidates of material images which can be arranged as the (N+1)-th image. The user selects an image from the candidates.

In the above-described Japanese Patent No. 4424389 and Japanese Patent No. 4343027, a user designates a template. However, if a material image that matches the condition of the template regarding the material image to be used does not exist, an output image that matches the content of the template cannot be generated. In addition according to these patent documents, designation of a template according to a theme is performed. However, in this case, since the feature quantity such as the composition of the material image is not considered, an output image that matches the material image may not be generated in some cases.

Further, according to the semiautomatic method discussed in Japanese Patent Application Laid-Open No. 2011-30159, only the correlation of the feature quantities between adjacent material images is considered. Thus, an output image of a scene with a story line including a plurality of cuts cannot be automatically generated. In this context, a cut is an interval where imaging is continuously performed in an output image. Further, a scene includes a plurality of cuts and has one meaning or one story.

The present invention is directed to realizing an image generation apparatus useful in easily generating an output image that matches a material image which a user desires to include and includes a plurality of scenes with an alignment order.

SUMMARY OF THE INVENTION

According to an aspect of the present disclosure, an image generation apparatus includes a storage unit configured to store a plurality of selection condition rows including a plurality of selection conditions used for selecting a material image from a plurality of material images, a first material image selection unit configured to select at least one material image from the plurality of material images as a first material image, a feature quantity obtaining unit configured to obtain a feature quantity of the first material image, a selection condition row selection unit configured to select a selection condition row including the obtained feature quantity from the plurality of selection condition rows stored in the storage unit, a second material image selection unit configured to select a second material image from the plurality of material images based on a selection condition of the selection condition row which has been selected, and an output image generation unit configured to generate an output image based on the first and the second material images which have been selected and the selection condition row which has been selected.

Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings.

DESCRIPTION OF THE EMBODIMENTS

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the invention and, together with the description, serve to explain the principles of the disclosure.

An image generation apparatus according to a first exemplary embodiment of the present disclosure is described below with reference to the drawings.FIG. 1illustrates a hardware configuration of an image generation apparatus according to the first exemplary embodiment.

A central processing unit (CPU)101is responsible for all processing of each unit of the image generation apparatus. The CPU101reads a command stored in a read-only memory (ROM)102or a random access memory (RAM)103in order, interprets the command, and executes processing according to the result of the interpretation. Processing in flowcharts described below is performed by the CPU101. A program, data, and a working area necessary for the processing performed by the CPU101are provided by the ROM102and the RAM103to the CPU101.

A storage medium104for storing image data is a hard disk, a CompactFlash (CF) card, a Secure Digital (SD) card, a universal serial bus (USB) memory, a memory card, or the like. An input device105is a device such as a touch screen and receives an instruction input by a user. An output device106displays images and characters. A liquid crystal display is widely used for such an output device. Further, the output device106can include a touch panel function. In that case, the output device can also be used as the input device105.

FIG. 7illustrates a user interface (UI) screen of the image generation apparatus according to the present exemplary embodiment. A window701is a UI screen of the image generation apparatus according to the present exemplary embodiment. A list of thumbnails703of material images is displayed in an area702. When a user selects a thumbnail he/her desires to use, a frame of the selected thumbnail is thickened as illustrated by a thumbnail704.

An output image generation start button705is used for starting output image generation processing according to the present exemplary embodiment. An area706is where an output image obtained as a result of the output image generation processing is reproduced. A button707is a playback button used for starting playback of the output image which is obtained as a result of the output image generation processing. The output image is a moving image or a slideshow. A slideshow is data for displaying a plurality of moving images or still images in a predetermined order and a time length.

FIG. 23is a schematic diagram of a data structure of a slideshow. A data structure2301of the slideshow includes slideshow attribute information2302and playback list information2303. The slideshow attribute information2302includes a name of the slideshow, a playback time, a pointer to the playback list information2303, a transition type, and a transition time. The playback list information2303includes a start time and an end time of a moving image or a still image and a file name of the moving image or the still image when one moving image or still image is displayed. The beginning of the slideshow is set to zero.

InFIG. 23, a slideshow with a name of “ABC” has a playback time of “3 minutes 12 seconds”. Its transition type is “crossfade” and its transition time is “2 seconds”. Further, files “C:\ footage\IMG—0001.jpg”, “C:\footage\IMG—0012.jpg”, and “C:\footage\MVI—0028.mov” are displayed from 0 to 2,999 milliseconds, 3,000 to 5,999 milliseconds, and 6,000 to 11,999 milliseconds, respectively.

When one moving image or still image is switched over to another image, image processing which is designated by the transition type and time is performed. InFIG. 23, cross-fade of two seconds is performed. The cross-fade is image processing for synthesizing images by gradually reducing the opacity of a previous moving image or still image while gradually increasing the opacity of a following moving image or still image. The data structure and the values described above are examples and the present invention is not limited to such examples.

FIG. 2illustrates a functional block diagram of the image generation apparatus. Processing of each unit will be described below. A material image201is a still image or a moving image data.

FIG. 3illustrates a data structure of the material image201. The material image201has an exchangeable image file format (Exif) including imaging information301and material image data302. The shooting date and time can be obtained from a clock integrated in the camera. Information of the aperture value and the focal length can be obtained from lens information. The camera orientation can be obtained from a value detected by a gyroscopic sensor integrated in the camera.

Referring back again toFIG. 2, the first exemplary embodiment is further described. An image generation apparatus202is an image generation apparatus according to the present exemplary embodiment. A first material image selection unit203selects a material image designated by a user as a first material image. Although the user directly designates the first material image according to the present exemplary embodiment, the user can designate a feature quantity of the material image as a selection condition. Then, based on the designated selection condition, the first material image can be selected.

The feature quantity is information indicating various features of the material image. The feature quantity information can be obtained from the Exif data attached to the material image or from an analysis result of the material image. Information pieces such as a shooting date and time, an imaging apparatus name, resolution, an imaging direction, a shutter speed, an aperture, an ISO sensitivity, a light metering mode, a focal length, and global positioning system (GPS) information can be obtained from the Exif data. Further, the feature quantities regarding a number of faces, a face size, human figure recognition, and a line of sight direction can be obtained from the analysis results based on the face detection, human figure recognition, and line of sight detection. The feature quantities can be obtained by a different analysis. For example, voice detection or smile detection can be used. Further, although the feature quantity can be obtained by performing the analysis processing each time, it can be obtained in advance and stored in the storage medium104in association with the material image.

A feature quantity obtaining unit204obtains the feature quantity of the first material image. A selection condition row205is data concerning a generation pattern of an output image. The selection condition row205includes at least a selection condition of the material image to be used in each cut of the output image and arrangement position information that defines the alignment order of the material images. The storage medium104includes a plurality of selection condition rows205. Although the selection condition is described below, it is, for example, an imaging location, a number of faces, or a face size.

FIGS. 4A to 4Cillustrate concrete examples of the output images, which are generated by applying the selection condition row.FIG. 4Ais a generation pattern for effectively expressing an action/reaction of an object. A first cut401is a long shot of an imaging location describing the state of the location. In a second cut402, the object gazes up at something. A third cut403shows the moon, which is what the object gazes up in the cut402. A fourth cut404is a close-up of the object and expresses the feeling of the object.

FIG. 4Bis a generation pattern which can be used for introducing a character. A cut411is a group photograph. Cuts412,413, and414are used for introducing the persons appeared in the cut411.

FIG. 4Cillustrates a generation pattern for realizing a natural output image by matching the directions of line of sight in cuts. A cut421is a long shot of a person A and a person B facing each other. In a cut422, the person A is looking at the same direction the person A is looking at in the cut421. In a cut423, the person B is looking at the same direction the person B is looking at in the cut421. These are merely examples and the present exemplary embodiment is not limited to such examples.

FIG. 5illustrates a data structure of the selection conditions. A table501includes the selection conditions of a material image. A CheckFlag item502is used for determining the selection condition for determining whether a target material image includes the feature quantity of the first material image in the processing described below. The selection condition having a CheckFlag value set to “1” is subjected to the determination. Details of the determination are described below.

A Position item503indicates a display position of each cut in a time sequence. “Start” indicates the start time of the cut and “End” indicates the end time of the cut. The time is expressed in milliseconds. A Condition item504indicates a condition for selecting the material image and shows the feature quantity required for the material image to be selected. For example, “Location” indicates the imaging location. If a value same as the one for the Cut4is to be used, “Cut4” is set to “Value”. “FaceNumber” indicates the number of faces. “FaceSize” indicates a face size. “Line of Sight” indicates the direction of the line of sight. “Elevation of Camera” indicates an elevation angle of the camera. If a value of 30 degrees or greater is to be set, it is set as “30<” using the inequality sign. These selection conditions are merely examples, and not limited to the above-described examples.

In a table505, effect information is described. An Effect Type item506indicates the type of the effect. “Blur” indicates the effect of blur and “Transform” indicates the effect of basic modification such as “enlarge”, “reduce”, “move”, and “rotate”. Other effects can also be used. A Position item507indicates a range to which the effect is applied. “Start” indicates a start position of the effect, and “End” indicates an end position of the effect. Time is expressed in milliseconds. A Parameter item508is the parameter of the effect. For example, “Type” indicates a parameter type. “Value” indicates a parameter value. “Time” indicates a time corresponding to the value of “Value” when the value of the parameter changes according to time passage.

A table509includes transition information. A Transition Type item510indicates a type of the transition. “BlackOutIn” indicates the transition regarding black out/in. “CrossFade” indicates the transition regarding cross-fade. Other transitions can also be used. A Position item511indicates a range to which the transition is applied. “Start” indicates a start position of the transition, and “End” indicates an end position of the transition.

These three tables501,505, and509configure one set of the selection condition rows. The storage medium104includes a plurality of generation patterns regarding the selection condition rows. Although a data structure of the generation pattern is described above, the present exemplary embodiment is not limited to such an example. For example, only the table501can be used as the selection condition row.

Referring back again toFIG. 2, the first exemplary embodiment is further described. A selection condition row selection unit206selects the selection condition row including the feature quantity of the first material image as the selection condition of which CheckFlag item502is “1”.

The above-described selection method of the selection condition row is a merely example, and the selection method is not limited to the above-described example. For example, the determination can be performed using all the selection conditions and then the selection condition row which satisfies at least one selection condition can be selected. Further, a selection condition row where all the material images that satisfy each condition row of the selection condition row may be selected.

The selection condition row may be selected by a user if a plurality of selection condition rows is selectable, and a selection condition row with the higher priority may be selected based on a priority added to the selection condition row in advance.

The priority may be changed according to a determination result of a selection frequency, a selection interval, and arrangement from the selection history of the selection condition rows. For example, if a same selection condition row is continuously selected, the priority of the selection condition row is lowered.

In addition, the priority may be set for each piece of theme information provided by the user. The theme information is, for example, atmospheric information such as a tempo of the cut change or a note of a background music (BGM) of the output image as a result of the generation. For example, if the tempo is fast, the priority of the selection condition row including many cuts which are changed quickly may be raised.

Further, the priority may be determined based on the position of the output image which is generated by applying the selection condition row in the time sequence of the output image as a result of the generation. For example, if the output image generated by applying the selection condition row is arranged in the early stage, the priority of the selection condition row such as the one illustrated inFIG. 4Bthat introduces the characters may be raised.

A second material image selection unit207selects a second material image based on a selection condition of the selection condition row205which has been selected. An output image generation unit208arranges the first and the second material images based on the selection condition row205which has been selected, generates an output image209, and stores the output image in the storage medium104or outputs it to the output device106. If a plurality of the first material images is selected, the output image generation unit208arranges the first material images in the order designated by the user to generate the output image. Such an arrangement is a merely example and the present exemplary embodiment is not limited to the above-described example. For example, the output image can be generated so that the first material images are arranged in the order of the shooting date and time.

FIG. 17is a flowchart illustrating processing corresponding to the UI operation of the image generation apparatus according to the first exemplary embodiment. In step S1701, when the power is supplied, the CPU101displays the window701. In step S1702, the CPU101obtains a user input to the input device105. In step S1703, the CPU101determines whether the user input is turning off the power. In step S1703, if the user input is not turning off the power (NO in step S1702), the processing proceeds to step S1704. In step S1704, the CPU101determines whether the user input is an operation regarding material image selection. In step S1704, if the user input is the material image selection operation (YES in step S1704), the processing proceeds to step S1705. In step S1705, the frame of the thumbnail of the selected material image is thickened. Then, the processing returns to step S1702to obtain the user input.

In step S1704, if the user input is not the material image selection operation (NO in step S1704), the processing proceeds to step S1706. In step S1706, the CPU101determines whether the user has pressed the output image generation start button705. In step S1706, if the CPU101determines that the user has pressed the output image generation start button705(YES in step S1706), the processing proceeds to step S1707. In step S1707, the CPU101starts the output image generation processing. Then, the processing returns to step S1702. Details of the output image generation processing will be described below.

In step S1706, if the CPU101determines that the user has not pressed the output image generation start button705(NO in step S1706), the processing proceeds to step S1708. In step S1708, the CPU101determines whether the user has pressed the button707being the output image playback button. If the CPU101determines that the user has pressed the output image playback button707(YES in step S1708), the processing proceeds to step S1709. In step S1709, the CPU101reproduces the output image which is generated by the output image generation processing just before step S1707. In step S1708, if the CPU101determines that the user has not pressed the output image playback button707(NO in step S1708), the processing returns to step S1702, and the user input is obtained. If the user input is power OFF (YES in step S1703), then the processing ends.

FIG. 6is a flowchart illustrating the output image generation processing of the image generation apparatus in step S1706according to the first exemplary embodiment. In step S601, after the output image generation processing is started, the CPU101receives a user input. In step S602, based on the user input obtained in step S601, the CPU101selects the material image selected by the user as a first material image. In step S603, the CPU101obtains a selection condition row based on the feature quantity of the first material image. Details of the selection condition row selection processing will be described below. In step S604, the CPU101selects a second material image based on the selected selection condition row. Details of the selection condition selection processing will be described below.

In step S605, the CPU101determines whether the selection condition rows of all the first material images have been selected. If the CPU101determines that the selection condition rows of all the first material images have not yet been selected (NO in step S605), the processing returns to step S602, and the next first material image selection processing is performed. If the CPU101determines that the selection condition rows of all the first material images have been selected (YES in step S605), the processing proceeds to step S606. In step S606, the CPU101performs the output image generation processing. The output image generation processing is to arrange the selected first and the second material images based on the selected selection condition rows to generate an output image, and store the generated output image in the storage medium104. Then, the output image generation processing ends.

FIG. 8is a flowchart illustrating the selection condition selection processing performed in step S603. In step S801, the CPU101obtains a feature quantity of the first material image. In step S802, the CPU101obtains a selection condition of which CheckFlag item502is “1” as a selection condition for the determination. In step S803, the CPU101determines whether the feature quantity of the first material image is included in the selection condition obtained in step S802. If the CPU101determines that the feature quantity of the first material image is not included in the selection condition (NO in step S803), the processing returns to step S802, and the next selection condition for the determination of the selection condition row is obtained. On the other hand, if the CPU101determines that the feature quantity of the first material image is included in the selection condition (YES in step S803), the processing proceeds to step S804. In step S804, the CPU101determines that the selected selection condition row is the selection condition row to be used, and then the selection condition row selection processing ends.

FIG. 9is a flowchart illustrating the second material image selection processing performed in step S604inFIG. 6. When the second material image selection processing is started, in step S901, the CPU101obtains the selection condition row selected in step S603. In step S902, the CPU101obtains a selection condition other than the selection condition determined as including the feature quantity of the first material image in step S803from the selection condition row. In step S903, the CPU101obtains a feature quantity of a material image.

In step S904, the CPU101determines whether the feature quantity of the material image is included in the selection condition. If the feature quantity of the material image is included in the selection condition (YES in step S904), the processing proceeds to step S905. In step S905, the CPU101determines that the material image is the second material image. In step S906, the CPU101determines whether checking of all the material images is finished. If all the material images is not yet checked (NO in step S906), the processing returns to step S903, and the feature quantity of the next material image is obtained.

In step S906, if it is determined that all the material images are checked (YES in step S906), the processing proceeds to step S907. In step S907, from among the material images which have been determined as the second material images, the CPU101selects a material image of which shooting date and time is closest to that of the first material image as the second material image.

The above described processing is an example and the present exemplary embodiment is not limited to such an example. For example, the material image which is firstly determined as the second material image can be determined as the second material image. For example, in another embodiment of the present invention the criteria for selecting which material image among the plurality of material images that were checked may be different. An exemplary criteria, may be to stop the search and select a second material image once a material image has been identified which meet the selection condition. In another alternative, all the material images are not checked, and the second material images is selected form an incomplete subset of images that meet the selection condition.

Further, priority of each material image can be calculated by performing weighting of the feature quantity of the material image. Then, the material image with a high priority can be selected as the second material image. In step S908, the CPU101determines whether all the selection conditions have been checked. If all the selection conditions have not yet been checked (NO in step S908), the processing returns to step S902, and the next selection condition is obtained. On the other hand, if all the selection conditions have been checked (YES in step S908), the second material image selection processing ends.

In step S606, the output image is generated by arranging all the selected material images in one sequence. Although the selection condition rows are connected in a normal order, the order of the selection condition rows may be determined in advance, or the order can also be changed according to a material image between two selection condition rows.

According to the above-described configuration and processing, an output image that matches the material images and includes a plurality of scenes with an alignment order can be easily generated.

Further, not all of the material images are selected by the user as the output image, and since other material images are automatically selected from the material images selected by the user, the number of operations performed by the user can be reduced.

A second exemplary embodiment will be described with reference to drawings. Descriptions of parts similar to those of the first exemplary embodiment are not repeated.

FIG. 10illustrates a configuration of an image generation apparatus according to the second exemplary embodiment. A network I/F1001performs communication control so that, for example, the CPU101can receive image data information from a material image server1003, which is an external server, via a network1002.

FIG. 11illustrates a functional block diagram of the image generation apparatus according to the second exemplary embodiment. If the material image201that satisfies the selection condition is not stored in the storage medium104, a second material image selection unit1101causes an external material image search unit1102to search the material image server1003via the network1002for a material image.

FIG. 12is a flowchart illustrating neighborhood material image selection processing according to the second exemplary embodiment. In step S1201, the CPU101determines whether a second material image is stored in the storage medium104. If a second material image is not stored in the storage medium104(NO in step S1201), the processing proceeds to step S1202. In step S1202, the CPU101searches the material image server1003based on the selection condition. In step S1203, the CPU101determines whether a material image that satisfies the selection condition is stored in the material image server1003. If a material image that satisfies the selection condition is stored in the material image server1003(YES in step S1203), the processing proceeds to step S1204. In step S1204, the CPU101determines the material image in the material image server as the second material image.

According to the above-described configuration, even if a material image that satisfies the selection condition does not exist, an output image that matches the material image and includes a plurality of scenes with an alignment order can be easily generated.

A third exemplary embodiment will be described with reference to drawings. Descriptions of parts similar to those of the first and the second exemplary embodiments are not repeated.

FIG. 13is a functional block diagram of an image generation apparatus according to the third exemplary embodiment. If the feature quantity of a material image201does not match the selection condition, a second material image selection unit1301causes a material image processing unit1302to edit the material image so that the material image satisfies the selection condition, and selects the edited material image as the second material image.

Regarding the material image after editing, if a material image after editing is newly generated and the older material image after editing is no longer necessary, the older material image is deleted. For the determination of editing, a table including an editing base material image condition and editing contents which are associated with the selection condition is prepared in advance. Whether a material image can be edited is determined by referencing the table. The editing base material image condition is a condition of a material image used as an editing base.

FIGS. 14A and 14Billustrate examples of processing results of the material image processing unit1302. A material image1401and a material image1404are material images before editing. Cropped regions1402and1405are respectively cropped from the material images1401and1404. Material images1403and1406are material images after editing. According to such editing, a close-up material image can be generated from a long shot material image.

FIG. 15is a table in which a condition of an editing base material image, an editing content, and a selection condition are associated with one another according to the third exemplary embodiment. A Condition item1501is a selection condition. A Base Condition item1502is a condition of the material image being the editing base. An Editing Function item1503is the editing content. The table inFIG. 15is an example and the present exemplary embodiment is not limited to such an example.

FIG. 16is a flowchart illustrating neighborhood material image selection processing according to the third exemplary embodiment. In step S1601, the CPU101obtains the condition1502which is a condition of the material image being the editing base and corresponds to the selection condition. In step S1602, the CPU101determines whether the feature quantity of the material image includes the condition1502of the editing base material image. If the feature quantity of the material image includes the condition1502(YES in step S1602), the processing proceeds to step S1603. In step S1603, the CPU101obtains the editing condition1503. In step S1604, based on the obtained editing condition, the CPU101edits the material image.

According to the above-described configuration and processing, even if the feature quantity of the material image does not match the selection condition, an output image that matches the material image and includes a plurality of scenes with an alignment order can be easily generated.

Next, a display screen of the image generation apparatus described in the first to the third exemplary embodiments above will be described as a fourth exemplary embodiment.

FIG. 18illustrates a configuration of a display screen of the image generation apparatus according to the fourth exemplary embodiment. This screen corresponds to the screen of the output device106illustrated inFIG. 1.FIG. 19illustrates a configuration of the display screen of the image generation apparatus according to the present exemplary embodiment.

A display device1911generates an output image and displays a display screen illustrated inFIG. 18. The display device1911includes a liquid crystal display or a cathode ray tube (CRT) display, and displays information when the user performs the selection operation and material images presented by the image generation apparatus. Although not illustrated, a computer1912includes a material image selection unit, a material image sequence generation unit, and an output image generation unit. The computer1912executes a program for generating an output image and controls processing of the above-described flow and processing inFIG. 20described below. A mouse1913is an operation device used by a user to perform the selection operation.

A storage medium1914corresponds to the storage medium104inFIG. 1. A plurality of selection condition rows including arrangements of a plurality of selection conditions are stored in the storage medium1914. The selection condition row includes conditions for selecting a material image such as a still image or a moving image used for generating an output image and a material image used for generating a candidate material image sequence presented by the image generation apparatus. Further, a program for generating the output image is stored in the storage medium1914. A material image stored in a server on the Internet can also be stored in the storage medium1914via the Internet line1915and used for the generation of the output image.

FIG. 20is a flowchart illustrating the output image generation processing.

The operation of the output image generation will be described with reference to the display screen illustrated inFIG. 18and the flowchart inFIG. 20. When the output image generation processing is started, in step S2001, the CPU101reads a material image stored in the storage medium1914. In step S2002, the CPU101generates a display image of the material image and displays it in a material image display area1820.

By pushing a left button of the mouse1913, the user selects a material image from display images1821of a plurality of material images which are displayed in a selectable manner in the material image display area1820(also referred to as a first display area) on the display screen. Accordingly, the material image selection unit selects the user-selected material image. In step S2003, when the image generation apparatus detects the input of the user-selected material image (YES in step S2003), the processing proceeds to step S2004. In step S2004, the image generation apparatus displays a display image1831of the user-selected material image which has been selected by the user in a user-selected material image display area1830on the display screen.

Processing in steps S2003and S2004is repeated by the user selecting the material images the user desires to use in generating the output image from the material image display area1820in order using the mouse1913. Accordingly, the display images1831to1833of a plurality of user-selected material images are displayed in the user-selected material image display area1830(also referred to as a third display area).

In step S2005, the image generation apparatus performs face detection processing with respect to the display images1831to1833of the user-selected material images. In the face detection processing, the size of each face is classified into big, medium, and small to distinguish the human figure captured in the display image. Then, by using the result of the face detection, a material image sequence is generated. The material image sequence is an output image which is generated by arranging each user-selected material image in a sequence. In step S2006, the material image sequence generation unit generates a material image sequence using the result of the face detection obtained in step S2005.

For example, the material image sequence is generated based on the selection condition row stored in the storage medium1914. In the selection condition row, the selection conditions of the material image and conditions of arrangements of the selected material images are described. A plurality of selection condition rows describing the arrangement order and the selection conditions is provided as illustrated inFIG. 21. The selection condition row describes the condition of the material images to be selected from the user-selected material images and their arrangement order to generate the material image sequence.

For example, material images including a face same as the face of a human figure in the user-selected material image identified by the face detection are selected. Then, a material image with a small face is arranged as the first image. The user-selected material image is arranged as the second image, and a material image with a big face is arranged as the third image. Then, these images are connected to form an image sequence. In this manner, based on the selection condition of the material image and the arrangement condition of the selected material images, a plurality of image sequences are generated for each material image which has been selected.

In step S2007, with respect to each of the display images1831to1833of the user-selected material images, a plurality of corresponding material image sequences are displayed in a material image sequence selection area1840(also referred to as a second display area) on the display screen. InFIG. 18, image material sequences for display1-A,1-B,1-C and2-A,2-B,2-C, and3-A,3-B,3-C are arranged and displayed in the material image sequence selection area1840.

The material image sequence for display which is displayed in the material image sequence selection area1840can be displayed by arranging thumbnail images of the material images in an arrangement order. If the material image includes a still image and a moving image, a moving image with a reduced size can be displayed next to a thumbnail of a still image. Although the material image sequence is generated and displayed after all the material images are selected according to the present exemplary embodiment, the generation and the display of the material image sequence can be performed each time a corresponding material image is selected.

Next, from the image material sequences for display1-A,1-B, and1-C in the material image sequence selection area1840on the display screen, the user selects a user-selected material image sequence corresponding to the display image1831of the user-selected material image by clicking the left button of the mouse1913. In step S2008, when the image generation apparatus detects the input of the user-selected material image sequence (YES in step S2008), the processing proceeds to step S2009. In step S2009, the image generation apparatus displays a display image sequence of the user-selected material image sequence1-B selected by the user in the user-selected material image display area1830on the display screen.

Similarly, the user selects a user-selected material image sequence corresponding to the display image1832of the user-selected material image from a plurality of material image sequences2-A,2-B, and2-C and a user-selected material image sequence corresponding to the display image1833of the user-selected material image from a plurality of material image sequences3-A,3-B, and3-C.

The user-selected material image sequences1-B,2-C, and3-A selected in this manner are displayed in a material image sequence arrangement designation area1850(also referred to as a fourth display area) in order. If the user desires to change the arrangement of the user-selected material image sequences, the user drags the material image the user desires to change from the material image sequence arrangement designation area1850by pressing the left button of the mouse1913and drops it at the desirable position by releasing the left button.

In step S2011, when the image generation apparatus detects the input of the arrangement of the user-selected material image sequence (YES in step S2011), the processing proceeds to step S2012. In step S2012, the image generation apparatus rearranges and displays the display image sequence of the user-selected material image sequence. In step S2013, the output image generation unit generates an output image by arranging the material image sequences designated by the user. In step S2014, the output image is displayed in a generation result display area1860on the display screen. Below the generation result display area1860, there are provided, from left to right, fast-rewind, stop, playback, and fast-forward buttons. By operating these buttons, the output image can be previewed.

As described above, if the user selects a material image from the material images, a plurality of material image sequences are presented in which each of the user-selected material images is connected and arranged. Since the user can select a material image sequence from the plurality of material image sequences, the output image can be easily generated.

Further, since each display area is provided on the same screen, the user can perform processing on the image by a drag-and-drop operation without changing the display screen. Accordingly, operability is good.

According to the present exemplary embodiment, the user selects a material image from the material image display area1820by clicking the left button of the mouse1913. If the user desires to change the once selected user-selected material image, the user may perform the following operation. For example, the user selects a user-selected material image1831to be deleted from the user-selected material images displayed in the user-selected material image display area1830illustrated inFIG. 18by clicking a right button of the mouse1913. Then, the material image is deleted.

Next, the user selects a user-selected material image to be replaced from the material image display area1820by clicking it with the left button of the mouse1913. Then, as illustrated inFIG. 22, a newly selected user-selected material image2234is arranged and displayed in place of the deleted display image1831of the user-selected material image in the user-selected material image display area1830. In association with the change of the user-selected material image, the material image sequences1-A,1-B, and1-C in the material image sequence selection area1840are replaced with material image sequences4-A,4-B, and4-C that correspond to the user-selected material image2234.

Further, the user-selected material image sequence1-B in the material image sequence arrangement designation area1850is replaced with the user-selected material image sequence4-B. Thus, if the user does not desire to change the material image sequence4-B displayed in the second position in the material image sequence selection area as the material image sequence that corresponds to the user-selected material image2234, the user does not need to select the material image sequence again.

Further, according to the present exemplary embodiment, a result of the face detection is used in the selection condition described in the selection condition row, however, other information can be used as the selection condition. For example, the shooting date and time, or position information obtained by the GPS information can be used as the selection condition. Furthermore, the selection condition is not limited to the result of the face detection and can be a combination of the above-described conditions.

As described above, an output image that matches the material image which the user desires to use and includes a plurality of scenes with an arrangement order can be easily generated by a drag-and-drop operation.

An image generation apparatus according to a fifth exemplary embodiment will be described based on drawings. Since the hardware configuration of the present exemplary embodiment is similar to the one described with reference toFIG. 1according to the first exemplary embodiment, the functional block diagram is similar to the one described with reference toFIG. 2according to the first exemplary embodiment, and the data structure of the material image is similar to the one described with reference toFIG. 3according to the first exemplary embodiment, their descriptions are not repeated. Further, since the selection condition and transition information about the selection condition row are similar to the information described with reference to the tables501and509inFIG. 5according to the first exemplary embodiment, the output image generation processing is similar to the processing described with reference toFIG. 6according to the first exemplary embodiment, and the UI screen is similar to the screen described with reference to the screen illustrated inFIG. 7according to the first exemplary embodiment, their descriptions are not repeated. Further, inFIGS. 24A,24B, and25, components and parts same as those described in other exemplary embodiments are denoted by the same reference numerals and their descriptions are not repeated.

FIG. 24Ais a schematic diagram illustrating a data structure of effect information of a selection condition row of the image generation apparatus according to the present exemplary embodiment. InFIG. 24A, a table2401that presents the effect information is similar to the table505inFIG. 5according to the first exemplary embodiment except that a Condition item is added. The Condition item is used for adding an effect to a material image. A corresponding effect is applied to a material image having the feature quantity in the condition. The “Condition” includes feature quantities of “Type” and “Value”. The Type indicates a type and the Value indicates a value of the feature quantity.

According to the example inFIG. 24A, if a material image from 0 to 2,999 milliseconds has a “Gender” type and its “Value” is “Female”, “Effect Type” of “Soft Focus” is applied. Further, if the feature quantity of “Type” is “Facial Expression” and “Value” is “Smile”, “Effect Type” of “Zoom-In” is applied. Furthermore, if the feature quantity of “Type” is “Season” and “Value” is “Summer” as well as “Type” is “Location” and “Value” is “Beach”, the effect of “Contrast Intensify” will be applied.

“Soft Focus” is image processing for synthesizing, for example, an original image and a blurred image obtained by Gaussian blur processing. Further, “Zoom-In” is image processing for cropping an image and gradually reducing the region according to time passage. Further, “Contrast Intensify” is image processing for enhancing the contrast of the image. These are merely examples and the present exemplary embodiment is not limited to such examples.

FIG. 24Bis a schematic diagram illustrating a data structure of information indicating whether effects regarding the selection condition row can be multiplied according to the image generation apparatus of the present exemplary embodiment. InFIG. 24B, a table2402includes information regarding whether effects can be multiplied. More precisely, the table2402includes items “Effect Type 1” and “Effect Type 2” that indicate the effect type and Multiply that indicates whether the effects of the Effect Type 1 and the effect of the Effect Type 2 can be used together. If the effects of the both types can be used together, “Multiply” will be “Enable”. If not, “Multiply” will be “Disable”.

According to the example inFIG. 24B, “Soft Focus” and “Zoom-In” can be used together, but “Soft Focus” and “Contrast Intensify” cannot be used together. Further, “Contrast Intensify” and “Zoom-In” can be used together. These are merely examples and the present exemplary embodiment is not limited to such examples.

FIG. 25is a flowchart illustrating the selection condition row selection processing of the image generation apparatus according to the present exemplary embodiment. In step S2501, the CPU101obtains effect information that matches the condition. More specifically, the effect information in which feature quantity of the material image selected from the selection condition row matches Condition in the table2401is obtained.

In step S2502, the CPU101determines whether a material image as a target of a plurality of effects exists. If such a material image does not exist (NO in step S2502), the processing proceeds to step S2503. In step S2503, the CPU101selects the effect information obtained in step S2501. In step S2504, the CPU101updates the effect information by replacing the old effect information with the effect information of the selection condition row.

On the other hand, in step S2502, if a material image as a target of a plurality of effects exists (YES in step S2502), the processing proceeds to step S2511. In step S2511, the CPU101determines whether a material image that does not allow the multiplying of the effects exists based on the table2402. If a material image that does not allow the multiplying of the effects does not exist (NO in step S2511), the processing proceeds to step S2503described above.

If a material image that does not allow the multiplying of the effects exists (YES in step S2511), the processing proceeds to step S2512. In step S2512, the CPU101selects a combination of the effect information pieces which can be applied together to the material image that does not allow the multiplying of the effects. In step S2513, with respect to material images other than the material image that does not allow the multiplying of the effects, the CPU101selects the effect information obtained in step S2501, and the processing proceeds to step S2504described above.

As described above, according to the present exemplary embodiment, an effect that matches a feature quantity of a selected material image can be selected and applied to the material image. Further, the image generation apparatus can be controlled such that effects which cannot produce normal effects or produce opposite effects when used together are not combined. Thus, effects can be appropriately provided to a selected material image.

This application claims priority from Japanese Patent Application No. 2012-047947 filed Mar. 5, 2012, which is hereby incorporated by reference herein in its entirety.