Method and apparatus for creating a summary video

An automated creation of a summary video based on video segments extracted from several videos by using a social network of viewers and a social curve specified by a user in order to select and order the video segments. A social network of viewers (influence scores, experts) may be used to establish metrics for video segments. A social curve may be defined with respect to the video segment metrics to allow for the automatic creation of a summary video.

This application claims the benefit, under 35 U.S.C. § 365 of International Application PCT/EP2015/064452, filed 25 Jun. 2015, which was published in accordance with PCT Article 21(2) on 30 Dec. 2015, in English, which claims the benefit of European Patent Application No. 14306017.6 filed 27 Jun. 2014, herein incorporated by reference.

FIELD OF INVENTION

The present disclosure generally relates to the automatic creation of a video summary based on a keyword search and social network analysis.

BACKGROUND

With the prevalence of online content sharing, collaboration, and social media, many users are continuously uploading media content (e.g., pictures and video) to the Internet. In some instances, several users attending a single event may upload several hundreds of pieces of content from various devices to several different online content sharing and social websites. In other instances, many different users may provide content around similar ideas or concepts that are actively being discussed.

Once this content is uploaded, communities of users may comment on the various pieces of content, where some content may be seen more popular or viewed at a higher frequency than others. At times, some users may scour the various pieces of content from a single event or content that discuss similar ideas and/or subjects to manually create a new aggregated piece of content, such as a video mix, to form summary of an event, idea, or subject, for example. Since this manual process may be long and burdensome, there is a need for the automatic creation of video summaries around an event, idea, subject, etc.

However, one drawback from an automatic creation of a video summary from potentially hundreds of pieces of content is that there may be little control in the selection of quality content and/or content created by known users that is used to create the video summary. Therefore, there is a need for the ability to automatically create a video mix summary from a set of videos that also enable the user to define the types of content to be included in the video mix, where the types of content may be based on varying metrics that may be defined by a video summary system, the user, or within a social network of users.

BRIEF SUMMARY

Some embodiments provide a method and apparatus for creating a summary video based on a social network analysis via a social network of users. Some embodiments provide for receiving a keyword query from a user wishing to create a summary video from several videos. Comments, which may or may not be synchronized to the video timeline, related to the several videos may be retrieved based on the keyword query. Pre-selected video segments may then be provided based on the retrieved comments. Social metrics may then be assigned to the pre-selected video segments. A social curve for the video summary may then be received by the user, and a summary video based on the social curve can then be created, where the summary video is a combination of a set of pre-selected video segments.

The preceding Summary is intended to serve as a brief introduction to some embodiments of the present disclosure. It is not meant to be an introduction or overview of all inventive subject matter disclosed in this document. The Detailed Description that follows and the Drawings (or “Figures” or “FIGs.”) that are referred to in the Detailed Description will further describe some of the embodiments described in the Summary as well as other embodiments. Accordingly, to understand all the embodiments described by this document, a full review of the Summary, Detailed Description and the Drawings is needed.

DETAILED DESCRIPTION

In the following detailed description, numerous details, examples, and embodiments are set forth and described. However, it will be clear and apparent to one skilled in the art that the disclosure is not limited to the embodiments set forth, and that the disclosed embodiments may be practiced without some of the specific details and examples discussed.

Throughout this disclosure several terms may be used to refer to different aspects of the automatic video summary creation process according to the present disclosure. A video segment refers to a continuous interval in a video (e.g., source video). Any reference to the term video is meant to convey a video signal with all its synchronized audio signals as a whole, where a video may also include animations, GIFs, and other similar multimedia content. A viewer is a user who posted one or more comments on one or more videos. Viewers may be organized into a social network in some embodiments. A comment is a document that is attached to a particular viewer and may be characterized by a set of text tags or keywords. Some comments may also be synchronized with a video's timeline, meaning that for each of these synchronized comments, there is a known link to a video segment. The user is the person who wants to build the summary video using video segments from all other videos. A social curve is a curve specifying the social score aspects of the video summary that is being built.

Some embodiments of the present disclosure provide a system and method that allows users to automatically generate a video summary from several videos, hereinafter source videos, based on video segments extracted from the several videos. The video summary can be produced automatically by a user specifying a keyword query and a social curve aligned to the desired video summary's timeline. To accomplish this, some embodiments may use textual data that is synchronized with the source videos' timelines. This data may be viewer comments that discuss particular time points or time periods in the source videos. A social network of these viewers may also be utilized in order to better select, rank, and/or order video segments to be included in the video summary.

For example, a web site that hosts videos and viewer comments associated with these videos may be a platform for executing the system and methods of the present disclosure. On such a video hosting platform, registered users may have the possibility to post comments in the feed of videos. Theses commenters may also explicitly specify the time point in the video that is being discussed. For example, the comment, “LOL 3:24,” may be provided to indicate that the commentator saw something funny at minute 3′24″.

Based on the several comments found on a video hosting website, a social network of viewers may be built based on viewers who co-comment on the same videos, viewers who subscribed or follow other viewers, friendship links, number of comments by a viewer, number of up-votes/down-votes received on comments by the viewer, etc. An exemplary implementation of how a video summary may be accomplished according to the present disclosure will now be described in reference toFIG. 1.

FIG. 1illustrates a block diagram of an exemplary system for automatically creating a summary video according to one embodiment of the present disclosure.FIG. 1includes a social network of viewers100, a user105, a search module110, a social network module115, a scoring module120, a sort module125, and an editing module130. The various inputs and outputs illustrated inFIG. 1include a set of videos135from one or more video hosting platforms, comments140made on the set of videos135, keyword search queries145and a social curve150provided by the user, pre-selected video segments with155, comment and social score pairings160, video segment and social score pairings165, ordered lists of video segments170, and the summary video175provided by the system.

The system of the present disclosure may make use of one or more video hosting platforms that provide videos with a corresponding viewer comment thread, where some comments may be synchronized to the timeline of the video. A user105that wants to build a video summary from a set of videos135may use the video hosting platform's traditional search engine to select a set of videos135about a particular subject and/or event, for example. In this search, keywords that may be in the description of the videos or in the video tags and/or titles may be searched.

The user105may then continue to the process of pre-selecting a set of video segments using a search module110. This may be accomplished by the search module110via a keyword search query145that matches comments that may or may not be synchronized to the timelines in the set of videos. Specifically, the search module110returns a set of matched comments and the video segments155based on the keyword search query145.

A social network module115may analyze a social network of viewers100and assign a scoring metric to each viewer (e.g., influence scores, expert scores, etc.). The scoring metric may interchangeably be referred to as a social score throughout this disclosure. The social network may be built by considering viewers100that have commented on the set of videos, or only viewers100who commented on the subset of pre-selected video segments155. As a result of this social network module115, each comment (not necessarily a synchronized comment) may be assigned the social score160of its viewer. In another embodiment, the social score160may be associated to each comment rather than to its creator and is evaluated as a function (e.g., weighted function) of its creator's social score and of the comment's own score. The final comment score160or metric may be evaluated according to its position within the discussion thread it belongs to (e.g. lead comment, response comment, ending comment, etc.), the length of the discussion thread, the number of comments that have been later posted and relating to this comment, the number of viewers who co-commented on the same videos, viewers who subscribed or follow other viewers, friendship links, number of comments by a viewer, number of up-votes/down-votes received on comments by the viewer, etc. One of ordinary skill in the art will recognize that various appropriate methods may be used to ascertain any type of metric to video segments and/or comments based on the available data points provided within a video hosting platform.

The scoring module120may then aggregate the social scores160of comments for each pre-selected video segment155and compute a social score for the video segment165. For example, a video segment social score165for any particular pre-selected video segment155may be the sum, the median or the mean of the scores160or of a part of the scores160(e.g., a function applied on the best 50%, or the worst 67%, or the part of the scores160that is in the range of the best 40%-90%.)

The user105may also pre-define a social curve150using a graphical interface, for example.FIG. 2illustrates an exemplary social curve200where a social score205is mapped on a y-axis and a time duration210of the video summary to be created is mapped along the x-axis. Using a social curve interface, the user150may pre-define the preferred duration205of the video summary, and draw freehand the general shape (i.e. social curve200) of social score the user105wants to use for the video summary. The social curve200enables the user to alternate between video segments

that were commented on by important people in the social network of viewers100and others that were not commented on by known and/or highly rated people within the social network of viewers100.

The sort module125may then use the social curve150provided by the user105and order the pre-selected video segments165so as to concur with the social curve150. For example, a higher value of the curve should indicate a video segment with a higher social score. The sort module125may also discard video segments in cases where the video segments are too numerous or too long. The sort module125then returns an ordered list of selected video segments170for the final video summary.

The ordering of video segments may be conducted in various ways. For example, a user105provides the desired social curve150, where the curve dictates the duration of the summary video along the x-axis. Then each video segment165may be compared with the social score specified by the social curve150at a corresponding time along the time axis. The analysis may be integrated over the duration of the considered video segments. The video segments165that best fit the integrated social curve level205(e.g., exact value±delta/difference) are then identified. When several video segments are identified as fitting the social curve150, one of the video segments165may be selected at random or based on a defined selection criteria for selecting one out of many potential video segments. The sorting may be an optimization where the goal may be to minimize the area under the absolute value of the social curve200minus the social curve of the ordered video segments170. An additional regularization parameter may assist in controlling the length of the selected video segments165.

Another exemplary method for selecting one of the matching video segments165is to select the video segment that best fits the curve displacement or, when several video segments165are best fits for the curve, selecting one of the best fits at random. In some embodiments, if the curve is sharp (high valued derivatives), the shortest video clips may be selected. If the curve is flat, a few short video segments and a longer video segment may be selected.

Eventually, the editing module130takes the ordered list of video segments170and builds the final video summary175. The editing module130may add transitions between video sequences so both the audio signal and the video signal between two video segments transition smoothly. For example, fading techniques may be applied from segment to segment.

FIG. 3illustrates a flow chart of an exemplary process300used in some embodiments for creating a summary video according to the present disclosure. The process300may begin by receiving (at step310) a keyword query from a user. The process300may then retrieve (at step320) all comments that are relevant to the keyword query using a search module110similar to the one described inFIG. 1. Some or all of the retrieved comments may be synchronized comments that identify a time or segment of the video for which the comment corresponds to. Then the module110returns (at step330) video segments that are related to the comments. For comments that are not synchronized comments, the entire video may be treated as a single video segment. Synchronized comments that identify a time may treat a pre-defined duration of the video as a video segment, where the video segment begins at the identified time and ends at a pre-defined duration. Synchronized comments that identify a complete segment (e.g., identify a start and end time or duration) will return the appropriate portion as a video segment in step330.

Using a social network of viewers, the process300may then infer a social metric or score for each viewer and assign (at step340) the social metric to each viewer's comments. The process300may then use a scoring module to assign (at step350) a social metric to each pre-selected video segment from step330based on the social metrics of the video segment's related comments assigned at step340.

The process300then receives (at step360) a social curve (e.g.,200) specified by a user for the video summary to be created. This social curve specifies the alternation between high scored and low-scored parts of the video summary. The process300may then use a sort module to return (at step370) an ordered list of the selected video segments based on the social metrics of the video segments so that the order of video segments follow the social curve received at360. Finally, the process300may create (at step380) a summary video using an editing module via the ordered list of video segments from370.

One of ordinary skill in the art will recognize that process300may be performed in various appropriate ways without departing from the scope of the disclosure. For instance, the process may not be performed as one continuous series of operations in some embodiments. In addition, the process may be implemented using several sub-processes or as part of a larger macro-process. Furthermore, various processes may be performed concurrently, sequentially, or some combination of sequentially and concurrently. Moreover, the operations of the process may be performed in different orders.

FIG. 4illustrates an exemplary block diagram of a system400for implementing a summary video creation process or engine according to some embodiments of the present disclosure. The system400includes a server410and one or more electronic devices such as smart phones420, personal computers (PCs) (e.g., desktops or laptops)430, and tablets440. The server410provides support for the summary video creation processes and/or engine according to the present disclosure as well as for hosting videos used in the video summary and/or the summary video itself via the Internet450. In some embodiments, users may access the summary video engine on the server410and provide a keyword search query and social curve via a browser or application on the electronic devices.

In some embodiments, the above-described operations may be implemented as on a particular machine such as a desktop computer, laptop, handheld device (e.g. smartphone or tablet), one or more servers accessible via the Internet, or any combination of such devices. Many of the processes and modules described may also be implemented as software processes that are specified as at least one set of instructions recorded on a non-transitory computer readable storage medium. When these instructions are executed by one or more computational elements (e.g., microprocessors, microcontrollers, Digital Signal Processors (“DSPs”), Application-Specific ICs (“ASICs”), Field Programmable Gate Arrays (“FPGAs”), etc.) the instructions cause the computational element(s) to perform actions specified in the instructions.

FIG. 5illustrates a schematic block diagram of a computer system500with which some embodiments of the disclosure may be implemented. For example, the system described above in reference toFIG. 1may be at least partially implemented using computer system500. As another example, the processes described in reference toFIG. 3may be at least partially implemented using sets of instructions that are executed using computer system500.

Computer system500may be implemented using various appropriate devices. For instance, the computer system may be implemented using one or more personal computers (“PC”), servers, mobile devices (e.g., a Smartphone), tablet devices, and/or any other appropriate devices. The various devices may work alone (e.g., the computer system may be implemented as a single PC) or in conjunction (e.g., some components of the computer system may be provided by a mobile device while other components are provided by a tablet device).

Computer system500may include a bus510, at least one processing element520, a system memory530, a read-only memory (“ROM”)540, other components (e.g., a graphics processing unit)550, input devices560, output devices570, permanent storage devices580, and/or a network connection590. The components of computer system500may be electronic devices that automatically perform operations based on digital and/or analog input signals.

Bus510may represent all communication pathways among the elements of computer system500. Such pathways may include wired, wireless, optical, and/or other appropriate communication pathways. For example, input devices560and/or output devices570may be coupled to the system500using a wireless connection protocol or system. The processor520may, in order to execute the processes of some embodiments, retrieve instructions to execute and data to process from components such as system memory530, ROM540, and permanent storage device580. Such instructions and data may be passed over bus510.

ROM540may store static data and instructions that may be used by processor520and/or other elements of the computer system. Permanent storage device580may be a read-and-write memory device. This device may be a non-volatile memory unit that stores instructions and data even when computer system500is off or unpowered. Permanent storage device580may include a mass-storage device (such as a magnetic or optical disk and its corresponding disk drive, flash memory/storage, SIM cards, etc).

Computer system500may use a removable storage device and/or a destination storage device as the permanent storage device. System memory530may be a volatile read-and-write memory, such as a random access memory (“RAM”). The system memory may store some of the instructions and data that the processor uses at runtime. The sets of instructions and/or data used to implement some embodiments may be stored in the system memory530, the permanent storage device580, and/or the read-only memory540. For example, the various memory units may include instructions for authenticating a client-side application at the server-side application in accordance with some embodiments. Other components (e.g.550) may perform various other functions. These functions may include interfacing with various communication devices, systems, and/or protocols.

Input devices560may enable a user to communicate information to the computer system and/or manipulate various operations of the system. The input devices may include keyboards, cursor control devices, audio input devices and/or video input devices. Output devices570may include printers, displays, and/or audio devices. Some or all of the input and/or output devices may be wirelessly or optically connected to the computer system.

Finally, as shown inFIG. 5, computer system500may be coupled to a network through a network adapter590. For example, computer system500may be coupled to a web server on the Internet such that a web browser executing on computer system500may interact with the web server as a user interacts with an interface that operates in the web browser.

As used in this specification and any claims of this application, the terms “computer”, “server”, “processor”, and “memory” all refer to electronic devices. These terms exclude people or groups of people. As used in this specification and any claims of this application, the term “non-transitory storage medium” is entirely restricted to tangible, physical objects that store information in a form that is readable by electronic devices. These terms exclude any wireless or other ephemeral signals.

It should be recognized by one of ordinary skill in the art that any or all of the components of computer system500may be used in conjunction with the disclosed embodiments. Moreover, one of ordinary skill in the art will appreciate that many other system configurations may also be used in conjunction with the disclosed embodiments or components of the embodiments.

Moreover, while the examples shown may illustrate many individual modules as separate elements, one of ordinary skill in the art would recognize that these modules may be combined into a single functional block or element. One of ordinary skill in the art would also recognize that a single module may be divided into multiple modules.

While the disclosure has been described with reference to numerous specific details, one of ordinary skill in the art will recognize that the disclosure can be embodied in other specific forms without departing from the scope of the disclosure. For example, several embodiments were described above by reference to particular features and/or components. However, one of ordinary skill in the art will realize that other embodiments might be implemented with other types of features and components, and that the disclosure is not to be limited by the foregoing illustrative details.