Patent ID: 12254036

DESCRIPTION OF EMBODIMENTS

Subject matter will now be described more fully hereinafter with reference to the accompanying drawings, which form a part hereof, and which show, by way of illustration, specific example embodiments. Subject matter may, however, be embodied in a variety of different forms and, therefore, covered or claimed subject matter is intended to be construed as not being limited to any example embodiments set forth herein; example embodiments are provided merely to be illustrative. Likewise, a reasonably broad scope for claimed or covered subject matter is intended. Among other things, for example, subject matter may be embodied as methods, devices, components, or systems. Accordingly, embodiments may, for example, take the form of hardware, software, firmware or any combination thereof (other than software per se). The following detailed description is, therefore, not intended to be taken in a limiting sense.

In the accompanying drawings, some features may be exaggerated to show details of particular components (and any size, material and similar details shown in the figures are intended to be illustrative and not restrictive). Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the disclosed embodiments.

Embodiments are described below with reference to block diagrams and operational illustrations of methods and devices to select and present media related to a specific topic. It is understood that each block of the block diagrams or operational illustrations, and combinations of blocks in the block diagrams or operational illustrations, can be implemented by means of analog or digital hardware and computer program instructions. These computer program instructions or logic can be provided to a processor of a general purpose computer, special purpose computer, ASIC, or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, implements the functions/acts specified in the block diagrams or operational block or blocks.

In some alternate implementations, the functions/acts noted in the blocks can occur out of the order noted in the operational illustrations. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments of methods presented and described as flowcharts in this disclosure are provided by way of example in order to provide a more complete understanding of the technology. The disclosed methods are not limited to the operations and logical flow presented herein. Alternative embodiments are contemplated in which the order of the various operations is altered and in which sub-operations described as being part of a larger operation are performed independently.

For the purposes of this disclosure the term “server” should be understood to refer to a service point which provides processing, database, and communication facilities. By way of example, and not limitation, the term “server” can refer to a single, physical processor with associated communications and data storage and database facilities, or it can refer to a networked or clustered complex of processors and associated network and storage devices, as well as operating software and one or more database systems and applications software which support the services provided by the server.

For the purposes of this disclosure a “network” should be understood to refer to a network that may couple devices so that communications may be exchanged, such as between a server and a client device or other types of devices, including between wireless devices coupled via a wireless network, for example. A network may also include mass storage, such as network attached storage (NAS), a storage area network (SAN), or other forms of computer or machine readable media, for example. A network may include the Internet, one or more local area networks (LANs), one or more wide area networks (WANs), wire-line type connections, wireless type connections, cellular or any combination thereof. Likewise, sub-networks, which may employ differing architectures or may be compliant or compatible with differing protocols, may interoperate within a larger network. Various types of devices may, for example, be made available to provide an interoperable capability for differing architectures or protocols. As one illustrative example, a router may provide a link between otherwise separate and independent LANs.

For example, a network may enable RF or wireless type communication via one or more network access technologies, such as Global System for Mobile communication (GSM), Universal Mobile Telecommunications System (UMTS), General Packet Radio Services (GPRS), Enhanced Data GSM Environment (EDGE), 3GPP Long Term Evolution (LTE), LTE Advanced, Wideband Code Division Multiple Access (WCDMA), Bluetooth, 802.11b/g/n, or the like. A wireless network may include virtually any type of wireless communication mechanism by which signals may be communicated between devices, such as a client device or a computing device, between or within a network, or the like

A communication link may include, for example, analog telephone lines, such as a twisted wire pair, a coaxial cable, full or fractional digital lines including T1, T2, T3, or T4 type lines, Integrated Services Digital Networks (ISDNs), Digital Subscriber Lines (DSLs), wireless links including satellite links, or other communication links, such as may be known to those skilled in the art. Furthermore, a computing device or other related electronic devices may be remotely coupled to a network, such as via a telephone line or link, for example.

A computing device may be capable of sending or receiving signals, such as via a wired or wireless network, or may be capable of processing or storing signals, such as in memory as physical memory states, and may, therefore, operate as a server. Thus, devices capable of operating as a server may include, as examples, dedicated rack-mounted servers, desktop computers, laptop computers, set top boxes, integrated devices combining various features, such as two or more features of the foregoing devices, or the like. Servers may vary widely in configuration or capabilities, but generally a server may include one or more central processing units and memory. A server may also include one or more mass storage devices, one or more power supplies, one or more wired or wireless network interfaces, one or more input/output interfaces, or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or the like.

Throughout the specification and claims, terms may have nuanced meanings suggested or implied in context beyond an explicitly stated meaning. Likewise, the phrase “in one embodiment” as used herein does not necessarily refer to the same embodiment and the phrase “in another embodiment” as used herein does not necessarily refer to a different embodiment. It is intended, for example, that claimed subject matter include combinations of example embodiments in whole or in part. In general, terminology may be understood at least in part from usage in context. For example, terms, such as “and”, “or”, or “and/or,” as used herein may include a variety of meanings that may depend at least in part upon the context in which such terms are used. Typically, “or” if used to associate a list, such as A, B or C, is intended to mean A, B, and C, here used in the inclusive sense, as well as A, B or C, here used in the exclusive sense. In addition, the term “one or more” as used herein, depending at least in part upon context, may be used to describe any feature, structure, or characteristic in a singular sense or may be used to describe combinations of features, structures or characteristics in a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again, may be understood to convey a singular usage or to convey a plural usage, depending at least in part upon context. In addition, the term “based on” may be understood as not necessarily intended to convey an exclusive set of factors and may, instead, allow for existence of additional factors not necessarily expressly described, again, depending at least in part on context.

In this age of content overload, few people have time to consume extensive audio/video information that is available from the various sources. The time required to listen to audio or watch longer videos to understand their content often deters users from entirely engaging with such content. Currently, users have to inspect the content item at randomly selected time points to determine if it suits their interests and warrants a more extensive review. Moreover, review of large multimedia files such as video content items can pose severe strain on computational resources. These problems can be exacerbated when users interact with resource-intensive content such as videos via mobile devices. In such resource and space constrained environments, providing summaries of multimedia content items is desirable. Users may be encouraged to review a short, condensed summary, the longer version of which they may otherwise ignore thereby resulting in improved user engagement.

The techniques described herein allow multimedia content, such as videos, accessible to the end users in digestible chunks, greatly reducing the storage size of the video and transmission speed across the network. Thus, users of mobile devices can increase their access and consumption of the multimedia content. Turning now to the figures,FIG.1is a schematic diagram of a multimedia summarization system100that combines features from different media channels in order to produce summaries of multimedia items in accordance with one embodiment. The multimedia summarization system100comprises an input module102, a summarization module104and an output module106stored in a non-transitory processor-readable storage device. When executed by the processor of a computing device (not shown), these modules receive a content item108and produce a multimedia summary150of the received content item108in accordance with embodiments described herein. If the received content item108is a video content item, a video summary150comprising portions of the content item108that convey the important portions of the content item108can be generated. On the other hand, if the received content item108is an audio content item, a corresponding audio summary150comprising its important portions or highlights can be generated.

The input module102receives a multimedia content item108such as an audio, a video content item to be summarized. The multimedia content108can be received by the input module102from both internal and external sources associated with the multimedia summarization system100. In an embodiment, the multimedia summarization system100can be associated with a content provider who receives content from various content partners. Based on the content provider's determination of the length of the content to be presented to the users, the content received from the content providers can be summarized in accordance with embodiments described herein. In an embodiment, the multimedia summarization system100can be made available to users in general, for example, via a publicly accessible server. The multimedia content108may thus include amateur or professionally generated content in any format. By the way of illustration and not limitation, the received content item108can pertain to, news, meetings, and videos of events, movies, radio programs or other recordings. In addition, the multimedia content108can be of any format.

The input module102can also receive a compression budget110for the multimedia content item108. The compression budget110can include the extent to which the multimedia content item108is to be summarized. The compression budget110can be received in terms of different criteria. In an embodiment, the compression budget110can be expressed as a percentage of the total running time of the received content item108. The compression budget110can also be expressed in terms of the actual running time of the summary150. Thus, if the compression budget110is received as a percentage, the input component102can be configured to calculate the total running time for the summary150and transmit it to other modules of the multimedia summarization system100.

In an embodiment, the input module102comprises an analyzing module1022that analyzes the received multimedia content108to determine if it can be summarized. If the received content item108can be summarized, it is transmitted to the summarization module104. If the received content item108cannot be summarized, a null result is returned. In an embodiment, the analyzing module1022comprises a statistical classifier. The classifier can be trained on summarizability labels provided by humans. It is based on a simple linear regression model that takes the length of the content item, source and various coarse-grained features for video and audio quality (such as the bitrate or whether speech is present) into account.

If it is determined that the received content item108can be summarized, the content item108along with the compression budget150is transmitted by the input module102to the summarization module104. The summarization module104processes the received content item108in accordance with the various procedures as described herein to identify portions of the content item108to be included in the summary150. In an embodiment, the summarization module104can be configured to provide a series of time intervals corresponding to the portions of the received content item108that are to be included in the summary150. In an embodiment, the summarization module104can provide the portions of the received content item108to be included in the summary150. The series of time intervals or portions of the received content item108are identified such that the resultant summary150conforms to the compression budget110provided with the content item108.

The output module106receives one of the series of time intervals corresponding to the portions of the received content item108or the portions of the received content item108to generate the summary150. In an embodiment, the series of time intervals can be in the form of a sequence of non-contiguous, non-overlapping sub-intervals of equal lengths. Each subinterval has a start-time and an end-time that corresponds to a clip or a portion of the received content item108. In an embodiment, the output module106can further comprise an assembly module1062that receives the time intervals. The assembly module1062generates a new content item by stitching together the segments or clips of the content item108that correspond to the time intervals thereby generating the summary150which is returned to a requesting entity. In an embodiment, the stitching can comprise more than just concatenation of content clips related to the subintervals. It can comprise, for example, transitions to smooth the junctures between video segments or even insertion of additional content such as advertisements.

In an embodiment, additional sources of information including closed-captioning and meta information associated with a content item can be used for summary generation in accordance with some embodiments described herein. For example, in the case of content items which may include songs, methods as described herein can be applied based on any closed-captioning data available for the content. By the way of illustration and not limitation, the additional sources of information can comprise audio processing components that recognize instances of laughter, clapping, etc. and video processing components that carry out boundary shot detection on video content, topic labels or tags associated with the content item108.

FIG.2shows the sub-modules of the summarization module in accordance with one embodiment. The sub-modules comprise, a sequence generating module202, a sequence scoring module204and a sequence identification module206. The sequence generating module202receives the content item108upon the determination that it is summarizable and processes it to generate a plurality of segment sequences. Each sequence comprises segments of audio data corresponding to a respective series of contiguous, non-overlapping time intervals. The plurality of segment sequences thus generated are processed by the sequence scoring module204to compute an informativeness score for a segment and a coherence score for a sequence in accordance with embodiments detailed further infra. In an embodiment, the informativeness score for each segment comprises a combination of a salience score and a diversity score. The plurality of segment sequences along with their respective informativeness scores for each sequence segment are transmitted to the sequence identification module206. The sequence identification module206is configured to identify and select a subsequence of segments from one of the plurality of segment sequences that satisfies the compression budget110while maximizing the informativeness and coherence scores. The selected subsequence of segments is transmitted to the output module106for the generation of the summary150as described herein.

FIG.3illustrates a segment diagram of the sequence generating module202in accordance with one embodiment. As described supra, the sequence generating module202is configured to generate a plurality of segment sequences from the received content item108. The sequence generating module202comprises an extraction module302, an automated speech recognition (ASR) module304and a partitioning module306. If the received content item108is an audio-only content item, it may be directly provided to the ASR module304without any action by the extraction module302. If the received content item108also includes video content, the audio from the content item108is ripped by the extraction module302and such audio is provided to the ASR module306. In an embodiment, the extraction module302can comprise an open source tool such as the VLC tools.

The ASR module304receives the audio input either directly or via the extraction module302and generates a text transcript of the received audio file. The output of the ASR module304comprises a text transcript of a standard form used by media players. In an embodiment, the transcript is associated with a word lattice that is then reduced to a Word Confusion Network (WCN). This network can be viewed as a graph of all distinct utterances possible within each successive time interval, along with their associated probabilities. Characteristics of spoken language such as pauses, hesitations, corrections, back-channel utterances and the like are part of the word information in the WCN. Thus, the ASR module304can hypothesize multiple alternatives for each audio segment along with a confidence score per alternative. In an embodiment, there may be no alternative segments hypothesized for a segment and hence, so a uniform confidence score is provided. In an embodiment speaker identifiers can be inferred by the ASR module3204via mapping particular speakers to respective identifiers. In an embodiment, the text transcript generated by the ASR module may not distinguish between different speakers or if a speaker has changed. In an embodiment, a time-aligned text transcript of the received content item108is generated so that each audio segment has a time interval associated with it.

The partitioning module306receives the text transcript from the ASR module304and divides it into a plurality of contiguous text segment sequences corresponding to time intervals marked by their respective start and end times. In one embodiment, the text segments of each sequence have the same size so that segments from different sequences having different sizes. It may be appreciated that the segment sizes for the various sequences need to be optimized for coherence and informativeness. Smaller size segments provide potentially more control over informativeness but assembling a larger number of segments together for the summary150can compromise coherence.

It may be appreciated that while the extraction module302and the ASR module304are shown as part of the sequence generating module202, it may be appreciated that this is not necessary and that one or more of these modules may be remote from the multimedia summarization system100and accessible via a network. For example, the ASR module304can comprise open source tools such as, Sphinx, HTC and the like.

In an embodiment, the computer readable medium further comprises instructions that cause the processor to determine an informativeness score of each segment of the plurality of segment sequences, where the informativeness score is determined using a supervised machine learning algorithm such as, regression models based on simple linear regression or support-vector-machine regression with linear or non-linear kernels. The informativeness score of a segment reflects the probability that the segment belongs in the summary150. The training data for the machine learning algorithm comprises transcripts of content items with labels for particular time spans pertaining to each content item, indicating if the particular time span should be included in a summary of the respective content item. The subsequence of segments that are included in a summary of a respective one of the content items is based on a machine-learned regression function that combines weights for each member of a set of features for computing the informativeness score. The feature set includes the segment's Page Rank and DivRank scores), the respective position of a segment in transcript order, terms (comprising words, or word-stems, or phrases) in the segment, length of the segment, similarity between the segment and the preceding and succeeding segments in the sequence (if any), and changes in feature values between the segment and the preceding and succeeding segments in the sequence (if any).

In an embodiment, no training data is used and hence an unsupervised machine learning methodology is implemented.FIG.4shows the details of the sequence scoring module204in accordance with this embodiment. The sequence scoring module204comprises a similarity module402, a salience module404, a diversity module406and an informativeness module408. The plurality of segment sequences from the sequence generating module202are transmitted for scoring to the sequence scoring module204. In an embodiment, each of the segment sequences can be represented as a graph wherein each segment is represented as a node of the graph and the directed edges between later nodes and earlier ones in the sequence represent links between them. To compute the weight of an edge between two nodes, the cosine similarity of the segments associated with the nodes has to be calculated. Accordingly, the similarity module402is configured to determine tf.idf term weights of the terms associated with each node (i.e., with each segment) and a cosine similarity calculation between segments is carried out. In some embodiments, the terms can be truncated to correspond to stems, so that frequency counts are higher than those of words that map to the same stem. In some embodiments, the features are semantic ones where the terms in the segment are extended to synonymous terms found in a thesaurus or found by a statistical measure of strength of association in a large collection.

In an embodiment, the similarity values between the various nodes as determined by the similarity module402are accessed by the salience module404in order to calculate a salience score for each segment represented as a node in a graph. To compute the salience score, the weights of nodes in the sequence graph as described earlier are updated based on a Page Rank algorithm that carries out a random walk of the graph. The weight of a node in the graph is initially assigned based on position of the corresponding segment in the sequence (i.e., earlier segments have more weight, with exponential decay). In each iteration of the Page Rank, the weight of each node is updated based on the weights of ‘incoming’ links from later nodes that point to it, normalized by the weights of ‘outgoing’ links that it points to, with a renormalization to represent transition probabilities. The output is an array of salience scores for each segment in the sequence.

In an embodiment, the diversity module406determines the diversity score for each of the nodes in the graph. The DivRank algorithm is used in a random walk on the graph similar to Page Rank except that at each iteration, the diversity is achieved by updating not only the node weights as in Page Rank but also the edge weights (i.e., transition probabilities) to reflect how often a particular node has been revisited in the random walk. The output is an array of diversity scores for each segment in the sequence. Including diversity in the computations for generating the summary150, results in non-redundant regions of the content item108being favored.

The informativeness module408combines the salience score and the diversity score for each node (or segment). In an embodiment, the salience and diversity scores make equal contributions to the final informativeness score. In an embodiment, the salience and the diversity scores can be combined so that one score is weighted more heavily than the other. The parameters for combining the two scores can also be learnt via machine learning. The informativeness module408can be trained to apply appropriate parameters for combining the salience and diversity scores via supervised data which comprises parameters as set by human trainers.

Each of the segments sequences are also scored for coherence by the coherence module410as detailed herein. The coherence of a sequence can be computed using weights for contiguity in transcript order and the time duration of the segments. In an embodiment, speaker change (if available) can also be used. In another embodiment, the presence of particular terms like pronouns or subordinating conjunctions can also be used. In another embodiment, general text segmentation techniques, such as TextTiling, based on segment similarity can be used to find coherence gaps in the segments. In an embodiment, coherence weights can be trained from a corpus.

FIG.5is a block diagram showing the details of a sequence identification module206in accordance with one embodiment. The informativeness scores generated by the sequence scoring module204for the segments in the sequences are received by the sequence identification module206. The segments in each of the sequences are added up by the addition module502to the target length based on the coherence score and compression budget110as detailed infra. The scores of the plurality of segment sequences are compared by the comparison module504. The segment sequence having the highest score is selected by the selection module506for generating the summary150.

In one embodiment, summarization is mathematically dealt with as an optimization problem. In a further embodiment, the methods disclosed herein are applied to text transcripts of audio tracks. A text is considered in this embodiment as a sequence of sentences V. Given a budget of length L words, the summarizing module104is configured to find a subsequence S of V that maximizes the function F(S) such that length(S) is approximately equal to L. If F is monotone submodular, the problem can be solved using a greedy algorithm near optimally. To solve this problem, F is decomposed into relevant features, reflecting informativeness (comprising aspects of salience and diversity) and coherence, based on relations between successive words that characterize the coherence of the subsequence S. The coherence of a subsequence S is a function, detailed infra, of the confidence of S and the weights of discourse features that relate successive terms in the subsequence (e.g., features such as the presence of conjunctions, pronouns, common word-stems, etc.).

If V is the time interval for the content item108and LTis the compression budget, for example, in seconds a greedy approach can be used for partitioning the content item108in accordance with embodiments described herein. A set of segment sizes B for the content item108, is determined as shown below based on the number of iterations λ1and size adjustment parameter λ2.

For i=1 to λ1doBlockSizes :- {int(λ2*|LT|/i)} ∪ BlockSizesdoneReturn BlockSizes

FIG.6illustrates an embodiment of two different partitionings of the content item108in accordance with one embodiment. At602, the content item108is partitioned into a sequence of segments of size b1. At604the content item108is partitioned into a sequence of segments of size b2. As seen fromFIG.6, each partitioning has three different layers corresponding to n=1, n=2 and n=3 in the WCN for each segment. The segment sizes as determined herein can be further adjusted slightly so that they do not end right after an inferred pause boundary. In order to obtain the summary160, Equation (1) shown below is solved for each layer n in each refined partitioning. This corresponds to filling in scores for all of the segments inFIG.6and the top scored segments are greedily selected up to length |LT|.

When the above procedure is employed by the summarization module104for the audio transcript of the content item108, fixed-size segments of audio as shown above are used. Accordingly, if V is the time interval (in seconds) or the total running time of the content item108, S is the sub-interval of V (S⊆V), b is a segment size in seconds chosen from a set B of segment sizes and n be the segment b's nth-ranked WCN hypothesis, then the summary as generated by the multimedia summarization system100should satisfy the Eq. (1) shown below:
S*∈argmaxbeB,Sbn≤vF(Sbn) where |Sbn|≈LTEqn. (1)

Sbn, is a subinterval of V of size b and ASR rank n and where F:2v→{0, 1} such that:

F⁡(Sbn)=(λ3Sbn⁢∑x∈Sbn⁢Sal⁡(x))+(λ4Sbn⁢∑x∈Sbn⁢Div⁡(x))+λ5⁢Coh⁡(Sbn)Eqn.⁢(2)⁢Coh⁡(Sbn)=λ6⁢Conf⁡(Sbn)+λ7⁢∑i=1Sbn-1⁢⁢∑j=1m⁢⁢σj⁡(wi,wi+1)Eqn.⁢(3)

In Eq. (3) Coh is the Coherence of the sequence corresponding to the subinterval Sbn, and Conf is the confidence of the nth-ranked Word Confusion Network (WCN) hypothesis for Sbn, and the σ1 . . . mare discourse features (e.g., presence of conjunctions, pronouns, common word stems, etc.) relating successive word pairs (bigrams) (wi, wi+1) in λbn. The λ's in Eqns. (2) and (3) are tuning parameters that can be set heuristically or else learned from training data using a supervised machine learning algorithm of the sort mentioned supra. It may also be noted that as discussed supra, segment size variation and segment partitioning also contributes to coherence.

Sal or Salience in Eq. (2) is calculated as a Page Rank:

Sal⁡(u)=(1-λ8)+λ8⁢∑v∈IN⁡(u)⁢p0⁡(u,v)⁢Sal⁡(v)Σw∈OUT⁡(v)⁢p0⁡(v,w)Eqn.⁢(4)

where u and v are graph nodes corresponding to segments and λ8is the damping factor. Again as discussed supra, each segment is represented as a node in a graph corresponding to the sequence of segments of a equal width and is weighted based on the transition probability p0(u, v) from node u to node v of ‘incoming’ (IN) links from later segments that point to it normalized by the weights of outgoing (OUT) links that it points to.

For the Salience calculation as shown in Eq. (4), the transition probability (edge weight from one node to another) remains stationary over time. In contrast, Div in Eq. (4) yields the diversity score DivRank, which is an extension to Page Rank that reinforces transition probabilities to the node based on the number of times the node is visited. In Eq. (5) shown below which is used for computing the diversity score, the transition probability (edge weight) from a node u to a node v at time t is dependent on (i) the prior probability of visiting node v, p*(v) (which is the same as the Page Rank jump probability if the prior is uniform) as well as (ii) the static transition probability p0(u, v) from node u to node v prior to any reinforcement, times NT(v), which is the number of times the vertex v has been visited at time T.

pT⁡(u,v)=(1-λ8)⁢p*⁡(v)+λ8⁢NT⁡(v)*p0⁡(u,v)Σw∈V⁢NT⁡(w)*p0⁡(u,w)Eqn.⁢(5)

FIG.7shows a flowchart700that details a method of generating a summary for a multimedia content item107in accordance with embodiments described herein. The method begins at702wherein a summarizable multimedia content item702is received in addition to a desired level of compression or compression budget for a summary. The compression budget may be expressed either in terms of the fraction of the total running time of the actual content item107or in terms of the absolute running time (in terms of seconds, minutes and the like) for the summary150. A received content item may not be summarizable if, for example, the compression budget is above the total running time of a received content item or over a certain threshold fraction of the total running time of the actual content item107. In this case, generating the summary may be more expensive and hence a received content item may be rejected as not being summarizable. Hence, the content item107and the desired compression rate or desired level of compression or compression budget is received at702after a pre-processing that determines if the content item107is summarizable.

At704, a text transcript is obtained from the ASR module304via processing of the audio portion of the content item107and the text transcript thus obtained is partitioned into sequences of segments at706. The segments in a particular sequence have respective fixed and equal widths (i.e., block size) different from segments of other sequences. In an embodiment, multiple hypotheses can be generated with the text transcript by the ASR module304. Accordingly, multiple layers can be generated at706for different ASR hypotheses for each segment. At708the ASR confidence scores are obtained. It can be appreciated that708can be omitted if only a single hypothesis is generated by the ASR module304. At710the informativeness scores comprising a combination of the salience score and the diversity score are obtained for the segments in each of the segment sequences. At712, segments with low WCN confidence scores are discarded. At714, segments are added so that the resulting sequence has the highest combination of informativeness and coherence scores per Eqn. (2), until the target length condition is satisfied. The result is a best candidate summary subsequence for the particular block size. When the compression budget is expressed in terms of the actual running time for the summary150, the target length will be equal to the compression budget. However, if the compression budget is expressed as a fraction, then:Target length (in time units)=compression budget*total running time of content item107.

At716, the candidate summary subsequences (each with a different block size) are compared and the subsequence with the highest score is selected for generating the summary. In an embodiment, the start and end times of each of the segments in the selected segment subsequence is transmitted to an assembly module1072for the generation of the summary content item150.

FIG.8shows a flowchart800detailing a method of obtaining the best candidate summary subsequence for each particular block size from a sequence in accordance with one embodiment (of step714inFIG.7). The method begins at802wherein a candidate segment is selected from the sequence based on the informativeness scores. If no segments are left in the current layer for selection at802, the method moves to804wherein the criteria for high ASR confidence is lowered and the candidate at that block address (i.e., time interval) in the next layer from the WCN is chosen (e.g., the layer drops from n=1 to n=2, etc.). If no layers in a particular sequence of segments remain, the method terminates on the end block.

If a candidate segment is selected at802, the length of the next candidate is added to the current summary output length and tested for whether the total is within the length budget as shown at806. If at806, it is determined that the length over the compression budget110, it indicates that the segments required for constructing the summary have been selected for that particular sequence of segments and hence, the method terminates on the end block. If at806, it is determined that the length is less than or equal to the compression budget110, the method determines the coherence of the summary that would result if the next candidate is added to the summary as shown at808. At810the candidate segment is added to the summary in input order and the method returns to802for selecting the next informative segment for processing as detailed herein. The method terminates if there is no next most informative candidate segment at802, at which point the summary, if any, is output.

FIG.9shows the experimental results obtained in an evaluation of the summaries generated by the multimedia summarization system100. A test set of source video items was created and summaries for the test set were generated in accordance with the various embodiments detailed herein. The summaries generated by the multimedia summarization system100for a compression budget of 10% were scored by auto-comparing informativeness of transcripts of the system-summarized segments with transcripts of human-summarized video items using the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) method. It computes the score shown below in Eqn. 6:

ROUGEN=ΣS∈REF⁢Σgramn∈S⁢Countmatch⁡(gramn)ΣS∈REF⁢Σgramn∈S⁢Count⁡(gramn)Eqn.⁢(6)

In Eqn. 6, let the human summaries be the set REF; the numerator counts n-grams that occur in both the system and human summary, while the denominator counts n-grams that occur in the human summary. Here ROUGE-1 considers n-grams which are unigrams whereas ROUGE-SU4 considers bigrams with internal gaps of up to 4 words.FIG.8shows Precision, Recall, and F-measure for summaries of the source videos generated by the multimedia summarization system100in accordance with embodiments described herein. As seen fromFIG.8, the informativeness of summarization can further improve with curated transcripts of the speech in the audio when compared with transcripts generated via ASR.

Generally, in order to get from audio to natural language, ASR must be availed of. However, ASR quality can be poor, as it is highly dependent on training data in the domain of the input audio. Further, audio output can be degraded by poor audio recording technologies, microphone quality, and ambient noise during recording, etc. The result is poor confidences associated with parts of the ASR output. Embodiments of the multimedia summarization system100as disclosed herein are therefore configured to be highly noise-tolerant.

The content items summarized by the embodiments disclosed herein can come from any source and is not restricted to news, meetings, videos tied to certain events, etc. This can cause problems for more advanced natural language techniques beyond finding words, as sentences may not be recognizable and thus traditional approaches to summarization based on deeper natural language processing (NLP) may not be suitable. Thus embodiments disclosed herein do not avail of such advanced NLP elements that identify parts-of-speech, sentences, syntactic structure, or particular conversational moves. As a result the multimedia summarization system100as disclosed herein is configured for processing various types of data (data heterogeneity).

Moreover, it may be appreciated that the multimedia summarization system100need not be restricted to any one language and is multilingual. To the extent that the ASR is available for a foreign language, audio in that language may be available, and the summarization techniques as detailed herein are largely language-neutral, except for language-specific lexical features in the Coherence scoring. Moreover, various embodiments disclosed herein are configured to combine machine-judged informativeness (how well the summary covers the content in the input) with machine judged coherence (how well the summary reads).

As shown in the example ofFIG.10, internal architecture of a computing device1000includes one or more processing units (also referred to herein as CPUs)1012, which interface with at least one computer bus1002. Also interfacing with computer bus1002are persistent storage medium/media1006, network interface1014, memory1004, e.g., random access memory (RAM), run-time transient memory, read only memory (ROM), etc., media disk drive interface1008, an interface1020for a drive that can read and/or write to media including removable media such as floppy, CD-ROM, DVD, etc., media, display interface1010as interface for a monitor or other display device, keyboard interface1016as interface for a keyboard, pointing device interface1018as an interface for a mouse or other pointing device, and miscellaneous other interfaces1022not shown individually, such as parallel and serial port interfaces, a universal serial bus (USB) interface, and the like.

Memory1004interfaces with computer bus1002so as to provide information stored in memory1004to CPU1012during execution of software programs such as an operating system, application programs, device drivers, and software modules that comprise program code or logic, and/or computer-executable process steps, incorporating functionality described herein, e.g., one or more of process flows described herein. CPU1012first loads computer-executable process steps or logic from storage, e.g., memory1004, storage medium/media1006, removable media drive, and/or other storage device. CPU1012can then execute the stored process steps in order to execute the loaded computer-executable process steps. Stored data, e.g., data stored by a storage device, can be accessed by CPU1012during the execution of computer-executable process steps.

Persistent storage medium/media1006is a computer readable storage medium(s) that can be used to store software and data, e.g., an operating system and one or more application programs. Persistent storage medium/media1006can also be used to store device drivers, such as one or more of a digital camera driver, monitor driver, printer driver, scanner driver, or other device drivers, web pages, content files, metadata, playlists and other files. Persistent storage medium/media1006can further include program modules/program logic in accordance with embodiments described herein and data files used to implement one or more embodiments of the present disclosure.

FIG.11is a schematic diagram illustrating an implementation of a computing device in accordance with embodiments of the present disclosure. A computing device1100capable of sending or receiving signals, such as via a wired or a wireless network, and capable of running application software or “apps”1110. A computing device may, for example, include a desktop computer or a portable device, such as a cellular telephone, a smart phone, a display pager, a radio frequency (RF) device, an infrared (IR) device, a Personal Digital Assistant (PDA), a handheld computer, a tablet computer, a laptop computer, a set top box, a wearable computer, an integrated device combining various features, such as features of the forgoing devices, or the like.

The computing device1100may vary in terms of capabilities or features. The computing device or a user device can include standard components such as a CPU1102, power supply1128, a memory1118, ROM1120, BIOS1122, network interface(s)1130, audio interface1132, display1134, keypad1136, illuminator1138, I/O interface1140interconnected via circuitry1126. Claimed subject matter is intended to cover a wide range of potential variations. For example, the keypad1136of a cell phone may include a numeric keypad or a display1134of limited functionality, such as a monochrome liquid crystal display (LCD) for displaying text. In contrast, however, as another example, a web-enabled computing device1100may include one or more physical or virtual keyboards1136, mass storage, one or more accelerometers, one or more gyroscopes, global positioning system (GPS)1124or other location identifying type capability, Haptic interface1142, or a display with a high degree of functionality, such as a touch-sensitive color 2D or 3D display, for example. The memory1118can include Random Access Memory1104including an area for data storage1108.

The computing device1100may include or may execute a variety of operating systems1106, including a personal computer operating system, such as a Windows, iOS or Linux, or a mobile operating system, such as iOS, Android, or Windows Mobile, or the like. A computing device1100may include or may execute a variety of possible applications1110, such as a client software application1114enabling communication with other devices, such as communicating one or more messages such as via email, short message service (SMS), or multimedia message service (MMS), including via a network, such as a social network, including, for example, Facebook, LinkedIn, Twitter, Flickr, or Google+, to provide only a few possible examples. The computing device1100may also include or execute an application to communicate content, such as, for example, textual content, multimedia content, or the like. The computing device1100may also include or execute an application to perform a variety of possible tasks, such as browsing1112, searching, playing various forms of content, including locally stored or streamed content, such as, video, or games (such as fantasy sports leagues). The foregoing is provided to illustrate that claimed subject matter is intended to include a wide range of possible features or capabilities.

For the purposes of this disclosure a computer readable medium stores computer data, which data can include computer program code that is executable by a computer, in machine readable form. By way of example, and not limitation, a computer readable medium may comprise computer readable storage media, for tangible or fixed storage of data, or communication media for transient interpretation of code-containing signals. Computer readable storage media, as used herein, refers to physical or tangible storage (as opposed to signals) and includes without limitation volatile and non-volatile, removable and non-removable media implemented in any method or technology for the tangible storage of information such as computer-readable instructions, data structures, program modules or other data. Computer readable storage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM, flash memory or other solid state memory technology, CD-ROM, DVD, or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other physical or material medium which can be used to tangibly store the desired information or data or instructions and which can be accessed by a computer or processor.

For the purposes of this disclosure a system or module is a software, hardware, or firmware (or combinations thereof), process or functionality, or component thereof, that performs or facilitates the processes, features, and/or functions described herein (with or without human interaction or augmentation). A module can include sub-modules. Software components of a module may be stored on a computer readable medium. Modules may be integral to one or more servers, or be loaded and executed by one or more servers. One or more modules may be grouped into an engine or an application.

Those skilled in the art will recognize that the methods and systems of the present disclosure may be implemented in many manners and as such are not to be limited by the foregoing exemplary embodiments and examples. In other words, functional elements being performed by single or multiple components, in various combinations of hardware and software or firmware, and individual functions, may be distributed among software applications at either the client or server or both. In this regard, any number of the features of the different embodiments described herein may be combined into single or multiple embodiments, and alternate embodiments having fewer than, or more than, all of the features described herein are possible. Functionality may also be, in whole or in part, distributed among multiple components, in manners now known or to become known. Thus, myriad software/hardware/firmware combinations are possible in achieving the functions, features, interfaces and preferences described herein. Moreover, the scope of the present disclosure covers conventionally known manners for carrying out the described features and functions and interfaces, as well as those variations and modifications that may be made to the hardware or software or firmware components described herein as would be understood by those skilled in the art now and hereafter.

While the system and method have been described in terms of one or more embodiments, it is to be understood that the disclosure need not be limited to the disclosed embodiments. It is intended to cover various modifications and similar arrangements included within the spirit and scope of the claims, the scope of which should be accorded the broadest interpretation so as to encompass all such modifications and similar structures. The present disclosure includes any and all embodiments of the following claims.