Automated Compositing of Content Compilations

A system includes a computing platform having processing hardware and a memory storing a software code. The processing hardware is configured to execute the software code to receive multiple content units each including a start descriptor for an initial content segment and an end descriptor for a last content segment, identify the start descriptor and the end descriptor for each of the content units, and select a first content unit for beginning a content compilation. The processing hardware is further configured to execute the software code to determine multiple similarity metrics each comparing the end descriptor of the first content unit with the start descriptor of a respective one of the other content units, rank, using the similarity metrics, the other content units with respect to one another, select, based on the rank, a second content unit, and composite the content compilation using the second content unit.

BACKGROUND

Joining or merging discrete units of content is a common task in content creation. For example, multiple video clips may be merged by a professional video editor in a studio, or by an amateur user via an online social media platform or utilizing a commercial software application. However, conventional solutions for merging discrete content units to form a content compilation achieve good results only when working with content that has a common theme or context running through the content units. For instance, where the content units are video clips, the clips being merged are typically from sporting events, celebrations, vacations, home videos, or educational tutorials, to name a few examples. In the conventional art, video clips selected from multiple different topics or themes tend to produce poor composite videos that may be jarring to watch and listen to due to abrupt transitions of visual style, thematic subject matter, and sound effects. Consequently, there is a need in the art for an automated solution for compositing disparate units of content from a broad range of topics, many of which may have different audio-visual characteristics or styles, to produce a compilation of that content that is appealing, especially where these compilations are unique for each user and human involvement cannot be afforded.

DETAILED DESCRIPTION

The following description contains specific information pertaining to implementations in the present disclosure. One skilled in the art will recognize that the present disclosure may be implemented in a manner different from that specifically discussed herein. The drawings in the present application and their accompanying detailed description are directed to merely exemplary implementations. Unless noted otherwise, like or corresponding elements among the figures may be indicated by like or corresponding reference numerals.

The present application is directed to automated systems and methods for compositing content compilations that address and overcome the deficiencies in the conventional art. According to the present automated compositing solution, information about features of content units, such as their audio features, the images they include, and their semantic features, for example, are extracted and used to automate the compositing of the content units in a manner that produces results comparable in quality to those produced by an expert human content editor. In contrast to conventional techniques for creating content compilations, the present solution can advantageously work with content that spans a wide range of themes and topics, as well as a host of visual styles and audio tracks created by different artists in different settings and for different purposes. In addition, according to the present compositing solution a user supplied template or layout, which is typically relied upon in conventional content compilation techniques, is neither sought nor utilized. As a result, the present solution advantageously enables the automated production of a coherent content compilation without abrupt switches in visual style, story arc, or audio effects.

As defined in the present application, the terms “automation,” “automated,” and “automating” refer to systems and processes that do not require human intervention. Although in some implementations a human editor may review the content compilations composited by the systems and using the methods described herein, that human involvement is optional. Thus, the methods described in the present application may be performed under the control of hardware processing components of the disclosed automated systems.

Moreover, as defined in the present application, the expression “machine learning model” may refer to a mathematical model for making future predictions based on patterns learned from samples of data or “training data.” Various learning algorithms can be used to map correlations between input data and output data. These correlations form the mathematical model that can be used to make future predictions on new input data. Such a predictive model may include one or more logistic regression models. Bayesian models, or neural networks (NNs).

A “deep neural network,” in the context of deep learning, may refer to an NN that utilizes multiple hidden layers between input and output layers, which may allow for learning based on features not explicitly defined in raw data. As used in the present application, a feature identified as an NN refers to a deep neural network. In various implementations, NNs may be trained as classifiers and may be utilized to perform image processing or natural-language processing.

FIG. 1shows a diagram of system100for performing automated compositing of content compilations, according to one exemplary implementation. As shown inFIG. 1, system100includes computing platform102having processing hardware104and system memory106implemented as a computer-readable non-transitory storage device. According to the present exemplary implementation, system memory106stores software code110, one or more machine learning models120(hereinafter “machine learning model(s)120”), and content and feature set database112storing features extracted from a curated set of content units by machine learning model(s)120.

As further shown inFIG. 1, system100is implemented within a use environment including communication network114providing network communication links116, recommendation engine122, usage database126, user system130including display132, and user118of user system130. Also shown inFIG. 1are multiple content units136, content compilation128composited by system100using software code110(hereinafter “composited content compilation128”), and usage data134corresponding to the engagement level of user118with composited content compilation128.

Although the present application refers to one or more of software code110, machine learning model(s)120, and content and feature set database112as being stored in system memory106for conceptual clarity, more generally, system memory106may take the form of any computer-readable non-transitory storage medium. The expression “computer-readable non-transitory storage medium.” as defined in the present application, refers to any medium, excluding a carrier wave or other transitory signal that provides instructions to processing hardware104of computing platform102. Thus, a computer-readable non-transitory storage medium may correspond to various types of media, such as volatile media and non-volatile media, for example. Volatile media may include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory may include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory storage media include, for example, optical discs, RAM, programmable read-only memory (PROM), erasable PROM (EPROM), and FLASH memory.

It is further noted that althoughFIG. 1depicts software code110, machine learning model(s)120, and content and feature set database112as being co-located in system memory106, that representation is also merely provided as an aid to conceptual clarity. More generally, system100may include one or more computing platforms102, such as computer servers for example, which may be co-located, or may form an interactively linked but distributed system, such as a cloud-based system, for instance. As a result, processing hardware104and system memory106may correspond to distributed processor and memory resources within system100. Consequently, in some implementations, one or more of software code110, machine learning model(s)120, and content and feature set database112may be stored remotely from one another on the distributed memory resources of system100.

Processing hardware104may include multiple hardware processing units, such as one or more central processing units, one or more graphics processing units, and one or more tensor processing units. By way of definition, as used in the present application, the terms “central processing unit” (CPU), “graphics processing unit” (GPU), and “tensor processing unit” (TPU) have their customary meaning in the art. That is to say, a CPU includes an Arithmetic Logic Unit (ALU) for carrying out the arithmetic and logical operations of computing platform102, as well as a Control Unit (CU) for retrieving programs, such as software code110, from system memory106, while a GPU may be implemented to reduce the processing overhead of the CPU by performing computationally intensive graphics or other processing tasks. A TPU is an application-specific integrated circuit (ASIC) configured specifically for artificial intelligence (AI) processes such as machine learning.

In some implementations, computing platform102may correspond to one or more web servers, accessible over a packet-switched network such as the Internet, for example. Alternatively, computing platform102may correspond to one or more computer servers supporting a private wide area network (WAN), local area network (LAN), or included in another type of limited distribution or private network. Moreover, in some implementations, communication network114may be a high-speed network suitable for high performance computing (HPC), for example a 10 GigE network or an Infiniband network.

Although user system130is shown as a desktop computer inFIG. 1, that representation is provided merely as an example as well. More generally, user system130may be any suitable mobile or stationary computing device or system that includes display132and implements data processing capabilities sufficient to implement the functionality ascribed to user system130herein. For example, in other implementations, user system130may take the form of a laptop computer, tablet computer, or smartphone, for example.

With respect to display132of user system130, display132may be implemented as a liquid crystal display (LCD), light-emitting diode (LED) display, organic light-emitting diode (OLED) display, quantum dot (QD) display, or any other suitable display screen that perform a physical transformation of signals to light. Furthermore, display132may be physically integrated with user system130or may be communicatively coupled to but physically separate from user system130. For example, where user system130is implemented as a smartphone, laptop computer, or tablet computer, display132will typically be integrated with user system130. By contrast, where user system130is implemented as a desktop computer, display132may take the form of a monitor separate from user system130in the form of a computer tower.

The functionality of software code110will be further described by reference toFIG. 2, which shows flowchart200presenting an exemplary method for use by system100to perform automated compositing of content compilations, according to one implementation. With respect to the actions outlined inFIG. 2, it is noted that certain details and features have been left out of flowchart200in order not to obscure the discussion of the inventive features in the present application.

Referring toFIG. 2in combination withFIG. 1, flowchart200begins with receiving multiple content units136, each one of the content units including a start descriptor for an initial content segment and an end descriptor for a last content segment (action210). Multiple content units136may span a wide range of themes and topics including wildlife and nature, fantasy, action-adventure, animation, live action, to name a few examples, as well as content having a host of different visual styles and audio tracks created by different artists in different settings and for different purposes. Moreover, multiple content units136may include clips of audio-video (AV) content, audio clips without video, or video clips without audio, for example. In addition, each content unit may include semantic features, such as a written synopsis, as well as metadata tags identifying keywords describing the included content. For instance, in the case of AV content including clips of episodic television (TV) or movie content, that metadata may identify genre type (action, adventure, comedy, fantasy, etc.), lead character type (pirate, princess, superhero, teacher, criminal, etc.), or subject type (educational, coming of age, sci-fi, sports, racing etc.) of the content clips, to name a few examples.

In some implementations, multiple content units136may be received by system100from user system130via communication network114and network communication links116, based on selection inputs to user system130by user118. For example, multiple content units136may be content units affirmatively selected by user118from a menu of predetermined curated content units that have been annotated to include the synopsis and keyword metadata tags described above. Alternatively, in some implementations multiple content units136may be received by system100from recommendation engine122, based on content consumption preferences of user118that are known or inferred by recommendation engine122. In those various implementations, multiple content units136may be received by software code110, executed by processing hardware104of computing platform102.

With respect to the start descriptor and end descriptor of each of multiple content units136, it is noted that those start and end descriptors are generated from a feature set previously extracted from each content unit and stored in content and feature set database112. Referring toFIGS. 3A and 3B,FIG. 3Ashows diagram300A depicting extraction of an exemplary feature set from content unit336in the form of an exemplary AV content clip, according to one implementation, whileFIG. 3Bshows diagram300B depicting generation of exemplary start descriptor350aand end descriptor350bof content unit336, according to one implementation. It is noted that content unit336corresponds in general to any or all of multiple content units136, and those corresponding features may share any of the characteristics attributed to content unit336by the present disclosure.

As shown inFIG. 3A, a variety of different analytical processes can be used to extract different features included in content unit336. For example, spectrogram analysis338acan be used to extract audio features342from content unit336, while one or more machine learning model based image extraction techniques338b, using one or more of machine learning model(s)120inFIG. 1for example, may be used to extract image features344. In addition, various natural language processing based (NLP-based) techniques338cmay be used to extract semantic features346and keyword features348from content unit336.

The analytical processes performed on content unit336result in generation of a multi-valued floating point vector representation (i.e., feature vectors) for each of audio features342, image features344, semantic features346, and keyword features348. By way of example, and as noted above, signal processing techniques like spectrogram analysis338aof the audio track included in content unit336may be used to identify audio features342. A trained NN, such as a convolutional NN (CNN) may be used to identify image features344. For example, VGG-19 is a trained CNN that is very effective at object recognition. The initial layers of this CNN may be used to generate efficient and compact image feature representations for individual video frames.

As described above, content unit336may include a written synopsis and metadata tags identifying keywords describing content unit336. Those text elements can be analyzed with NLP-based techniques338csuch as Global Vectors for Word Representation (GloVe) and Bidirectional Encoder Representations from Transformers (BERT) word embeddings, WordNet, and term frequency-inverse document frequency (TF-IDF) feature generation. These NLP-based techniques enable representation of the clip description and its metadata in a feature vector space where they can be compared, clustered, and classified very efficiently. In addition, the word embedding techniques used allow for handling various categories of ontologies and can handle nuances when synonyms, hyponyms, hypernyms and even contranyms are present.

Rather than utilizing a single set of feature vectors describing the entirety of content unit336, the present approach contemplates using two sets of feature vectors for each content unit336, a start feature vector for an initial content segment of content unit336and an end feature vector for a last content segment of the content unit. In some implementations, start and end feature vectors are computed only for the audio and image features of content unit336. In those implementations, one feature vector each is computed for the semantic and keyword features using all of the available textual features of content unit336.

As a specific example in use cases involving AV content units, video frames from a predetermined time duration or frame count from the beginning and end of content unit336, such as fifteen percent (15%) of the total time duration or frame count of content unit336for example, may be used to compute the start and end image feature vectors. For the audio feature vectors, another predetermined subset of the front half and the end half of the audio track of content unit336may be used, such as 50% of each of the front half and back half of the audio track, for example. It is noted that the exemplary percentages described above are hyperparameters that can be set and selectably modified by an administrator of system100.

Once the feature vector sets for content unit336have been identified, a start descriptor for content unit336may be generated by concatenating or joining two or more of the start audio feature vector, the start image feature vector, the semantic feature vector for the entire content unit, and the keyword feature vector for the entire content unit together, end-to-end. Analogously, an end descriptor for content unit336may be generated by using one or both of the end audio and image feature vectors in similar combination with one or both of the same semantic and keyword feature vectors used to generate the start descriptor. This process is depicted inFIG. 3B, which shows start descriptor350aand end descriptor350bof content unit336.

As shown inFIG. 3B, start descriptor350aincludes start audio feature vector352ain combination with start image feature vector354a, semantic feature vector356, and keyword feature vector358, while end descriptor350bincludes end audio feature vector352bin combination with end image feature vector354b, semantic feature vector356, and keyword feature vector358. Thus, for each content unit336, start descriptor350aand end descriptor350bmay take the form of respective vectors in a multi-dimensional feature space. Moreover, as noted above, in some implementations, the content units corresponding to content unit336may be AV clips from a variety of different genres. In some of those implementations, as shown inFIG. 3B, each of start descriptor350aand end descriptor350bmay include two or more of an audio feature vector component, an image feature vector component, a keyword feature vector component, and a semantic feature vector component based on a written synopsis of the AV clip.

Flowchart200further includes identifying start descriptor350aand end descriptor350bfor each one of content units136/336(action220). Action220may be performed by software code110, executed by processing hardware104of computing platform102, by accessing content and feature set database112.

Flowchart200further includes selecting a first content unit of multiple content units136/336for beginning a content compilation (action230). The selection of the first content unit in action230may be performed by software code110, executed by processing hardware104of computing platform102. In various implementations, the selection of the first content unit for beginning the content compilation may be performed randomly, or may be based on a first content identifier included in multiple content units136/336. That is to say, in some implementations the first content unit selected in action230may be selected from within a set of contents suggested by recommendation engine122or by user118. However, in other implementations, selection of the first content unit for beginning the content compilation may be performed by software code110, executed by processing hardware104of computing platform102, in the manner described below by reference toFIGS. 4A and 4B.

FIG. 4Ashows a portion400A of exemplary multi-dimensional feature space460including feature space landmarks464represented inFIG. 4Aby exemplary feature space landmarks464a,464b, and464c, according to one implementation. Also shown inFIG. 4Aare clusters of content units mapped onto multi-dimensional feature space460and represented inFIG. 4Aby exemplary content clusters462aand462b. It is noted that the content units mapped into multi-dimensional feature space460, inFIG. 4A, correspond in general to content unit336. Consequently, the content units included in content the clusters represented by exemplary content clusters462aand462bmay share any of the characteristics attributed to content unit336by the present disclosure, and vice versa. It is further noted that the dimensions of multi-dimensional feature space460may correspond respectively to some, or all, of the feature vectors characterizing content unit336. The multi-dimensional feature space may have 3 modes or modalities (audio, visual, and textual), for example with many dimensions in each modality.

As part of the process of training machine learning model(s)120, inFIG. 1, a corpus of expertly composed content compilations may be aggregated. The content compilations included in this corpus may be produced by expert human content editors, and are known to be well formed and suitable for use as “best-of-class” exemplars for the content compilations to be composited by system100. The corpus of expertly composed content compilations is aggregated so as to be sizeable and diverse enough to cover substantially all of the visual styles, audio themes, topics, and semantic features that content unit336would typically depict. There may also be diversity in how the content units are composited together to form a satisfying narrative arc embodying qualities of good storytelling. For example a good content compilation may have an intriguing start, suspenseful middle, and an exciting end.

Referring toFIG. 4A, the content units included in each expertly composited content compilation are plotted in multi-dimensional feature space460. A clustering algorithm may then be run that finds content clusters. e.g., content clusters462aand462b, as well as cluster centers. i.e., superclusters of the content clusters (hereinafter “feature space landmarks”) represented inFIG. 4Aby exemplary feature space landmarks464. It is noted that the number of content clusters and feature space landmarks to be identified by the clustering algorithm is a hyperparameter that can be set and selectably modified by an administrator of system100.

Each expertly composited content compilation can be visualized as a trajectory among the feature space landmarks464of multi-dimensional feature space460. That is to say, in some implementations, each of content units136/336may further include a feature space landmark464identifier corresponding to one or more of multiple content clusters in multi-dimensional feature space460, and each expertly composited content compilation can be visualized as a trajectory along the sequence of feature space landmarks464identified by the content units composited into that content compilation. An example of such a trajectory, with a merely exemplary length of five content units in the interests of conceptual clarity, is shown by diagram400B inFIG. 4B.FIG. 4Bshows content compilation trajectory466starting at feature space landmark464a, continuing sequentially to respective feature space landmarks464b.464c, and464d, and ending at feature space landmark464e.

In some implementations, selection of the first content unit in action230may be based on the first content units in the expertly composited content compilations serving as exemplars. That is to say, action230may include determining a first desirable feature space landmark for the first content unit, and selecting the first content unit based on the first desirable feature space landmark.

Merely by way of example, let it be assumed that each expertly composited content compilation includes a first content unit having a feature space landmark identifier for one of feature space landmarks464a,464b, or464c, but no other feature space landmarks. Let it be further assumed that feature space landmark464ais identified most often chosen by the first content element in an expertly composited content compilation, followed by feature space landmark464b, which in turn is identified more frequently than feature space landmark464c.

To select the first content unit in action230, the feature space landmark identifier of each of content units136/336may be obtained, and the weighted distance of each feature space landmark identified by those content units from each of the expert selected starting feature space landmarks464a,464b, and464cmay be determined. It is noted that the weights applied to the distances described above may be the inverse ratios of the number of times each of feature space landmarks464a,464b, and464cwas identified by the first content unit in the expertly composited content compilations. Finally, the weighted distances of each of the feature space landmarks identified by the content units from each of feature space landmarks464a.464b, and464ccan be averaged. In this particular use case, the characteristic determining the desirability of the feature space landmark of the content unit to be selected as the first content unit for beginning the content compilation in action230may be its averaged weighted distance from feature space landmarks464a,464b, and464c. The content unit that identifies the feature space landmark having that shortest averaged weighted distance from feature space landmarks464a,464b, and464cmay then be selected as the first content unit in action230.

Flowchart200further includes determining similarity metrics each of which compares end descriptor350bof the first content unit selected in action230with start descriptor350aof each of the other content units of multiple content units136/336(action240). Action240may be performed by software code110, executed by processing hardware104of computing platform102. It is noted that although the exemplary implementations described below refer to the similarity metrics determined in action240as corresponding to Euclidean distances, that characterization is merely provided by way of example. In other implementations, the similarity metrics determined in action240may include Manhattan distances (also known as L1-distances), or may include other similarity measures, such as cosine similarity for example. In various implementations, these similarity metrics determined in action240measure the distance between vectors in a multi-dimensional vector space as a means of quantifying their similarity.

Referring toFIG. 5A.FIG. 5Ashows diagram501A depicting compositing of content compilation556, according to one exemplary implementation. Content compilation556is shown to be partially composited and to include first content unit536a, second content unit536b, and third content unit536c. Also shown inFIG. 5Aare other, not yet composited content units536d,536e.536f, and536g(hereinafter “content units536d-536g”), as well as distances538d,538e,538f, and538g(hereinafter “distances538d-538g”) of a start descriptor of each of respective content units536d-536gfrom the end descriptor of third content unit536c.

It is noted that, according to the exemplary implementation shown inFIG. 5A, distances538d-538gcorrespond to the similarity metrics determined in action240. Thus, referring toFIGS. 2, 3B, and 5Ain combination, action240may include determining distances538d-538gof end descriptor350bof third content unit356cfrom start descriptor350aof each of respective content units536d-536g. Distances538d-538gmay be the Euclidean distances of the start descriptor350aof each of respective content units536d-536gfrom the end descriptor350bof third content unit536cin multi-dimensional feature space460described above. It is further noted that each of first content unit536a, second content unit536b, third content unit536c, and content units536d-536gcorrespond in general to content unit336, inFIG. 3A. Thus, each of first content unit536a, second content unit536b, third content unit536c, and content units536d-536gmay share any of the characteristics attributed to content unit336by the present disclosure, and vice versa.

Flowchart200further includes ranking, using the similarity metrics determined in action240, content units536d-536gwith respect to one another (action250). For example, where the similarity metrics determined in action240are distances538d-538g, the ranking performed in action250may rank content units536d-536gbased on respective distances538d-538g, with content unit536dassociated with least distance538dbeing ranked highest. i.e., first, and content unit536gassociated with greatest distance538gbeing ranked lowest, i.e., last. The ranking of content units536d-536gin action250may be performed by software code110, executed by processing hardware104of computing platform102.

Flowchart200further includes selecting, based on the ranking performed in action250, a second content unit of multiple content units136/336(action260). Action260may be performed by software code110, executed by processing hardware104of computing platform102, and in some implementations may be performed using machine learning model(s)120. The specific process depicted by diagram501A is the selection of a fourth content unit for continuation of the compositing of content compilation556from among content units536d-536g. According to the exemplary implementation shown inFIG. 5A, the content unit among content units536d-536ghaving the shortest distance among distances538d-538g, i.e., content unit536d, is selected as the fourth content unit for continuing compositing of content compilation556.

However, it is noted that a process analogous to that described by reference to diagram501A may be used to select second content unit536bbased on the distance between the end descriptor of first content unit536aand the start descriptor of second content unit536b, as well as to select third content unit536cbased on the distance between the end descriptor of second content unit536band the start descriptor of third content unit536c.

As seen in the approach described above by reference toFIG. 5A, at each step in the process software code110, executed by processing hardware104, must make a heuristic choice to select the next content unit of content compilation556. In some implementations, software code110may utilize machine learning model(s) to produce content compilation556and to learn over time to improve its own selection performance. In some such implementations, machine learning model(s)120receives as input the end descriptor of the last content unit already in partially composited content compilation556, i.e., third content unit536c, and all of the start descriptors of a predetermined number “k” of ranked content units536d-536gjoined end-to-end as one complete input feature vector. It is noted that the integer value k is a hyperparameter for the system that can be set and selectably modified by an administrator of system100. It is further noted that top k candidate content units are selected on the same distance based approach described above. Thus, the top k content units are the k closest clips to the last clip already composited into content compilation556in feature vector space460.

Machine learning model(s)120produces k outputs, representing the 1-of-k choices that must be made in order to select the next content unit for continuing content compilation556. The learning objective for machine learning model(s)120is to pick a best next content unit from among the best k candidates available to it.

As noted above by reference toFIGS. 4A and 4B, in some implementations, each of first content unit536a, second content unit536b, third content unit536c, and content units536d-536gfurther includes a feature space landmark identifier corresponding to one or more of the content clusters represented by content clusters462aand462bin multi-dimensional feature space460. In those implementations, processing hardware104may further execute the software code110to identify the feature space landmark for each of first content unit536a, second content unit536b, third content unit536c, and content units536d-536g, and to select sequential content units for compositing of content compilation556further based on those feature spaced landmarks. That is to say, processing hardware104may further execute the software code110to determine, based on the first feature space landmark for first content unit536a, a desirable second feature space landmark for second content unit536b, and to select second content unit536bfurther based on that desirable second feature space landmark in action260. Referring toFIG. 5B,FIG. 5Bdepicts compositing of content compilation556using such an approach.

FIG. 5Bshows diagram501B depicting compositing of content compilation556, according to another exemplary implementation. Content compilation556is shown to be partly composited and to include first content unit536a, second content unit536b, and third content unit536c. Also shown inFIG. 5Bare other, not yet composited content units536d.536e,536f.536g, and536h(hereinafter “content units536d-536h”), as well as distances538d.538e,538f,538g, and538h(hereinafter “distances538d-538h”) of a start descriptor of each of respective content units536d-536hfrom the end descriptor of third content unit536c. In addition, diagram501B shows trajectory566of content compilation556. Trajectory566begins at feature space landmark564aidentified by first content unit536a, continues through feature space landmarks564band564cidentified by respective second and third content units536band536c, and continues through feature space landmarks564d,564e.564f, and564g, respectively, before ending at feature space landmark564h.

Each of feature space landmarks564a.564b,564c.564d,564e,564f.564g, and564h(hereinafter “feature space landmarks564a-564hcorresponds in general to any of feature space landmarks464a,464b.464c,464d, and464e(hereinafter “feature space landmarks464a-464e”) shown inFIG. 4A or 4B. Thus, each of feature space landmarks564a-564hmay share any of the characteristics attributed to any of feature space landmarks464a-464eby the present disclosure, and vice versa.

Referring toFIGS. 1 and 5Bin combination, in some implementations, machine learning model(s)120have been trained as sequence prediction model(s) using the corpus of expertly composited content compilations described above by reference toFIGS. 4A and 4B. When implemented as sequence prediction model(s), machine learning model(s) may take the form of one or more of a recurrent neural network (RNN), having a long short-term memory (LSTM) architecture, for example, or as any finite-state transducer. Machine learning model(s) are configured to predict the next feature space landmark in trajectory566of content compilation556, given the previous partial order of feature space landmarks.

The prediction takes the form of probabilities, one each for each of the possible feature space landmarks. The feature space landmark with the highest probability is identified as the next feature space landmark in the sequence of trajectory566. Consider the last content unit composited into content compilation556thus far, i.e., third content unit536c. In order to select the next clip, i.e., the fourth content unit for content compilation556, feature space landmark564cidentified by third content unit536c, as well as, in some implementations, one or both of feature space landmarks564aand564bidentified by respective first content unit536aand second content unit536b, can be analyzed by machine learning model120to predict a list of desirable feature space landmarks for use as next feature space landmark564dof trajectory566. From that list, the feature space landmark having the highest probability may be selected as feature space landmark564d. Then, from the ranking of the content units described above by reference to action250andFIG. 5A, based on distances538a-538h, the highest ranked content unit including the feature space landmark identifier of the feature space landmark selected as feature space landmark564dmay then be selected as the next content unit. i.e., the fourth content unit.

As shown inFIG. 5B, the feature space landmark identifier of feature space landmark564dselected as the next feature space landmark of trajectory566is Lmk-B. Of ranked content units536d-536h, only content unit536gincludes feature space landmark identifier Lmk-B. Consequently, despite being ranked lower than content units536d.536e, and536f, content unit536gmay be selected as the fourth content unit of content compilation556.

It is possible that none of content units536d-536hincludes the feature space landmark identifier of the feature space landmark having the highest probability predicted by machine learning model(s)120. Under those circumstances, the feature space landmark having the second highest probability may be selected as feature space landmark564dof trajectory566, and the fourth content unit may then be selected based on being the highest ranked of content units536d-536eincluding the feature space landmark identifier for the second highest probability feature space landmark may be selected as the fourth content unit of content compilation556. This selection process may continue until the fourth content unit of content compilation is successfully identified.

It is noted that although the specific example shown and described by reference toFIG. 5Brefers to selection of the fourth content unit of content compilation556, a process analogous to that described by reference to diagram501B may be used to select second content unit536b, third content unit536c, and so forth. Thus, in some implementations, action260may include determining, based on the first feature space landmark identified by first content unit536a, desirable second feature space landmark564bfor second content unit536b, and selecting second content unit536bfurther based on desirable second feature space landmark564b. Moreover, in some implementations, second content unit536bmay be selected by software code110, executed by processing hardware104of computing platform102, and using machine learning model(s)120.

In some implementations, flowchart200can continue and conclude with compositing of content compilation556using second content unit536b(action270). As noted above, the automated method outlined by flowchart200follows a heuristic approach to composite content compilation556beginning with first content unit536aselected in action230. The process of compositing of content compilation556can further include selecting third content unit536c, selecting content unit536das the fourth content unit of content compilation556, selecting one of content units536e,536f, or536gas the fifth content unit of content compilation556, and so forth. Action270may be performed by software code110, executed by processing hardware104of computing platform102, and results in production of composited content compilation128.

Although in some implementations flowchart200can conclude with action270, in implementations in which second content unit536bis selected using machine learning model(s)120, flowchart200may further include obtaining usage data134corresponding to an engagement level of user118with composited content compilation128(action280), and training machine learning model(s)120using usage data134, resulting in an improved selection performance by machine learning model(s)120(action290). As noted above, usage data134corresponds to the engagement level of user118with composited content compilation128. Such an engagement level may be determined based on how much of composited content compilation128user118consumes: either user118consumes composited content compilation128in its entirety, which can be taken as a positive sign, or if user118abandons it part way through, that could be construed as some measure of dislike or non-engagement.

FIG. 6Ashows diagram601A depicting production of an implicit signal for improving the selection performance of the system shown inFIG. 1, based on usage data, whileFIG. 6Bshows diagram601B depicting production of an implicit signal for improving content sequencing performance for compositing of a content compilation, based on usage data. As shown inFIG. 6A, content compilation656is shown to be partly composited and to include first content unit636a, second content unit636b, and third content unit636c. Also shown inFIG. 6Aare other, not yet composited content units636d,636e,636f,636g, and636h(hereinafter “content units636d-636h”), as well as distances638d.638e.638f,638g, and638h(hereinafter “distances638d-638h”) of a start descriptor of each of respective content units636d-636hfrom the end descriptor of third content unit636c. In addition, diagram601A shows exemplary reinforcement based machine learning model620.

It is noted that each of first content unit636a, second content unit636b, third content unit636c, and content units636d-636hcorrespond in general to content unit336, inFIG. 3A. Thus, each of first content unit636a, second content unit636b, third content unit636c, and content units636d-636hmay share any of the characteristics attributed to content unit336by the present disclosure, and vice versa. It is further noted that distances638d-668g, inFIG. 6A, correspond respectively in general to distances538d-538g, inFIGS. 5A and 5B, while distance638hcorresponds in general to distance538hinFIG. 5B. Moreover, reinforcement based machine learning model620corresponds in general to machine learning model(s)120, inFIG. 1, and those corresponding features may share any of the characteristics attributed to either corresponding feature by the present disclosure.

FIG. 6Bshows content compilation628composited by system600for automated compositing of content compilations, and usage data obtained by system600. Also shown inFIG. 6Bis corpus615of content clips available for compositing. System600, content compilation628, and usage data634correspond respectively in general to system100, content compilation128, and usage data134, inFIG. 1. Thus, system600, content compilation628, and usage data634may share any of the characteristics attributed to system100, content compilation128, and usage data134by the present disclosure, and vice versa.

Using a predetermined threshold for the engagement level, usage data134/634may be used to produce an implicit signal that system100/600can use to improve the sequencing performance of reinforcement based machine learning model620. As discussed above, at each step in the process of compositing content compilation128/628, system100/600must make a heuristic choice to select the next clip in the sequence. According to the exemplary implementation shown inFIG. 6Areinforcement based machine learning model620can, over the course of time, learn the best clip to insert at the present position that works over and above that of the basic heuristic process.

For example, reinforcement based machine learning model620model may receive as input the end descriptor of the last clip already in partially constructed sequence so far, e.g., third content clip636c, and all of the start descriptors of the top k candidate clips (e.g., content clips636d-636h), joined end-to-end as one complete input feature vector. Here, the number k is a hyperparameter for system100/600that can be set and selectably modified by an administrator of system100/600. It is noted that the top k candidate clips may be selected on the same Euclidean or other distance based approach outlined above by examining all content clips that are not already include in partially composited content compilation656. This means that the top k content clips are the k closest content clips to the last clip already in the sequence (e.g., third content clip636c), in feature vector space. The input feature vector applied to reinforcement based machine learning model620model represents the ‘current state of the world’ in reinforcement based machine learning parlance. Reinforcement based machine learning model620has k outputs, representing the 1-of-k choices that must be made in selecting the next content clip in the sequence. The learning objective for reinforcement based machine learning model620is to select a next best content clip from the k candidates available to it.

After the process depicted inFIG. 6Ahas constructed the full sequence using a combination of heuristics for short-listing the top-k candidate content clips (e.g. as referenced inFIG. 5A) and a trainable landmark traversal model to select a best next content clip from among those candidates at each step in the sequence (e.g. as referenced inFIG. 5B), the final composited content compilation128/628is presented to user118. Based on usage data134/634describing how user118engages with the composition, for example the percentage of content compilation128/628viewed by user118before user118abandons content compilation128/628, an implicit signal can be generated and can be used for further training or retraining of reinforcement based machine learning model620. For example, as depicted inFIG. 6B, a reward or penalty may be assessed for composited content compilation128/628, characterized by real numbers for instance, e.g., +1.0 for reward. −1.0 as a penalty. This reward or penalty can be applied or back-propagated through a reinforcement learning model of machine learning model620so that reinforcement based machine learning model620can, over the course of multiple iterations, learn to composite content compilations that are sequenced such that user118is likely to consume them in their entirety and possibly return to again and again.

As shown inFIG. 1, usage data134/634may be obtained in action280from usage database126or user system130, via communication network114and network communication links116. Actions280and290may be performed by software code110, executed by processing hardware104of computing platform102. One advantage of the approach described by reference toFIGS. 6A and 6Bis that even an initially untrained machine learning model can make effective choices from the candidate clips, because the candidate clips are already short-listed based on desirable characteristics. e.g., closeness to the last clip already in the sequence. Over the course of time as the machine learning model receives feedback based on usage data134/634, the machine learning model can continue to improve.

Referring toFIG. 7,FIG. 7shows flowchart700presenting an exemplary method for use by system100to perform automated compositing of content compilations, according to another implementation. With respect to the actions outlined inFIG. 7, it is noted that certain details and features have been left out of flowchart700in order not to obscure the discussion of the inventive features in the present application.

Referring toFIGS. 1, 3A, and 3Bin conjunction withFIG. 7, flowchart700begins with receiving multiple content units136/336, each one of the content units136/336including start descriptor350afor an initial content segment and end descriptor350bfor a last content segment (action710). Action710may be performed by software code110, executed by processing hardware104of computing platform102, in the manner described above by reference to action210of flowchart200.

Flowchart700further includes identifying start descriptor350aand end descriptor350bfor each one of content units136/336(action720). Action720may be performed by software code110, executed by processing hardware104of computing platform102, by accessing content and feature set database112.

Flowchart700further includes identifying multiple candidate content compilations, each of the candidate compilations including multiple content units136/336in a different temporal sequence (action730). For example, where multiple content units136/336include three content units: e.g., content unit A, content unit B. and content unit C, action730may result in identification of three factorial (3!), i.e., six, candidate compilations: ABC, ACB, BAC. CAB. BCA. CBA, while any integer number “n” of multiple content units136/336may result in identification of up to n! content compilations. Thus, the candidate compilations identified in action730may include all n! permutations of multiple content units136/336, or any subset thereof. Moreover, the different permutations of candidate compositions can be used to compile a set of similarity metrics with which each candidate composition can be ranked and the best one be identified, as described below by reference to actions740and750. It is noted that such similarity metrics may be compiled using one or both of the baseline heuristic approach described above by reference toFIG. 5A, or the landmark traversal approach described above by reference toFIG. 5B. Action730may be performed by software code110, executed by processing hardware104of computing platform102.

Flowchart700further includes, for each one of the candidate content compilations identified in action730, determining, by comparing the end descriptor of each of the content units to the start descriptor of the next content unit in the temporal sequence, a score or a continuity score for that candidate content compilation, resulting in multiple continuity scores (action740). In one implementation in which each candidate content compilation includes “n” content units, for example, the distance in a multi-dimensional feature space separating the end descriptor of each of the first 1 to (n−1) content units to the start descriptor of the next content unit in the temporal sequence of each candidate content compilation may be computed, in a manner analogous to that described above by reference toFIG. 5A. In that implementation, the continuity score may be determined to be the sum of those distances for each candidate content compilation.

As noted above by reference toFIGS. 4A, 4B, and 5B, in some implementations, each of multiple content units136/336further includes a feature space landmark identifier corresponding to one or more content clusters in a multi-dimensional feature space. In those implementations, the continuity score for each of the candidate content compilations may further be determined by comparing the feature space landmark identifier of each content unit of the candidate content compilation to the feature space landmark of the next content unit in the temporal sequence of that candidate content compilation. For example, based on the comparison of the feature space landmark identifiers, a trajectory of each candidate content compilation in the multi-dimensional feature space may be determined, and that trajectory could be compared to the trajectories of the corpus of expertly composited content compilations described above. Action640may be performed by software code110, executed by processing hardware104of computing platform102, and in some implementations, using machine learning model(s)120.

In some implementations, flowchart700may continue and conclude with selecting, based on the continuity scores determined in action740, a best content compilation from among the candidate content compilations (action750). It is noted that, depending on the specific way in which the continuity score is determined, the best candidate content compilation may be the candidate content compilation having the highest, or the lowest, continuity score. For example, in implementations in which the sum of the distances separating the end descriptor of each of the first 1 to (n−1) content units to the start descriptor of the next content unit in the temporal sequence of each candidate content compilation determines the continuity score, the candidate content compilation having the lowest score. i.e., smallest sum of distances, may be selected as the best content compilation. Action750may be performed by software code110, executed by processing hardware104of computing platform102.

Although in some implementations flowchart700can conclude with action750, in implementations in which the best content compilation is selected in action750using machine learning model(s)120, flowchart700may further include obtaining usage data134corresponding to an engagement level of user118with the selected best content compilation (action760), and training machine learning model(s)120using usage data134, resulting in an improved selection performance by machine learning model(s)120(action770). Actions760and770may be performed by software code110, executed by processing hardware104of computing platform102, in a manner analogous to actions280and290described above.

With respect to the methods outlined by flowcharts200and700, it is noted that actions210,220,230,240,250,260, and270, or actions210,220,230,240,250,260,270,280, and290, or actions710,720,730,740, and750, or actions710,720,730,740,750,760, and770, may be performed in an automated process from which human involvement can be omitted.

Thus, the present application discloses automated systems and methods for compositing content compilations that address and overcome the deficiencies in the conventional art. As described above, according to the present automated compositing solution, information about features of content units, such as their audio features, the images they include, and their semantic features, for example, are extracted and used to automate the compositing of the content units in a manner that produces results comparable in quality to those produced by an expert human content editor. In contrast to conventional techniques for creating content compilations, the present solution can advantageously work with content that spans a wide range of themes and topics, as well as a host of visual styles and audio tracks created by different artists in different settings and for different purposes. In addition, according to the present compositing solution a user supplied template or layout, which is typically relied upon in conventional content compilation techniques, is neither sought nor utilized. As a result, the present solution advantageously enables the automated production of a coherent content compilation without abrupt switches in visual style, story arc, or audio effects.