Automatic video generation using auto-adaptive video story models

A video processing system generates and automatically updates videos. A client inputs a smart script that defines general preferences and information to include in a video. The video processing system generates a model of the story from the smart script. A story model comprises story beats that define order and characteristics of information that is presented in a story. The video processing system accesses client content (e.g., video clips, images, and testimonials from a client website) and assigns content items to story beats. The story model and associated content is rendered into a video. The video processing system can then adapt the story model in response to viewer feedback, external events, or other client-supplied parameters. Adapting the story model may include reordering, adding, or removing story beats, or altering parameter values associated with story beat characteristics. Thus, the video processing system can iteratively improve and update a video automatically.

BACKGROUND

Field of Art

This disclosure relates generally to automatic video generation, and in particular to generating iterations of video narratives by improving story models using machine learning.

Description of Art

The creation of videos can be an arduous un-scalable task, especially if a video needs to be updated regularly to remain relevant. In particular, after a video has been produced, there are few available methods for automatically updating the video. As a result, most individuals or small businesses who want a video for promotional, informative, or other purposes purchase a single expensive video and have no way of adjusting the content of the video to reflect changes (e.g., if a small restaurant updates its menu and wants the new menu to show in the video).

In some cases, developers have addressed this issue by specifically creating templated videos. Such a solution involves a single video template with specific areas to which different content can be inserted to customize and update the video. However, all instances of videos created and updated with this technique look similar, rendering them uninteresting to viewers since the overall structure of the videos does not change. For example, thousands of templated videos promoting real estate agents look exactly the same except for the photo of the realtor and an address of the agency.

SUMMARY

Embodiments relate to a video processing system that automatically generates and updates videos that are based on adaptable video story models. To avoid the generation of videos for different clients that all look the same, a client, upon first requesting a video, provides the video processing system with a smart script document that generally describes a variety of possible video segments (also referred to as “story beats”) that can be used in a video. The client also provides the video processing system with content in the form of media asset items that may be used in the video, or provides the video processing system with access information (e.g., resource locators on the Internet) where media assets related to the client may be found.

The video processing system collects media asset items that are needed to render video sections as described in the smart script. The video processing system also constructs a story model based on the information in the smart script. In one embodiment, a story model comprises a sequence of story beats with associated parameter values that define characteristics of the story as it is to be rendered in a video. The video processing system may use trained algorithms to adjust the values of parameters for individual story beats and to adjust the number and order of the story beats within the story model.

Collected media assets are linked to story beats in the story model. The video processing system renders a video based on the story model using the linked media assets and may publish the video or provide the video for publication by the requesting client. The video processing system can automatically update the content of the video over time by collecting and linking newly available media asset items to the story model and re-rendering the video. The video processing system can also update the story model in response to viewer feedback metrics. Thus the video narrative may be improved over time to remain relevant and to be more engaging to viewers.

DETAILED DESCRIPTION

Overview

Videos are a versatile and engaging medium for conveying information to an audience. A video can distribute information using many presentation formats including still images, moving images, audio, and text. Videos can also be used to present different kinds of information, for example, news, entertainment, advertisements, and the like. However, producing video content can require time, money, and resources that are not readily available. This problem is exacerbated for situations in which video content needs to be highly personalized or frequently updated. For example, a real estate agent may want a promotional video that shows houses that are currently available in a community.

In some cases, individuals and small to medium-sized businesses may use a system that populates a simple video template with content. Although they may be able to manually update the content over time, all videos based on the same template look similar, which makes it difficult for the content to stand out and interest viewers. For example, a thousand videos designed with a simple template to promote realtors may look almost identical except for the photo of the realtor and the address of the real estate agency.

The video processing system described herein automatically generates custom videos without requiring undue time, money, and effort to produce each new video. In particular, the video processing system may be provided with a selection of content and general parameters for a video. The video processing system generates a video that conforms with the parameters using provided content. The video processing system can also automatically update the video content and structure when new content is available and can also adapt the video in view of viewer feedback metrics.

A client provides preferences and parameters for a video. The client may also provide content or locations (e.g., on the Internet) where the video processing system can locate content, for example, a universal resource locator for a client website.

The video processing system adjusts values for story beat parameters and reorders story beats within the story model. To generate a video, the video processing system accesses content that matches the parameters of each story beat and assembles the content to create a video that matches the story model. The number, length, order, and parameter values of story beats may be adjusted when a video is updated. For example, a machine model may be used to update values associated with story beat parameters based on viewer feedback metrics. In this way a video can be updated automatically and improved over time.

A story model, as described herein, refers to a sequence of story beats with associated parameter values that define characteristics of the story as it is to be rendered in a video. A story model may be embodied as an n-dimensional vector with values corresponding to each of n story characteristics. That is, a story model may be an ordered sequence of story beats, for example, where each story beat is represented by parameter values for each of the n dimensions of story characteristics. The story model may include information about types of content to use in the video, parameters affecting aspects of the video, and an order in which story beats occur within the video.

A story beat, as described herein, refers to a generalized data representation of a video segment that may be used in a part of a story model. That is, a story beat may include parameter values associated with characteristics of a point in the video. For example, parameter values in a story beat may indicate that a segment of the video should last for 3 seconds, should be animated, and should include sad music.

System Architecture

FIG. 1is a high level block diagram of a system environment for a video processing system130, in accordance with an embodiment.FIG. 1includes client devices110, a network120, and the video processing system130. For clarity, only one client device110is shown inFIG. 1. Alternate embodiments of the system environment can have any number of client devices110as well as multiple video processing systems130. The functions performed by the various entities shown inFIG. 1may vary in different embodiments.

The video processing system130may generate and modify videos in response to input parameters and viewer feedback. The video processing system130includes a story engine140, a semantic engine150, an input data store160, a story model data store170, and a media data store180.

The story engine140develops and updates story models. The story engine140also collects and assembles media content that can be used to render videos based on story models. In some embodiments, the story engine includes machine learned models that are trained to alter story model parameters according to information such as past story models, viewer feedback metrics to past videos based on the story models, current events, and client preferences. Additional details about the story engine140are included in the description ofFIG. 2A.

The semantic engine150evaluates media assets for use in videos that are generated by the story engine140. In some embodiments, the semantic engine150is used to determine whether non-text content items include applicable subject matter for use in the video. For example, the semantic engine150may use machine learning techniques (e.g., convolutional neural networks) to identify the subject of an image and to identify the center of action within the image. The semantic engine150can also perform other functions for the video processing system130. In some embodiments, the semantic engine150finds useable content that was not provided by the client, for example, if additional material is needed to generate a video based on a story model. Additional information about the semantic engine150is provided in the description ofFIG. 2B.

The input data store160is software, hardware, firmware or a combination thereof for storing information that is provided to the video processing system130by a client for whom the video processing system130is producing a video. In some embodiments, the video processing system may not receive this information directly from the client, for example, a system administrator of the video processing system130may provide input information in formatted configuration files. Information stored in the input data store160may include a smart script, a host source list, and a render profile list. The smart script is a configuration file that describes an abstracted and parameterized version of the story that the client wants presented in a video. In particular, smart scripts describe general concepts and characteristics for story beats that may be used in a story model. Smart scripts are discussed in further detail in the description of the story engine140atFIG. 2A. Another type of input stored in the input data store is a host source list. The host source list may be embodied as a document that includes records about accessing content provided by a client. For example, the host source list may include uniform resource locators (URLs) at which the video processing system130can access client content (e.g., text, audio, video, and other assets) for use in the video. The input data store160also stores a render profile list. The render profile list includes a schedule for when the video processing system130is to update the video and may also include a list of adaptation triggers, that is, special circumstances, conditions, or actions that will prompt the video processing system130to update the video.

The video processing system130is software, hardware, firmware or a combination thereof for storing story models in the story model data store170. In the process of updating videos, the story engine140adapts story models. In some cases, the story engine140may provide previous story models as input to a machine model that will be used to create a new story model. Story models stored in the story model data store170may also be used to generate new videos.

The media data store180is a repository that stores content that the video processing system130may include in a video. The media data store180may be embodied as a database. The content may include text, audio, video, and images. In one embodiment, the video processing system130obtains the content from sources listed in the host source list (e.g., a website associated with a client who requested a video). In some embodiments, the media data store180may also contain stock media provided by the video processing system130or other media assets that are available for use in a video.

A viewer, client, or other user may be able to access a video through a client device110. In some embodiments, various client devices may serve different functions for the video processing system130. A client device110may provide a way for a user to view a video produced by the video processing system130, a client device110may be a way for the client who requested the video to provide preferences or feedback, or a client device may be a content server from which the video processing system130can retrieve content for use in a video. Client devices110can be personal or mobile computing devices such as smartphones, tablets, or notebook computers. In some embodiments, client devices110may be web servers or other servers that host content.

Client devices110communicate with the video processing system via a network120. The network120may comprise any combination of local area and wide area networks employing wired or wireless communication links. In some embodiments, all or some of the communication on the network120may be encrypted.

FIG. 2Ais a block diagram of a system architecture for a story engine140, in accordance with an embodiment. The story engine140generates and updates story models and links content to story beats within story models to produce videos. The story engine140includes, among other components, an ontology mapper204, a story model shaper206, a source linker208, a render module210, a panel dispatch module212, an adaptation dispatcher214, and a publisher216. Components of the story engine140may be embodied as software modules. In various embodiments, the story engine140may contain more, fewer, or different components than those shown inFIG. 2Aand the functionality of the components as described herein may be distributed differently from the description herein.

To produce a video, the story engine140generates a story model that is based on a smart script. A smart script may be provided by a client, or on behalf of a client, as input to the video processing system130when the client requests a new automatable video set. In one embodiment, a smart script is a configuration file that describes, at an abstract level, characteristics for all story beats that are to be made available to a story model. The smart script also describes parameters for video narratives that are generated from story models and scopes to which the video narratives are to be constrained.

When developing a smart script, a client first considers what kind of story the video is going to tell. As several examples, a client may want a video advertising a restaurant, a video describing the life of a historical figure, or a video that shows the weather for a particular area. If a similar video type has been developed before, then the video processing system130may supply the client with an existing smart script that the client can alter to suit her needs. Otherwise, the client will need to develop a new smart script to provide to the video processing system130. A smart script describes individual story beats that may be used in a video narrative. For example, a smart script for a video introducing a small business may include an indication that the video could include the business name, the city where the business is located, the vocation of the proprietor, and promoted skills of employees. That is, the smart script tells the story in a parameterized format using abstracted data classes rather than explicitly listing specific information. The smart script may also include data presentation preferences of the client. For example, the smart script may indicate that the video should include a storefront photo, a testimonial video, and an animated logo. Further, the smart script may include transition, ordering preferences, and general rule sets for the video. For example, the smart script may be organized into classes such as “story open”, “dissolve transition”, “animation”, “identify subject within first 5 seconds”, “close with a flying logo”, “insert hyperlinks”, and the like. A smart script thus describes portions of a story by listing general parameters that story beats can have.

In some cases, a client may develop a smart script by creating an example story, for example, by storyboarding a specific video narrative. The example story is then tokenized into individual story beats that describe the different portions of the video. A resulting smart script comprises descriptions of the story beats.

The ontology mapper204is a software module that parses information associated with the host source list and collects content that may be used in the video from the host sources. The ontology mapper204extracts content from sources indicated in the host source list, which is kept in the input data store160. For example, if the host source list includes source URLs or other data sources, the ontology mapper204accesses the URLs and other sources (e.g., via the network120) and extracts the content. Such content may include text, images, video, and audio. In particular, the ontology mapper204collects content in formats that are listed in the smart script. For example, the smart script may describe a story beat in which a still image of a cat is shown, and the ontology mapper204would then access the sources to find a still image of a cat. Content extracted by the ontology mapper204is processed and then may be stored by the video processing system, for example, in the media data store180.

The ontology mapper204is a software module that passes non-text components to the semantic engine150for further processing and analysis. The sematic engine150is further described inFIG. 2B. The ontology mapper204identifies portions of the text components that are relevant to story beats included in the smart script. The ontology mapper204may use natural language processing techniques to categorize information in text components of the content. Natural language processing techniques used by the ontology mapper204may include sentence segmentation, tokenization, part of speech tagging, entity detection, and relationship detection. The ontology mapper204may also associate each text component with a story beat to which it may be relevant. For example, content with text may be in the form of a review left on a website. The ontology mapper204may categorize the content as a testimonial and may associate the content with a story beat in the smart script that indicates that the video could include written testimonials.

The story model shaper206is a software module that constructs and adapts story models. That is, the story model shaper206develops a story model upon receipt of a new smart script and also adapts existing story models based on a variety of viewer feedback metrics, described below in detail with reference to a panel dispatch module212. A story model can be thought of as a sequence of story beats, the contents, length, characteristic parameters, and order of which are determined by the original smart script. The story model shaper206creates such a sequence of story beats to form an initial story model.

A story model may be represented by any number of dimensions of parameters. That is, for a story model described by n parameters, each story beat may be characterized by n values. Examples of parameters that may be associated with a story beat include market target, depth, popularity, seasonality, location specificity, target age demographic, target income demographic, timeliness, style type, irreverence level, color scheme, length of time within the story, etc. Some parameters may be represented by relative value (e.g., length of story beat in video) while other parameter values may represent categorization (e.g., where specific parameter values are assigned to different color schemes, styles, or target demographic groups).

In the initial story model, parameters associated with story beats are instantiated with equal parameter values for all story beats. The story model shaper206then amplifies or attenuates values assigned to parameters for each story beat, for example, according to preferences indicated in the smart script. Adjusting the parameters associated with story beats alters the shape of the story model. As such, a story model can be conceptualized as having a “wave shape” that represents the changes in story beat parameters over the course of a story.

In one embodiment, the story model shaper206adjusts parameters of the initial story model according to vectors whose values may be assigned in the smart script. In some cases, the vectors are a business class vector, a customer class vector, and an engagement class vector. These three vectors may include adjustments to make to parameters of each story beat in a story model, effectively tuning the wave shape of the story model according to target viewer demographics. A business class vector is a set of parameter values representing the kind of interaction that is to be described in the video, a customer class vector is a set of parameter values representing the profiles of viewers in the target demographic, and an engagement class vector is a set of parameter values indicating whether target viewers are new or returning viewers with respect to the topic of the video (e.g., if target viewers have accessed a website associated with the video subject before).

For example, if the story model shaper206is used to shape a video for a 24-hour plumbing business that is aimed at low-middle-income males who are new to the business, the story model shaper206may amplify values of story beat parameters associated with low cost, basic services, certification, and positive reviews, and may attenuate parameters associated with custom plumbing, sales of luxury fixtures, photos of large homes, etc.

In addition to generating new story models, the story model shaper206can adapt existing story models. For example, the story model shaper206can change the shape of a story model in accordance with client-scheduled parameters (e.g., a client schedules a video update for holidays, or every two months). The story model shaper206may also adapt a story model in response to viewer reactions to the video produced based on the story model (e.g., to avoid a tendency to lose viewers at a specific point in the video). Such a process of dynamic adaptation allows story models to evolve as the model is altered to have different wave shapes over time.

To adapt an existing story model, the story model shaper206accepts data about viewer feedback metrics from the adaptation dispatcher214. The viewer response data is used to instantiate new iterations of existing videos by improving past story models and generating new videos. The story model shaper206includes machine learning algorithms that evaluate and alter an existing wave shape. Examples of semi-supervised algorithms that may be used by the story model shaper include Ordinary Least Squares Regression and Multivariate Adaptive Regression Splines. The semi-supervised machine learning algorithms are used to train machine models of the story model shaper to recognize successful story wave shapes. For example, the algorithms may train machine models using story wave shapes (e.g., from past iterations of videos) that are labeled as successful or unsuccessful at engaging viewers. The semi-supervised machine learning algorithms may also train machine models for adjusting wave shape parameters for story models based on inputs including viewer feedback metrics. The description ofFIG. 4further discusses use of machine learned models and algorithms to adjust a story model.

Examples of unsupervised machine learning algorithms that may be used by the story model shaper206include the Apriori algorithm and k-Means Clustering. The unsupervised algorithms may be used by the story model shaper206to analyze relationships between viewer feedback metrics and wave shapes of story models. In some embodiments, such relationship information may be used to train machine models from which the semi-supervised algorithms can learn information about how a story wave shape can affect viewer engagement. As one example, an unsupervised algorithm may classify story models into different classes of wave shapes and may identify similarities and differences in viewer feedback metrics for each class of wave shape which can then be used to train another machine model to generate story models with wave shapes that have a likelihood of producing videos of interest to a particular demographic.

After a story model has been generated and shaped (or re-shaped in the case of story model adaptation), the source linker108, which may be embodied as a software module, links media assets from the media data store180to the individual story beat elements within the story model. To link content to a story beat element, the source linker108identifies a content item that matches the parameter values of the story beat and that content item may be assigned to the story beat. For example, if a story beat includes parameters describing use of a positive text review of a restaurant as a part of a promotional video for the restaurant, the source linker208may access the media data store180to search for positive testimonials, as identified by the ontology mapper204. In some embodiments, the content in the media data store180may already be suggested for use with a particular story beat in view of initial analysis from the ontology mapper204.

The render module210is a software module that assembles the content identified by the source linker208and renders it into a video. In some embodiments, video rendering may be handled by a separate render farm system, by a virtual machine cluster, or by a single computing device, such as the video processing system103. The render module210may render the video into multiple machine-dependent resolutions as may be stipulated in the render profile list stored in the input data store160. After it is rendered, physical video files may be stored, and served, by the video processing system130or on a separate server platform. The physical video files can be played back by video-enabled client devices110(e.g., cell phones, tablets, personal computers, smart televisions, etc.).

The publisher216is a software module that can deliver a universal embed code to the client who requested the video (or to a representative content management system). A universal embed code may be embodied as a block of hypertext markup language (HTML) that includes an object (e.g., the video) in a webpage. The universal embed code allows a client device110to access the video. The client can include the embed code in locations, such as on a website, where they want the video to be accessed by viewers. In other embodiments, the publisher216may provide the embed code or the video file itself to an outside video distribution system. Such a distribution method may be supported by third party programmatic service providers that offer direct publishing of auto-generated videos.

The first time a story model is rendered into a video, the panel dispatch module212, which is a software module, may send the video to an outside review service to receive a first set of viewer feedback metrics about the video. A video based on a newly generated story model has not been improved in view of reactive input and thus may not be ready for release to public viewers. The panel dispatch module212assigns a video to an outside panel service, for example, to a focus group of users who will provide feedback. The video is viewed by a pre-selected group of humans who interact with an application on a client device110to view and react to the video. The application on the client device110may be designed to collect viewer feedback metrics such as points where the video was paused, points where the video was stopped, whether portions of the video were replayed, etc. The panel dispatch module212receives the viewer feedback metrics generated by the viewing users and transmits it to the metrics interpreter215in the adaptation dispatcher214for analysis.

The adaptation dispatcher214is a software module that indicates when and how a story model is to be adapted or a new video is to be generated. The adaptation dispatcher214accesses information stored in the input data store160related to source web pages and schedules from the render profile list to determine when a story model or video is to be updated. The adaptation dispatcher214may trigger updates in view of a client update schedule that may request, for example, daily, weekly, holiday, or other situational updates.

The adaptation dispatcher214also includes a metrics interpreter215. The metrics interpreter215receives viewer feedback metrics and evaluates whether the story model of a video ought to be adapted or whether the video ought to be re-rendered based on the existing story model. In one embodiment, the metrics interpreter215uses viewer feedback metrics to determine whether a story model or video is to be updated. For example, the metrics interpreter215may be programmed to determine that a story model needs to be adjusted whenever a certain percentage of viewers do not complete watching the associated video. When the adaptation dispatcher214determines that that a story model needs to be updated, it triggers a response from the story model shaper206which will update the story model based on the viewer feedback metrics. Similarly, when the adaptation dispatcher214determines that a video needs to be re-rendered (but that the video does not need a new story model) the source linker208may collect new sources for use in an updated video (e.g., to create a video with content from an updated client website).

FIG. 2Bis a block diagram of a system architecture for a semantic engine150, in accordance with an embodiment. The semantic engine150analyzes non-textual content that was collected by the ontology mapper204. The semantic engine150may also access and collect additional content if the ontology mapper204was not able to find enough or adequate content at sources listed in the host source list. The semantic engine150includes, among other components, a grading processor218, a media resolver220, an external search module222, a providence verification module224, and an internal media data store226. Components of the semantic engine150may be embodied as software modules. In various embodiments the story engine140may contain more, fewer, or different components than those shown inFIG. 2Band the functionality of the components as described herein may be distributed differently from the description herein.

The grading processor218is a software module that identifies specific subjects of videos, images, and audio. To do so, the grading processor218may use image processing techniques including convolutional neural networks. The neural networks or other image processing models may be trained using a set of training images prior to use in the grading processor218. In addition to identifying the subjects of images, the grading processor218may also determine the center of action of a content item and other information such as level of exposure or effective resolution. This information can be used later by the source linker208, for example to determine portions of an image to display most prominently in a rendered video.

The media resolver220is a software module that evaluates each individual content asset and either accepts or rejects the content item for use in a video. In one embodiment, the media resolver220analyzes content after it has been analyzed by the grading processor218, and the media resolver220may use data generated by the grading processor218when analyzing content. To determine whether a piece of content is useable, the media resolver220may employ image processing techniques including trained neural networks. In the case of the media resolver220, the image processing algorithms are trained with training data including content labeled as having a success or failure state. For example, the image processing models may be trained to identify blurry images as failure states and to reject them for use in videos. Content that is not accepted for use in the video is marked and may be used as an example of a failure state when the image processing algorithms are trained in the future.

The external search module222is a software module that finds content items that were not found in host sources by the ontology mapper204. Additionally, the media resolver220may alert the external search module222to find content items if the content that the ontology mapper204identified from a host source was not accepted for use in a video. The external search module222executes directed asset searches to find content that has available rights control and providence data and which matches story beat requirements that were unmet in the content search conducted by the ontology mapper204. For example, if a story beat defined in a smart script includes a still image of a dog and the ontology mapper204was not able to find a still image of a dog on the client website, the external search module222may search the Internet for a still image of a dog that is available for use.

The providence verification module224is a software module that determines the usage rights or providence of media content. In one embodiment, the providence verification module224only verifies usage rights for media assets accessed by the external search module222, as other content is known to originate from client source locations listed in the host source list. Input to the providence verification module224may be in the form of images or sets of still frames exported from motion video clips. For example, the providence verification module224may perform a combination of automated search by image and reverse image searches. If usage rights can be determined for the content and if the usage rights allow for use of the content, then the media asset is marked as usable. Otherwise the content is deleted from the video processing system130.

Usable content collected from locations other than the client sources may be stored in the internal media data store226. The internal media data store226may be embodied as a database. That is, the content (e.g., video clips, images, and audio) stored in the internal media data store226has been analyzed and identified by the grading processor218, and has been allowed for use in a video by the media resolver220and the providence verification module224. Content in the internal media data store226may be accessed by modules of the story engine140to link to story beats in a story model and to subsequently include in videos.

Story Models

FIG. 3is an example illustration of a story model300, in accordance with an embodiment. As mentioned previously, a story model300is a data set that defines planned characteristics of a video. Such characteristics may be more generally defined by a smart script. The example ofFIG. 3is just one possible way of conceptualizing the components of a story model300. In particular, a story model300comprises a sequence of story beats310. One example story beat310is emphasized inFIG. 3with a dotted line that passes through several lines indicating different parameter values. A story beat310is a set of parameter values that represent a portion of a video. A story model300may be defined by multiple dimensions315of parameters. In the example ofFIG. 3, three dimensions315are shown. Each dimension315represents values of one parameter of the video and each story beat310includes defined parameter values for each of the dimensions315. Examples of parameters that may be used to characterize a story model300include length of story beat, color scheme, style, audio type, volume, mood, content type, seasonality, etc. In some dimensions, parameter values may represent relative amounts, for example, in the case of seasonality, story beat length, or volume. In other dimensions, parameter values may represent categorizations, for example, style, color scheme, mood, content type (e.g., show a still image, video, audio, etc.) and the like.

When the story model shaper206adjusts a story model300, it is changing the values associated with parameters for each dimension315for each story beat310. In some cases, the story model shaper206may also reorder story beats within the story model300. Rearranging the order of the story beats310causes content to be presented in a different order when a video is generated based on the story model300. For example, in a story model300for a video advertising a company, a story beat that defines an animated logo may be moved to come before a story beat that defines a short video clip of an employee of the company.

A story wave shape is defined by the way values of parameters in all dimensions315change over the course of the story. Curves are drawn along the parameter values inFIG. 3to illustrate story wave shapes. A full story wave shape could be conceptualized as an overlay of all the wave shapes for every dimension315used in a story model300.

Generating and Updating the Story Model

FIG. 4is an example illustration of automatic story model generation using machine models, in accordance with an embodiment. As mentioned previously in the description ofFIG. 2A, the story model shaper206generates and adapts story models300. In one embodiment, the story model shaper206includes trained machine models to which machine learned algorithms may be applied to produce a new story model300. That is, the story model shaper206includes trained machine models that can be used to update parameters and ordering of story beats310within a story model300in view of a variety of inputs.

In one example embodiment, a machine model of the story model shaper206is trained using labeled training data. The training data may include versions of past story models300as well as features that were associated with the story models300. Additionally, the training data may be labeled, in the case of supervised or semi-supervised learning algorithms. For example, a past story model300used for training a machine model to generate story models300may include a label indicating whether the video generated based on the past story model300was successful or not (e.g., whether viewers watched and liked the video).

Prior to generating a story model300, the story model shaper206receives features410as input. The features410may include information about one or more previous iterations of the story model300, viewer feedback metrics, identifications of target demographics, and other client preferences. The first time a story model is shaped, the features410may not include information such as previous iterations of the story model or feedback from viewers.FIG. 4shows a few example input features410that may be provided to the story model shaper206, including the old story model300A that was previously presented to viewers, the smart script410A, the number of views a video based on the previous story model300A received, an average time viewers spent watching the video410C, time points at which viewers paused the video410D, and a target age of viewers for whom the video is being produced410E.

The features410are applied to a trained machine model by the story model shaper206. The story model shaper206uses algorithms associated with the trained machine models to generate a new story model300B based on the input features410. For example, the story model shaper206algorithms may attenuate and amplify parameter values for each story beat310and may also reorder story beats310within the story model300. The new story model300B is then linked to content, rendered, and published as an updated version of the video. In some embodiments, additional machine learning algorithms are used by the story model shaper206to classify successful story models300. For example, a particular story wave shape may be especially effective at producing videos that engage the teenage demographic, and a machine learning algorithm may notice such a pattern. In those cases, classifications and correlations between features410and successful story models300may be used to train future machine learned models for generating effective story models300.

Process

FIG. 5is a flowchart illustrating a process for generating a video using a story model, in accordance with an embodiment. The video processing system130collects510media asset items. The media asset items may be content collected from sources associated with a client (e.g., a client website) as listed in the host source list. In some cases, the client sources may not have all of the types of media asset items needed to populate story beat concepts that are described in the smart script. In such cases, the video processing system may access content from other sources.

The video processing system130generates520a story model that includes story beats. The story model describes a general idea of a story and includes an ordered sequence of story beats (i.e., portions of the story). The individual story beats may be generally described in the smart script, although the story model may use only a subset of the story beats from the smart script in some embodiments. The video processing system adjusts530story beat parameter values. In one embodiment, values associated with story beat parameters are determined by machine learning algorithms using machine models that have been trained to improve story models given input features related to story preferences and target demographics. In some embodiments, the algorithms also reorder story beats within the story model.

After a story model has been adjusted, the video processing system130links540collected media asset items to story beats within the story model. A media asset item may be linked to a story beat if its characteristics meet the criteria of the story beat parameters. A video is rendered550based on the story model. The video includes the media assets that are linked to the story beats in the story model. The process may be repeated to improve and update the video over time.

In various embodiments, the video processing system130may generate videos by an alternative process to the example shown inFIG. 5, or may perform the steps ofFIG. 5in a different order. For example, in one embodiment, the video processing system130may adjust story beat parameters of an existing story model without collecting new media asset items. In another example embodiment, the video processing system130may collect new media asset items and link the newly collected media asset items to the matching story beats in an existing story model without adjusting the story beat parameter values.

Hardware Components

FIG. 6is a block diagram illustrating components of an example machine able to read instructions from a machine-readable medium and execute them in one or more processors (or controllers). Specifically,FIG. 6shows a diagrammatic representation of video processing system130in the example form of a computer system600. The computer system600can be used to execute instructions624(e.g., program code or software) for causing the machine to perform any one or more of the methodologies (or processes) described herein. In alternative embodiments, the machine operates as a standalone device or a connected (e.g., networked) device that connects to other machines. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

The example computer system600includes one or more processing units (generally processor602). The processor602is, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a controller, a state machine, one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these. The computer system600also includes a main memory604. The computer system may include a storage unit616. The processor602, memory604, and the storage unit616communicate via a bus608.

In addition, the computer system600can include a static memory606, a graphics display610(e.g., to drive a plasma display panel (PDP), a liquid crystal display (LCD), or a projector). The computer system600may also include alphanumeric input device612(e.g., a keyboard), a cursor control device614(e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a signal generation device618(e.g., a speaker), and a network interface device620, which also are configured to communicate via the bus608.

The storage unit616includes a machine-readable medium622on which is stored instructions624(e.g., software) embodying any one or more of the methodologies or functions described herein. For example, the instructions624may include instructions for implementing the functionalities of the story engine140, and the semantic engine150. The instructions624may also reside, completely or at least partially, within the main memory604or within the processor602(e.g., within a processor's cache memory) during execution thereof by the computer system600, the main memory604and the processor602also constituting machine-readable media. The instructions624may be transmitted or received over a network626via the network interface device620.

Additional Considerations