Patent ID: 12231747

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the invention described herein relate to systems and methods for algorithmic editing of digital video for optimising audience engagement therewith.

General System Architecture

With reference toFIG.1, there is shown a schematic representation of a computer system100in accordance with an embodiment of the present invention.

The computer system100comprises a video clip engagement system102that in turn comprises a video analysis server104and an algorithmic editing server106. The video analysis server104is communicable over a communications network (in this instance taking the form of the Internet108) with a source video server110for receiving source video files stored thereby. The source videos are stored by the video server110in any suitable file format, including, but not limited to, MOV and MP4 file format.

As will be described in more detail in subsequent paragraphs, the video analysis server104is configured to analyse video files received from the video server110for extracting various forms of metadata. The extracted metadata is subsequently stored in aggregated form in a video metadata data store112(i.e. with a reference to the corresponding video file). The data store112may be implemented using any suitable data stored, for example, a relational or non-relational database application program, such as, e.g., Oracle, Sybase and the like. According to the embodiment described herein, the data store112takes the form of a SQL database.

The algorithmic editing server106implements one or more video clip generating algorithms for generating a selection of candidate video clips for each source video (i.e. based on input metadata stored in the data store112). The candidate video clips are subsequently communicated to a video clip distribution network114that stores and delivers the video clips when requested by viewer devices. The video clip distribution network may take on any suitable physical form, for example including one or more streaming servers that are configured to stream video clips to viewer devices using techniques well understood in the art. In addition, the algorithmic editing server106is configured to store the relevant aggregated metadata for each candidate video clip, which is a subset of the metadata for the source video, in the data store112with reference to the corresponding stored video clip.

When a viewer device116requests a video, the algorithmic editing server106implements a video clip selection algorithm that applies the aggregated metadata for the stored video clips for selecting a candidate video clip to deliver to the viewer device.

As will become evident from subsequent paragraphs, the clip selection algorithm is configured to select the candidate video clip that is most likely to result in the highest viewer engagement (and in turn increase the likelihood of the viewer accessing the complete source video, and/or take some subsequent action desired by the system102, as a result of having viewed the video clip). The selected candidate video clip is subsequently delivered to the viewer device116by the video clip distribution network114. It will be understood that the selected video clip may be streamed, sent as a complete file, or otherwise electronically delivered to the viewer device, depending on the digital medium.

The viewer devices116may take on various forms, including but not limited to, a general-purpose computer, laptop, smartphone, tablet, smart TV, or any suitable network-enabled computing device that implements a display screen and the appropriate software for playing a video clip delivered by the video clip distribution network114.

Further, the mode of digital delivery of the video clip (i.e. digital medium) may vary, depending on the desired implementation. For example. one mode of delivery may be via e-mail. In this example configuration, the video clip engagement system102may include an image or video with a source URL referencing the engagement system102in the HTML content of an email sent to a viewer or group of viewers. Opening the email using an email application resident on the viewer device116causes the email application to parse the HTML content and send a content request to the URL maintained by the engagement system102to retrieve the image or video, which triggers the algorithmic editing server106to dynamically select an appropriate video clip and in turn respond with an HTTP re-direct instruction causing the email application to retrieve the corresponding video clip from the content distribution network114.

Another mode of delivery may be via a mobile application retrieving video content using a streaming protocol. In this example, a streaming manifest that specifies the video files to be delivered to a viewer device is provided by the engagement system102and is dynamically adjusted to specify the location of the chosen video clip on the content distribution network114. Another mode of delivery may be via a number of Facebook video advertisements targeted at audience segments. In this example, a number of video clips are uploaded to Facebook along with targeting criteria for each (e.g. located in a particular country, aged between a predefined range, or any other ascertainable profile data), and Facebook systems show the relevant candidate video clip to each viewer that matches the specified targeting criteria.

Finally, the engagement system102can be communicable with a customer relationship management database111for receiving data relating to the intended audience (e.g. the comedy version of a video clip is to be delivered to one or more viewers that are known to have a preference for comedy movies).

Process Flow for Video Analysis

FIG.2shows a schematic representation of a process flow for analysing a source video for metadata extraction, according to an embodiment of the invention.

In step S1, the video analysis server104communicates with the source video server110for receiving a source video. Alternatively, the source video may be retrieved from local file storage, uploaded from a portable storage drive, or received via any other suitable wired or wireless digital communication means.

At step S2, the video analysis server104implements a program configured to process the source video to extract various forms of metadata (steps S2ato2e).

More particularly, at step S2a,any basic metadata embedded within the source video is extracted by the program. By way of example, the embedded metadata may comprise: title; description; tags; source; resolution; duration and the like. The basic metadata may also be obtained from the source video server110, where available. Also, at step S2a,a video processing engine implemented by the video analysis server104is operable to analyse each frame of the video to measure brightness levels and variance from the previous frame. Recorded brightness and variance metrics are timestamped and labelled. As an example, brightness levels may be used to exclude clips with fade-in or fade-out durations that exceed a predefined optimal duration for the target media.

At step S2b,computer vision labelling is employed by the video analysis server104to identify objects, text overlays, events and/or scene changes within the source video. The identified objects, text overlays, events and scene changes are timestamped and labelled (based on label taxonomy data stored by the analysis server104). In a particular embodiment, the video analysis server104implements the Google VideoIntel service (see http://https://cloud.google.com/video-intelligence/) which evaluates pixels within each frame of the source video using deep-learning models for object, event and scene change labelling. Another suitable form of evaluation and identification software application is the Amazon Rekognition Service (see https://aws.amazon.com/rekognition).

At step S2c,an audio analysis is carried out by the video analysis server104for detecting predefined audio characteristics within the source video, such as a quiet scene, noisy scene, explosion, indoor scene, outdoor scene, speech, laughter, ambient noise or any other desired predefined audio characteristic. Again, the detected characteristics are timestamped and labelled. In a particular embodiment, the server104implements the SensAI service (see http://audeering.com/technology/sensai/) for identifying the audio characteristics.

At step S2d,a facial detection, recognition and interpretation procedure is carried out by the video analysis server104for detecting and recognising faces that appear in different frames and scenes, and interpreting the emotional expressions (for example, surprise, excitement, anger) on each face in each frame. Any identified faces and expressions are timestamped and labelled. Again, in a particular embodiment, the server104implements the Google VideoIntel service which evaluates pixels within each frame for facial detection and recognition. As an example, facial recognition metadata may be used to generate candidate video clips focussed on the parts of a source video including a particular face or which include a particular facial expression.

At step S2e,a speech recognition module employing one or more speech recognition engines is implemented by the video analysis server104for identifying speech within the source video, using techniques well understood in the art. In a particular embodiment, the speech recognition module is configured to identify selected predefined words or utterances within the source video. Any identified words/utterances are timestamped and labelled.

At step S3, the metadata extracted/determined in step S2is normalized into a consistent data structure independent of the source of the metadata, and stored, in aggregated form in the data store112in association with the source video (step S4).

At step S5the metadata is processed to identify higher-level labels for the labelled metadata (step S5a); infer video genres/sub-genres that are associated with individual labels or groupings of labels in the labelled metadata (step S5b); and for statistical clustering (step S5c). The results of any additionally derived metadata resulting from step S5is stored in the data store112.

In more detail, step S5acomprises applying codified videography knowledge to determine, for example, applicable genres/subgenres based on individual labels or label groupings that occur in a particular sequence.

For example, the labels “bonfire”, “combat” and “dance” may have been identified and associated with a particular source video in step S2(e.g. embedded within the source video file, identified as a result of the speech recognition, or object detection processes S2d/S2e). As shown in Table 1 below, the codified videography knowledge is applied to recognise the label “bonfire” as implying a historical drama with a weighting of 0.7, or a romantic comedy with a weighting of 0.5, or a historical fantasy genre with a weighting of 0.7. The label “combat” is recognised as implying a historical drama with a weighting of 0.5, a historical fantasy with a weighting of 0.8, or a superhero action film with a weighting of 0.6. Finally, the label “dance” is recognised as implying a romantic comedy with a weighting of 0.7.

TABLE 1LabelInferred Genre/SubgenreWeightingBonfireDrama/Historical Drama0.7BonfireComedy/Romantic Comedy0.4BonfireFantasy/Historical Fantasy0.7CombatDrama/Historical Drama0.5CombatFantasy/Historical Fantasy0.8CombatAction/Superhero0.6DanceComedy/Romantic Comedy0.7

The software algorithm evaluates the combined weightings and records the genre/subgenre that has the highest cumulative weighting. In an embodiment, the codified videography knowledge is embodied by software which is programmed to automatically input any labels identified in step S4, run an algorithm that sums the associated weightings (which may be stored in table format) and output the most likely genre and/or subgenre. In an alternative embodiment, each label is additionally weighted by the duration of its detected presence in the source video, and/or a confidence metric representing how certain the label appears.

Step S5bcomprises determining higher-level labels for the labelled metadata identified in steps S2and S3using one or more semantic clustering algorithms. By way of example, metadata associated with the label “cat” may also be associated with the label “animal”. This semantic clustering is achieved using a semantic knowledge graph such as that provided by Google Knowledge Graph (see URL: https://developers.google.com/knowledge-graph/).

Step S5ccomprises clustering labels that are in close proximity to other labels, either within a single source video, or across a sample of many videos. In this regard, the video analysis server104is configurable to extract and process metadata for other video sources (112ato112n) for determining label clusters.

In a particular embodiment, clusters are determined using a k-medoids unsupervised learning algorithm, as will be understood by persons skilled in the art (although it will be understood that any suitable clustering algorithm could be utilised, depending on the desired implementation). Further, according to embodiments described herein, the “proximity” between two labels is implemented by a statistical proximity software algorithm implemented by the video analysis server104. The statistical proximity algorithm is programmed to statistically calculate proximity between labels based on the frequency and proportion of time that two or more labels are detected alone and simultaneously in the source video (and optionally other videos). Further, the selection of the optimal number of clusters to be formed may be implemented by calculating the optimal Calinski-Harabasz Index for a range of possible values of k.

By way of example, the statistical proximity algorithm may determine the number of times and the corresponding time periods (i.e. determined from the timestamps recorded at step S2) that the label “bonfire” was detected in the source video. Similarly. the algorithm may determine the number of times and corresponding time periods that a particular actor was detected in the source video. If multiple instances of both the bonfire and actor were detected, but neither occurred at the same time as the other (i.e. there was no overlap in their corresponding time stamps), then the proximity algorithm may determine that the labels are not statistically close (i.e. are “independent”). On the other hand, if each instance of bonfire occurred at the same time each instance of the actor was determined, then the algorithm may determine that the two labels are very close in distance. It will be understood that the distance may be represented in any suitable form, such as percentage (e.g. 0% reflects no proximity, whereas 100% represents the closest relationship achievable), an absolute score or rating (e.g. 0 to 10), etc.

Process Flow for Video Editing and Clip Selection

FIG.3shows a schematic representation of a process flow for editing a source video for generating multiple clips that are dynamically selectable for delivery to viewers.

At steps S1aand S1bofFIG.3, a source video and the corresponding metadata is retrieved and temporarily stored by the algorithmic editing server106. At step S1c,medium data for the video clip presentation is determined. By way of example, if the digital medium is Instagram, the data may indicate that the video clip will be initially muted, played as an infinite loop, have a square aspect ratio, and support subtitles. For e-mail presentation, the medium data may indicate that the video clip will be muted, have a single loop, have a landscape aspect ratio and be of 10 second duration.

If there is any audience data available it is retrieved from the CRM store111(step S1d). By way of example, audience data may indicate that the target viewers have a preference for movies that are in the “horror” genre.

Step S1einvolves retrieving any engagement data associated with the source video or related source videos. Engagement data is representative of one or more engagement metrics that are recorded from previous aggregate audience interactions with video clips edited from the source video, or from other source videos.

As will be discussed in more detail below, engagement data is collected by various methods specific to each medium, and normalized to be independent of the medium, such that all engagement data can be aggregated. For example, engagement data resulting from video clips presented via email includes click-to-open-rate, which is collected by diverting clicks on the video clip and other content within the email via the algorithmic editing server106. An example of aggregate engagement data is that previous edits (i.e. corresponding to individual video clips) from the same source video have resulted in higher engagement when the edits (video clips) contain more coverage of the lead actor.

Finally, at step S1f,if aggregate context data is available, it is also retrieved. By way of example, aggregate context data may be that previous video clips delivered to viewers located in a particular State engage higher when they are a shorter duration (i.e. as determined from an evaluation of aggregated engagement metrics for that video, as will be described in more detail below).

At step S2, the algorithmic editing server106implements a clip generating algorithm for editing the source video to generate a plurality of candidate video clips, which selectively applies the metadata retrieved at step S1. Each candidate video clip comprises a subset of the source video, consisting of selected time sequences and/or pixel crops derived therefrom. The clip generating algorithm may re-order the time sequences and pixel crops.

In one implementation of the clip generating algorithm, only the video related metadata may be applied. Examples of how the clip generating algorithm may be configured to apply video metadata alone are outlined below:Include all time sequences from the source video that include the most prevalent face derived from facial recognition (step S2dofFIG.2), &/or the most prevalent label derived from computer vision labelling (step S5a). Order the sequences in the same chronological order as they occur in the source video.Calculate relationship strength between all pairs of recognized faces in the source video, using a distance metric that considers the overlap of screen times that each face is present in the source video (as per step5cofFIG.3). Then determine the pair of faces that have the strongest relationship (based on frequency and overlap). and include all time sequences from the source video that include either or both face. Order the sequences in a repeating pattern of both faces/face #1/both faces/face #2.
Other implementations of the clip generating algorithm that additionally apply any available medium data, audience data, context data and engagement data:A multi-dimensional feature vector is constructed from the medium, audience and context data, and one or more machine learning algorithms are used to learn and predict the relationships between the feature vector and the video metadata.

By way of example, the engagement data may reflect that Red Bull's biking fans on Facebook watching on a mobile phone in California are two times more likely than average to engage with a clip labelled as “mud”, 0.5 time more likely than average to engage with a clip labelled as “parachute”.

Based on this knowledge, the clip generating algorithm may be configured to pick the time sequences from the source video that include the computer-vision labels (e.g. “mud”) that are related to the highest engagement for the target medium, audience and expected contexts. Multiple candidate video clips that are derived from these time sequences may then be generated.

It will be understood that the clip generating algorithm can be programmed in various different ways to optimise audience engagement, depending on the desired implementation and available metadata.

At step S4an authorised user (e.g. a content expert employed by the engagement system102, or a marketer employed by a customer) may optionally review and selectively edit the candidate video clips (e.g. to filter or re-mix the selected time sequences, pixel crops). According to a particular embodiment, this is carried out by way of a web-based video editor.

At step S5, any differences between the algorithmic clips and the manually approved clips are analysed and used to improve the clip generating algorithm. By way of example, one specific customer of the engagement system102may want to ignore any source content that the computer vision labelling has identified (at step S2bofFIG.2) as lens flare and the algorithm may be configured accordingly. Thus, it will be understood that the clip generating algorithm may be customised for individual clients of the engagement system102or applied more generally.

Once the final candidate video clips have been generated, they are communicated to the content distribution network114for subsequent delivery. The metadata associated with each candidate video clip is stored in the metadata store112with a reference to the corresponding video clip.

At step S6, responsive to determining a clip selection trigger, the algorithmic editing server106retrieves the metadata for the candidate video clips, as well as any available audience data, viewer data, engagement data and context data.

According to embodiments described herein, the clip selection trigger takes the form of a content request message placed by the viewer device116. The content request message is received by the algorithmic editing server106and can take on different forms (and be triggered by different actions), depending on the digital medium. According to one embodiment, the content request is triggered by the viewer opening the email/web page/app. Alternatively, as discussed above, the content request may be generated as a result of a viewer selecting a URL included in the email/web page/app (i.e. that directs the application to a resource hosted by the engagement system102).

At step S7, responsive to receiving the request message, the algorithmic editing server106implements a clip selection algorithm that applies the retrieved metadata for selecting a video clip. An instruction is then sent to the video content distribution network114for delivering the selected clip to the requesting viewer device.

In a particular embodiment, the clip selection algorithm evaluates context data associated with the content request message. This may include, for example, the device type (e.g. phone, laptop), application type, network type, screen density, location, local time, language preferences and the like.

The algorithmic editing server106may determine device type and application type from the HTTP User Agent header present in the content request. Location, network type, and time zone are identified by the server106by evaluating the source IP address of content request, and lookup databases. Local time is calculated from the time of the content request and the derived time zone. Screen density may be detected using an HTML snippet that is triggered differently according to varying screen densities. Language preferences are determined from the HTTP Accept Language header present in the content request. In cases where HTTP request headers or IP address are obscured due to network proxies, the context data is marked as “Unknown” until it can be attributed from subsequent related requests (such as a click) that are not obscured.

Device type, application type, network type and screen density may be applied by the clip selection algorithm for selecting a video clip of a particular encoding format—for example, MP4, animated GIF, animated PNG, WEBP—that works best for engagement.

In a particular embodiment, engagement data may by representative of the number of previous video plays and one or more engagement metrics (e.g. dwell time, number of clicks on the video, basket events) for each candidate clip. Table 2 below provides an illustrative example.

TABLE 2Engagement Metric-Number of PlaysNumber of ClicksClip #11560124Clip #2180090. . . Clip #n162595

The engagement metrics shown above should not be seen as limiting and it will be understood that any suitable behavioural or event driven metric and associated engagement point could be recorded and utilised, depending only on the desired implementation.

The selection of the best clip from the retrieved data is a classic problem of probability theory, referred to as the “multi-armed bandit” problem—ref. https://en.wikipedia.org/wiki/Multi-armed_bandit

In a particular embodiment, the Thompson Sampling model is employed by the clip selection the algorithm for selecting best clip (see URL: https://en.wikipedia.org/wiki/Thompson_sampling).

These strategies remain valid even when the universe of candidate clips increases over time based on the candidate clip editor algorithm and/or the manual video editor creating additional candidate clips based on additional real-time aggregate engagement data.

When audience data, viewer data and context data is also available, the clip selection algorithm becomes what is known in probability theory as a “contextual multi-armed bandit problem”. A multi-dimensional feature vector is constructed from one or more of the audience, viewer and context data, and a machine learning algorithm—e.g. a Neural Bandit algorithm or a Bandit Forest algorithm—is used to learn how the feature vector and engagement metrics relate to each other.

At step S8, the selected candidate video clip is delivered to the triggering viewer device(s)116by the content distribution network114. According to embodiments described herein, a unique user identifier is generated for each viewer. As will be described in more detail below, the algorithmic editing server106records the output of the algorithm and the choice of video clip selected for that viewer so that subsequent engagement and context data can be attributed to that selection.

As discussed above, engagement data related to the particular viewer and the selected clip is captured from subsequent behaviour and events, specific to the particular medium (step S9). For example, in email, a “clickthru” event (used in aggregate to calculate click-to-open-rates for each video clip and which is a known online advertising performance metric representing an individual responding to an object by clicking on it) is captured from subsequent clicks on the video clip or surrounding content. The click is attributed to the chosen clip by embedding the unique user identifier into the clickthru link for each recipient of the email, to thereby allow the algorithmic engagement server106to associate the clickthru event to the specific recipient and the video clip that was previously selected for that recipient.

In addition, engagement data may include dwell duration, whether the viewer clicked on the video content, whether they executed a downstream event (e.g. placing a product in a cart or buying a product at checkout subsequent to viewing the presented video clip), or any other determinable downstream event representative of a user engagement with the video clip.

Finally, the engagement and context data is aggregated and stored by the algorithmic editing server106. In an embodiment, as soon as the aggregated engagement data is stored it can be dynamically applied to any new content requests, or can be used for dynamically adjusting an engagement system102response to any current viewer interactions (e.g. to adjust a downstream event triggered though a viewer watching a candidate video clip).

Further Detail of System Configuration

According to the illustrated embodiment, one form of viewer device116comprises a smartphone. It will be understood that the smartphone is equipped with the necessary hardware and software to communicate with the video clip distribution network114for receiving the video clips for display to the viewer. The smartphone and network114communicate over a mobile broadband network. Although not illustrated inFIG.1, the broadband network includes standard network elements including a base station controller, home location register, mobile switching centre, message centre, equipment identity register, and message gateway (although it will be appreciated that any suitable network connection may be utilised including private/public wireless networks, etc.).

As previously stated, the viewer device116should not be seen as being limited to any one type of device and can include any suitable computing device that is capable of receiving streaming video from the video clip distribution network114.

With additional reference toFIG.4, there is shown a typical configuration for a server computer suitable for implementing any one of the video analysis, clip editing and video serving capabilities (i.e. as performed by server computers104,106and110). The server computer400includes a processor410, memory412, storage device414, network device416, and power supply420. In one implementation, memory412is a volatile memory unit or units. In another implementation, memory412is a non-volatile memory unit or units. Memory412also can be another form of computer-readable medium (e.g., a magnetic or optical disk. Memory412may be non-transitory.

The storage device414can be any suitable form of device capable of providing mass storage. In one implementation, the storage device414can be or contain a computer-readable medium (e.g., a hard disk device, a flash memory or other similar solid-state memory device, or an array of devices, such as devices in a storage area network or other configurations.) A computer program product can be tangibly embodied in a data carrier. The computer program product also can contain instructions that, when executed, perform one or more methods (e.g., those described above.) The data carrier is a computer or machine-readable medium.

The processor410can execute instructions within the server computer400, including instructions stored in memory412. The processor410can be implemented as a chipset of chips that include separate and multiple analogue and digital processors. The processor410can provide, for example, for implementation of the various video analysis, algorithmic editing, proximity distance, or clip selection algorithms as described in preceding paragraphs.

It will be understood that the various server systems104,106,110,114need not be implemented as separate servers and certain or all functions performed by those servers could be integrated (e.g. such that, if desired, a single server could perform all functions). While the invention has been described with reference to the present embodiment, it will be understood by those skilled in the art that alterations, changes and improvements may be made and equivalents may be substituted for the elements thereof and steps thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt the invention to a particular situation or material to the teachings of the invention without departing from the central scope thereof. Such alterations, changes, modifications and improvements, though not expressly described above, are nevertheless intended and implied to be within the scope and spirit of the invention. Therefore, it is intended that the invention not be limited to the particular embodiment described herein and will include all embodiments described and envisioned herein.

In the claims which follow and in the preceding description of the invention, except where the context requires otherwise due to express language or necessary implication, the word “comprise” or variations such as “comprises” or “comprising” is used in an inclusive sense, i.e. to specify the presence of the stated features but not to preclude the presence or addition of further features in various embodiments of the invention.