Corroborating video data with audio data from video content to create section tagging

Systems and methods for tagging video content are disclosed. A method includes: receiving a video stream from a user computer device, the video stream including audio data and video data; determining a candidate audio tag based on analyzing the audio data; establishing an audio confidence score of the candidate audio tag based on the analyzing of the audio data; determining a candidate video tag based on analyzing the video data; establishing a video confidence score of the candidate video tag based on the analyzing of the video data; determining a correlation factor of the candidate audio tag relative to the candidate video tag; and assigning a tag to a portion in the video stream based on the correlation factor exceeding a correlation threshold value and at least one of the audio confidence score exceeding an audio threshold value and the video confidence score exceeding a video threshold value.

BACKGROUND

The present invention relates generally to tagging video content and, more particularly, to a system and method for corroborating video data with audio data to automatically tag video content.

With the advent of and sharing of live-stream video, user computer devices (e.g., mobile smartphones, tablets, etc.) may be used to record video of an event and may be used to tag specific portions of the video. For example, during a live-stream video recording of a sporting event, the user computer device may tag certain portions of the video recording (e.g., the players at the event) which contains information (e.g., the name of the player) about the video recording. Existing systems either require the user to create singular tags or generate tags that are not always relevant to the consumption of the video content.

SUMMARY

In an aspect of the invention, a computer implemented method includes: receiving, by a computer device, a video stream from a user computer device, the video stream comprising audio data and video data; determining, by the computer device, a candidate audio tag based on analyzing the audio data; establishing, by the computer device, an audio confidence score of the candidate audio tag based on the analyzing of the audio data; determining, by the computer device, a candidate video tag based on analyzing the video data; establishing, by the computer device, a video confidence score of the candidate video tag based on the analyzing of the video data; determining, by the computer device, a correlation factor of the candidate audio tag relative to the candidate video tag; and assigning, by the computer device, a tag to a portion in the video stream based on the correlation factor exceeding a correlation threshold value and at least one of the audio confidence score exceeding an audio threshold value and the video confidence score exceeding a video threshold value.

In another aspect of the invention, a computer program product comprises a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computing device to cause the computing device to receive a video stream from a user computer device, the video stream comprising audio data and video data; determine a candidate audio tag based on analyzing the audio data and determine a candidate video tag based on analyzing the video data; establish an audio confidence score of the candidate audio tag based on the analyzing of the audio data and a video confidence score of the candidate video tag based on the analyzing of the video data; determine a correlation factor between the candidate audio tag and the candidate video tag based on the audio confidence score relative to the video confidence score; and assign a tag to a portion in the video stream on the user computer device based on the correlation factor exceeding a correlation threshold value and at least one of the audio confidence score exceeding an audio threshold value and the video confidence score exceeding a video threshold value.

In another aspect of the invention, there is a system for verification of reports from sources. The system includes: a CPU, a computer readable memory and a computer readable storage medium associated with a mobile computer device program instructions to receive a video stream from a user computer device, the video stream comprising audio data and video data; program instructions to determine a candidate audio tag based on analyzing the audio data and upload audio information onto to an ontology database; program instructions to establish an audio confidence score of the candidate audio tag based on the analyzing of the audio data and downloading ontology information from the ontology database; program instructions to determine a candidate audio tag based on analyzing the video data and upload video information onto to an ontology database; program instructions to establish a video confidence score of the candidate video tag based on the analyzing of the video data and downloading ontology information from the ontology database; program instructions to determine a correlation factor of the candidate audio tag and the candidate video tag; and program instructions to assign a tag to a portion in the video stream based on the correlation factor exceeding a correlation threshold value and at least one of the audio confidence score exceeding an audio threshold value and the video confidence score exceeding a video threshold value.

DETAILED DESCRIPTION

The present invention relates generally to tagging video content and, more particularly, to a system and method for corroborating video data with audio data to automatically tag video content. Currently, significant difficulty exists in creating relevant, comprehensive and instantaneous tags for video content (e.g., during live-stream video recordings or previously recorded video content) without manual intervention from a user. Due to the significant quantity of untagged content that may be potentially relevant to the user, technologies that tag video content lack wide-ranging functionality. Aspects of the present invention provide a system for generating tags for video content of a video stream without user intervention.

Aspects of the present invention may process video content to provide comprehensive and instantaneous creation of relevant tags within the video content which will ensure higher level of user engagement during live-stream video recordings or previously recorded video content. Advantageously, the present invention provides a system for generating tags (e.g., segmentation tags or highlighted tags) for video content using natural language understanding (NLU) processing and image recognition processing without user intervention. By using NLU processing in cooperation with image recognition processing, tagging of certain (e.g., relevant) images within the video content during the progression of the video content may be comprehensive and instantaneous.

As described herein, embodiments utilize a correlation factor of the NLU processing relative to the image recognition processing to assign a tag to the video content. For example, as the video content progresses, a video content tagging device continually generates a correlation factor by comparing the NLU processing data for an ontology term and the image recognition data for the ontology term. Once the correlation factor exceeds a correlation threshold value, the video content tagging device assigns the tag in the video content relating to the ontology term.

Referring now toFIG. 1, a schematic of an example of a computing infrastructure is shown. Computing infrastructure10is only one example of a suitable computing infrastructure and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein. Regardless, computing infrastructure10is capable of being implemented and/or performing any of the functionality set forth hereinabove.

FIG. 2shows an exemplary environment in accordance with aspects of the invention. In embodiments, the present system uses a confidence score between NLU processing and image recognition processing to assign a tag to video content. For example, as the video content progresses, the video content tagging device continually generates an audio and video confidence score. Once the audio and video confidence score cross their respective threshold values, the video content tagging device assigns the tag to the video content. The video content tagging device de-assigns the tag to the video content when the audio and video confidence score falls below their respective threshold values.

As described herein, assigning the tag refers to the video content refers to highlighting a section of video content based on the NLU processing and the image processing of the video content. For example, assigning may refer to modifying an image of a car in a video image by placing a generic highlight, such as a box, around an image of a car. The assigned or highlighted section of the video content includes information (e.g., data, content information, Hypertext Markup Language HMTL information, etc.) that allows the assigned video content to be searchable, stored, and indexed.

In aspects, de-assigning the tag refers to removing the highlight of a section of video content based on the NLU processing and the image processing of the video content. For example, de-assigning may refer modifying the image of the car in the video image by removing the generic highlight, such as the box, around the image of the car.

The exemplary environment includes a video content tagging device60which may comprise a computer system12ofFIG. 1, and may be connected to a network50(e.g., via the network adapter20ofFIG. 1). The network50may be any suitable communication network or combination of networks, such as a local area network (LAN), a general wide area network (WAN), a public network (e.g., the Internet), a 3G network, a Long-Term Evolution (LTE) network, and/or a 5G network.

The video content tagging device60may comprise a plurality of modules configured to perform one or more functions described herein. In aspects, the video content tagging device60may include additional or fewer modules than those shown inFIG. 2. In aspects, separate components may be integrated into a single computing component or module. Additionally, or alternatively, a single module may be implemented as multiple computing components or modules (e.g., program modules42ofFIG. 1).

As described herein, the video content tagging device60includes a receiving module70, an audio analyzing module72, a video analyzing module74, a scoring module76, a correlation module77, a tagging module78, and a notification module80. In embodiments, the video content tagging device60is configured for communication via the network50with an ontology database100, a user computer device90, and a prospective user computer device98. The user computer device90and the prospective user computer device98may be, for example, personal digital assistants (PDA), cellular telephones, laptop computers, tablet computers, and/or computer systems that may communicate over the network50.

In embodiments, the ontology database100includes an ontology tree110which defines a representational foundational connection (e.g., ontologies) between different terms (e.g., nodes) of formal language. Ontologies are available for many concepts and are represented using the Web Ontology Language. Using the ontology database100, the video content tagging device60determines which portions or segments (e.g., cars, truck, flowers, etc.,) in video content to tag.

In aspects, the video content tagging device60communicates with the user computer device90via the network50. The user computer device90transmits a video stream92(e.g., a video recording) of an event98(e.g., a car show) to the video content tagging device60over the network50. The video stream92includes audio data94(e.g., audio data in the video recording of the car show) and video data96(e.g., video data in the video recording of the car show) of the event98for processing by the video content tagging device60.

Still referring toFIG. 2, the receiving module70in the video content tagging device60receives the video stream92from the user computer device90. For example, the receiving module70receives the video stream92of the car show from the user computer device90to be analyzed and tagged.

Based on receiving the video stream92, the audio analyzing module72analyzes the audio data94in the video stream92using NLU processing. The audio analyzing module72determines a candidate audio tag in the video stream92based on NLU processing of the audio data94. In some aspects, the audio analyzing module72may determine multiple candidate audio tags in the video stream92.

In aspects, the audio analyzing module72uses the NLU processing and determines that the audio data94includes a recitation of a specific type of car. Based on determining that the audio data94includes a recitation of the specific type of car, the audio analyzing module72determines that the specific type of car is the candidate audio tag in the audio data94.

For example, the analyzing module72uses the NLU processing on the audio data94in the video stream92to determine that audio data94refers to a specific type of car (e.g., a 1976 sedan) and that the 1976 sedan is the candidate audio tag. By way of another example, the analyzing module72uses the NLU processing on the audio data94in the video stream92to determine that audio data94refers to multiple types of cars (e.g., a 1976 sedan and a 1970 convertible) and that the 1976 sedan and the 1970 convertible are both candidate audio tags based on NLU processing of the audio data94.

Based on the audio analyzing module72analyzing the audio data94, the scoring module76establishes an audio confidence score of the candidate audio tag in the audio data94as a percentage score. In aspects, the audio confidence score may be based on an analysis of the audio data94(e.g., the frequency the audio data94refers to the candidate audio tag). The percentage scores, as described herein, established by the scoring module76are non-limiting examples and other confidence percentage scores may be established. The scoring module76continually uses NLU processing and updates the audio confidence score of the candidate audio tag on the audio data94as the video stream92progresses.

For example, based on the scoring module76determining the audio data94refers to a 1976 sedan, the audio analyzing module72may determine the audio confidence score of the candidate audio tag of the 1976 sedan, as a non-limiting example, is 95% (e.g., the frequency in the audio data94that the audio data94refers to the 1976 sedan). In another example, based on the scoring module76determining the audio data94refers to a 1956 sedan, the audio analyzing module72may determine the audio confidence score of the candidate audio tag of the 1956 sedan, as a non-limiting example, is 85% (e.g., the frequency in the audio data94that the audio data94refers to the 1956 sedan). However, based on the scoring module76determining the audio data94later in the video stream92refers to a 1976 sedan, the audio analyzing module72may determine the audio confidence score of the candidate audio tag of the 1976 sedan, as a non-limiting example, is 90% (e.g., the frequency in the audio data94that the audio data94refers to the 1976 sedan).

Based on establishing the audio confidence score for the candidate audio tag, the video analyzing module74analyzes the video data96using image recognition processing. The video analyzing module74determines a candidate video tag in the video stream92based on the analysis of the video data96. In some aspects, the video analyzing module may determine multiple candidate video tags in the video stream.

In aspects, the video analyzing module74uses the image recognition processing and determines that the video data96includes an image of a specific type of car. Based on determining that the video data96includes a image of the specific type of car, the video analyzing module74determines that the specific type of car is the candidate video tag in the video data96.

For example, the video analyzing module74uses image recognition processing of the video data96to determine that the video data96has an image of a specific type of car (e.g., the 1976 sedan). Based on determining that the video data96has the image of a specific type of car, the video analyzing module74determines that the image of a specific type of car is a candidate video tag (e.g., the 1976 sedan may be determined as the candidate video tag). In aspects, the video analyzing module74may determine that the video data96has an image of multiple types of cars (e.g., the 1976 sedan and the 1970 convertible) and determines that the video data96has multiple candidate video tags (e.g., both the 1976 sedan and the 1970 convertible may be determined as candidate video tags).

In accordance with other aspects, the video analyzing module74processes the video stream92at a frame rate level (e.g., scanning an index of all frames per second contained in the video stream92). Specifically, the video analyzing module74scans the individual frames (e.g., i-frame) and uses image recognition processing in determining the exact video content in each i-frame. As the number of i-frames processed increases, the certainty of the resulting percentage score of the candidate video tag increases.

Based on the video analyzing module74analyzing the video data96, the scoring module76establishes a video confidence score of the candidate video tag in the video data96as a percentage score. In aspects, the video confidence score may be based on an analysis of the video data96(e.g., the frequency for each i-frame of the video data96having an image of the candidate video tag). The percentage scores, as described herein, established by the scoring module76are non-limiting examples and other confidence percentage scores may be established. The scoring module76continually uses image recognition processing and updates the video confidence score of the candidate video tag on the video data96as the video stream92progresses.

For example, based on the scoring module76determining the video data96in the video stream92refers to the 1976 sedan, the video analyzing module74may determine the video confidence score of the candidate video tag, as a non-limiting example, is 95% (e.g., the frequency for each i-frame of the video data96having an image of the 1976 sedan). In another example, based on the scoring module76determining that the video data96refers to a 1956 sedan, the video analyzing module74may determine the video confidence score of the candidate video tag of the 1956 sedan, as a non-limiting example, is 85% (e.g., the frequency for each i-frame of the video data96having an image of the 1956 sedan). However, based on the scoring module76determining that the video data96later in the video stream92refers to a 1976 sedan, the video analyzing module74may determine the video confidence score of the candidate video tag of the 1976 sedan, as a non-limiting example, is 90% (e.g., the frequency for each i-frame of the video data96having an image of the 1976 sedan).

Based on the scoring module76establishing the audio confidence score for the candidate audio tag and the video confidence score for the candidate video tag, the correlation module77determines a correlation factor between the candidate audio tag and the candidate video tag. The correlation module77determines the correlation factor based on the audio confidence score (e.g., via the NLU processing) relative to the video confidence score (e.g., via the image recognition processing) as a percentage score.

For example, based on the audio analyzing module72determining the audio confidence score of a 1976 sedan in the video stream92is 94% and the video analyzing module74determining that the video confidence score of the 1976 sedan in the video stream92, as a non-limiting example, is 90%, the correlation module77may determine the correlation factor between the audio confidence score relative to the video confidence score, as a non-limiting example, is 95%. In another example, based on the audio analyzing module72determining the audio confidence score of a 1976 sedan in the video stream92, as a non-limiting example, is 70% and the video analyzing module74determining that the video confidence score of the 1976 sedan in the video stream92is 95%, the correlation module77may determine the correlation factor determined by the correlation module, as a non-limiting example, is 80%.

In aspects, based on establishing the audio confidence score, establishing the video confidence score and determining the correlation factor, the tagging module78of the video content tagging device60assigns a tag to a portion or segment in the video stream92. Specifically, the tagging module78tags the portion of the video stream92based on the correlation factor exceeding a correlation threshold value, and the audio confidence score exceeding an audio threshold value or the video confidence score exceeding a video threshold value. The audio threshold value, the video threshold value, and the correlation threshold value are predetermined values generated prior to processing by the video content tagging device60.

For example, the tagging module78assigns the tag to the portion of the video stream92which has an image of the 1976 sedan based on the correlation factor for the 1976 exceeding a correlation threshold value and the candidate audio tag for the 1976 sedan exceeding the audio threshold value or the candidate video tag for the 1976 exceeding the video threshold value. By way of another example, the tagging module78assigns tags to multiple portions (e.g., the 1976 sedan and a 1970 convertible) of the video stream92based on the correlation factor for the multiple candidate portions exceeding their respective correlation threshold values and the multiple candidate audio tags exceeding their respective audio threshold values, or the multiple candidate video tags exceeding their respective video threshold values.

In accordance with other aspects, the tagging module78de-assigns the tag to the portion in the video stream92based on the audio confidence score falling below the audio confidence threshold value, the video confidence score falling below the video confidence threshold value while the correlation factor experiences a delta (e.g., change) in the audio confidence score relative to the video confidence score. For example, when the audio confidence score of the 1976 sedan falls below the audio threshold value, the video confidence score of the 1976 sedan falls below the audio threshold value while the correlation factor of the audio confidence score of the 1976 sedan relative to the video confidence score of the 1976 sedan changes dramatically (e.g., delta), the tagging module78closes the tag to the 1976 sedan on the video stream92.

Based on the tagging module78assigning the tag to the portion in the video stream92, the notification module80may notify the prospective user computer device98that the tag has been assigned to the video stream92. The prospective user computer device98and other user devices (e.g., via the network50) may be informed by the notification module80that the tag has been assigned. In aspects, a prospective user may select notification options on the prospective user computer device98based on content on the video stream92being posted on websites.

For example, notification options are selected by the prospective user of the prospective user computer device98in order for the prospective user to be alerted when tags relevant to the prospective user are assigned to the video stream92and posted on a website. In particular, the prospective user computer device98may receive an automatic notification at the time of a broadcast of a particular tag during the event98for topics that are relevant to the prospective user.

To avoid overwhelming the prospective user computer device98, the notification module80analyses the tag assigned by the tagging module78using natural language processing (NLP). Based on the context of key words on the tag, the notification module80applies ontology-based algorithms on the data and metadata of the tag to find the appropriate semantical closest meaning to the keyword of the tag. In aspects, the ontology-based algorithms refer to NLP algorithms applied to an object ontology as described with reference to the ontology tree of the ontology database100. By using the ontology-based algorithms, the prospective user computer device98will receive notifications that are relevant to the prospective user and not receive notifications that are not relevant to the prospective user.

Based on prospective user selecting the notifications options on the prospective user computer device98and the notification module80applying the ontology-based algorithms on the tag assigned by the tagging module78, the prospective user computer device98will receive automatic notifications from the notification module80without the need for the prospective user to declare the precise scope of the notification. In other aspects, the prospective user computer device98may receive the notification time of the tag and may receive the video stream92starting from the specific moment the tagging module assigns the tag to the video stream92.

For example, based on the tagging module78assigning a tag (e.g., a tag to a 1976 sedan at a car event) to the video stream92at a certain time in the video stream92and the video stream92being posted on a website, the prospective user computer device98will receive a notification from the notification module80that the tagging module78has assigned the tag to the 1976 sedan at the car event. The prospective user computer device98may also receive the time the tagging module78assigned the tag for the 1976 sedan, and receive the video stream92of the car event starting at the certain time in which the tagging module78assigned the tag for the 1976 sedan.

In other aspects, the prospective user computer device98may not receive a notification from the notification module80when the tagging module78assigns a tag to 1960 truck at the car event, but may receive a notification when the tagging module78assigns a tag to a 1977 sedan at the car event. In yet other aspects, based selecting a notification option of receiving notifications when tag occurs only for the 1976 sedan, the user computer device90will receive a notification from the notification module80when the 1976 sedan is tagged and not when a 1970 sedan is tagged.

In aspects, to avoid overwhelming the processing system of the user computer device90, the notification module80may apply notification algorithms to the video stream92in order to find the appropriate semantical closest meaning to the user input of the notification options. The notification algorithms may be defined as ontology matching algorithms that use ontology based meaning of NLU processing and image recognition processing to determine the appropriate semantical closest meaning to the user input. In this manner, based on the context of the keyword inputted by the user, the user computer device90receives only notifications for interested topics.

FIG. 3depicts an exemplary ontology database100in accordance with aspects of the invention. The ontology database100includes the ontology tree110for terms of formal language, and defines the foundational connection between different terms of formal language. By way of example, the ontology tree110represents a hierarchal presentation of semantical concepts with a parent node (e.g., a generalized concept) compared to a child node (e.g., a specified concept). Each node (e.g., child and parent) has a lengthy textual description defining the precise meaning and additional structured attributes of the node such as data type, length, related concepts to the node, previous versions of the concepts of the node, etc.

As described herein,FIG. 3illustrates the ontology tree110as a graph of formal language which utilizes nodes to connect the individual terms of the formal language with solid or dashed lines. Every node of this graph stands for a concept which may be correspond to words or short phrases (e.g., noun or noun phrases). A solid line connecting two nodes represents an actual foundation connection of the formal language terms that have determined. The dotted line represents a possible foundational connection from a known node to unknown node(s).

For example, the ontology database100ofFIG. 3illustrates an ontology tree110for a vehicle. The vehicle node has a foundational connection to an un-motorized vehicle node and a motorized vehicle node. The un-motorized vehicle node has a foundational connection to a bicycle node. The bicycle node has a possible foundation connection (e.g., dotted line) to unknown nodes.

Still referring toFIG. 3, the motorized vehicle node has a foundation connection to a bus node, a car node, a truck node, and a motorbike node. The car node has a foundational connection to a convertible node, a sedan node, and to sports utility vehicle (SUV) node. The sedan node has a foundational connection to a Brand A node, a Brand B node, and a Brand C node. The Brand A node has a foundation connection to a 1964 year node, 1965 year node, and a 1966 year node. In aspects, the bus node, the truck node, the motorbike node, the 1964 year node, the 1965 year node, and the 1966 year node has a possible foundation connection (e.g., dotted line) to an unknown nodes.

In aspects, some of the modules (e.g., the audio analyzing module72, the video analyzing module74, and the scoring module76) in the video content tagging device60connect to the ontology database100and upload information (e.g., metadata, data, etc.,) to the ontology database100to create the ontology tree110and download information from the ontology database100to perform the action of the module. The audio analyzing module72performs analysis of the audio data94by using NLU processing on the audio data94and uploads audio information (e.g., metadata and the audio data94) to generate the individual nodes in the ontology tree110. The video analyzing module74performs analysis of the video data96by using image recognition processing on the video data96and uploads video information (e.g., metadata and the video data96) to generate video information for individual nodes in the ontology tree110.

In other aspects, the scoring module76and the correlation module77download ontology information (e.g., ontology data) from the ontology database100about individual nodes in the ontology tree110. The scoring module76uses the ontology information from the ontology database100to establish the audio confidence score and the video confidence score for the individual nodes of the ontology tree110. The correlation module77uses the ontology information from the ontology database100to determine the correlation factor for the individual nodes of the ontology tree110.

FIGS. 4-5show an exemplary use in accordance with aspects of the invention. Referring toFIG. 4, the user computer device90live streams a car show which includes multiple types of cars (e.g., a convertible424, a SUV422and a sedan420). On the video screen400(e.g., an individual frame of the video stream92) of the user computer device90, there is an image of an individual410(e.g., an image of a celebrity or a friend) who may narrate about the cars at the car show. On approaching the cars in the car show, the individual410begins narrating features of the cars or anecdotes about the cars (e.g., the year and make of the cars).

In aspects, a car ontology is established based on the narration of the individual, individual interest, or existing metadata. As the individual continues to narrate about the car, the video content tagging system defines the confidence in the individual's speech patterns relevant to the car and assigns a tag to the video content for the 1976 car when the confidence exceeds a threshold value.

As described herein, the video content tagging device60receives the video stream92of the narration by the individual410of the cars. Upon reception and analysis of the audio data94and the video data96in the video stream92, the audio analyzing module72and the video analyzing module74uploads information to an ontology database100. Running in the background, the ontology database100generates a car ontology which includes, but is not limited to, the type of the car, the brand of the car, the year of the car, etc.

As the individual410continues to narrate about a specific type of car, such as the convertible424, the audio analyzing module72and the video analyzing module74creates the candidate audio tag and the candidate video tag for the convertible424. The scoring module76establishes an audio confidence score (e.g., variable S) and a video confidence score (e.g. variable I) for the convertible424.

Additionally, the scoring module76marks two time variables: timeS (TS) which represents the first instance of NLU processing of the audio data94for convertible424, and timeI (TI) which represents the first instance of image recognition processing of the video data96for the convertible424. The correlation module77determines the correlation factor (e.g., variable C) for the convertible424.

Each variable (e.g., S, I, and C) has a respective confidence threshold value (e.g., audio confidence threshold value (TS), video confidence threshold value (TI), and correlation threshold value (TC) for the convertible424). The tagging module78assigns a convertible tag440for the convertible424according to equation 1:
if {C≥TC&&S≥TS} OR if {C≥TC&&I≥TI}=>assign tag at MIN(timeS,timeI).

Equation 1 describes that based on the correlation factor (C) exceeding the correlation threshold value (TC), and the audio confidence score (S) exceeding the audio confidence threshold value (TS) or the video confidence score (I) exceeding the video confidence threshold value (TI), the tagging module78assigns the tag for the particular portion of the video content in the video stream92at the earliest timeS or timeI. For example, the tagging module78assigns the convertible tag440for the convertible424based on the correlation factor (C) for the convertible424exceeding the correlation threshold value (TC) and the audio confidence score (S) or the video confidence score (I) for the convertible424exceeding their respective threshold value (e.g., TS or TI). The time in which tagging module78assigns the convertible tag440at the instance when either the audio confidence score (S) or the video confidence score (I) exceed their respective threshold value, TS or TI, whichever is earliest.

In aspects, the convertible tag440remains assigned as long the audio confidence score (S) for the convertible424exceeds the audio confidence threshold value (TS) or the video confidence score (I) for the convertible424exceeds the video confidence threshold value (TS). For example, during the interruption, the narrating individual may refer to the convertible424by a nickname. In this case, the tagging module continues to assign the tag to the convertible424as the narrating individual continues to narrate about the convertible424by the nickname.

For example, based on the audio confidence score (S) of the convertible424falling below the audio confidence threshold value (TS) and the video confidence score (I) of the convertible424falling below the video confidence threshold value (TI) while the correlation factor (C) experiences a delta (e.g., a change in the audio confidence score (S) relative to the video confidence score (I) of the convertible424), the tagging module78de-assigns the convertible tag440at the last of time EndS or EndI. In aspects, the tagging module saves the assignment time and the de-assignment time of the convertible tag440for the convertible424.

Referring toFIG. 5, during the middle of the narration of the car show, another individual approaches the individual410performing the narration in mid-sentence. Due to the interruption by the other individual, the individual410stops narrating about the convertible424and responds to the other individual.

By way of example, based on the individual410continuing to narrate about cars at the car show and the audio confidence score (S) for the convertible424exceeding the audio confidence threshold value (TS) or the video confidence score (I) for the convertible424exceeding the video confidence threshold value (TS), the tagging module78may continue to assign the tag to the convertible tag440. By way of another example, based the individual410narrating about other types of cars (e.g., a sedan420) at the car show after the interruption, the tagging module78may de-assign the convertible tag440and may assign a sedan tag540for sedan420. By way of yet another example, based on the individual410narrating about all types of cars (e.g., the convertible424and the sedan420) at the car show after the interruption, the tagging module78may continue to assign the convertible tag440and, in addition, may assign a sedan tag540.

FIG. 6illustrates a flowchart of steps of a method in accordance with aspects of the invention. At step600, the video content tagging device60receives a video stream92from the user computer device90. In aspects, the video stream92is about an event98. For example, as described with respect toFIG. 2, the receiving module70receives live-stream recording of a car event from the user computer device90.

At step602, the video content tagging device60analyzes the audio data94in the video stream92received from the user computer device90. In aspects, based on receiving the video stream92from the user computer device90at step600, the audio analyzing module72analyzes the audio data94of the video stream92to determine a candidate audio tag in the video stream92.

For example, as described with reference toFIG. 2, the audio analyzing module72analyzes the live-stream recording video of the car event to determine candidate audio tags for specific types of cars (e.g., sport utility vehicles, sedans, convertibles, etc.) which are at the car event or other relevant objects at the car event. The audio analyzing module72analyses the live-stream recording of the car event by using NLU processing on the audio data94of the live-stream recording.

At step604, the video content tagging device60establishes an audio confidence score of the candidate audio tag in the audio data94received from the user computer device90. In aspects, based on analyzing the audio data94in the video stream92at step602, the scoring module76establishes the audio confidence score of the candidate audio tag in the audio data94received from the user computer device90.

For example, as described with reference toFIG. 2, the scoring module76establishes the audio confidence score of the specific types of cars at the car event based on the analysis of the video data96of the video stream92. The scoring module76uses NLU processing and updates the audio confidence score of the candidate audio tag as the video stream92progresses.

At step606, the video content tagging device60analyzes the video data96in the video stream92received from the user computer device90. In aspects, based on establishing the audio confidence score of the candidate audio tag in the audio data94at step604, the video analyzing module74analyzes the video data96of the video stream92to determine a candidate video tag in the video stream92.

For example, as described with reference toFIG. 2, the video analyzing module74analyzes the live-stream recording of the car event to determine candidate video tags for specific types of cars (e.g., sport utility vehicles, sedans, convertibles, etc.) which are at the car event. The video analyzing module74analyses the live-stream recording of the car event by using image recognition processing on the video data96of the live-stream recording.

At step608, the video content tagging device60establishes a video confidence score of the candidate video tag in the video data96received from the user computer device90. In aspects, based on analyzing the video data96in the video stream92at step606, the scoring module76establishes the video confidence score of the candidate video tag in the video data96received from the user computer device90.

For example, as described with reference toFIG. 2, the scoring module76establishes the video confidence score of the specific types of cars at the car event based on the analysis of the video data96of the video stream92. The scoring module76uses image recognition processing and updates the video confidence score of the candidate video tag as the video stream92progresses.

At step610, the video content tagging device60determines a correlation factor between the candidate audio tag and the video audio tag. In aspects, based on establishing the audio confidence score of the candidate audio tag at step604and the establishing the video confidence score of the candidate audio tag at step608, the correlation module determines a correlation factor. For example, based on the audio confidence score the specific types of cars being high percentage value and the video confidence score of the specific types of cars being high percentage value, the correlation factor for the specific types of cars may be a high percentage value.

At step612, the video content tagging device60assigns a tag to a portion in the video stream92received from the user computer device90. In aspects, based on establishing the audio confidence score at step604, establishing the video confidence score at step608, and determining the correlation factor at610, the tagging module78assigns the tag to the portion in the video stream92.

In aspects, the tagging module78assigns the tag based on the correlation factor exceeding a correlation threshold value, and the audio confidence score exceeding an audio threshold value or the video confidence score exceeding a video threshold value. The tagging module78assigns the tag to the portion in the video stream92in which the candidate audio tag and the candidate video tag has been established. For example, as described with reference toFIG. 2, based on the correlation factor exceeding the correlation threshold value for a specific type of car, and the audio confidence score exceeding the audio threshold value for the specific type of car or the video confidence score exceeding the video threshold value for the specific type, the tagging module78assigns a tag to the specific type of car (e.g., sport utility vehicles, sedans, convertibles, etc.) in the video stream92.

At step614, the video content tagging device60notifies the prospective user device98that the tagging module78assigned a tag to the video stream92. In aspects, based on the tagging module78assigning the tag and the video being posted on a website, the notification module80notifies the prospective user computer device98. For example, based on the tagging module78assigning a tag to a specific type of car, the prospective user computer device98may receive a notification from the notification module80that the tagging module78has assigned the tag to the specific type of car. In other aspects, the prospective user computer device98may receive the notification time of the tag of the specific type of car and may receive the video stream92starting from the specific moment the tagging module assigns the tag to the specific type of car in the video stream92.