Generating and providing topic visual elements based on audio content and video content of a digital video

The present disclosure relates to methods, systems, and non-transitory computer-readable media for generating a topic visual element for a portion of a digital video based on audio content and visual content of the digital video. For example, the disclosed systems can generate a map between words of the audio content and their corresponding timestamps from the digital video and then modify the map by associating importance weights with one or more of the words. Further, the disclosed systems can generate an additional map by associating words embedded in one or more video frames of the visual content with their corresponding timestamps. Based on these maps, the disclosed systems can identify a topic for a portion of the digital video (e.g., a portion currently previewed on a computing device), generate a topic visual element that includes the topic, and provide the topic visual element for display on a computing device.

BACKGROUND

Recent years have seen significant advancements in hardware and software platforms for presenting digital videos to users. For example, many conventional video presentation systems implement a platform accessible by users via a client device in order to upload, search for, interact with, comment on, and/or view digital videos. Such conventional systems often present, along with a digital video, information about the video (e.g., a title, a brief description, etc.) to inform a user of the contents of the video. In many cases, as a cursor hovers over a scrub bar (i.e., a seek bar or video progress bar) associated with a digital video, a conventional system may provide a preview thumbnail depicting a scene associated with the position of the cursor. Accordingly, the conventional system enables the user to preview the contents of a portion of the digital video based on the provided preview thumbnails. Although conventional systems can provide preview thumbnails for informing users, such systems often fail to flexibly provide additional detail when a preview thumbnail is uninformative, resulting in an inaccurate portrayal of the contents of a portion of the digital video and wasted resources due to inefficient video navigation.

SUMMARY

One or more embodiments described herein include systems, methods, and non-transitory computer-readable media that generate video topic summary elements based on the audio content and visual content of a digital video. The system can provide the video topic summary elements in connection with video thumbnails or as part of a table or contents to allow for informed efficient navigation and exploration of digital videos. For example, in one or more embodiments, the system converts the audio of the digital video to text and indexes the converted text with the frames of the digital video. The system can further determine an importance of words of the converted text. Additionally, the system can analyze the visual content of the digital video to extract words embedded in the video frames. The system can then map words from the converted text with high importance to the extracted words to identify a topic for a portion of the digital video. The system can then provide a visual element (e.g., a textual representation) for the topic, for example, when a user hovers a cursor over a position of a scrub bar associated with the portion of the digital video. In this manner, the system can flexibly generate visual elements that accurately convey the topic associated with a given portion of a digital video.

DETAILED DESCRIPTION

One or more embodiments described herein include a video topic generation system that generates visual elements that accurately convey the topic of a given portion of a digital video based on the audio content and visual content of the digital video. For example, in one or more embodiments, the video topic generation system converts the audio of the digital video to text and indexes the converted text with the frames of the digital video. The video topic generation system can further determine an importance of words of the converted text. Additionally, the video topic generation system can analyze the visual content of the digital video to extract words embedded in the video frames. The video topic generation system can then map words from the converted text with high importance to the extracted words to identify a topic for a portion of the digital video.

To provide an example, in one or more embodiments, the video topic generation system identifies a digital video that includes visual content and audio content. The video topic generation system can generate a transcript map based on the audio content by associating words of the audio content with timestamps of the digital video. The video topic generation system can further modify the transcript map by determining and associating importance weights with the words of the audio content. Additionally, the video topic generation system can generate an embedded text map based on the visual content by associating embedded words from video frames of the digital video with corresponding timestamps. Based on the modified transcript map and the embedded text map, the video topic generation system can generate a topic visual element for a portion of the digital video and provide the visual topic element for display on a computing device.

As just mentioned, in one or more embodiments, the video topic generation system generates a topic visual element based in part on audio content of a digital video. Indeed, in one or more embodiments, the video topic generation system uses speech-to-text conversion to generate text corresponding to the words of the audio content. The video topic generation system can generate a transcript map by associating the words of the audio content (i.e., the text representing the words) with timestamps of the digital video. In particular, for a given word, the video topic generation system can associate that word with the timestamp of the digital video at which the audio of the word was provided.

The video topic generation system can further generate quantitative values representing the importance of the words of the audio content (i.e., importance weights). For example, in one or more embodiments, the video topic generation system analyzes the transcript map to determine a frequency of the words of the audio content using a term frequency-inverse document frequency model. The video topic generation system can then associate importance weights with the words based on the frequency. For example, in one or more embodiments, based on the frequency of the words (e.g., for words that satisfy a frequency threshold), the video topic generation system utilizes a summarization machine learning model to generate importance weights for the words. The video topic generation system can modify the transcript map by associating the importance weights with those words within the transcript map.

As also mentioned, in one or more embodiments, the video topic generation system generates the topic visual element further based on the visual content of the digital video. In particular, the video topic generation system can analyze the visual content to identify and extract words that are embedded in (e.g., displayed in) the video frames of the digital video. In one or more embodiments, the video topic generation system analyzes the visual content using optical character recognition. The video topic generation system can generate an embedded text map by associating the embedded words from the video frames to their corresponding timestamps.

As further mentioned above, in one or more embodiments, the video topic generation system generates a topic visual element for a portion of the digital video based on the modified transcript map and the embedded text map. In particular, for a give portion of the video, the video topic generation system can identify a match between a word from the modified transcript map and a word from the embedded text map. In one or more embodiments, the video topic generation system identifies the match by comparing words from the modified transcript map that correspond to the portion of the digital video and have a high importance weight to words from the embedded text map that correspond to the portion of the digital video. The video topic generation system can generate the topic visual element based on the identified match(es).

In one or more embodiments, upon identifying multiple matches, the video topic generation system can determine a subtopic for the portion of the digital video. In particular, the video topic generation can assign the matching word with the highest importance weight as the topic and the matching word with the next highest importance weight as the subtopic. Accordingly, the video topic generation system can generate the topic visual element to include the topic and the subtopic.

In some embodiments, the video topic generation may not identify any matches, for a give portion of the video, between words from the modified transcript map and words from the embedded text map. In such embodiments, the video topic generation system can generate a topic based on a top-weighted word from the modified transcript map that corresponds to the portion of the digital video.

In one or more embodiments, the video topic generation system generates the topic visual element for a portion of a digital video in real time. For example, the video topic generation system can generate the topic visual element in response to detecting that an input indicator (e.g., a cursor or touch gesture) is positioned at a particular point on a scrub bar corresponding to the digital video. Based on the particular point of the scrub bar, the video topic generation system can identify a corresponding range of times stamps. The video topic generation system can then determine a topic for the portion of the video corresponding to the range of time stamps. The video topic generation system can provide a topic visual element for display while the input indicator is positioned at the point of the scrub bar associated with the portion of the digital video. In some embodiments, however, the video topic generation system generates the topic visual element at some time before a user views the digital video.

In some embodiments, the video topic generation system generates a table of contents that includes the topics (and subtopics, if applicable) of the digital video. The video topic generation system can provide the table of contents for display, for instance, as an initial frame of the digital video. In one or more embodiments, the video topic generation system further configures the table of contents so that a user can navigate the digital video upon selecting one of the included topics (or subtopics).

As mentioned above, conventional video presentation systems suffer from several technological shortcomings that result in inflexible, inaccurate, and inefficient operation. For example, conventional video presentation systems typically rigidly provide preview thumbnails that include a visual representation a frame of the video. As these thumbnails are small they make it difficult to determine the theme or topic of the video. These shortcomings are exacerbated when the digital video does not include a significant change in scenery across large portions of the video (e.g., where the digital video presents an academic lecture or other presentation where a speaker stands at a podium throughout the duration of the digital video). Thus, a user attempting to navigate the digital video using the scrub bar may be presented with various preview thumbnails that look similar and do not allow the user to determine a topic at a given point in the video.

Some conventional video presentation systems attempt to address these concerns by providing URLs (e.g., within a description area for the digital video) that enable a user to navigate to a desired portion of the digital video. Some conventional systems provide tags (e.g., within the digital video itself) to inform the user of the current topic of the digital video. However, these methods often require a user (e.g., the creator of the digital video) to manually create and insert the URLs or tags. Further, the conventional systems often rigidly limit the availability of these approaches to the time-of-creation of the digital videos. In other words, conventional systems may prevent a user from providing URLs or tags to a preexisting digital video.

In addition to the foregoing problems, conventional video presentation systems can also operate inefficiently. In particular, as conventional systems typically provide inefficient means for navigating or exploring the digital video. For example, without an accurate portrayal of the topics, the conventional systems typically require a user to watch the entire digital video or to navigate to various portions of the digital video aimlessly before finding the portion that discusses a desired topic. Accordingly, such conventional systems often require a significant amount of computing resources to navigate or find a desired portion of a digital video.

The video topic generation system provides several advantages over conventional systems. For example, the video topic generation system can operate more flexibly than conventional systems. In particular, by generating and providing topic visual elements based on the audio content and visual content of a digital video, the video topic generation system can flexibly convey the topic for a given portion where the scenes of the digital video are largely similar and the preview thumbnails do not change significantly. Additionally, by generating topic visual elements based on the audio content and visual content of digital videos, the video topic generation system can flexibly provide the topic visual elements without requiring manual user input. Further, as mentioned above, the video topic generation system can flexibly generate a topic visual element for a portion of a digital video after the creation of the digital video (e.g., in real time, as a user interacts with the scrub bar).

Further, the video topic generation system can operate more accurately than conventional systems. Indeed, by generating and providing topic visual elements, the video topic generation system can accurately portray the topic for a given portion of the digital video. For example, the video topic generation system can accurately convey at what point the digital video changes topics and the substance of those topics even when the preview thumbnails for the digital video look similar. Additionally, by providing topic visual elements the video topic generation system avoids the ambiguity that may be present within the visual image of the preview thumbnail.

Additionally, the video topic generation system improves efficiency. Indeed, by accurately portraying the topic for a given portion of a digital video via a topic visual element, the video topic generation system enables a user to navigate to a desired portion of the digital video without having to watch large portions (or even the entire video). Accordingly, the video topic generation system reduces the amount of computing time and power used in navigating digital videos.

The video topic generation system can generate topics and topic visual elements for a digital video. A digital video can include visual content (visual data) and audio content (audio data). In some embodiments, a digital video includes other data (e.g., metadata) or features for viewing or otherwise interacting with the visual content and/or audio content. Indeed, in some embodiments, a digital video includes a digital file that contains the visual content, audio content and/or other data.

In one or more embodiments, the video topic generation system can generate topics and topic visual elements from visual content and audio content. Visual content can include digital data representing the frames of the digital video. In some embodiments, the visual content includes one or more digitally-captured or digitally-reproduced visual elements of a digital video. In some embodiments, visual content includes one or more digitally-rendered visual elements, such as visual elements generated using computer-generated imagery.

In one or more embodiments, audio content can include digital data associated with noise or sound. In some embodiments, audio content includes digital data representing words or other discernible noises generally understood by humans as spoken communication. For example, audio content can include digital data representing words (or noises) vocalized by a person, presented via a communications device, presented via audio recording, or generated and presented by a machine. In one or more embodiments, audio content includes digital data representing an audio channel of a digital video.

Additionally, in one or more embodiments, a portion of a digital video includes a segment of the digital video. Indeed, a portion of a digital video can include a segment of the digital video that is shorter in length than the entirety of the digital video. To illustrate, a portion of a digital video can include, but is not limited to, a segment of the digital video delineated by a scene of the digital video, a window of time within the digital video (e.g., a range of timestamps associated with the digital video), or a topic or theme discussed or otherwise portrayed by the digital video. In some embodiments, however, a portion of a digital video can include the entirety of the digital video.

Additionally, in one or more embodiments, a scrub bar includes a visual element representing a time element of a digital video. In some embodiments, a scrub bar includes a progress bar that indicates the current progress of a digital video (e.g., how much of the digital video has played). Indeed, a scrub bar can update as the digital video plays. In some embodiments, however, a scrub bar includes a visual element representing a portion of the digital video that is currently playing relative to the time frame of the digital video. Indeed, the scrub bar can update as a user navigates to different portions of the digital video. The scrub bar can be part of the digital video itself (e.g., included within the digital data corresponding to the digital video). In some instances, the scrub bar is provided by a video presentation system.

Further, in one or more embodiments, a topic visual element includes a visual element portraying a topic or theme of a portion of a digital video. For example, a topic visual element can include a user interface element that displays (e.g., textually or symbolically) a topic of a portion of a digital video. In one or more embodiments, the topic visual element can further display one or more subtopics corresponding to the topic of the digital video.

In one or more embodiments, an input indicator includes a visual element used to indicate a position of input or potential input within a user interface. For example, an input indicator can include a pointer or cursor (e.g., an arrow or finger) displayed on a user interface to show the placement of input or potential input. An input indicator can be associated with various methods of input that include, but are not limited to, an input peripheral (e.g., a mouse, a keyboard, a joystick, a stylus), a touch gesture, eye tracking, body movement, or voice command.

Additionally, a preview thumbnail (also referred to as a scrub-bar-preview thumbnail or simply thumbnail) includes a visual element representing a portion of a digital video. For example, in one or more embodiments, a preview thumbnail includes a graphical user interface element that displays a video frame associated with a portion of a digital video. In some embodiments, the digital video (or a video presentation system) displays the preview thumbnail in response to detecting an input indicator at a point of a scrub bar associated with the corresponding portion of the digital video.

In one or more embodiments, a transcript map includes an association between characteristics of audio content. For example, in some embodiments, a transcript map includes a mapping between words of the audio content and timestamps of the audio content corresponding to those words (e.g., indicating the time of the video at which the words were spoken or otherwise produced). A transcript map can include a data structure (e.g., a table or vector) that stores the mapping but is not so limited. For instance, a transcript map can include one or more pointers in memory associating stored characteristics of the audio content (e.g., a pointer associated with a word that points to a memory location storing the corresponding timestamp).

In one or more embodiments, an embedded word includes a word displayed by a digital video. In some embodiments, an embedded word includes a word having a visual representation included in the visual content of a digital video. For example, an embedded word can include a word having a textual representation (e.g., the spelling of the word or an abbreviation of the word) or a symbolic representation displayed within one or more video frames of the digital video.

In one or more embodiments, an embedded text map includes an association between characteristics of visual content. For example, in some embodiments, an embedded text map includes a mapping between a word from the visual content (e.g., an embedded word) and a timestamp of the video corresponding to that word (e.g., indicating the time of the video at which the word was displayed). An embedded text map can include a data structure (e.g., a table or vector) that stores the mapping but is not so limited. For instance, an embedded text map can include one or more pointers in memory associating stored characteristics of the visual content (e.g., a pointer associated with an embedded word that points to a memory location storing the corresponding timestamp).

In one or more embodiments, a timestamp includes an indication of time. Indeed, in some embodiments, a timestamp includes a time of an occurrence of a digital video relative to the time frame of the digital video (e.g., where an initializing timestamp corresponds to the beginning of the digital video). For example, a timestamp can indicate a time at which a particular video frame is displayed or particular audio content is provided. A timestamp can be measured using various intervals, such as minutes, seconds, milliseconds, etc.

Additionally, in one or more embodiments, a term frequency-inverse document frequency model includes a computer algorithm or model that analyzes the frequency with which a word occurs within a segment of text (e.g., document or transcript map). For example, a term frequency-inverse document frequency model can analyze a segment of text to determine a frequency with which a word appears in the segment of text. In one or more embodiments, the term frequency-inverse document frequency model offsets the frequency value (e.g., using a weighting factor) based on the appearance of the word in one or more other segments of text (e.g., other documents). A term frequency-inverse document frequency model can represent the frequency with which a word appears using integers or as a fraction or percentage (offset by the appearance of the word in one or more other documents).

Further, in one or more embodiments, a summarization machine learning model includes a computer algorithm or model that generates a value that indicates the importance of a word within a segment of text. For example, a summarization machine learning model can include a computer algorithm that is trained to generate importance weights for words based on training words or training groups of words (e.g., training sentences or other segments of text used for training). After training, the summarization machine learning model can analyze a word and assign the word an importance weight. To illustrate, a summarization machine learning model can include a machine learning model. In particular, a summarization machine learning model can include a classifier.

Moreover, in one or more embodiments, a machine learning model includes a computer representation or an algorithm that can be tuned (e.g., trained) based on inputs to approximate unknown functions. In particular, a machine learning model can include to a model that utilizes algorithms to learn from, and make predictions on, known data by analyzing the known data to learn to generate outputs that reflect patterns and attributes of the known data. For instance, a machine learning model can include, but is not limited to, a neural network (e.g., a convolutional neural network and/or deep learning neural network), decision tree, association rule learning, inductive logic programming, support vector learning, Bayesian network, regression-based model, principal component analysis, and/or a combination thereof.

In one or more embodiments, an importance weight includes an output generated by a summarization machine learning model. For example, in some embodiments, an importance weight includes a value or set of values indicative of an importance or relevance of a word. An importance weight can include a single value, a vector of values, or a matrix of values. In one or more embodiments, a top-weighted word includes a word from a set of words that has an importance weight that is higher than, or at least equal to, the importance weight of the other words from the set of words.

Further, in one or more embodiments, a table of contents includes a set of topics. For example, in some embodiments, a table of contents includes an organized list showing a progression of topics associated with a digital video. In one or more embodiments, a table of contents also includes one or more subtopics that are relevant to a given topic.

Additional detail regarding the video topic generation system will now be provided with reference to the figures. For example,FIG. 1illustrates a schematic diagram of an exemplary system100in which a video topic generation system106can be implemented. As illustrated inFIG. 1, the system100can include a server(s)102, a network108, client devices110a-110n, and a third-party system114.

Although the system100ofFIG. 1is depicted as having a particular number of components, the system100can have any number of additional or alternative components (e.g., any number of servers, client devices, third-party systems, or other components in communication with the video topic generation system106via the network108). Similarly, althoughFIG. 1illustrates a particular arrangement of the server(s)102, the network108, the client devices110a-110n, and the third-party system114, various additional arrangements are possible.

The server(s)102, the network108, the client devices110a-110n, and the third-party system114may be communicatively coupled with each other either directly or indirectly (e.g., through the network108discussed in greater detail below in relation toFIG. 11). Moreover, the server(s)102, the client devices110a-110n, and the third-party system114may include a variety of computing devices (including one or more computing devices as discussed in greater detail with relation toFIG. 11).

As mentioned above, the system100includes the server(s)102. The server(s)102can generate, store, receive, and/or transmit data, including topic visual elements. For example, the server(s)102can identify a digital video (e.g., a digital video hosted on the third-party system114) accessed by the client device110aand transmit a topic visual element (e.g., either to the third-party system114or directly to the client device110a) for display on the client device110a. In one or more embodiments, the server(s)102comprises a data server. The server(s)102can also comprise a communication server or a web-hosting server.

As shown inFIG. 1, the server(s)102can include a video presentation system104. In particular, the video presentation system104can provide functionality by which a user (e.g., a user of one of the client devices110a-110n) can access, view, search for, and/or interact with digital videos. For example, a client device can access a digital video stored on the video presentation system104or access, via the video presentation system104, a digital video stored on the third-party system114. In response to determining that the client device has accessed the digital video, the video presentation system104can provide options that a user of the client device can use to view and/or interact with the digital video. The video presentation system104can further provide options the user can use to search for one or more additional digital videos.

Additionally, the server(s)102include the video topic generation system106. In particular, in one or more embodiments, the video topic generation system106utilizes the server(s)102to generate topic visual elements corresponding to portions of digital videos. For example, the video topic generation system106can utilize the server(s)102to identify a digital video accessed by a client device (e.g., one of the client devices110a-110n) and then provide, to the client device, a topic visual element corresponding to a portion of the digital video.

For example, in one or more embodiments, the video topic generation system106, via the server(s)102, identifies a digital video (e.g., accessed by a client device) that includes audio content and visual content. The video topic generation system106can, via the server(s)102, generate a transcript map based on the audio content by associating words of the audio content with timestamps of the digital video. The video topic generation system106can further modify the transcript map by associating importance weights with the words of the audio content. The video topic generation system106can further generate an embedded text map, via the server(s)102, based on the visual content by associating embedded words from video frames of the digital video with corresponding timestamps. Via the server(s)102, the video topic generation system106can generate a visual topic element based on the modified transcript map and the embedded text map and provide the topic visual element for display with the digital video on the client device.

In one or more embodiments, the third-party system114stores and/or presents digital videos. For example, in one or more embodiments, the third-party system114can be accessed by a client device (e.g., one of the client devices110a-110n) and can provide, in response, functionality by which the client device can access, view, search for, and/or interact with digital videos. In other words, in one or more embodiments, the third-party system114operates similarly to the video presentation system104. In some embodiments, the third-party system114operates as a digital video database, and a client device can access digital videos stored on the third-party system114via the video presentation system104.

In one or more embodiments, the client devices110a-110ninclude computing devices that can access and display digital videos, as well as topic visual elements. For example, the client devices110a-110ncan include smartphones, tablets, desktop computers, laptop computers, or other electronic devices. The client devices110a-110ncan include one or more applications (e.g., the client application112) that can access and display digital videos, as well as topic visual elements. For example, the client application112can include a software application installed on the client devices110a-110n. Additionally, or alternatively, the client application112can include a software application hosted on the server(s)102, which may be accessed by the client devices110a-110nthrough another application, such as a web browser.

The video topic generation system106can be implemented in whole, or in part, by the individual elements of the system100. Indeed, althoughFIG. 1illustrates the video topic generation system106implemented with regard to the server(s)102, different components of the video topic generation system106can be implemented in a variety of the components of the system100. For example, one or more components of the video topic generation system106—including all components of the video topic generation system106—can be implemented by a computing device (e.g., one of the client devices110a-110n). Example components of the video topic generation system106will be discussed in more detail below with regard toFIG. 9.

As mentioned above, the video topic generation system106can generate a topic visual element for a portion of a digital video.FIG. 2illustrates a block diagram of the video topic generation system106generating a topic visual element214in accordance with one or more embodiments. While the discussion regardingFIG. 2(as well as many of the subsequent figures) may discuss the digital video202including or displaying certain features (e.g., a scrub bar, a digital video indicator, play button, a volume button, a preview thumbnail) it will be understood that, in one or more embodiments, some or all of these features are provided by a video presentation system (e.g., the video presentation system104or the third-party system114).

As illustrated inFIG. 2, the video topic generation system106identifies a digital video202. In one or more embodiments, the video topic generation system106identifies the digital video202by receiving the digital video202from an external source, such as a computing device. In some embodiments, the video topic generation system106identifies the digital video202from a database storing digital videos. In still further embodiments, the video topic generation system106identifies the digital video202upon detecting access of the digital video202. For example, the video topic generation system106can store the digital video202in a database and detect when a computing device accesses the digital video202. In some embodiments, a third-party system stores the digital video202, and the video topic generation system106detects when a computing device accesses the digital video202via the third-party system.

As shown inFIG. 2, the digital video202displays a video frame204. Indeed, the video frame204represents a video frame of the digital video202currently displayed as part of a portion of the digital video202displayed while a computing device plays the digital video202or in response to the computing device pausing the digital video202. Further, the digital video202includes a digital video indicator206and a scrub bar208. The digital video indicator206can move across the scrub bar208while the computing device plays the digital video202or as a user navigates the digital video202to indicate the portion of the digital video202that is currently playing. Indeed, as shown inFIG. 2, the digital video indicator206is associated with the video frame204, indicating the point of the digital video202at which the video frame204occurs relative to the time frame of the digital video202.

As further shown inFIG. 2, the digital video202includes the preview thumbnail210. In particular, the preview thumbnail210displays a video frame of the digital video corresponding to a portion of the digital video202associated with a position of an input indicator212at the scrub bar208. Thus, the preview thumbnail210can provide a preview of the contents of the digital video202based on a position of the input indicator212at the scrub bar208. Indeed, the preview thumbnail210can provide a preview of a portion of the digital video202that occurs before or after a currently displayed portion of the digital video202. The preview thumbnail210can also provide a preview of the currently displayed portion of the digital video202, such as when the input indicator212is positioned at or near the digital video indicator206.

As illustrated inFIG. 2, the video topic generation system106can analyze the digital video202to generate and provide a topic visual element214. Indeed, as will be discussed in more detail below, the video topic generation system106can analyze audio content and visual content of the digital video202in order to generate the topic visual element214. The topic visual element214displays the topic of a portion of the digital video202. In particular, the topic visual element214displays the topic of the portion of the digital video202corresponding to the preview thumbnail210.

By generating and providing topic visual elements, the video topic generation system106can provide a clear representation of the topic associated with that portion of the digital video202. Indeed, the video topic generation system106can improve upon the inaccuracies of conventional systems by more clearly conveying, via a topic visual element, the contents of a portion of the digital video where the preview thumbnails representing the contents of the digital video may look the same. By improving upon the accuracy of conveying the contents of a digital video, the video topic generation system106can further improve upon the inefficiencies of conventional systems. Indeed, the video topic generation system106improves the navigability of a digital video, reducing the amount of computing resources and/or time required to do so.

As mentioned above, the video topic generation system106can generate a topic visual element for a portion of a digital video based, at least in part, on the audio content of the digital video. Indeed, the video topic generation system106can analyze the audio content and generate the topic visual element based on the analysis. By generating topic visual elements based, at least partly, on the audio content of a digital video, the video topic generation system106can accurately convey the topics of the digital video even when the visual elements of the digital video remain the same.FIG. 3illustrates a block diagram of the video topic generation system106analyzing audio content304of a digital video302in accordance with one or more embodiments.

As shown inFIG. 3, the video topic generation system106generates a transcript map306based on the audio content304of the digital video302(i.e., based on an analysis of the audio content304). To illustrate, in one or more embodiments, the video topic generation system106analyzes the audio content304utilizing speech-to-text conversion. Indeed, the video topic generation system106can utilize the speech-to-text conversion to generate text based on words of the audio content304(e.g., speech included in the audio content). The video topic generation system106can use various different algorithms or models to perform the speech-to-text conversion. For example, in one or more embodiments, the video topic generation system106utilizes a library of algorithms, such as STT Lib, to process the audio content304and generate the corresponding text.

As further shown inFIG. 3, the video topic generation system106generates the transcript map306by associating the words of the audio content304with timestamps of the digital video302(e.g., <word>, <timestamp>). In particular, the video topic generation system106can associate the textual representation of the words from the audio content304as generated by the speech-to-text conversion with the corresponding timestamps from the digital video302. In one or more embodiments, the video topic generation system106associates every word of the audio content304with corresponding timestamps. In some embodiments, however, the video topic generation system106associates a subset of the words of the audio content304with corresponding timestamps. For example, in some embodiments, the video topic generation system106excludes pre-determined words—such as “the” or “in”—from the transcript map306.

In one or more embodiments, the video topic generation system106modifies the transcript map (e.g., generated as described above with regard toFIG. 3) by associating importance weights with the words of the audio content of a digital video (e.g., the audio content304). In one or more embodiments, the video topic generation system106associates importance weights with the words based on the frequency of the words.FIGS. 4A-4Billustrate a block diagram of associating importance weights with the words of audio content in accordance with one or more embodiments.

In particular,FIG. 4Aillustrates a block diagram of the video topic generation system106determining a frequency of the words of audio content of a digital video. As shown, the video topic generation system106utilizes a term frequency-inverse document frequency model402to determine the frequency of the words. Indeed, as illustrated inFIG. 4A, the video topic generation system106can utilize the term frequency-inverse document frequency model402to analyze the transcript map404in order to determine the frequency of words. In some embodiments, however, the video topic generation system106utilizes the term frequency-inverse document frequency model402to analyze other data related to the words of the audio content. For example, the term frequency-inverse document frequency model402can analyze a raw transcript of the audio content to determine the frequency of the words.

In one or more embodiments, the term frequency-inverse document frequency model402determines the frequency of a word TF based on a raw count TC of the appearance of the word in the transcript map404. Indeed, the term frequency-inverse document frequency model402can determine the frequency of the word TF as follows:
TF(w,d)=TC(w,d)/|d|(1)

In algorithm 1, w corresponds to the word and |d| corresponds to the number of words included in the audio content (e.g., the number of words in the transcript map404). In one or more embodiments, the term frequency-inverse document frequency model402determines the frequency of the various words of the audio content (e.g., the words included in the transcript map404) using algorithm 1. In some embodiments, the term frequency-inverse document frequency model402offsets the value of TF for a particular word (e.g., using a weighting factor) based on the appearance of that word in one or more other segments of text (e.g., other transcripts or transcript maps).

FIG. 4Billustrates a block diagram of the video topic generation system106determining and associating importance weights with the words of the audio content based on the determined frequencies of the words. As shown inFIG. 4B, the video topic generation system106can determine whether the frequency of a given word satisfies a frequency threshold406. In one or more embodiments, the frequency threshold406includes a predetermined frequency value. The video topic generation system106can compare the frequency of a given word to the frequency threshold406to determine whether the frequency of the word satisfies the frequency threshold406(e.g. equals or exceeds the frequency value). Based on determining that the frequency of a word satisfies the frequency threshold, the video topic generation system106generates an importance weight for the word. Indeed, as shown inFIG. 4B, the video topic generation system106generates the importance weight utilizing the summarization machine learning model408. In one or more embodiments, the video topic generation system106generates importance weights for the words having the highest frequency as determined by the term frequency-inverse document frequency model402(e.g., the ten words having the highest frequency). In some embodiments, the video topic generation system106generates importance weights for the words that satisfy the frequency threshold406and are among the words having the highest frequency.

In one or more embodiments, the video topic generation system106trains the summarization machine learning model408using a set of training documents and corresponding training extracts. In particular, a training extract can include one or more sentences that have been extracted from a corresponding training document.

In one or more embodiments, the video topic generation system106trains the summarization machine learning model408by converting sentences from the training documents into feature vectors. For example, the video topic generation system106can convert the sentences into the feature vectors using a word2vec algorithm or a Global Vectors (GloVe) algorithm. For example, in one or more embodiments, the video topic generation system106converts sentences into feature vectors as described by C. McCormick, Word2Vec Resources, 2016, http://mccormickml.com/2016/04/27/word2vec-resources/, which is incorporated herein by reference in its entirety.

The summarization machine learning model408can process the feature vectors corresponding to a training document and generate a binary label (i.e., 0 or 1) that predicts whether the sentence corresponding to a particular feature vector is included in the extracted summary for that training document. The video topic generation system106can compare the predicted binary label with a ground truth (i.e., the training extract corresponding to the training document) to determine a loss. Based on the loss, the video topic generation system106can modify parameters of the summarization machine learning model408(e.g., via back propagation). The video topic generation system106can train the summarization machine learning model408using several iterations of providing a training document, comparing the resulting predicted binary label to a ground truth, and modifying the model parameters. With each iteration of training, the video topic generation system106gradually increases the accuracy with which the summarization machine learning model408can determine a quantitative value corresponding to the importance of words.

After training, the video topic generation system106can utilize the summarization machine learning model408to generate importance weights for those words having a frequency that satisfies the frequency threshold406. Indeed, in one or more embodiments, the video topic generation system106trains the summarization machine learning model408based on sentences of training documents but uses the summarization machine learning model408to generate importance weights for individual words.

The video topic generation system106can convert those words satisfying the frequency threshold406into feature vectors (e.g., using the word2vec algorithm). Based on the feature vector corresponding to a word, the video topic generation system106can utilize the summarization machine learning model408to generate an importance weight for that word. In one or more embodiments, the importance weight for a word includes a value between 0 and 1. The video topic generation system106can modify the transcript map404by associating the importance weights with the words (e.g., <word>, <timestamp>, <importance weight>, as shown by the modified transcript map410). In one or more embodiments, the video topic generation system106assigns a low importance weight or an importance weight of 0 to those words having a frequency that failed to satisfy the frequency threshold406. In some embodiments, the video topic generation system106excludes words failing to satisfy the frequency threshold406from the modified transcript map410entirely.

The video topic generation system106can use various implementations of a summarization machine learning model408. In particular, the video topic generation system106can use an extractive model or an abstractive model. For example, the summarization machine learning model408can include an implementation of various models, such as Gensim, TextTeaser, PyTeaser, or pytextrank.

As mentioned, the video topic generation system106can generate a topic visual element for a portion of a digital video further based on the visual content of the digital video. Indeed, the video topic generation system106can analyze the visual content and generate the topic visual element based on the analysis.FIG. 5illustrates a block diagram of the video topic generation system106analyzing visual content502of the digital video302in accordance with one or more embodiments.

As shown inFIG. 5, the visual content502includes a video frame504of the digital video302. Indeed, though not shown in the figure, the visual content502can include a plurality of additional video frames of the digital video302. Further, as shown inFIG. 5, the video frame504includes a plurality of embedded words. Indeed, generally speaking, visual content of a digital video can include one or more video frames that include at least one embedded word.

As shown inFIG. 5, the video topic generation system106generates an embedded text map506based on the visual content502of the digital video302(i.e., based on an analysis of the visual content502). In particular, in one or more embodiments, the video topic generation system106analyzes the visual content502to extract embedded words from video frames (e.g., the video frame504) of the digital video302. For example, in some embodiments, the video topic generation system106analyzes the visual content502using optical character recognition to identify the embedded words from the video frames of the digital video302(i.e., of the visual content502).

To provide an example, in one or more embodiments, the video topic generation system106utilizes adaptive optical character recognition to accurately fix the video frames of the digital video302as per the content geometry. The video topic generation system106can then run optical character recognition to extract the textual content (i.e., the embedded words) from the fixed video frames. In one or more embodiments, while running the optical character recognition, the video topic generation system106determines whether a word returned by the optical character recognition is present in a localized dictionary. For example, upon determining that the embedded words are generally provided in English, the video topic generation system106can determine whether a given embedded word is included in an English dictionary. In one or more embodiments, upon determining that an embedded word is not included in the localized dictionary, the video topic generation system106excludes that word from the embedded text map506.

As further shown inFIG. 5, the video topic generation system106generates the embedded text map506by associating the embedded words from the video frames of the digital video302with corresponding timestamps of the digital video302(e.g., <embedded word>, <timestamp>). For example, in one or more embodiments, the video topic generation system106associates the embedded words form the video frames as identified and extracted by the optical character recognition with the corresponding timestamps from the digital video302. In one or more embodiments, the video topic generation system106associates every embedded word from the video frames with corresponding timestamps. In some embodiments, however, the video topic generation system106associates a subset of the embedded words with corresponding timestamps. For example, in some embodiments, the video topic generation system106excludes pre-determined words—such as “the” or “in”—from the embedded text map506.

As previously mentioned, in one or more embodiments, the video topic generation system106identifies a topic for a portion of a digital video based on the modified transcript map and the embedded text map corresponding to the digital video.FIG. 6illustrates an overview of a sequence of acts that the video topic generation system106performs for identifying a topic for a portion of a digital video in accordance with one or more embodiments.

For instance, as shown inFIG. 6, the video topic generation system106performs an act602of identifying a timestamp of a digital video associated with a position of an input indicator at a scrub bar corresponding to the digital video. Indeed, the video topic generation system106can detect, on a user interface displayed on a computing device, a position of an input indicator (e.g., a cursor) at a point of a scrub bar corresponding to the digital video. In response to detecting the position of the input indicator, the video topic generation system106can identify a timestamp (denoted Tu) of the digital video associated with the position of the input indicator at the point of the scrub bar. In one or more embodiments, the point of the scrub bar at which the input indicator is positioned is associated with a particular portion of the digital video. Accordingly, by identifying the timestamp Tu, the video topic generation system106can identify the portion of the digital video associated with the point of the scrub bar at which the input indicator is positioned.

Furthermore, as illustrated inFIG. 6, the video topic generation system106performs an act604of determining a range of timestamps that includes the timestamp Tu. In one or more embodiments, the video topic generation system106determines (e.g., identifies) a range of timestamps that includes the timestamp Tu of the digital video associated with the position of the input indicator at the point of the scrub bar, a set of timestamps occurring prior to the timestamp, and a set of timestamps occurring subsequent to the timestamp (i.e., Tu±n). For example, the video topic generation system106can determine the range of timestamps to include timestamps preceding the timestamp Tu by up to ten seconds and timestamps following the timestamp Tu by up to ten seconds (i.e., n=10).

As further shown inFIG. 6, the video topic generation system106performs an act606of determining a set of words from the modified transcript map based on the range of timestamps. In particular, the video topic generation system106can determine a set of words that includes words that are associated with the range of timestamps (i.e., are associated with a timestamp falling within the range of timestamps). In one or more embodiments, the video topic generation system106determines the set of words to include a pre-determined number of words. For example, the video topic generation system106can determine the set of words to include the k top-weighted words from a portion of the modified transcript map associated with the range of timestamps (e.g., having words that fall within the range of timestamps) where k includes a pre-determined value. In one or more embodiments, the video topic generation system106stores the set of words within a vector.

Additionally, as shown inFIG. 6, the video topic generation system106performs an act608of identifying a topic based on the set of words and the embedded text map. For example, in one or more embodiments, the video topic generation system106performs a reverse search in the embedded text map to compare words from the embedded text map and words from the set of words. In particular, the video topic generation system106can compare words from the set of words with words from a portion of the embedded text map associated with the range of timestamps. In one or more embodiments, the video topic generation system106identifies the topic by identifying a match between a word from the set of words and a word from the portion of the embedded text map associated with the range of timestamps. The video topic generation system106can identify a match between words using various methods (e.g., determining that the words are exactly equivalent, determining that the words satisfy a similarity threshold etc.)

In one or more embodiments, the video topic generation system106can determine that multiple words from the set of words match with words from the portion of the embedded text map associated with the range of timestamps. In some embodiments, the video topic generation system106determines that an additional match corresponds to a subtopic. For example, when comparing words from the set of words and the portion of the embedded text map, the video topic generation system106can begin with the first word from the set of words and progress through the subsequent words up to the last word from the set of words. The video topic generation system106can determine that a first match between words corresponds to the topic and that a second match corresponds to a subtopic.

In some embodiments, the video topic generation system106determines that there is no match between words from the set of words and words form the portion of the embedded text map associated with the range of timestamps. Indeed, the video topic generation system106can determine that the portion of the embedded text map does not include a matching word corresponding to a word from the set of words. Consequently, the video topic generation system106can determine that the top-weighted word from the set of words corresponds to the topic. Thus, while much of the present disclosure discusses identifying topics and generating topic visual elements based on the audio content and visual content of a digital video, the video topic generation system can identify topics and generate topic visual elements based on the audio content alone in some embodiments.

By identifying a topic for a portion of a digital video based, at least partly, on the audio content of the digital video, the video topic generation system106can operate more flexibly than conventional systems. Indeed, by analyzing the audio content, the video topic generation system106can identify a topic based on more information than provided by the visual content of the digital video alone. Thus, while the visual content may not change significantly in appearance throughout the duration of the digital video (resulting in similar-looking preview thumbnails), the video topic generation system106can identify a topic for one portion of the digital video that may be different than another portion of the digital video. Accordingly, the video topic generation system106can further operate more accurately by identifying topics based on a greater collection of available information.

As mentioned above, the video topic generation system106can generate a topic visual element for a portion of a digital video.FIG. 7illustrates an overview of generating a topic visual element in accordance with one or more embodiments. As shown inFIG. 7, the video topic generation system106generates the topic visual element704for a portion of the digital video706based on an identified topic702(e.g., a topic identified as described above with reference toFIG. 6). As further shown, the topic visual element704can include a textual representation of the identified topic702.

As further shown inFIG. 7, the video topic generation system106can append the topic visual element704to a scrub-bar-preview thumbnail708associated with a portion of the digital video706(i.e., associated with the position of an input indicator710at a point of a scrub bar712corresponding to the digital video706). Where the video topic generation system106has identified one or more subtopics corresponding to the topic of the topic visual element704, the topic visual element704can include the subtopic(s).

As illustrated inFIG. 7, the video topic generation system106can generate and provide the topic visual element704in real time. Indeed, the video topic generation system106can detect that the input indicator710is positioned at a particular point at the scrub bar712corresponding to the digital video706. The video topic generation system106can, in response to detecting the position of the input indicator, and in real time, determine the timestamp corresponding to the position of the input indicator710, determine a range of timestamps that includes that timestamp, generate a set of words from the modified transcript map based on the range of timestamps, compare those words to the words from the embedded text map, identify a topic based on the comparison, and generate the topic visual element704. Accordingly, the video topic generation system106can provide the topic visual element704for display with the digital video706on the computing device while the input indicator710is positioned at the point of the scrub bar712associated with the portion of the digital video. In one or more embodiments, the video topic generation system106generates the modified transcript map and the embedded text map at some previous time but generates and provides topic visual elements in real time as the input indicator moves about the scrub bar712of the digital video706.

In some embodiments, however, the video topic generation system106identifies topics and generates topic visual elements for the digital video706at some time prior to a user viewing or a computing device displaying the digital video706. For example, in one or more embodiments, the video topic generation system106segments the digital video706into several intervals (e.g., 20 second intervals). The video topic generation system106can then identify a topic and generate a corresponding topic visual element for one or more of the intervals. The video topic generation system106can then provide a topic visual element for a portion of the digital video706for display upon detecting the input indicator710positioned at a point at the scrub bar712associated with the portion of the digital video706.

It should be noted that, although the present disclosure generally discusses analyzing individual words of the audio content and visual content of a digital video, the video topic generation system106can analyze groups of words (e.g., phrases and/or sentences) in one or more embodiments. Indeed, the video topic generation system106can identify entire groups of words as a topic. In some embodiments, the video topic generation system106analyzes the words of the audio content and visual content individually but identifies a group of words as the topic upon determining that the matching word is associated with a group of words (e.g., is part of a common phrase used in the digital video).

Thus, in one or more embodiments, the video topic generation system106can identify a digital video having audio content and visual content. The video topic generation system106can analyze the audio content and the visual content in order to identify a topic for a portion of the digital video and generate a corresponding topic visual element. The algorithms and acts described with reference toFIGS. 2-7can comprise the corresponding structure for performing a step for generating a topic visual element for a portion of the digital video based on the audio content and the visual content of the digital video.

By generating and providing topic visual elements based on the audio content and visual content of a digital video, the video topic generation system106can operate more flexibly than conventional systems. Indeed, by generating and providing topic visual elements based on the audio content and visual content of a digital video, the video topic generation system106can identify and convey the topics of a digital video without the need for manual user input. Further, by generating and providing topic visual elements in real time (e.g., as a user positions an input indicator over a scrub bar corresponding to the digital video) the video topic generation system106can flexibly avoid the limitations of providing such information at the time the digital video is created.

As previously mentioned, the video topic generation system106can generate a table of contents for a digital video based on the audio content and the visual content of the digital video. For example, the video topic generation system106can generate the table of contents based on the modified transcript map generated based on the audio content and the embedded text map generated based on the visual content.FIG. 8illustrates a block diagram of generating a table of contents806in accordance with one or more embodiments. As shown inFIG. 8, the video topic generation system106identifies topics (and subtopics) based on the modified transcript map802and the embedded text map804and compiles the topics (and subtopics) to generate the table of contents806. In one or more embodiments, the table of contents806includes topics corresponding to the topic visual elements generated for various portions of the digital video.

In one or more embodiments, the video topic generation system106provides the table of contents806for display on a computing device. In particular, the video topic generation system106can provide the table of contents806for display with the digital video. Indeed, as mentioned above, the video topic generation system106can identify topics for a digital video at a time prior to a user viewing the digital video. Accordingly, the video topic generation system106can generate the table of contents806, which includes those identified topics, before a user views the digital video. In one or more embodiments, the video topic generation system106provides the table of contents806for display within an initial frame of the digital video. In some embodiments, the video topic generation system106provides the table of contents806for display within a description area corresponding to the digital video.

In one or more embodiments, the video topic generation system106configures the table of contents806to facilitate navigation of the corresponding digital video. For example, in some embodiments, the video topic generation system106generates the table of contents806so the included topics (and subtopics) comprise links to the corresponding portions of the digital video. Accordingly, the video topic generation system106can detect a user selection of a topic from the table of contents806. In response to detecting the user selection of the topic, the video topic generation system106can provide the portion of the digital video for display on a computing device (i.e., the computing device with which the user made the selection).

Turning now toFIG. 9, additional detail will now be provided regarding various components and capabilities of the video topic generation system106. In particular,FIG. 9illustrates the video topic generation system106implemented by the server(s)102and the video presentation system104. As shown, the video topic generation system106can include, but is not limited to, an input indicator detector902, a speech-to-text conversion engine904, a transcript map generator906, an embedded text map generator908, a term frequency-inverse document frequency model application manager910, a summarization machine learning model training engine912, a summarization machine learning model application manager914, a topic identification engine916, a topic visual element generator918, and data storage920(which includes training documents922, training extracts924, a summarization machine learning model926, and a term frequency-inverse document frequency model928).

As just mentioned, and as illustrated inFIG. 9, the video topic generation system106includes the input indicator detector902. In particular, the input indicator detector902can detect the positioning and interaction of an input detector with respect to a computing device, such as the positioning and interaction with a user interface displayed on a computing device. For example, the input indicator detector902can detect the position of an input detect with respect to a scrub bar corresponding to a digital video (e.g., a point of the scrub bar at which the input indicator is positioned).

Additionally, as shown inFIG. 9, the video topic generation system106includes the speech-to-text conversion engine904. In particular, the speech-to-text conversion engine904can analyze audio content and generate text corresponding to speech included in the audio content. For example, the speech-to-text conversion engine904can generate text based on the speech included in the audio content of a digital video.

Further, as shown inFIG. 9, the video topic generation system106includes the transcript map generator906. In particular, the transcript map generator906can generate a transcript map based on the audio content of a digital video. For example, the transcript map generator906can generate a transcript map based on the text generated by the speech-to-text conversion engine904. The transcript map generator906can generate a transcript map by associating the words of the audio content of the digital video with timestamps of the digital video. In one or more embodiments, the transcript map generator906can further modify the transcript map by associating importance weights with the words of the audio content. For example, the transcript map generator906can associate importance weights generated by the summarization machine learning model application manager914with the words of the audio content within the transcript map.

As shown inFIG. 9, the video topic generation system106further includes the embedded text map generator908. In particular, the embedded text map generator908can generate an embedded text map based on the visual content of a digital video. For example, the embedded text map generator908can analyze one or more video frames of the visual content (e.g., using optical character recognition) and identify words embedded in the video frames. The embedded text map generator908can generate the embedded text map by associating the embedded words from the video frames with corresponding timestamps of the digital video.

As shown inFIG. 9, the video topic generation system106also includes the term frequency-inverse document frequency model application manager910. In particular, the term frequency-inverse document frequency model application manager910can determine the frequency of words of the audio content. For example, the term frequency-inverse document frequency model application manager910can analyze the transcript map generated by the transcript map generator906to determine a frequency for the words of the audio content.

Further, as shown inFIG. 9, the video topic generation system106includes the summarization machine learning model training engine912. In particular, the summarization machine learning model training engine912trains a summarization machine learning model to generate weights for audio content words. In one or more embodiments, the summarization machine learning model training engine912trains the summarization machine learning model using training documents and corresponding training extracts. The summarization machine learning model training engine912can use the summarization machine learning model to predict whether a sentence from a training document appears in the corresponding training extract, compare the prediction to the training extract, and modify parameters of the summarization machine learning model based on the comparison.

Additionally, as shown inFIG. 9, the video topic generation system106includes the summarization machine learning model application manager914. In particular, the summarization machine learning model application manager914can utilize the summarization machine learning model trained by the summarization machine learning model training engine912to generate importance weights for the words of the audio content included in the transcript map. In one or more embodiments, the summarization machine learning model application manager914utilizes the summarization machine learning model to generate importance weights for words based on the frequency of those words determined by the term frequency-inverse document frequency model application manager910(e.g., based on whether a given word satisfies a frequency threshold).

As shown inFIG. 9, the video topic generation system106also includes the topic identification engine916. In particular, the topic identification engine916can identify a topic for a portion of a digital video. For example, the topic identification engine916can compare words from the modified transcript map generated by the transcript map generator906and the embedded text map generated by the embedded text map generator908to identify a match. In one or more embodiments, the topic identification engine916specifically compares words from a portion of the modified transcript map and a portion of the embedded text map that correspond to the portion of the digital video.

Indeed, as discussed above, the video topic generation system106can identify a topic for the portion of a digital video in real time. Accordingly, based on the input indicator detector902detecting that an input indicator is positioned at a point of a scrub bar that is associated with the portion of the digital video, the topic identification engine916can identify a timestamp of the digital video associated with the position of the input indicator (i.e., associated with the portion of the digital video). The topic identification engine916can further determine a range of timestamps that includes the timestamp associated with the position of the input indicator and generate a set of words from the modified transcript that are associated with the range of timestamps. The topic identification engine916can identify a topic for the portion of the digital video by comparing the words from the set of words with words from the embedded text map that fall within that range of timestamps.

Additionally, as discussed above, the video topic generation system106can identify the topic for the portion of the digital video a time prior to a user viewing the digital video. As an example, the topic identification engine916can segment the digital video into intervals (e.g., twenty-second intervals) and identify a topic for each interval. Whether identifying topics in real time or at some previous time, the topic identification engine916can further identify one or more subtopics corresponding to the identified topics.

Further, as shown inFIG. 9, the video topic generation system106includes the topic visual element generator918. In particular, the topic visual element generator918can generate a topic for the portion of the digital video based on the topic identified by the topic identification engine916. Where the topic identification engine916has identified one or more subtopics corresponding to the topic, the topic visual element generator918can generate the topic visual element to include the subtopic(s). The topic visual element generator918can provide the topic visual element for display on a computing device.

In one or more embodiments, the topic visual element generator918generates a table of contents for the digital video. Indeed, the topic visual element generator918can generate a table of contents that includes all topics and subtopics identified for a digital video. The topic visual element generator918can provide the table of contents for display with the digital video (e.g., as an initial frame of the digital video).

Additionally, as shown inFIG. 9, the video topic generation system106includes data storage920. In particular, data storage920includes training documents922, training extracts924, a summarization machine learning model926, and a term frequency-inverse document frequency model928. The training documents922and the training extracts924can store the training documents and training extracts, respectively, used by the summarization machine learning model training engine912to train the summarization machine learning model926. The summarization machine learning model926can store the summarization machine learning model trained by the summarization machine learning model training engine912and used by the summarization machine learning model application manager914to generate importance weights for words of audio content. The term frequency-inverse document frequency model928can store the term frequency-inverse document frequency model utilized by the term frequency-inverse document frequency model application manager910to determine the frequency of words of the audio content.

Each of the components902-928of the video topic generation system106can include software, hardware, or both. For example, the components902-928can include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices, such as a client device or server device. When executed by the one or more processors, the computer-executable instructions of the video topic generation system106can cause the computing device(s) to perform the methods described herein. Alternatively, the components902-928can include hardware, such as a special-purpose processing device to perform a certain function or group of functions. Alternatively, the components902-928of the video topic generation system106can include a combination of computer-executable instructions and hardware.

Furthermore, the components902-928of the video topic generation system106may, for example, be implemented as one or more operating systems, as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components902-928of the video topic generation system106may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components902-928of the video topic generation system106may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components902-928of the video topic generation system106may be implemented in a suite of mobile device applications or “apps.” For example, in one or more embodiments, the video topic generation system106can comprise or operate in connection with digital software applications such as ADOBE® CREATIVE CLOUD® or ADOBE® FLASH® PLAYER. “ADOBE,” “CREATIVE CLOUD,” and “FLASH” are either registered trademarks or trademarks of Adobe Inc. in the United States and/or other countries.

FIGS. 1-9, the corresponding text and the examples provide a number of different methods, systems, devices, and non-transitory computer-readable media of the video topic generation system106. In addition to the foregoing, one or more embodiments can also be described in terms of flowcharts comprising acts for accomplishing the particular results, as shown inFIG. 10.FIG. 10may be performed with more or fewer acts. Further, the acts may be performed in different orders. Additionally, the acts described herein may be repeated or performed in parallel with one another or in parallel with different instances of the same or similar acts.

As mentioned,FIG. 10illustrates a flowchart of a series of acts1000for generating a topic visual element for a portion of a digital video in accordance with one or more embodiments. WhileFIG. 10illustrates acts according to one embodiment, alternative embodiments may omit, add to, reorder and/or modify any of the acts shown inFIG. 10. The acts ofFIG. 10can be performed as part of a method. For example, in some embodiments, the acts ofFIG. 10can be performed, in a digital medium environment for presenting digital videos, as part of a computer-implemented method. Alternatively, a non-transitory computer-readable medium can store instructions that, when executed by at least one processor, cause a computing device to perform the acts ofFIG. 10. In some embodiments, a system can perform the acts ofFIG. 10. For example, in one or more embodiments, a system includes at least one memory device comprising a digital video comprising audio content and visual content and a summarization machine learning model trained to generate weights for audio content words. The system can further include at least one server device that causes the system to perform the acts ofFIG. 10.

The series of acts1000includes an act1002of identifying a digital video. For example, the act1002involves identifying a digital video comprising audio content and visual content. In one or more embodiments, the video topic generation system106identifies the digital video by receiving the digital video from a client device or a third-party system. In some embodiments, the video topic generation system106identifies the digital video within a database. In still further embodiments, the video topic generation system106identifies the digital video by detecting access of the digital video by a computing device.

The series of acts1000also includes an act1004of generating a transcript map based on audio content of the digital video. For example, the act1004involves generating a transcript map, based on the audio content of the digital video, by associating words of the audio content with timestamps of the digital video. The act1004includes the sub-act1006of generating text based on the audio content. Indeed, in one or more embodiments, the video topic generation system106generates the text utilizing speech-to-text conversion. The act1004further includes the sub-act1008of associating the words of the audio content based on the text. Indeed, the video topic generation system106can generate the transcript map by associating the textual representation of the words of the audio content with the corresponding timestamps of the digital video.

Further, the series of acts includes an act1010of generating an embedded text map based on visual content of the digital video. For example, the act1010involves generating an embedded text map, based on the visual content of the digital video, by associating embedded words from video frames of the digital video with corresponding timestamps of the digital video. The act1010includes the sub-act1012of analyzing the visual content using optical character recognition. For example, the sub-act1012involves analyzing the visual content using optical character recognition to identify the embedded words from the video frames of the digital video. The act1010further includes the sub-act1014of generating the embedded text map based on the analysis. For example, the sub-act1014can include associating the embedded words from the video frames of the digital video with the corresponding timestamps of the digital video based on the analysis

Additionally, the series of acts includes an act1016of modifying the transcript map. For example, the act1016involves modifying the transcript map by associating importance weights with the words of the audio content. The act1016includes the sub-act1018of determining a frequency of the words of the audio content. For example, the sub-act1018can involve determining a frequency of the words of the audio content using a term frequency-inverse document frequency model. The video topic generation system106can associate the importance weights with the words of the audio content based on the frequency. Indeed, in one or more embodiments, based on the frequency of the words, the video topic generation system106utilizes a summarization machine learning model to generate importance weights for the words For example, the act1016includes the sub-act1020of determining that a frequency of a given word satisfies a frequency threshold. The act1016further includes the sub-act1022of generating an importance weight for the given word. For example, the sub-act1022can involve generating an importance weight for the given word utilizing a summarization machine learning model, trained to generate weights for audio content words, based on determining that the frequency of the given word satisfies the frequency threshold. Further, the act1016includes the sub-act1024of associating the importance weights with the words in the transcript map, which can include associating the importance weight—that was generated for the given word—with the given word.

The series of acts further includes an act1026of generating a topic visual element. For example, the act1026involves generating a topic visual element for a portion of the digital video based on the modified transcript map and the embedded text map.

In one or more embodiments, the video topic generation system106detects, on a user interface displayed on a computing device, a position of an input indicator at a point of a scrub bar corresponding to the digital video, the point of the scrub bar being associated with the portion of the digital video. Accordingly, the video topic generation system106can generate the topic visual element for the portion of the digital video in response to detecting the position of the input indicator at the point of the scrub bar associated with the portion of the digital video. For example, the act1026includes the sub-act1028of identifying a timestamp of the digital video. In particular, the sub-act1028involves identifying a timestamp of the digital video associated with the position of the input indicator at the point of the scrub bar associated with the portion of the digital video. Further, the act1026includes the sub-act1030of determining a range of timestamps that includes the timestamp. In particular, the sub-act1030involves determining a range of timestamps comprising the timestamp of the digital video associated with the position of the input indicator at the point of the scrub bar. Indeed, in one or more embodiments, the range of timestamps can include the timestamp of the digital video associated with the position of the input indicator at the point of the scrub bar, a set of timestamps occurring prior to the timestamp, and a set of timestamps occurring subsequent to the timestamp. Additionally, the act1026includes the sub-act1032of generating a set of words from the modified transcript map. For example, the sub-act1032involves generating a set of words from the modified transcript map that are associated with the range of timestamps. In one or more embodiments, generating the set of words includes generating a set of words comprising one or more top-weighted words from a portion of the modified transcript map associated with the range of timestamps. The act1026also includes the sub-act1034of generating the topic visual element based on the set of words and the embedded text map. In particular, the sub-act1034involves generating the topic visual element for the portion of the digital video based on the set of words and the embedded text map.

In one or more embodiments, the video topic generation system106generates the topic visual element for the portion of the digital video based on the modified transcript map and the embedded text map by identifying a match between a word from the modified transcript map and a word from the embedded text map, the word from the modified transcript map and the word from the embedded text map both being associated with the portion of the digital video; and generating the topic visual element based on the identified match. In some embodiments, the video topic generation system106generates the topic visual element for the portion of the digital video based on the modified transcript map and the embedded text map by determining that a portion of the embedded text map does not include a matching word corresponding to a word from a portion of the modified transcript map, the portion of the embedded text map and the portion of the modified transcript map corresponding to the portion of the digital video; and, based on determining that the portion of the embedded text map does not include the matching word, generating the topic visual element for the portion of the digital video based on a top-weighted word from the portion of the modified transcript map.

In one or more embodiments, the video topic generation system106can determine a subtopic for the portion of the digital video based on the modified transcript map and the embedded text map. Accordingly, the visual topic element can include the subtopic. To provide an example, the video topic generation system106can identify a first match between a word from the modified transcript map and a word from the embedded text map, the word from the modified transcript map and the word from the embedded text map both being associated with the portion of the digital video; and generate the topic visual element based on the first match. Further, the video topic generation system106can identify a second match between an additional word from the modified transcript map and an additional word from the embedded text map, the additional word from the modified transcript map and the additional word from the embedded text map both being associated with the portion of the digital video; and determine a subtopic for the portion of the digital video based on the second match. Accordingly, the topic visual element can include the subtopic.

The series of acts also includes an act1036of providing the topic visual element for display. For example, the act1036involves providing the topic visual element for display with the digital video on the computing device. In one or more embodiments, providing the topic visual element for display with the digital video on the computing device comprises appending the topic visual element to a scrub-bar-preview thumbnail associated with the portion of the digital video for display on the computing device. In one or more embodiments where the video topic generation system106generates the topic visual element in response to detecting a position of an input indicator at a point of a scrub bar associated with the portion of the digital video, the video topic generation system106can provide the topic visual element for display with the digital video on the computing device while the input indicator is positioned at the point of the scrub bar associated with the portion of the digital video.

In one or more embodiments, the series of acts1000further includes acts for generating and providing a table of contents for the digital video. For example, in one or more embodiments, the acts include generating a table of contents for the digital video based on the modified transcript map and the embedded text map, the table of contents comprising a topic corresponding to the topic visual element for the portion of the digital video; and providing the table of contents for display on the computing device. In some embodiments, the acts further include detecting a user selection of the topic corresponding to the topic visual element from the table of contents; and, in response to detecting the user selection of the topic, providing the portion of the digital video for display on the computing device.

FIG. 11illustrates a block diagram of an example computing device1100that may be configured to perform one or more of the processes described above. One will appreciate that one or more computing devices, such as the computing device1100may represent the computing devices described above (e.g., the server(s)102, the client devices110a-110n, and/or the third-party system114). In one or more embodiments, the computing device1100may be a mobile device (e.g., a mobile telephone, a smartphone, a PDA, a tablet, a laptop, a camera, a tracker, a watch, a wearable device, etc.). In some embodiments, the computing device1100may be a non-mobile device (e.g., a desktop computer or another type of client device). Further, the computing device1100may be a server device that includes cloud-based processing and storage capabilities.

As shown inFIG. 11, the computing device1100can include one or more processor(s)1102, memory1104, a storage device1106, input/output interfaces1108(or “I/O interfaces1108”), and a communication interface1110, which may be communicatively coupled by way of a communication infrastructure (e.g., bus1112). While the computing device1100is shown inFIG. 11, the components illustrated inFIG. 11are not intended to be limiting. Additional or alternative components may be used in other embodiments. Furthermore, in certain embodiments, the computing device1100includes fewer components than those shown inFIG. 11. Components of the computing device1100shown inFIG. 11will now be described in additional detail.

In particular embodiments, the processor(s)1102includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, the processor(s)1102may retrieve (or fetch) the instructions from an internal register, an internal cache, memory1104, or a storage device1106and decode and execute them.

The computing device1100includes memory1104, which is coupled to the processor(s)1102. The memory1104may be used for storing data, metadata, and programs for execution by the processor(s). The memory1104may include one or more of volatile and non-volatile memories, such as Random-Access Memory (“RAM”), Read-Only Memory (“ROM”), a solid-state disk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage. The memory1104may be internal or distributed memory.

The computing device1100includes a storage device1106including storage for storing data or instructions. As an example, and not by way of limitation, the storage device1106can include a non-transitory storage medium described above. The storage device1106may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive or a combination these or other storage devices.

As shown, the computing device1100includes one or more I/O interfaces1108, which are provided to allow a user to provide input to (such as user strokes), receive output from, and otherwise transfer data to and from the computing device1100. These I/O interfaces1108may include a mouse, keypad or a keyboard, a touch screen, camera, optical scanner, network interface, modem, other known I/O devices or a combination of such I/O interfaces1108. The touch screen may be activated with a stylus or a finger.

The computing device1100can further include a communication interface1110. The communication interface1110can include hardware, software, or both. The communication interface1110provides one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices or one or more networks. As an example, and not by way of limitation, communication interface1110may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI. The computing device1100can further include a bus1112. The bus1112can include hardware, software, or both that connects components of computing device1100to each other.