SYSTEM AND METHOD TO REVIEW ONLINE VIOLENCE AND EDUCATION

A computing system is configured to obtain a video that includes text elements and visual elements. The computing system is further configured to generate a plurality of text tokens representative of audio spoken in the video and a plurality of frame tokens representative of one or more frames of the video. The computing system is further configured to generate a set of features that includes a text feature, a frame feature, and a multi-modal feature, wherein the multi-modal feature is representative of multi-modal elements of the video, and wherein generating the set of features is based on the plurality of text tokens and the plurality of frame tokens. The computing system is further configured to associate the set of features with one or more labels to generate a multi-label classification of the video. The computing system is further configured to output an indication of the multi-label classification of the video.

TECHNICAL FIELD

This disclosure relates to computing systems and, more specifically, to a content filtering system.

BACKGROUND

Online platforms make available a variety of content, such as videos. For example, an online video platform may enable users to stream videos created and uploaded by other users of the platform. The online video platform may organize videos by a variety of different categories. For example, the online video platform may organize videos that are appropriate for children into a particular collection of “kids” videos.

SUMMARY

In general, this disclosure describes video classification techniques for generating multi-label classifications of videos by generating frame, text, and multimodal features of a video and organizing the videos based on one or more classifications. An individual or organization may seek to identify videos for a child to watch that not only contain appropriate content for children but that also contain educationally relevant content.

Rather than manually watching, classifying, and approving individual videos, an individual or organization may employ the video classification techniques to classify videos and subsequently identify videos that are appropriate for another individual or group of individuals (e.g., a child/children) to view. For example, a parent may find the process of manually reviewing every video that their child watches for inappropriate content unduly time-consuming and tedious. In addition, the parent may struggle to select videos that are tailored to a child's particular educational level and needs. An alternative approach is to classify videos based on one or more classifications to generate multi-label classifications of the videos.

In an example, an analysis system may generate and/or extract tokens from a video and process the tokens to create features of the video which are associated with one or more labels in a multi-label classification. The analysis system may generate tokens that include multimodal tokens using a fusion encoder to encode both text and visual cues from a video into the multimodal tokens. The analysis system may process the multimodal tokens into features representative of the text, video, and multi-modal cues of the video for use in a multi-label contrastive processing of the features. As part of the multi-label contrastive processing, the analysis system may associate the features with one or more class prototypes that are based on educational content codes among other characteristics. The analysis system may output an indication of the multi-label classification for use by one or more recipients, such as a video platform and/or user system.

In an example, a method includes obtaining, by a computing system, a video that includes text elements and visual elements; generating, by the computing system and based on the text elements and the visual elements, a plurality of text tokens representative of audio spoken in the video and a plurality of frame tokens representative of one or more frames of the video; generating, by the computing system using a machine learning model, a set of features that includes a text feature, a frame feature, and a multi-modal feature, wherein the multi-modal feature is representative of multi-modal elements of the video, and wherein generating the set of features is based on the plurality of text tokens and the plurality of frame tokens; associating, by the computing system, the set of features with one or more labels to generate a multi-label classification of the video; and outputting, by the computing system, an indication of the multi-label classification of the video.

In another example, a computing system includes a memory and one or more programmable processors in communication with the memory and configured to obtain a video that includes text elements and visual elements; generate, based on the text elements and the visual elements, a plurality of text tokens representative of audio spoken in the video and a plurality of frame tokens representative of one or more frames of the video; generate, using a machine learning model, a set of features that includes a text feature, a frame feature, and a multi-modal feature, wherein the multi-modal feature is representative of multi-modal elements of the video, and wherein generating the set of features is based on the plurality of text tokens and the plurality of frame tokens; associate the set of features with one or more labels to generate a multi-label classification of the video; and output an indication of the multi-label classification of the video.

In yet another example, non-transitory computer-readable media includes instructions configured to cause one or more processors to obtain a video that includes text elements and visual elements; generate, based on the text elements and the visual elements, a plurality of text tokens representative of audio spoken in the video and a plurality of frame tokens representative of one or more frames of the video; generate, using a machine learning model, a set of features that includes a text feature, a frame feature, and a multi-modal feature, wherein the multi-modal feature is representative of multi-modal elements of the video, and wherein generating the set of features is based on the plurality of text tokens and the plurality of frame tokens; associate the set of features with one or more labels to generate a multi-label classification of the video; and output an indication of the multi-label classification of the video.

DETAILED DESCRIPTION

FIG.1is a block diagram illustrating an example computing environment100for classification, in accordance with the techniques of this disclosure. Computing environment100includes analysis system102, user system130, media source140, and network150.

Analysis system102may be one or more types of computing system and/or device, such as a server, mainframe, supercomputer, cloud computing environment, distributed computing environment, virtualized computing environment, desktop computer, laptop computer, tablet computer, smartphone, or other type of computing device. In some examples, analysis system102may be integrated with one or more other systems, such as user system130and/or media source140. Analysis system102may include one or more processors that execute instructions of one or more processes of analysis system102. For example, analysis system102may include one or more of Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Reduced Instruction Set Computer (RSIC) processors, multi-core processors, single-core processors, virtualized processors, and/or other types of processors. Analysis system102may include one or more software components that are executed by one or more processors of analysis system102. For example, analysis system102may include one or more programs executed by a processor of analysis system102that communicate with user system130and/or media source140.

User system130may be a computing system and/or device associated with a user. For example, user system130may be smartphone, smartwatch, augmented reality (AR) glasses/goggles, virtual reality (VR) glasses/goggles, tablet computer, smartwatch, vehicle entertainment system, gaming system, streaming device, smart television, set-top box, laptop computer, desktop computer, and/or other type of computing device/system. User system130may communicate with one or more other computing devices/systems. For example, user system130may communicate with media source140via network150.

Network140may include one or more types of networks, such as cellular networks, Wide Area Networks (WAN), Local Area Networks (LAN), and other types of networks. Network140may represent the internet. Network140may communicatively interconnect one or more computing systems and/devices. For example, network140may enable communication between user system130and media source140.

Media source140stores video content and presents the video content to consumers. Media source140may represent an online video platform. Media source140may present the video content by streaming the video content to a device for real-time video consumption, by making the video available to a device for download and/or caching, or by otherwise outputting the video content to another device. Media source140may store many thousands or even millions of videos of various lengths, each video including video content. Video content includes image data but may also include audio data. The audio data may include speech that can be transcribed into text using speech recognition. Each video may also be associated with metadata for the video. The metadata may include a Uniform Resource Locator (URL) at which the video is available, an identity of the source, an identity of the creator, a length, a primary language, a content description, or other metadata for the video.

Media source140may be a computing platform and/or computing system. Media source140may be a collection of one or more computing systems, storage systems, cloud computing environments, virtualized computing environments, and/or other types of computing systems.

Media source140may enable computing systems and devices to obtain videos from media source140. Media source140may enable a computing system, such as user system130, to search for and obtain videos from media source140. For example, media source140may provide data indicating one or more videos available from media source140in response a query received from user system130and/or analysis system102. Media source140may stream, via network150, videos to user system130for consumption by a user of user system130or for processing by analysis system102. Additionally, media source140may generate and output video recommendations for users. For example, media source140may use one or more algorithms to recommend videos based on previously viewed videos. Media source140may output the video recommendations via one or more interfaces such as a website associated with media source140, a companion application executed by user system130such as media application132, etc.

Media source140may organize videos by one or more categories such as recommended age, topic, source (e.g., the user or entity that uploaded or created a particular video), and other types of categorizations. Media source140may organize the videos based on the one or more categories to assist users with finding videos that they wish to view. For example, media source140may enable a user to find videos that are algebra tutorials. In another example, media source140may enable a user to find videos that teach the alphabet. Media source140may include a platform for child-appropriate videos that limits the videos available to view to those that meet one or more content requirements (e.g., restrictions on content such as violence advertising, etc.). For example, media source140may include a children-focused video platform that includes a collection of videos selected as meeting the one or more content requirements.

In general, user system130may interact with media source140to obtain videos for consumption. An individual (e.g., a parent/guardian) who manages the use of user system130by a user may wish to obtain videos for consumption by the user (e.g., a child using user system130) that are not only safe/appropriate for consumption by the user but that also include educational content that is relevant for the user. The individual may struggle to identify videos that are safe for consumption by the user and that also include educational content that is relevant/tailored to the needs of the user. For example, the individual may struggle to select videos for viewing via user system130for a child in elementary school that are tailored to the current educational level of the child and appropriate for the child to watch. Additionally, media source140may be unable to filter out every video uploaded to a child-focused portion of media source140that includes inappropriate content.

In accordance with the techniques of this disclosure, analysis system102may analyze videos obtained from media source. Analysis system102may obtain videos from media source140and process the videos to generate multi-label classifications for the videos. Analysis system102may generate text tokens that are representative of audio in a video and video tokens representative of one or more frames of the video. Analysis system102may generate a set of features that includes a text feature, a frame feature, and a multimodal feature. Analysis system102may associate the set of features with one or more labels to generate a multi-label classification of the video. Analysis system102may generate a multi-label classification based on labels that are educational and content labels. For example, analysis system102may generate multi-label classifications that are based on educational content standards (e.g., grade-levels, subject matters, etc.).

Analysis system102may preemptively obtain videos from media source140to enable analysis system102to preemptively identify videos that are appropriate for a user of user system130. Analysis system102may obtain videos from media source140using one or more techniques. Analysis system102may crawl for and/or scrape videos from media source140. For example, analysis system102may obtain videos using a search function of the media platform. For example, analysis system102may generate search terms for videos (e.g., search terms consistent with particular types of videos, such as educational videos) and search for videos on media source140using the generated search terms. In some examples, analysis system102may obtain videos from media source140in response to a request from user system130. Analysis system102may provide video and metadata for the videos to one or more components, such as classification module104.

Analysis system102includes classification module104. Classification module104may be a software component of analysis system102, such as a program, process, module, plugin, or other type of software component. Classification module104may be executed by one or more processors of analysis system102. Classification module102may process videos obtained from media source140. Classification module102may process the videos to generate and/or extract tokens from the videos.

In an example, analysis system102provides a video to classification module104for processing. Classification module104processes the video to generate text tokens representative of audio spoken in the video and frame tokens representative of image data (e.g., frames) of the video. Classification module104may generate tokens that are mathematical representations of cues from the videos, such as vectorized representations of the cues. Audio spoken in a video (i.e., speech) can include recitation, songs, or other forms of verbal communication. Classification module104may use automatic speech recognition (ASR) to extract text from audio of the video and generate text tokens using a text encoder and based on the extracted text. Classification module104may generate the frame tokens using an image encoder. Classification module104may provide the tokens to a machine learning model such as ML model106.

Classification module104includes ML model106. ML model106may include one or more machine learning (ML) models, such as neural networks, deep learning network, transformer models, encoders, feed-forward networks, perceptrons, time delay neural networks (TDNN), reinforcement learning networks, Q-learning networks, and/or other types of ML models. For example, ML model106may include an encoder that applies multi-head cross attention to one or more tokens, and ML model106may include one or more multi-layer perceptrons (MLPs).

ML model106may include a fusion encoder that generates multi-modal tokens. The fusion encoder may process the text tokens and the frame tokens and generate multi-modal tokens that are representative of multi-modal elements of a video (e.g., a combination of video and audio elements of the video). For example, ML model106may generate multi-modal tokens that are vectors or other type of mathematical representation representative of the multi-modal elements of the video. The performance of fusion encoder may improve with more layers. ML model106may receive a multi-modal classification token in addition to and/or in lieu of the multi-modal tokens from the fusion encoder.

ML model106may generate sets of features for videos. ML model106may generate features that are representative of different cues of the videos. ML model106may generate sets of features that include a text feature, a video feature, and a multi-modal feature for each video processed by ML model106. For example, ML model106may generate a text feature based on a pooled group of one or more text classification tokens, a video feature based on a frame classification token of the frame tokens, and a multi-modal feature based on a multi-modal classification token.

ML model106may generate multi-label classifications of videos. ML model106may generate multi-label classifications that are classifications with one or more labels using the generated features. ML model106may generate the multi-label classifications using one or more techniques. For example, ML model106may associate the features of a video with one or more class prototypes, apply a contrastive loss function to the features, determine a cosine distance between the features and the class prototypes, and/or other techniques or combinations of techniques.

ML model106may generate multi-label classifications of videos where the labels are based on educational content requirements. ML model106may use labels that correspond to one or more educational requirements or standards (e.g., Common Core State Standards in the United States, local educational requirements, etc.). For example, ML model106may use labels that are based on one or more educational codes of an education standard. ML model106may generate multi-label classifications that are representative of the educational value of a video (e.g., the content of the video corresponding to one or more educational topics such as mathematics, literature, science, etc.). For example, ML model106may use sub-domains of educational content and/or a research-based content rubric. Classification module104may provide the multi-label classifications to one or more other components of analysis system102, such as recommendation module108. ML model106may classify educational content into domains, subdomains, and difficulty levels aligned to standards, such as Common Core State Standards and Head Start Early Learning Outcomes Framework. Using these classifications, analysis system102may recommend a personalized, sequenced progression of educational videos. Analysis system102may personalize recommendations by matching educational content the child has recently watched. Analysis system102may sequence recommendations to encourage repeated exposure to the same learning standards while slowly introducing more advanced standards. Analysis system102may follow children from grades PreK-5, adjusting to their viewing habits and learning trajectories over time. Analysis system102may use one or more components, such as recommendation module108to generate recommendations.

Analysis system102includes recommendation module108. Recommendation module108may be a software component, such a program, process, module, plugin, or other type of software component. Recommendation module108may generate recommendations for videos based on multi-label classifications generated by classification module104. Recommendation module108may additionally use information regarding a particular user of user system130to generate recommendations. Recommendation module108may use information regarding a particular user entered by the user and/or entered by an individual associated with the user (e.g., a parent or guardian of the user). For example, recommendation module108may use information, such as a user's educational level and/or educational requirements to generate recommendations.

Recommendation module108may generate recommendations of videos for a user to watch. Recommendation module108may generate recommendations by comparing multi-label classifications to information regarding a user. In an example, recommendation module108receives information regarding educational content requirements associated with a user. Recommendation module108compares multi-label classifications of a plurality of videos classified by classification module104to information regarding educational content requirements associated with the user. Recommendation module108determines a set of videos that meet the educational content requirements of the user. For example, recommendation module108may receive an indication of educational content requirements associated with a user (e.g., an educational level of a user). Recommendation module108may use fine-grained content categories and content difficulty levels in generating recommendations.

Recommendation module108may tailor videos to one or more needs of a user. Recommendation module108may receive an indication of one or more needs of a user beyond that of educational content requirements of the user such as educational areas where the user needs tutoring. For example, recommendation module108may receive an indication that a user requires further mathematical education at their educational level and determine one or more videos that have multi-label classifications consistent with the mathematical education. Recommendation module108may determine, based on the multi-label classification of a video, whether a particular video meets one or more content requirements for viewing by a user. Recommendation module108may output an indication of whether the particular video meets one or more content requirements for viewing by the user. Recommendation module108may generate a recommendation for a user to view a particular video. For example, recommendation module108may generate, based on the multi-label classification of the video and the educational content requirements associated with the video, a recommendation for the user to view the video. Recommendation module108may output an indication of the recommendation. Further, recommendation module108may generate media use insights and parenting tips. Recommendation module108may output an indication of the use insights and parenting tips.

Recommendation module108may revise and/or update video recommendations. Recommendation module108may use information, such as parent preferences, popular video content, and/or child viewing habits, to revise video recommendations. In addition, recommendation module108may update video recommendations as a child ages (e.g., as the child moves into more advanced educational content codes). Recommendation module108may output an indication of the revised video recommendations.

Analysis system102includes blocking module110. Blocking module110may be a software component, such a program, process, module, plugin, or other type of software component. Blocking module110may filter and/or block videos from viewing by user system130. Blocking module110may use the multi-label classifications and/or content restrictions to determine one or more videos that should not be viewable or otherwise accessible by a user of user system130.

In some examples, analysis system102is integrated into media source140and used by a provider for media source140to tailor videos to users. In some examples, analysis system102proxies requests sent by user system130for videos available from media source140. The requests can include a request to obtain a particular video and/or a search request for recommended videos available from media source140. Analysis system102may generate and output a recommendation in response to a request, block the request for a particular video, or take other action to process the request. In some examples, analysis system102proxies the videos.

Analysis system102may provide an indication of one or more videos to user system130. Analysis system102may provide the indication based on generating multi-label classifications for a plurality of videos obtained by analysis system102. Analysis system102may provide the indication for user system130to obtain videos that are consistent with educational requirements of a user of user system130. A media application of user system130may obtain those videos based on the indication.

User system130includes media application132. Media application132may be an application, module, program, process, or other type of software component executed by user system130. Media application132may provide media player functionality for user system130. In some examples, media application132may be a companion application of media source140. Media application132may enable a user of user system132to select and play videos from one or more sources such as media source140. Media application132may display videos via GUI134.

User system130includes GUI134. GUI134may be a graphical user interface (GUI) generated and displayed by one or more components of user system130. GUI134may include one or more visual elements generated by one or more components of user system130. For example, GUI134may be generated by an application, e.g., media application132or web browser, and/or operating system of user system130and output by a hardware component (e.g., a display) of user system134.

Media application132may obtain one or more videos from media source140. Media application132may obtain one or more video in response to a request from a user and/or in response to receiving an indication of one or more videos from analysis system102. Media application132may obtain a particular set of one or more videos that is based on the indication of one or more videos from analysis system102. In an example, media application132receives a request from a user to play videos. Media application132determines, based on the indication of one or more videos from analysis system102, which videos to request from media source140. Media application132requests a number of videos from media source based on determining which videos to request.

In some examples, media source140may determine which videos to provide to media application132. Media source140may receive an indication from analysis system102of a selection of videos that analysis system102has selected for a user of user system130. Media source140may provide one or more videos from the selection of videos to user system130in response to a request for videos from user system130. In an example, media source140receives a request for videos from user system130. Media source140determines a set of videos to provide to user system130, where the set of videos is based on an indication of a selection of videos from analysis system102.

Media application132may display a selection of videos obtained from media source140via GUI134. Media application132may display a selection of videos that is based on the indication received from analysis system102. For example, media application132may display a selection of videos obtained from media source140, where the selection of videos is based on the indication from analysis system102(e.g., media application132obtains videos indicated by analysis system102as meeting one or more educational content requirements). In some examples, media application132may display a selection of videos where the videos of the selection of videos are determined by media source140(e.g., media source140provides a selection of videos to analysis system102based on the indication from analysis system102).

Analysis system102may generate a dataset of curated videos annotated with educational content. Analysis system102may follow a standard such as Common Core State Standards to select education content suitable for the kindergarten or prekindergarten level. Analysis system102may consider two high-level classes of educational content: literacy and math. For each of these content classes, analysis system102may select a set of codes. For the literacy class, Analysis system102may select 7 codes and for the math class, analysis system102may select 13 codes. Analysis system102may associate each video with multiple labels corresponding to these codes. Analysis system102may enable annotation of the videos by trained education researchers following standard validation protocol to ensure correctness. Analysis system102may generate the dataset as consisting of carefully chosen background videos, i.e., without educational content, that are visually similar to the videos with educational content. In an example, analysis system102may generate the data as including expert-annotated videos with multiple classes (e.g., x literacy codes, y math codes, and a background).

Analysis system102may use one or more classes of educational content in generating multi-label classifications. For example, analysis system102may consider two high-level classes of educational content: literacy and math. For each of these content classes, analysis system102may select a set of codes. For the literacy class, analysis system102may select codes including, e.g., letter names, letter sounds, following words left to right when reading, sight words, letters in words, sounds in words, and rhyming. For the math class, analysis system102may select codes including, e.g., counting, written numerals, cardinality, comparing groups, subitizing, addition and subtraction, measurable attributes, sorting, spatial language, shape identification, building and drawing shapes, analyzing and comparing shapes, and patterns.

Analysis system102may validate multi-label classifications of video. To ensure the quality and correctness of the annotations, analysis system102may consider educational researchers to annotate the videos and follow a standard validation protocol. Analysis system102may use annotators trained by an expert. In addition, analysis system102may examine annotations on a selected set before engaging the annotator for the final annotation. Analysis system102may enable annotators to start once they reach 90% agreement with the expert. Further, analysis system102may estimate inter-annotator consistency to re-train annotators. Analysis system102may allow a period of time such as a month to train an education researcher to match expert-level coding accuracy.

Analysis system102may curate videos from sources, such as media source140, and enable annotation by the trained annotators to determine educational content in them. Analysis system102may generate a multi-label classification for each video, for each video may have multiple class labels that are quite similar, making the task a multi-label and fine-grained classification problem. For example, ‘letter names’ and ‘letter sounds’ where visual letters are shown in both but in ‘letter sounds’, analysis system102may emphasize the phonetic sound on the letter. Similarly, in both ‘build and draw shapes’ and ‘analyzing and comparing shapes’, multiple shapes can appear but analysis system102may focus the latter by comparing multiple shapes by shape and size. Analysis system102may conduct this task as different from common video classification setups where either multi-label or fine-grained aspects are dealt separately. Analysis system102may use single-label datasets such as HMDB51, UCF101, Kinetics700 and multi-label ones such as Charades as benchmarks for this problem. In addition, analysis system102may use YouTube-Birds and YouTube-Cars as analogous datasets for object recognition from videos and Multi-Sports and FineGym as labeled fine-grained action classes for sports. Analysis system102may use HVU as adding scenes and attributes annotations along with action and objects. Analysis system102may use the multi-label classification as action, object and scene recognition may not be enough for fine-grained video understanding. For instance, videos from a given education provider may share similar objects (person, chalkboard, etc.) and actions (writing on chalkboard) while covering different topics (counting, shape recognition etc.) in each video.

The techniques of this disclosure may include one or more technical advantages that realize one or more practical applications. For example, analysis system102may enable a parent and/or guardian of a user of user system130to ensure that videos viewed by the user are appropriate for the user (e.g., do not contain objectionable content and are age-appropriate). Analysis system102may more accurately classify videos, particular with regarding to educational content classifications, than existing classification techniques. For example, the use of multi-modal tokens and multi-label classification techniques may enable analysis system to more accurately classify videos based on their educational content compared to the existing classification techniques. In addition, the use of multi-label classifications may enable analysis system to provide more comprehensive classifications across a wide range of labels (e.g., educational content codes) instead of being limited to classifying to a single label. Further, analysis system102may enable the parent/guardian to tailor videos to the current educational content requirements of the user. For example, analysis system102may enable the parent/guardian to tailor the videos shown to the user based on an educational level of the user while avoiding the onerous and time-consuming need to manually and preemptively review each video before consumption.

FIG.2is a block diagram illustrating an example analysis system202, in accordance with one or more techniques of this disclosure. Analysis system202may be similar to analysis system102as illustrated inFIG.1and provide similar functionality. For example, analysis system202may include one or more types of computing system.

Analysis system202includes one or more of processors260. Processors260may include one or more types of processors. For example, processors260may include one or more of FPGAs, ASICs, graphics processing units (GPUs), central processing units (CPUs), reduced instruction set (RISC) processors, and/or other types of processors or processing circuitry. Processors260may execute the instructions of one or more programs and/or processes of analysis system202. For example, processors260may execute instructions of a process stored in memory268.

Analysis system202includes memory268. Memory268may include one or more types of volatile data storage such as random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art. Memory268may additionally or alternatively include one or more types of non-volatile data storage. Memory268may store data such as instructions for one or more processes of analysis system202. For example, memory268may store instructions of an operating system for execution by processors260. Memory268may store data provided by one or more components of analysis system202. For example, memory268may store information provided by communication units264.

Analysis system202includes one or more of communication units264. Communication units264may include one or more types of communication units/components such radios, modems, transceivers, ports, and/or other types of communication components. Communication units264may communicate using one or more communication protocols such as WIFI, BLUETOOTH, cellular communication protocols, satellite communication protocols, Asynchronous Transfer mode (ATM), ETHERNET, TCP/IP, optical network protocols such as Synchronous Optical Networking (SONET) and Synchronous Digital Hierarchy (SDH), and other types of communication protocols. Communication units264may enable analysis system202to communicate with one or more computing systems and devices. For example, communication units may enable analysis system202to communicate with a user system such as user system130via network170as illustrated inFIG.1.

Analysis system202includes one or more of input devices262. Input devices262may include one or more devices and/or components capable of receiving input such as touchscreens, microphones, keyboards, mice, and other types of input devices. Input devices262may enable a user of analysis system202to provide input to analysis system202. For example, input devices262may enable a user of analysis system202to type input via a keyboard.

Analysis system202includes one or more of output devices266. Output devices266may include one or more devices and/or components capable of generating output such as displays, speakers, haptic engines, light indicators, and other types of output devices. Output devices266may enable analysis system202to provide output to a user of analysis system202.

Analysis system202includes power source270. Power source270may include one or more sources of power for analysis system202such as solar power, battery backup, generator backup, and power from an electrical grid. For example, analysis system202may be powered by power source270that includes a connection to an electrical grid and a generator backup.

Analysis system202includes one or more of communication channels272(illustrated as “COMM. CHANNELS272” inFIG.2). Communication channels272may include one or more communication channels that interconnect one or more components of analysis system202. Communication channels272may include one or more types of communication channels such as hardware interconnects and/or software interconnects. For example, communication channels272may include a hardware interconnect between memory268and storage devices274.

Analysis system202includes one or more of storage devices274. Storage devices274may include one or more devices and/or components capable of storing data. Storage devices274may include one or more types of non-volatile storage devices such as magnetic hard drives, magnetic tape drives, cloud storage, remote storage, solid state drives, NVM Express (NVMe) drives, optical media, and other types of non-volatile storage. In some examples, storage devices274may include one or more types of volatile storage devices.

Storage devices274includes OS276. OS76may include one or more types of operating system (OS) such as desktop, enterprise, mobile, or other type of OS. OS276may provide an execution environment for one or more programs and/or processes of analysis system202. For example, OS276may provide an execution environment for one or more software components of analysis system202such as classification module276.

Storage devices274includes classification module204. Classification module204may be similar to classification module104as illustrated inFIG.1and provide similar functionality. For example, classification module204may be a software component that generates multi-label classifications of videos from one or more sources of media.

Classification module204may cause analysis system202to obtain videos from one or more sources. For example, classification module204may cause analysis system202to obtain videos from media source140as illustrated inFIG.1. Classification module204may cause one or more components of analysis system202to obtain videos. For example, classification module204may cause communication units264to transmit a request for videos to media source140. Classification module204may cause one or more components of analysis system202to obtain videos in response to a request from user system130. In an example, analysis system202receives a request for videos from user system130. Responsive to the receipt of the request, classification module204cause communication units264to provide a request for videos to media source140. Classification module204may generate the request for videos as including one or more requirements such as search terms, categories of the videos, publishing date ranges, and other requirements.

Classification module204may process videos. Classification module204may process obtained videos to generate multi-label classifications of the videos. Classification module204may generate multi-label classifications where each label of a given multi-label classification corresponds to a characteristic/cue of the video. For example, classification module204may generate a multi-label classification where each label corresponds to an educational content standard (e.g., an educational content code from the Common Core standards). Classification module204may use one or more ML models such as ML models206to generate the multi-label classifications.

Classification module204includes one or more of ML models206. ML models206may be similar to ML model106as illustrated inFIG.1and provide similar functionality. For example, ML models206may include one or more machine learning models such as neural networks, deep learning network, transformer models, encoders, feed-forward networks, perceptrons, time delay neural networks (TDNN), reinforcement learning networks, Q-learning networks, and/or other types of ML models. ML models206may include one or more ML models in one or more stages. For example, ML models206may include a first stage of an encoder model and a second stage of multi-layer perceptrons.

ML models206may use an initial processing of the obtained videos. ML models206may initially process the videos to generate data for use encoding tokens of the videos. ML models206may apply ASR to process the videos and generate text data from the videos (e.g., generate transcripts for the videos). ML models206may additionally and/or alternatively generate frame data of the videos. ML models206may generate frame data of the videos by “breaking up” the videos into the individual frames of the videos.

ML models206may generate text tokens and one or more text classification (CLS) tokens. ML models206may generate tokens for a video using the text data and the frame data. ML models206may generate text tokens that each denotes a unit of meaning within the text data. Text tokens may include, e.g., representations of words within the video transcripts (such as words or sub-words), characters, or vectorized representations of portions of the text data. ML model206may generate text tokens based on the text data and using a text encoder. For example, ML model206may use a text encoder to generate text tokens based on the text extracted from the videos. ML model206may use a transformer to generate word embeddings and the text CLS token representative of a sequence of text tokens for a given video. For example, ML model206may use a Bidirectional Encoder Representations from Transformers (BERT)-based text transformer to generate the one or more text CLS tokens.

ML model206may generate frame tokens. ML model206may generate frame tokens based on the frame data and using a frame encoder. ML models206may generate frame tokens. Frame tokens may be, e.g., image features of the frame, region-based image features of the frame, or vectorized representations of frames, based on the frame data. For example, ML model206may use an image encode that includes a vision transformer (ViT) to generate frame tokens based on the frame data. ML models206may generate one or more frame CLS tokens for the video tokens. For example, ML models206may generate a frame CLS token for each frame token for a given video. ML models206may generate a frame CLS token that is representative of a sequence of frame tokens for a given video. For example, ML models206may generate one or more frame CLS tokens that are corresponding vectors of one or more frames and based on one or more of the frame tokens. ML models206may pool the frame CLS tokens to generate a representation of the frame tokens.

ML models206may generate multi-modal tokens. ML models206may generate multi-modal tokens that are representative of a fusion of the text and video of a video. ML models206may use one or more ML models and/or techniques to generate the multi-modal tokens. For example, ML models206may use fusion encoder276to generate multi-modal tokens for a video.

ML models206include fusion encoder276. Fusion encoder276may be a module of classification module204that includes one or more ML models and/or other software components. For example, fusion encoder276may include a transformer model and a feed forward network among other software components. Fusion encoder276may process text tokens, image tokens, and an initial multi-modal CLS token of a video. In an example, fusion encoder276receives a plurality of text tokens, a plurality of frame tokens, and an initial multi-modal CLS token from classification module204. Fusion encoder276processes the received tokens through a transformer model and a feed forward network and outputs a multi-modal CLS token. Fusion encoder276may output a multi-modal CLS token representative of multi-modal characteristics of a given video.

ML models206may process the text, frame, and multi-modal CLS tokens into features. ML models206may process the text CLS token for a given video into text features of the video, the pooled frame CLS tokens into frame features of the video, and the multi-modal CLS token into multi-modal features of the video. ML models206may generate features that are representations of audio, video, and/or multi-modal cues of the video. For example, ML models206may generate a representation of a video, z, that includes video feature zv, text features zt, and multi-modal (e.g., fusion) features zfsuch that z={zv, zt, zf}. ML models206may use one or more ML models to generate the features for a video. For example, ML models206may use a first encoder to generate the video features, a second encoder to generate the text features, and a third encoder to generate the multi-modal features. ML models206provide the features to contrastive module204.

Contrastive module204includes contrastive module278. Contrastive module278may be a software component of classification module204. Contrastive module278may process features generated by ML models206and generate multi-label classifications of videos. Contrastive module278may associate the features with one or more class prototypes representative that are each representative of one or more characteristics such as educational codes. For example, contrastive module278may associate features of a video with one or more class protypes, where each class prototype is a representative of an educational code. Contrastive module278may use one or more techniques to associate the features of videos with the class prototypes such a determining a distance between each feature and the class protypes, using a contrastive learning model, and/or other techniques. Contrastive module278may use one or more distance metrics, such as cosine distance, Euclidian distance, and/or other types of distance metrics.

Contrastive module278may associate a set of features with one or more labels. Contrastive module278may associate the set of features using one or more techniques. For example, contrastive module278may apply a contrastive loss function to the set of features. Contrastive module278may determine, using the contrastive loss function, a distance between each class prototype of one or more class prototypes and a corresponding feature of the set of features, where each class prototype of the one or more class prototypes are representative of a corresponding classification. Contrastive module278may determine the distance as a cosine distance between the class prototypes and corresponding features.

Contrastive module278may generate and/or learn one or more class prototypes. Contrastive module278may use one or more ML models to generate (e.g., learn) the one or more class prototypes. For example, contrastive module278may use an ML model to generate class prototypes based on education data280, where each class prototype is representative of a corresponding education standard or code. Contrastive module278may learn the one or more class prototypes based on maximized distance between each of the one or more class prototypes.

Storage devices274include education data280. Education data280may be a database, data structure, or other type of information storage scheme. Education data208may include information regarding educational standards of a plurality of locales and for a range of education levels. For example, education data208may include information regarding educational standards for one or more states of the US and associated local jurisdictions (e.g., county and city education requirements) for K-12 grades. Education data208may additionally or alternatively include educational information and standards of other countries. Education data208may include information regarding educational content codes. For example, education data208may include information regarding educational content codes that are representative of educational topics and grade levels.

Contrastive module278may use information obtained from education data280to generate class prototypes. Contrastive module278may generate one or more class prototypes based on an associated educational content code obtained from education data280. In an example, contrastive module278obtains information from education data280regarding seventh grade-level mathematical topics and generates one or more class prototypes based on the obtained information. Contrastive module278may store information regarding class prototypes, multi-label classifications, and other information in media data282.

Storage devices274include media data282. Media data282may be a database, data structure, and/or other type of data storage. Media data282may maintain information such as class prototypes, identifiers of videos, multi-label classifications of videos, and other information. One or more components of analysis system202may use information stored by media data282and/or store data in media data282. For example, recommendation module208may obtain data regarding videos and corresponding multi-label classifications from one or more sources such as media data282.

Storage devices274include recommendation module208. Recommendation module208may be similar to recommendation module108as illustrated inFIG.1and provide similar functionality. For example, recommendation module208may be a software component of analysis system202that generates video recommendations. Recommendation module208may process requests for video received by analysis system202. In an example, analysis system202receives a request for videos and provides information regarding the request that includes a current educational level of an individual to recommendation module208. Recommendation module208processes the information regarding the request to determine one or more videos to recommend for the individual. Recommendation module208may recommend and/or filter videos based on a current educational status of an individual (e.g., current grade, topics requested by the individual, tutoring requests for particular topics, etc.), multi-label classifications of videos, information regarding requests for videos, information regarding the user, content restrictions, requests from the user (e.g., a request for more or less challenging videos, historical use information, and other information. Recommendation module208may use the multi-modal analysis framework to automatically detect fine-grained content categories and difficulty level.

In some examples, recommendation module208may preemptively generate recommendations for a user. Recommendation module208may generate recommendations for videos based on information regarding a user maintained by analysis system202. For example, recommendation module208may generate a recommendation based on the educational status of a user. Recommendation module208may store recommendations such as the preemptively generated recommendations in media data282.

Recommendation module208may provide video recommendations to one or more recipients. Recommendation module208may provide video recommendations in response to a request for video recommendations and/or preemptively provide recommendations prior to a request. Recommendation module208may provide video recommendations to one or more recipients such as user system130and/or media platforms such as media source140. For example, recommendation module208may provide video recommendations to media source140for media source140to filter which videos it provides to a user system.

Storage devices274include blocking module210. Blocking module210may be a software component of analysis system202that prevents and/or blocks an individual and/or user from viewing videos. For example, blocking module210may prevent a user of user system130from viewing videos with inappropriate content and/or insufficient educational content. Blocking module210may determine whether a video should be blocked based on a multi-label classification of the video. For example, blocking module210may determine that a video has insufficient educational content and prevent a user of user system130from viewing the video. Blocking module210may block a video based on determining that a video contains content that is inappropriate for a user of system130and/or that the user is otherwise blocked from viewing (e.g., the video contains content that a parent/guardian of the user has indicated that the user should be prevented from viewing). Blocking module210may block videos based on determining that the video contains inappropriate content such as violence, sexual content, coarse language, consumption of alcohol and/or tobacco products, consumerism (e.g., the video is an unboxing of a children's toy), and other content. Blocking module210may provide an indication to user system130and/or media source140to prevent viewing of the video.

FIG.3is a block diagram illustrating an example user system330, in accordance with one or more techniques of this disclosure. User system330may be similar to user system130as illustrated inFIG.1and provide similar functionality. For example, user system130may be a computing device such as a smartphone, tablet computer, laptop computer, desktop computer, virtual machine, AR goggles/glasses, VR goggles/glasses, and another type of computing device.

User system330includes one or more of processors350. Processors350may include one or more types of processors. For example, processors350may include one or more of FPGAs, ASICs, graphics processing units (GPUs), central processing units (CPUs), reduced instruction set (RISC) processors, and/or other types of processors or processing circuitry. Processors350may execute the instructions of one or more programs and/or processes of user system330. For example, processors350may execute instructions of a process stored in memory360.

User system330includes memory360. Memory360may include one or more types of volatile data storage such as random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art. Memory360may additionally or alternatively include one or more types of non-volatile data storage. Memory360may store data such as instructions for one or more processes of user system330. For example, memory360may store instructions of an operating system for execution by processors350. Memory360may store data provided by one or more components of user system330. For example, memory360may store information provided by communication units352.

User system330includes one or more of communication units352. Communication units352may include one or more types of communication units/components such radios, modems, transceivers, ports, and/or other types of communication components. Communication units352may communicate using one or more communication protocols such as WIFI, BLUETOOTH, cellular communication protocols, satellite communication protocols, Asynchronous Transfer mode (ATM), ETHERNET, TCP/IP, optical network protocols such as Synchronous Optical Networking (SONET) and Synchronous Digital Hierarchy (SDH), and other types of communication protocols. Communication units352may enable user system330to communicate with one or more computing systems and devices. For example, communication units may enable user system330to communicate with analysis system102and/or media source140view network150as illustrated inFIG.1.

User system330includes one or more of input devices354. Input devices354may include one or more devices and/or components capable of receiving input such as touchscreens, microphones, keyboards, mice, and other types of input devices. Input devices354may enable a user of user system330to provide input to user system330. For example, input devices354may enable a user of user system330to type input via a keyboard.

User system330includes one or more of output devices356. Output devices356may include one or more devices and/or components capable of generating output such as displays, speakers, haptic engines, light indicators, and other types of output devices. Output devices356may enable user system330to provide output to a user of user system330. For example, user system330may provide a graphical visualization of one or more videos that may be played.

User system330includes power source358. Power source358may include one or more sources of power for user system330such as solar power, battery backup, generator backup, and power from an electrical grid. For example, user system330may be powered by power source358that includes a battery internal to user system330.

User system330includes one or more of communication channels362(illustrated as “COMM. CHANNELS362” inFIG.3). Communication channels362may include one or more communication channels that interconnect one or more components user system330. Communication channels362may include one or more types of communication channels such as hardware interconnects and/or software interconnects. For example, communication channels362may include a hardware interconnect between memory360and storage devices364.

User system330includes one or more of storage devices364. Storage devices364may include one or more devices and/or components capable of storing data. Storage devices364may include one or more types of non-volatile storage devices such as magnetic hard drives, magnetic tape drives, cloud storage, remote storage, solid state drives, NVM Express (NVMe) drives, optical media, and other types of non-volatile storage. In some examples, storage devices364may include one or more types of volatile storage devices.

Storage devices364includes OS338. OS338may be an operating system (OS) of user system330such as a mobile OS, desktop OS, virtual machine, or other type of OS. OS338may provide an execution environment for one or more programs and/or processes of user system330. For example, OS338may provide an execution environment for one or more software components of user system330such as media application332.

Storage devices364include media application332. Media application332may be an application such as a mobile application, desktop application, browser-based application, or other type of application. In some examples, media application332may be a companion application of a media platform or source such as media source140. Media application332may enable a user of user system330to obtain and view videos. For example, media application332may enable user system330to obtain and play a video from media source140.

Media application332may manage which videos are displayed to a user of user system330. Media application332may manage which videos should be obtained from a media platform or source such as media source140. For example, media application332may refrain from obtaining videos that include objectionable content and/or that contain insufficient educational content. Media application332may obtain videos from media source332that are determined by media source140(e.g., analysis system102provides an indication to media source140of what videos are allowed to be provided to user system330). Media application332may include one or more components that manage which videos should be obtained.

Media application332includes selection module336. Selection module336may be a software component such as a plugin, module, subcomponent, subprocess, and/or another type of software component. Selection module336may manage which videos media application332obtains from media source140. For example, selection module336may cause media application332to obtain a particular set of videos from media source140. Selection module336may cause media application332to obtain videos based on indications received from analysis system102. In an example, media application332receives an indication of a set of videos from analysis system102. Selection module336determines which videos should be obtained from media source140based on the indication from analysis system102. In some examples, selection module336may cause media application332to retrieve a particular set of videos indicated by analysis system102(e.g., analysis system102indicates a particular set of videos that are to be shown instead of a selection of videos from which selection module336may select from).

Media application332may enable a user of user system330(e.g., a parent/guardian) to set criteria for videos displayed to another user (e.g., a child of the parent/guardian). Media application332may enable a user to enter information of the another user such as age, education level (e.g., grade level), topics for focusing/further tutoring (e.g., mathematics, literature, history, etc.), content restrictions (e.g., objectionable content, types of videos, etc.) and other information. Media application332may enable a user to enter the information for analysis system102to tailor the selection of videos to the another user.

Storage devices364include GUI334. GUI334may be a GUI of user system330that includes visual elements of one or more software components of user system330. For example, GUI332may include visual elements generated by OS338and media application332. Media application332may display one or more videos via GUI332.

FIG.4is a diagram illustrating an example operation of a machine learning model400, in accordance with one or more techniques of this disclosure. For the purposes of clarity,FIG.4is discussed in the context ofFIG.1. For example, machine learning (ML) model400may represent or be included in ML model106of analysis system102illustrated inFIG.1.

ML model400may receive data from analysis system102. ML model400may receive data such as text data402and/or video frames404from analysis system202. ML model400may receive data that has been processed by analysis system202. For example, analysis system102may capture audio cues by extracting speech from an audio track of a video. Analysis system102may extract speech by removing background audio (e.g., instruments, noise, etc.). Analysis system102may process the extracted speech using one or more techniques such as ASR to transcribe the text from the speech and generate text data402. Analysis system102may generate video frames404and provide video frames440to ML model400. Analysis system102may generate video frames404may extract frames from a video processed by analysis system102.

Analysis system102may test the performance of text encoder406. Analysis system102may test DistilBERT and T5 backbones for the text encoder. Text encoder406may use BERT as trained to predict masked spans of text. Text encoder406may use T5's unsupervised objective as similar, however, text encoder406may train T5 on predicting the entire sequence instead of the masked spans. Text encoder406may use GPT2 as taking an autoregressive approach to language modeling

ML model400may process video frames using image encoder408. Image encoder408may be a ML model, process, plugin, module, or other type of software component. For example, image encoder408may include a vision transformer that learns frame embeddings (e.g., frame tokens412) from video frame in addition to a CLS token for each frame. Image encoder408may generate a plurality of frame token412in addition to one or more CLS tokens. Image encoder may pool the generated CLS tokens into a compact video representation (e.g., video CLS token416). In some examples, image encoder408may use Random Resized Crop and RandAugment augmentations from torchvision. In addition, image encoder408may use ImageNet pretrained vision encoders ResNet50, ViT-B/32 (224×224 resolution) and ViT-B/16 (384×384 resolution). Image encoder408may include ResNet and/or ViT variants. For example, image encoder408may include a ViT-B/16-384 encoder for a larger COIN dataset. In addition, image encoder408may include a ViT-B/32-224 encoder for APPROVE.

ML model400includes encoders422A-422C (hereinafter “encoders422”). Encoders422may be one or more types of machine learning models and/or layers such as a multi-layer perceptrons. Encoders422may process text CLS token414, video CLS token416, and/or multi-modal CLS token420. Encoder422may process text CLS token414, video CLS token416, and/or multi-modal CLS token420into one or more representations (alternatively referred to as “features” throughout) of text, video, and multi-modal cues of a video. For example, ML model400may generate a text feature using a first neural network, a frame feature using a second neural network, and the multi-modal feature using a third neural network and based on the multi-modal classification token. Encoder422A may process text CLS token414into representation Ztof text cues of a video. Encoder422C may process video CLS token416into representation Zvof video cues of a video. Encoder422B may process multi-modal CLS token into representation Zfof multi-modal cues of a video. ML model may combine, aggregate, and/or otherwise create a representation of a video that is comprised of the three representations such as the video representation Z is comprised of {Zv, Zv, Zf}. Multi-label contrastive loss may be used along with shared prototypes to align the representations across both modalities

ML model400may generate multi-label classifications of a video based on one or more class prototypes such as class prototypes428A-428N (hereinafter “class prototypes428”). ML model400may learn class prototypes428as representations of class labels. For example, analysis system102may train one or more machine learning models based on a plurality of class labels to generate and/or learn class prototypes428. Class prototypes428may be based on information such as educational content standards, objectionable content identifiers (e.g., identifiers of particular types of objectionable content), and/or other information. For example, ML model400may generate class prototype428A as based on an educational standards code for fourth grade-level mathematics.

ML model400may generate multi-label classifications using multi-label contrastive framework426. Multi-label contrastive framework426may include using one or more techniques and/or ML models. Multi-label contrastive framework426may be executed and/or facilitated by a component such as contrastive module278as illustrated inFIG.2. For example, multi-label contrastive framework426executed by contrastive module278may determine a distance between each of class prototypes428and features424. Multi-label426may use the distance to perform inference on features424and generate multi-label classifications. Multi-label contrastive framework426may use one or more types of distance metrics, such as cosine distance, Euclidean distance, and/or other types of distance metrics to perform inference. ML model400may output the generated multi-label classifications to one or more components of analysis system102.

Analysis system102may train ML model400during one or more training processes. Analysis system102may train ML model400using a joint end-to-end learning of one or more components of ML model400(e.g., fusion encoder418). In addition, analysis system102may further refine class prototypes428during a multi-modal training phase. Analysis system102may follow a two-stage training process: during the initial unimodal training phase, analysis system102may utilize fixed prototypes in each modality to align the representations. Then in the second stage, analysis system102may train the unimodal encoders and the multi-modal fusion encoder (e.g., fusion encoder418) end-to-end. ML model400may use cross-modal alignment learned during the first stage to improve the learning of the multi-modal representation. ML model400may use a multi-modal learning phase that includes alternating optimization steps of training the network using contrastive loss and refining class prototypes428.

Analysis system102may optimize ML model400. Analysis system102may employ an optimized such as AdamW for training with a learning rate of 0.0005. Analysis system102may use a weight decay of 1e-6 on the MLP head during contrastive training and the classifier during BCE/Focal/Asym. loss. Analysis system102may use pre-trained vision and text backbones and set the backbone learning rate to 1/10th of the learning rate for the head. Analysis system102may use Exponential Moving Averaging every 10 steps with a decay of 0.999 for the model parameters.

ML model400may process one or more videos to determine whether the videos contain educational content and to characterize the content. ML model400may overcome challenges as the education codes by such as Common Core Standards can be similar such as ‘letter names’ and ‘letter sounds’, where the former focuses on the name of the letter and the latter is based on the phonetic sound of the letter. In addition, ML model400may understand education content that requires analyzing both visual and audio cues simultaneously as both signals are to be present to ensure effective learning. Conversely, standard video classification benchmarks such as other video services may use visual cues to detect the different classes. Finally, unlike standard well-known action videos, ML model400may use education codes that are more structured and not accessible to common users. ML model400may use a carefully curated set of videos and expert annotations to create a dataset to enable a data-driven approach. For example, ML model400may focus on two widely used educational content classes: literacy and math. For each class, ML model400may choose prominent codes (sub-classes) based on the Common Core State Standards that outline age appropriate learning standards. For example, ML model400may use literacy codes that include ‘letter names’, ‘letter sounds’, ‘rhyming’, and math codes include ‘counting’, ‘addition and subtraction’, ‘sorting’, ‘analyzing shapes’.

ML model400may formulate the problem as a multilabel fine-grained video classification task as a video may contain multiple types of content that can be similar. ML model400may employ multi-modal cues since besides visual cues, audio cues may provide important cues to distinguish between similar types of educational content. ML model400may use class prototypes based supervised contrastive learning approach. ML model400may learn a prototype embedding for each class. ML model400may employ a loss function to minimize the distance between a class prototype and the samples associated with the class label. Similarly, ML model400may maximize the distance between a class prototype and the samples without that class label. ML model400may extended classification for the proposed multilabel setup as samples may not be identified as positive or negative due to the multiple labels. ML model400may jointly learn the embedding of the class prototypes and the samples. ML model400may use embeddings that are learned by a multi-modal transformer network (MTN) that captures the interaction between visual and audio cues in videos. ML model400may employ automatic speech recognition (ASR) to transcribe text from the audio. ML model400use an MTN that consists of video and text encoders that learn modality-specific embedding and a cross-attention mechanism is employed to capture the interaction between them. ML model400may use MTN that is end-to-end learned through the contrastive loss.

FIG.5is a diagram illustrating an example operation of a fusion encoder518, in accordance with one or more techniques of this disclosure. Fusion encoder518may be similar to fusion encoder276as illustrated inFIG.2and/or fusion encoder418as illustrated inFIG.4. For example, fusion encoder518may generate multi-modal tokens530.

Fusion encoder518may receive a plurality of tokens. Fusion encoder518may receive a one or more of text tokens510and one or more of frame tokens512. Fusion encoder518may receive text tokens510that are representative of audio of a video. Fusion encoder518may receive frame tokens512that are representative of frames of a video. Fusion encoder518may receive text tokens510and frame tokens512from one or more other components of an analysis system such as analysis system202as illustrated inFIG.2.

Fusion encoder518may process the received tokens using attention module514. Attention module514may a software component of fusion encoder that includes one or more ML models, modules, plugins, processes, and/or other types of software component. For example, attention module514may include a multi-head cross attention module that processes the received tokens and applies multi-head cross attention. In some examples, attention module514may include a module that applies multi-head self-attention. Attention module514may include one or more attention layers. Attention module514may process the obtained tokens and provide an output to a feed forward network and/or other operation of fusion encoder518. For example, attention module514may apply the one or more attention layer to text token510and frame tokens514.

Fusion encoder518may include a component that generates a Kronecker product of the output of attention module514and the tokens received by fusion encoder518. Fusion encoder518may include one or more software and/or hardware components that generate the Kronecker product of the output of attention module514and the received tokens. For example, fusion encoder may include a software product that generates the Kronecker product and provides the output to FFN516.

Fusion encoder518includes feed forward network516(illustrated as and hereinafter referred to as “FFN516” inFIG.5). FFN516may be one or more types of feed forward networks. FFN516may process tokens received by fusion encoder516and tokens processed by attention module514. FFN516may output the processed tokens to one or more recipient components. For example, FFN516may output to an XOR component or operation of fusion encoder518.

Fusion encoder518may include a component that generates an XOR product of the output of FFN516and the Kronecker product. Fusion encoder518include one or more hardware and/or software components that generate an XOR product. For example, fusion encoder518may include an FPGA or software component that generates the XOR product. Fusion encoder518may generate an XOR product that includes one or more multi-modal tokens530and multi-modal CLS token520. Fusion encoder518may output the XOR product to one or more recipient components such as one or more encoders. For example, fusion encoder518may output the XOR product to one or more recipient components for generation of the multi-label classification.

FIG.6is a diagram illustrating an example operation of multi-label contrastive framework600, in accordance with one or more techniques of this disclosure. For the purposes of clarity,FIG.6is discussed in the context ofFIG.4. For example, multi-label contrastive framework600(hereinafter “MCF600”) may be similar to multi-label contrastive framework426as illustrated inFIG.4and provide similar functionality.

MCF600may include a plurality of class prototypes628A-628N (hereinafter “class prototypes628”). Class prototypes628may be learned and/or generated by one or more components of a machine learning model such as ML model400. MCF600may learn class prototypes as the representative for each class and consider these as anchors while determining positive and negative samples. Specifically, for a specific class prototype, a representation is learned to minimize distances between the prototype and samples with this class label and maximize the distances between the prototype and samples without this class label. MCF600may use multi-label contrastive learning instead of c single-label contrastive learning. MCF600may iteratively update the class prototypes while learning the feature representations. MCF600may define C={c1, . . . , cK} as the set of classes where K is the number of classes. For a sample x, MCF600may define Pml(x)={c+k}, c+k∈C as the set of multiple class labels associated with x (positive classes) and c−k∈C\Pml(x) denotes the missing classes (negative classes). MCF600may define CP={cp1, . . . , cpK} as the set of class prototypes. MCF600may use z as the representation for the sample x. MCF600may define multi-label contrastive loss as

MCF600may use one or more techniques for initializing class prototypes628. MCF600may compare the two strategies where after initializing the class prototypes, 1) keeping class prototypes628fixed and learn only the multi-modal embedding of the samples, and 2) class prototypes628and sample embedding are learned iteratively. MCF600may initialize class prototypes628either randomly, and/or with orthogonal constraints. MCF600may use orthogonal initialization when orthogonal initializing performs best in experiments and iterative adjusting the class prototypes achieves better performance. MCF600may consider hierarchical prototypes, for APPROVE, using a 2-level hierarchy where the first level consists of 18 classes, and the second level is the 3 super-classes: math, literacy, and background. MCF600may use 180 task categories of COIN that are organized into 12 domains in the taxonomy provided with the dataset. MCF600may use a hierarchy that imposes an additional constraint on learning the embeddings during training.

MCF600may minimize the loss of the positive class prototype and instance pairs in the numerator minimizes the distance between the representation z and the class prototypes corresponding to the sample, and vice versa for negative classes. MCF600may also utilize negative sampling to account for the class imbalance between positives and negatives

MCF600may learn class-specific prototypes such that the multilabel samples can be thought of as the combinations of the class prototypes selected based on the associated labels. For example, MCF may generate Zt as an N×d matrix of d-dimensional representations, i.e., zs), of N samples and L∈{0, 1}N×K is corresponding labels matrix with K classes. MCF600may denote CPt is a matrix of size K×d of K class prototypes at a training iteration t. Then, Zt=L×CPt+ε, where ε is the residual noise term. MCF600may assume a Gaussian noise that is unbiased and uncorrelated with the labels L and approximate class prototypes as CPt*≈(LT L)−1LT Zt, where operation (LT L) results in a square matrix amenable to inversion. For single labels, MCF600may imply averaging the features of the instances belonging to a given class as the prototype for that class. In a multi-label setup, MCF600may consider the co-occurrence between the labels. MCF may update class prototypes628with learning iterations such as

MCF600may use inference based on class prototypes. MCF600may rely on class prototypes628to carry out inference by utilizing the distance between the learned prototypes and test features. MCF600may use one or more types of distance metrics, such as cosine distance, Euclidean distance, and/or other types of distance metrics. Given the prototype loss-based training, MCF600may determine an estimated probability of a given class proportional to the normalized temperature scaled distance, such as cosine distance. MCF600may normalize the cosine distance such that −1 and 1 correspond to a confidence of 0 & 1 respectively. MCF600may generate a prediction such as:

p^(k❘x)∝exp⁡(sim⁡(z,cpk)τ)where z is a multi-modal representation of the sample x.

MCF600may process inputs using one or more encoders such as encoder622. Encoder622may include one or more machine learning models. Encoder622may include a single encoder model that processes one or more types of tokens. In some examples, encoder622includes an encoder model for processing a corresponding type of token. Encoder622may process the tokens and generate multi-label classifications. In the example ofFIG.6, MCF600may process each sample and the class prototypes corresponding to the labels associated with the sample and treat them as positive pairs. Similarly, MCF600may determine negative pairs based on the missing class labels. MCF600may generate prototypes represented by stars (★) and inputs as circles (∘) colored with all their relevant labels.

MCF600may use one or more datasets. MCF600may use one or more datasets to train and/or evaluate the performance of the multi-label classifications. For example, MCF600may evaluate the approach on datasets such as a subset of Youtube-8M and COIN datasets. YT-8M may include of a diverse set of videos with video and audio modalities. MCF600may consider a subset of YT-8M dataset with 46K videos and 165 classes. MCF600may use a database such as COIN that consists of instructional videos covering a wide variety of domains and spanning over 180 classes.

MCF600may compare efficacy of multi-label classifications against one or more baselines. MCF600may compare against one or more baselines such as:1) Binary cross-entropy. MCF600may compute loss for multiple labels by combining the binary cross-entropy losses for individual classes.2) Focal loss. MCF600may consider a modified binary cross-entropy to assign a higher weight to hard samples by adjusting a focusing parameter γ. MCF600may down-weight negative samples by using a weight α. MCF600may give a positive label such as:

ℒfocal(p)=-α⁡(1-p)γ⁢log⁡(p)where γ=23 and α=0.2.3) Asymmetric loss. MCF600may build upon focal loss by utilizing different focusing parameters such as γ+and γ−for positive and negative samples, respectively. MCF600may ignore native samples with a prediction probability lower than a margin m. For example, MCF600may give asymmetric loss for prediction p corresponding to a label y as:

L-=(max⁡(p-m,0))γ-*log⁡(1-max⁡(p-m,0))MCF600may follow a five step procedure to train a baseline. For example, MCF600may experimentally set {γ−=2, γ+=1, m=0.1} corresponding to the best performance on a dataset such as APPROVE.4) Metrics. MCF600may achieve relatively high precision as part of developing a reliable education content detection framework. MCF600may use a metric such as Recall@80% Precision (hereinafter “R@80”) as a primary metric. MCF600may also consider the standard area under the precision-recall curve (AUPR) that is not sensitive to a specific threshold for making the final prediction. In addition, MCF600may consider a label ranking average precision (LRAP) metric that may be more suitable for the multilabel setup. MCF600may use LRAP to estimate whether the ground truth classes are predicted with higher scores than the rest:

MCF600may compare proposed approaches with one or more baselines. For example, MCF600may determine that a particular approach outperforms the strongest baselines by 3.1% and 2.3% with respect to R@80 and AUPR, respectively. MCf600may determine results for separate models trained on Math and Literacy subsets of APPROVE, respectively. In an example MCF600may determine that results on a Math subset are higher compared to a Literacy subset, which may indicate that the literacy classes are harder to distinguish mostly due to the high inter-class similarity. MCF600may determine that the top three hardest classes are follow words, letters in words, and sounds in words and these are from the literacy set.

MCF600may test the proposed approaches one or more datasets such as public datasets. For example, MCF600may test one or more approaches on public datasets such as YT-46K and COIN. As YT-8M was primarily collected with the intention of visual classification, MCF600may determine that additional use of text data leads to a smaller improvement compared to APPROVE. MCF600may map each video from COIN to a single task. MCF600consider the Top-1 accuracy as the metric. On COIN MCF600may compare an approach with SupCon which may be effective for single labels. MCF600may determine that an approach outperforms SupCon and may justifies the effectiveness of the class prototypes based training in a generic contrastive learning framework.

MCF600may determine the robustness of an approach. For example, MCF600may compare one or more approaches using one or more metrics to evaluate the robustness. MCF600may use metrics such as:1) Noisy modality. MCF600may process videos that have noisy modalities where some of the video frames are missing, or ASR transcription is noisy. MCF600may determine whether an approach is robust against the cases where a percentage of video frames or text words are missing (e.g., due to noise in a video.2) Run-to-Run variance. MCF600may determine the variance across runs. For example, MCF600may determine that a low variance across runs indicates that an approach is not sensitive to random initialization of class prototypes628.3) Initialization the encoders. MCF600may one or more types of pretraining such as ImageNet pretraining for the image encoder. In addition, MCF600may use English Wikipedia+Toronto Book Corpus is used to pre-train the text encoder (e.g., an encoder such as encoder622). MCF600may generate results where the backbones are initialized with CLIP, which may provide a more aligned vision-text representation. MCF600may determine that the results are better with the CLIP initialization. In addition, MCF600may determine that improvements are more significant on COIN than APPROVE as CLIP models may not be exposed to educational videos.

FIG.7is a flow chart illustrating an example operation of an analysis system, in accordance with one or more techniques of this disclosure. For the purposes of clarity,FIG.7is described in the context ofFIG.1.

An analysis system, such as analysis system102, obtains a video that includes text elements and visual elements (702). Analysis system102may obtain the video from one or more sources, such as media source140. Analysis system102may obtain the video in response to a request from a user system, such as user system130, and/or preemptively obtain the video. For example, analysis system102may preemptively obtain a plurality of videos from media source140.

Analysis system102generates, based on the text elements and the visual elements, a plurality of text tokens representative of audio spoken in the video and a plurality of frame tokens representative of one or more frames of the video (704). Analysis system102may generate the text tokens and the frame tokens using one or more techniques. For example, analysis system102may generate the text tokens using a text transformer and the frame tokens using an image encoder. Analysis system102may generate a text classification token such as a text CLS token representative of text tokens. Analysis system102may generate a frame CLS token that is a representation of the video.

Analysis system102generates, using a machine learning model such as ML106, a set of features that includes a text feature, a frame feature, and a multi-modal feature, wherein the multi-modal feature is representative of multi-modal elements of the video, and wherein generating the set of features is based on the plurality of text tokens and the plurality of frame tokens (706). Analysis system102may use ML106that includes one or more layers and/or components to generate the set of features. Analysis system102may process the text tokens and the frame tokens to generate the multi-modal features. For example, ML model106may process the text tokens and the frame tokens using a fusion encoder to generate the features.

Analysis system102associates the set of features with one or more labels to generate a multi-label classification of the video (708). Analysis system102may generate a multi-label classification of the video where the labels are based on one or more educational content codes. Analysis system102may use one or more techniques to associate the set of features with the one or more labels. For example, analysis system102may determine a distance between each feature and a corresponding label, such as a cosine distance.

Analysis system102output an indication of the multi-label classification of the video (710). Analysis system102may output an indication of the classification to one or more recipients. For example, analysis system102may output an indication of the classification to media source140. Analysis system102may output the indication to user system130for a user (e.g., a parent/guardian of a child who uses user system130) to select one or more videos for the child to view.

The above examples, details, and scenarios are provided for illustration, and are not intended to limit the disclosure in any way. Those of ordinary skill in the art, with the included descriptions, should be able to implement appropriate functionality without undue experimentation. References in the specification to “an embodiment,” “configuration,” “version,” etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed to be within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly indicated.

Examples in accordance with the disclosure may be implemented in hardware, firmware, software, or any combination thereof. Modules, data structures, function blocks, and the like are referred to as such for ease of discussion and are not intended to imply that any specific implementation details are required. For example, any of the described modules and/or data structures may be combined or divided into sub-modules, sub-processes or other units of computer code or data as may be required by a particular design or implementation. In the drawings, specific arrangements or orderings of schematic elements may be shown for ease of description. However, the specific ordering or arrangement of such elements is not meant to imply that a particular order or sequence of processing, or separation of processes, is required in all embodiments.

In general, schematic elements used to represent instruction blocks or modules may be implemented using any suitable form of machine-readable instruction, and each such instruction may be implemented using any suitable programming language, library, application programming interface (API), and/or other software development tools or frameworks. Similarly, schematic elements used to represent data or information may be implemented using any suitable electronic arrangement or data structure. Further, some connections, relations or associations between elements may be simplified or not shown in the drawings so as not to obscure the disclosure. This disclosure is to be considered as exemplary and not restrictive in character, and all changes and modifications that come within the spirit of the disclosure are desired to be protected.

Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various operations and functions described in this disclosure. In addition, any of the described units, modules, engines, or components may be implemented together or separately as discrete but interoperable logic devices. Depiction of different features as modules, engines, or units is intended to highlight different functional aspects and does not necessarily imply that such modules, engines or units must be realized by separate hardware or software components. Rather, functionality associated with one or more modules, engines, or units may be performed by separate hardware or software components or integrated within common or separate hardware or software components.

The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. Instructions embedded or encoded in a computer-readable storage medium may cause a programmable processor, processing circuitry, or other processor, to perform the method, e.g., when the instructions are executed. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), Flash memory, a hard disk, a CD-ROM, a floppy disk, a cassette, magnetic media, optical media, or other computer readable media. A computer-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine. For example, a computer-readable medium may include any suitable form of volatile or non-volatile memory. In some examples, the computer-readable medium may comprise a computer-readable storage medium, such as non-transitory media. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).