Patent ID: 12217493

DETAILED DESCRIPTION

In accordance with some embodiments of the disclosed subject matter, mechanisms (which can include methods, systems, and media) for generating video classifications using multimodal video analysis are provided. More particularly, the disclosed subject matter relates to classifying videos as being safe or unsafe for advertisers using information from video frames, audio, and textual data.

In some embodiments, the mechanisms include receiving a video identifier associated with a video content item. For example, the video identifier can be associated with a video being presented on a computing device (e.g., a video shared on a social media application). In another example, the video identifier can be associated with a video that has been uploaded to a social media service.

In some embodiments, the mechanisms can parse and/or otherwise extract an audio portion and a plurality of image frames from the video corresponding to the video identifier. For example, the mechanisms can include one or more base models that analyze the video in parallel. In a more particular example, the mechanisms can include (i) an optical character recognition model that obtains text information corresponding to text appearing in at least one of the image frames, (ii) an image classification model that obtains, for each object appearing in at least one of the image frames, a probability that an object appearing in the image frame falls within a particular image class. In another more particular example, the mechanisms can include an automated speech recognition model that obtains text information corresponding to words that are being spoken in the video.

It should be noted, however, that the mechanism can contain any suitable model and can incorporate any suitable additional models. For example, in some embodiments, an audio tagging model can be used to analyze the audio portion of the video to detect one or more sounds appearing in at least one of the image frames.

In some embodiments, the mechanisms can combine the information obtained from the models applied to the audio portion and image frames extracted from the video to generate a combined analysis output for the video.

In some embodiments, the mechanisms can input the combined analysis output for the video into a trained multimodal neural network that determines a safety score for each of multiple categories that the video contains content belonging to a category of the plurality of categories. For example, a safety score can be generated by the trained multimodal neural network for each of eleven categories including (1) adult and explicit sexual content, (2) arms and ammunition, (3) crime and harmful acts to individuals and society, (4) death and injury, (5) online piracy, (6) hate speech and acts of aggression, (7) obscenity and profanity, (8) illegal drugs/tobacco/e-cigarettes/vaping/alcohol, (9) spam or harmful content, (10) terrorism, and (11) debated sensitive social issues. In a more particular example, the safety score can be a binary classification as to whether the video contains or does not contain content falling within one of the eleven categories.

It should be noted that the multimodal neural network can be trained in any suitable manner. For example, the multimodal neural network can be trained on video examples that have been classified as being unsafe in one or more categories. In another example, the multimodal neural network can be trained on video examples that have been classified as being unsafe in one or more categories and video examples that have been classified as being safe in one or more categories. In yet another example, the multimodal neural network can be trained on video examples selected by an advertiser as being unsafe for the advertiser's brand.

These mechanisms can be used in a variety of applications. For example, an advertiser can receive these safety scores and/or binary classifications to determine whether a particular video meets safety requirements. In continuing this example, the advertiser can determine whether to place an advertisement in connection with the video (e.g., a pre-roll advertisement, a mid-roll advertisement, or a post-roll advertisement). Additionally or alternatively, the mechanisms can provide the advertiser with an indication as to how many advertisements have been placed with a video that is deemed to be unsafe or otherwise unsuitable for a brand associated with the advertiser.

These and other features for generating video classifications using multimodal video analysis are described further in connection withFIGS.1-6B.

Turning toFIG.1, an illustrative example of a system100for generating video classifications using multimodal video analysis in accordance with some embodiments is shown. As illustrated, system100can include a coordination server102, analysis servers103,104, and105, a classification server108, a communication network110, and one or more user devices116.

Coordination server102can be any suitable server(s) for storing information, data, programs, media content, and/or any other suitable content. In some embodiments, server102can perform any suitable function(s). In some embodiments, coordination server102can send and receive messages using communication network110. For example, in some embodiments, coordination server102can combine analysis outputs from analysis servers103,104, and105and/or any other suitable analysis servers into a combined analysis record associated with an input video for transmission to classification server108. In a more particular example, as shown inFIGS.6A and6B, in response to inputting a video having multiple image frames into analysis server104for performing automated speech recognition, analysis server104for performing automated speech recognition, and analysis server105for performing image classification, coordination server102can combine the outputs from each analysis server and transmit the combined analysis information to a multimodal neural network executing on classification server108for classifying the content of the video into each of eleven Global Alliance for Responsible Media (GARM) categories and for indication which GARM categories that the video may be deemed unsafe for providing content, such as an advertisement.

Analysis servers103,104, and105can be any suitable servers for storing information, data, programs, media content, and/or any other suitable content. In some embodiments, analysis servers103,104, and105can send and receive messages using communication network110.

In some embodiments, analysis servers103,104, and105can each be configured to run and/or train a machine learning model (e.g., neural networks, decision trees, classification techniques, Bayesian statistics, and/or any other suitable technique) to perform image and/or audio analysis techniques.

For example, in some embodiments, analysis server103can be configured to run and/or train a machine learning model to perform optical character recognition (OCR). In this example, in some embodiments, analysis server103can train a machine learning model on a dataset such as images from social media which contain metadata and/or text overlaid on video frames. Continuing this example, in some embodiments, analysis server103can additionally run a trained machine learning model to output a transcript of metadata and/or text overlaid on a video frame when given a video outside of the training dataset as input. For example, as shown inFIG.6B, in response to inputting a video having multiple image frames into analysis server103for performing automated speech recognition, analysis server103can output a transcript of text that appears within the image frames of the video (e.g., “How to know if you're a POS”).

In another example, in some embodiments, analysis server104can be configured to run and/or train a machine learning model to perform automated speech recognition (ASR). In this example, in some embodiments, analysis server104can train a machine learning model on a dataset containing speech in any suitable language. Continuing this example, in some embodiments, analysis server104can additionally run a trained machine learning model to output a transcript of an audio record when given a video and/or audio track outside of the training dataset as an input. In another example, in some embodiments, analysis server104can be configured to run and/or train a machine learning model to tag an audio track. In this example, in some embodiments, analysis server104can train a machine learning model to recognize sounds relevant for advertising brand safety (e.g., explosions, gunshots). Continuing this example, in some embodiments, analysis server104can additionally run a trained machine learning model to output a record of audio tags identified in an audio track when given a video and/or audio track outside of the training dataset as input. For example, as shown inFIG.6B, in response to inputting a video having multiple image frames and an audio portion into analysis server104for performing automated speech recognition, analysis server104can output a transcript of the audio portion spoken in each of the image frames of the video (e.g., “How to know if you are a piece of s*** . . . it was better when the bottles were made of glass”).

In another example, in some embodiments, analysis server105can be configured to run and/or train a machine learning model to perform image classification. In this example, in some embodiments, analysis server105can train a machine learning model to classify images across any suitable number of categories. In particular, in some embodiments, analysis server105can train a machine learning model to classify images across 100 or more categories relevant for advertising brand safety (e.g., alcohol, drugs, nudity, extremist symbols). In some embodiments, given an image input to a trained machine learning model, analysis server105can output a probability for each category corresponding to the likelihood that the input image can be classified into each of the categories used to train the machine learning model. For example, as shown inFIG.6B, in response to inputting a video having multiple image frames into analysis server105for performing image classification, analysis server105can extract multiple frames from the video (e.g., each frame, a frame every five seconds, etc.) and output a probability, for each image class, as to whether an object appears within the image frame (e.g., “Person 100%,” “Beer 0%,” “Blood 2%,” “Nudity 2%,” etc.). It should be noted that, as shown inFIG.6B, the image classes having a higher probability can be ranked at the top of the list of image class probabilities for the video.

In another example, in some embodiments, analysis server105(or any other suitable analysis server) can be configured to run and/or train a machine learning model to perform object detection. In this example, in some embodiments, analysis server105can train a machine learning model to detect objects within an image. Continuing this example, in some embodiments, analysis server105can additionally run a trained machine learning model to output a record of objects detected when given an image outside of the training dataset as input.

It should be noted that, although the embodiments described herein include analysis server103for performing optical character recognition, analysis server104for performing automated speech recognition, and analysis server105for image classification, this is merely illustrative and any suitable number of analysis servers can be used. For example, a single analysis server can, in parallel, perform optical character recognition of text appearing in a video, automated speech recognition to detect words being spoken in the video, and image classification to detect objects appearing in the video. In another example, an analysis server can perform analyses on the image frames of the video, such as optical character recognition and image classification, and another analysis server can perform analyses on the audio portion of the video, such as automated speech recognition and audio tagging. In yet another example, additional analysis servers or additional models can be incorporated into system100, such as an analysis server for audio tagging that recognizes sounds occurring in the video (e.g., explosions or gunshots).

Classification server108can be any suitable server for storing information, data, programs, media content, and/or any other suitable content in some embodiments. In some embodiments, classification server108can send and receive messages using communication network110. For example, in some embodiments, classification server108can receive analysis results from coordination server102through communication links112.

In some embodiments, classification server108can run and/or train a multimodal classification machine learning model. For example, classification server108can include a combination of convolutional neural networks and text vectorizers. In a more particular example, the multimodal classifier can be a neural network that receives multiple inputs such as at least one of transcripts or text information from an optical character recognition model that detects text appearing within image frames of the video, transcripts or text information from an automated speech recognition model that detects speech spoken in an audio portion of the video, text based image descriptions generated by social media users, a list of probabilities generated by a pretrained image classifier that images within the image frames of the video fall within particular image classes, a list of audio tags, and/or a list of objects detected in the image frames of the video. In continuing this example, the neural network can process OCR transcripts using tokenization and word embedding. Additionally, in some embodiments, the neural network can process ASR transcripts using tokenization and word embedding. Additionally, in some embodiments, the neural network can process any other suitable text using tokenization and term-frequency inverse-document-frequency weighting. For example, video descriptions can be processed by tokenization and term-frequency inverse-document-frequency (TFIDF) weighting, where the TFIDF values can then be submitted to a fully connected layer. In some embodiments, classification server108can process image classifier predictions in a one-dimensional convolutional layer. For example, image classifier predictions can be padded to a standard length, and then submitted to a one-dimensional convolutional layer. Across all image predictions, the multimodal neural network can then select the maximum value of each dimension of the convolutional output.

In continuing this example, the classification head of the multimodal neural network begins by concatenating the final outputs of the ASR, OCR, description, and image classifier components. The output of this concatenation can then be successively processed by several alternating dropout and fully connected layers. A final fully connected classification layer can then compute the probability of the input video containing each binary GARM category.

In some embodiments, classification server108can store and/or access training data for use with the multimodal classification machine learning model. In some embodiments, the training data can include media content item(s) with audio track(s), video track(s), video description(s), text overlay on video frame(s), and/or any other suitable features. In some embodiments, the training data can include labels indicating a category, classification and/or any other suitable identifier to the audio track, video track, video description, text overlay, and/or any other suitable media content feature. In some embodiments, classification server108can use any suitable amount of training data to train the multimodal classification machine learning model. In some embodiments, classification server108can use a portion of available data to train the multimodal classification machine learning model.

Communication network110can be any suitable combination of one or more wired and/or wireless networks in some embodiments. For example, in some embodiments, communication network can include any one or more of the Internet, an intranet, a wide-area network (WAN), a local-area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and/or any other suitable communication network. In some embodiments, user devices116can be connected by one or more communications links (e.g., communications links114) to communication network110that can be linked via one or more communications links (e.g., communications links112) to coordination server102. The communications links can, in some embodiments, be any communications links suitable for communicating data among user devices116and coordination server102such as network links, dial-up links, wireless links, hard-wired links, any other suitable communications links, or any suitable combination of such links.

Servers102,103,104,105, and108can be implemented using any suitable hardware in some embodiments. For example, in some embodiments, coordination server102can be implemented using any suitable general-purpose computer or special-purpose computer and can include any suitable hardware. For example, in some embodiments, as illustrated in example hardware200ofFIG.2, such hardware can include hardware processor202, memory and/or storage404, an input device controller206, an input device208, display/audio drivers210, display and audio output circuitry212, communication interface(s)214, an antenna216, and a bus218.

Hardware processor202can include any suitable hardware processor, such as a microprocessor, a micro-controller, digital signal processor(s), dedicated logic, and/or any other suitable circuitry for controlling the functioning of a general-purpose computer or a special-purpose computer in some embodiments. In some embodiments, hardware processor202can be controlled by a computer program stored in memory and/or storage204. For example, in some embodiments, the computer program can cause hardware processor202to perform functions described herein.

Memory and/or storage204can be any suitable memory and/or storage for storing programs, data, documents, and/or any other suitable information in some embodiments. For example, memory and/or storage204can include random access memory, read-only memory, flash memory, hard disk storage, optical media, and/or any other suitable memory in some embodiments.

Input device controller206can be any suitable circuitry for controlling and receiving input from one or more input devices208in some embodiments. For example, input device controller206can be circuitry for receiving input from a touchscreen, from a keyboard, from a mouse, from one or more buttons, from a voice recognition circuit, from a microphone, from a camera, from an optical sensor, from an accelerometer, from a temperature sensor, from a near field sensor, and/or any other type of input device in some embodiments.

Display/audio drivers210can be any suitable circuitry for controlling and driving output to one or more display/audio output devices212in some embodiments. For example, display/audio drivers210can be circuitry for driving a touchscreen, a flat-panel display, a cathode ray tube display, a projector, a speaker or speakers, and/or any other suitable display and/or presentation devices in some embodiments.

Communication interface(s)214can, in some embodiments, be any suitable circuitry for interfacing with one or more communication networks, such as network112as shown inFIG.1. For example, interface(s)214can include network interface card circuitry, wireless communication circuitry, and/or any other suitable type of communication network circuitry in some embodiments.

Antenna216can be any suitable one or more antennas for wirelessly communicating with a communication network (e.g., communication network112) in some embodiments. In some embodiments, antenna216can be omitted.

Bus218can be any suitable mechanism for communicating between two or more components202,204,206,210, and214in some embodiments.

Any other suitable components can be included in hardware200in accordance with some embodiments.

Turning toFIG.3, an illustrative example of a process300for generating video classifications in accordance with some embodiments is shown. In some embodiments, process300can be wholly or partially performed by coordination server102, analysis servers103,104, and105, and/or classification server108.

In some embodiments, process300can begin in any suitable manner. In some embodiments, process300can begin when coordination server102receives a request from user device116for analysis of one or more video(s). For example, as shown inFIG.5, process300can begin when a video classification system receives a video identifier from a computing device (e.g., a computing device associated with an advertiser).

At302, process300can receive a video in some embodiments. In some embodiments, process300can receive the video in any suitable manner. For example, process300can receive a media file, a video identification label, and/or a storage location corresponding to a media file in some embodiments. In some embodiments, the video received at302can be a video generated on a social media site, one or more short video clips uploaded by a user to a social media and/or video sharing site, and/or any other suitable video.

At304, process300can parse the video received at302into an audio portion and multiple image frames corresponding to the frames of the video in some embodiments. For example, process300can extract the entire audio portion of the video for analysis and can extract a particular number of image frames from the video (e.g., a video frame that occurs at every 1 second, every frame of a video uploaded at a frame rate of 30 frames per second, etc.).

At306, process300can analyze the audio portion and the image frames using at least one of a plurality of audio and/or image analysis techniques in some embodiments. For example, in some embodiments, analysis server103can analyze the series of images using OCR, image classification, object detection, and/or any other suitable technique. In some embodiments, analysis server104can analyze the audio track using ASR, audio tagging, and/or any other suitable technique. In a more particular example, analysis server103can be configured to run and/or train a machine learning model to perform optical character recognition (OCR) on the extracted image frames. As shown inFIG.6B, in response to inputting a video having multiple image frames into analysis server103for performing automated speech recognition, analysis server104can output a transcript of text that appears within the image frames of the video (e.g., “How to know if you're a POS”). In continuing this example, analysis server104can, in parallel with analysis server103, be configured to run and/or train a machine learning model to perform automated speech recognition (ASR). As shown inFIG.6B, in response to inputting a video having multiple image frames and an audio portion into analysis server104for performing automated speech recognition, analysis server104can output a transcript of the audio portion spoken in each of the image frames of the video (e.g., “How to know if you are a piece of s*** . . . it was better when the bottles were made of glass”). In continuing this example, analysis server105can, in parallel with analysis server103and/or analysis server104, analysis server105can be configured to run and/or train a machine learning model to perform image classification. As shown inFIG.6B, in response to inputting a video having multiple image frames into analysis server105for performing image classification, analysis server105can extract multiple frames from the video (e.g., each frame, a frame every five seconds, etc.) and output a probability, for each image class, as to whether an object appears within the image frame (e.g., “Person 100%,” “Beer 0%,” “Blood 2%,” “Nudity 2%,” etc.).

In some embodiments, each analysis technique used at306can be implemented with a machine learning model, as described above inFIG.1in connection with analysis servers103,104, and105. In some embodiments, any suitable number and/or combination of analysis techniques can be used at306. In some embodiments, process300can use or can abstain from the use of any analysis technique (e.g., OCR) without affecting the results from any other analysis technique (e.g., image classification).

In some embodiments, at306, each analysis technique can produce an output as described in connection with analysis servers103,104, and105inFIG.1above and information flow diagrams inFIGS.4and5below.

At308, process300can combine results from the audio analysis and image frame analysis outputs at306in some embodiments. For example, in some embodiments, process300can write results from analysis server103and analysis server104to the same file and/or location in memory in some embodiments. In some embodiments, process300can use any suitable amount of data and/or metadata which is contained in the analysis output from analysis servers103,104, and105in some embodiments. In some embodiments, process300can combine any other suitable information with the results from306. For example, at308, process300can include a textual description from the metadata of the video and/or any other suitable metadata with the analysis results in some embodiments.

In some embodiments, process300can additionally format analysis results from any and/or all of analysis server103,104, and105at308for use as input to a multimodal machine learning model. For example, in some embodiments, process300can perform tokenization and word embedding on ASR transcripts at308. In another example, in some embodiments, process300can perform tokenization and word embedding on ORC transcripts at308. In another example, in some embodiments, process300can perform tokenization and term-frequency inverse-document-frequency (TDIF) weighting on textual description(s) of the video at308. In another example, in some embodiments, process300can submit predictions from the image classifier analysis to a 1-dimensional convolutional layer at308.

At310, process300can determine a probability that the video contains content from a plurality of categories using the combined and/or formatted analysis results in some embodiments. In some embodiments, process300can use the combined and formatted analysis results from308in any suitable manner at310. In some embodiments, process300can input the combined and formatted analysis results to a trained neural network at310.

For example, as described above, a multimodal neural network can receive multiple inputs such as at least one of transcripts or text information from an optical character recognition model that detects text appearing within image frames of the video, transcripts or text information from an automated speech recognition model that detects speech spoken in an audio portion of the video, text based image descriptions generated by social media users, a list of probabilities generated by a pretrained image classifier that images within the image frames of the video fall within particular image classes, a list of audio tags, and/or a list of objects detected in the image frames of the video. In continuing this example, the neural network can process OCR transcripts using tokenization and word embedding. Additionally, in some embodiments, the neural network can process ASR transcripts using tokenization and word embedding. Additionally, in some embodiments, the neural network can process any other suitable text using tokenization and term-frequency inverse-document-frequency weighting. For example, video descriptions can be processed by tokenization and term-frequency inverse-document-frequency (TFIDF) weighting, where the TFIDF values can then be submitted to a fully connected layer. In some embodiments, the neural network can process image classifier predictions in a one-dimensional convolutional layer. For example, image classifier predictions can be padded to a standard length, and then submitted to a one-dimensional convolutional layer. Across all image predictions, the multimodal neural network can then select the maximum value of each dimension of the convolutional output.

In continuing this example, the classification head of the multimodal neural network begins by concatenating the final outputs of the ASR, OCR, description, and image classifier components. The output of this concatenation can then be successively processed by several alternating dropout and fully connected layers. A final fully connected classification layer can then compute the probability of the input video containing each binary Global Alliance for Responsible Media category or GARM category.

In some embodiments, process300can train a neural network with a set of training data labeled with categories from the plurality of categories. In some embodiments, at310, process300can run a trained neural network with alternating dropout and fully connected layers. In some embodiments, the neural network can include a fully connected classification layer at310.

In some embodiments, at310, the trained neural network can output a probability for each category in the plurality of categories. For example, in some embodiments, at310, process300can output a set of eleven numbers [0.28, 0.01, 0.05, 0.00, 0.00, 0.33, 0.66, 0.70, 0.10, 0.05, 0.13] where each number corresponds to the probability that a video (e.g., social media post) is classified in the corresponding eleven categories set by the Global Alliance for Responsible Media in a framework for responsible brand safety (listed below):1. Adult and Explicit Sexual Content2. Arms and Ammunition3. Crime and Harmful Acts to Individuals and Society4. Death and Injury5. Online Piracy6. Hate Speech and Acts of Aggression7. Obscenity and Profanity8. Illegal Drugs, Tobacco, eCigarettes, Vaping, and Alcohol9. Spam or Harmful Content10. Terrorism11. Debated Sensitive Social Issues

In some embodiments, at312, process300can determine a threshold probability for each category in the plurality of categories in some embodiments. In some embodiments, process300can determine a threshold probability using any suitable mechanism. In some embodiments, process300can use a subset of training data which was reserved from training the neural network at310(“holdout data”). In some embodiments, process300can use a machine learning model, statistical model (e.g., F-score), and/or any suitable mathematical function to determine threshold probabilities. In some embodiments, process300can determine a different threshold for each category in the plurality of categories at312.

At314, process300can, for each category, compare the probabilities determined at310to the thresholds determined at312in some embodiments. In some embodiments, process300can assign a positive binary indicator to a probability that is equal to or above the threshold value (e.g., “yes” or “1”). Similarly, in some embodiments, process300can assign a negative binary indicator (e.g., “no” or “0”) to a probability that is less than the threshold value.

At316, process300can associate any category with a positive indicator from314with the video in some embodiments. In some embodiments, process300can associate any number of categories from the plurality of categories with the video in some embodiments. In some embodiments, process300can associate the categories to the video in any suitable manner. For example, process300can add the positive indicated categories to the metadata of the video in some embodiments.

For example, an advertiser can receive categories associated with a positive indicator to determine whether a particular video meets safety requirements. In continuing this example, the advertiser can determine whether to place an advertisement in connection with the video (e.g., a pre-roll advertisement, a mid-roll advertisement, or a post-roll advertisement).

In another example, an advertiser can receive categories associated with a positive indicator to determine how many advertisements have been placed with a video that is deemed to be unsafe or otherwise unsuitable for a brand associated with the advertiser.

Turning toFIG.4, an example400of an information flow diagram for generating video classifications in accordance with some embodiments is shown. In some embodiments, portions of information flow diagram400can be implemented on coordination server102, analysis servers103,104, and105, classification server108, and user device116.

At410, user device116can transmit a request to coordination server102for analysis of one or more videos in some embodiments. The request can be in any suitable format in some embodiments. For example, as shown inFIGS.4and5, the request can include a video identifier and/or file location in some embodiments. In some embodiments, coordination server102can use the video identifier, file location, or any other form of request to retrieve the video indicated by the request from user device116. It should be noted that user device116can be, for example, a server associated with a social media website that requests a classification of a video. In a more particular example, user device116can correspond with a server the receives uploaded video content items and transmits the video content items to coordination server102or any other server for obtaining a classification as to whether an advertisement should be associated with one or more of the video content items (e.g., safe for advertising or unsafe for advertising). In another more particular example, user device116can correspond with an advertising device that transmits a video identifier to coordination server102for obtaining a classification as to whether the advertising device should proceed with placing an advertisement in connection with the video corresponding to the video identifier (e.g., placing a bid, selecting an advertisement for placement with the video, etc.).

At420, coordination server102can send components of the video to a temporary storage location (e.g., a video storage bucket) in some embodiments. In some embodiments, as shown inFIGS.4and5, coordination server102can transmit an audio portion and multiple image frames sampled or otherwise extracted from a video to a video bucket, as described above in connection with process300inFIG.3.

At430,440, and450, coordination server102can alert analysis servers103,104, and105in some embodiments. Note that only three analysis servers are shown in diagram400, however, any suitable number of analysis servers can be implemented in some embodiments. In some embodiments, coordination server102can send a video id, synchronization key, temporary storage location, and/or any other suitable instructions to analysis servers103,104, and105.

In some embodiments, analysis server103can run a machine learning model trained in OCR as described above in connection withFIG.1. In some embodiments, analysis server103can retrieve the series of images from temporary storage at432. In some embodiments, analysis server103can analyze the series of images as described above in connection with process300inFIG.3. In some embodiments, analysis server103can send a return message to coordination server102at430. For example, analysis server103can send a message including the video id, a synchronization key, and/or text file with OCR transcript to coordination server102at430in some embodiments.

In some embodiments, analysis server104can run a machine learning model trained in ASR as described above in connection withFIG.1. In some embodiments, analysis server104can retrieve the audio track from temporary storage at442. In some embodiments, analysis server104can analyze audio track to produce a transcript as described above in connection with process300inFIG.3. In some embodiments, analysis server104can send a return message to coordination server102at440. For example, analysis server104can send a message including the video id, a synchronization key, and/or text file with an ASR transcript to coordination server102at440in some embodiments.

In some embodiments, analysis server105can run a machine learning model trained in image classification as described above in connection withFIG.1. In some embodiments, analysis server105can retrieve the series of images from temporary storage at452. In some embodiments, analysis server105can analyze the series of images as described above in connection with process300inFIG.3. In some embodiments, analysis server105can send a return message to coordination server105at450. For example, analysis server105can send a message including the video id, a synchronization key, and/or text file with analysis results to coordination server102at450in some embodiments.

In some embodiments, coordination server102can store return messages from analysis servers103,104, and105in a temporary cache at460. In some embodiments, coordination server102can receive return messages from analysis servers103,104, and105at any suitable time and in any suitable order in some embodiments.

In some embodiments, at470, coordination server102can notify classification server108. In some embodiments, coordination server102can send the plurality of analysis results from return messages to classification server108at470in any suitable format.

In some embodiments, classification server108can run and/or train a multimodal machine learning model with inputs received at470as described above in connection withFIGS.1and3.

In some embodiments, at470, classification server108can send a return message to coordination server102. In some embodiments, at470, classification server108can send the binary classification outputs from the multimodal classification model as described above in connection withFIG.3.

In some embodiments, coordination server102can use the return message from470to associate classification outputs from the classification server108with the video. In some embodiments, at480, coordination server102can return a report of classification results for the video to user device116.

In some embodiments, at least some of the above described blocks of the processes ofFIGS.3-5can be executed or performed in any order or sequence not limited to the order and sequence shown in and described in connection with the figures. Also, some of the above blocks ofFIGS.3-5can be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. Additionally or alternatively, some of the above described blocks of the processes ofFIGS.3-5can be omitted.

In some embodiments, any suitable computer readable media can be used for storing instructions for performing the functions and/or processes described herein. For example, in some embodiments, computer readable media can be transitory or non-transitory. For example, non-transitory computer readable media can include media such as non-transitory forms of magnetic media (such as hard disks, floppy disks, etc.), non-transitory forms of optical media (such as compact discs, digital video discs, Blu-ray discs, etc.), non-transitory forms of semiconductor media (such as flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), etc.), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and/or any suitable tangible media. As another example, transitory computer readable media can include signals on networks, in wires, conductors, optical fibers, circuits, any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and/or any suitable intangible media.

Accordingly, methods, systems, and media for generating video classifications using multimodal video analysis are provided.

Although the invention has been described and illustrated in the foregoing illustrative embodiments, it is understood that the present disclosure has been made only by way of example, and that numerous changes in the details of implementation of the invention can be made without departing from the spirit and scope of the invention, which is limited only by the claims that follow. Features of the disclosed embodiments can be combined and rearranged in various ways.