DETERMINING AUDIO AND VIDEO REPRESENTATIONS USING SELF-SUPERVISED LEARNING

Embodiments are disclosed for training a system to generate audio and video representations using self-supervised learning. The method may include receiving a video signal including an audio component and a video component. A first machine learning model is trained to determine a representation of the audio component using a contrastive learning task and a temporal learning task. A second machine learning model to determine a representation of the video component using the contrastive learning task and the temporal learning task. By training the machine learning models using both contrastive learning tasks and temporal learning tasks, the machine learning models learn short term features, long term features, and semantic features of input data.

BACKGROUND

Machine learning is a sub-area of artificial intelligence in which a machine learning model is trained to perform one or more specific tasks. For instance, a machine learning model can be trained to perform a target task by relying on patterns and inferences learned from training data, without requiring explicit instructions to perform the task. Some machine learning models can use the learned patterns and inferences to transform an input into a representation. The representation is an encoded representation of the input data used in downstream processing. Downstream processing tasks that may use the representation include including video processing tasks (including video retrieval tasks, action recognition tasks, classifying frames of a video, tagging video frames, searching video frames for objects, video fingerprinting, etc.), audio processing tasks (including audio retrieval tasks, action recognition tasks, classifying audio data, tagging audio data, searching audio data for words/speakers, audio fingerprinting, etc.), and the like. The accuracy of such downstream tasks is dependent on the ability of the machine learning model learning the patterns and inferences from training data and creating the representation of the input.

SUMMARY

Introduced here are techniques/technologies that train a system to generate representations of video data. Specifically, an audio encoder is trained to learn audio components of video data, and a video encoder is trained to learn visual components of the video data. The system is trained using a combination of contrastive learning with temporal pretext tasks. Specifically, temporal pretext tasks are applied to the audio modality, the video modality, and the video/audio modality. For example, each encoder of the system is trained to perform unitary intra-modal tasks such as classifying a playback speed and classifying a playback direction. Moreover, each encoder of the system is trained to perform pairwise intra and inter modal tasks such as temporal clip ordering.

Additionally, contrastive learning is applied to the video modality, and the video/audio modality. Positive and negative pairs for contrastive learning are determined using an evolving feature space. Specifically, prior feature vectors determined from the system are used to create sample-dependent positive and negative pairs. Training the system includes the expanded set of positive and negative pairs.

The loss determined from training the temporal pretext tasks is combined with the loss from training the contrastive tasks resulting in a total self-supervised learning loss. The total self-supervised learning loss is optimized over time, resulting in the generation of robust audio and video features extracted from audio components and video components of video data respectively.

Additional features and advantages of exemplary embodiments of the present disclosure will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of such exemplary embodiments.

DETAILED DESCRIPTION

One or more embodiments of the present disclosure include a training manager for the training of a system to learn to generate representations of video data. Video data includes both audio components and visual components to capture a scene. Specifically, a segment of video data includes temporal scene dynamics (e.g., object motion) and audio of the scene. Accordingly, video data includes a visual dimension, a temporal dimension, and an audio dimension.

In one conventional approach, training systems use only image data to train a system to generate a representation of video data. For example, conventional approaches train a system using intra-modal contrastive learning in the image domain. Other conventional approaches train a system to generate a representation of video data using intra-modal pretext classification tasks. However, as described, video data includes both audio and visual components. As a result, the conventional approaches are limited in the robustness of the determined representations of video data by excluding additional dimensions such as the audio dimension and the temporal dimension.

To address these and other deficiencies in conventional systems, the training manager of the present disclosure trains a video encoder and an audio encoder to learn representations of video data. The training manager of the present disclosure is used to train a representation system including the video encoder and the audio encoder using image components, audio components, and temporal dynamics of video data.

Specifically, the training manager captures short-term features of the video data by training the representation system using temporal pretext classification tasks such as classifying a playback speed and classifying a playback direction. Such temporal pretext classification tasks are unitary tasks as a single video window and audio segment are classified by the video encoder and the audio encoder respectively. The training manager captures longer-term features of video data by training the representation system using learning tasks at the video level such as clip ordering tasks. Such learning tasks are pairwise temporal pretext classification tasks as two inputs are classified by the video encoder and audio encoder of the representation system respectively.

Additionally, the training manager captures the relationship between the video data, audio data, and temporal dynamics, by training the representation system using contrastive learning. By training the audio encoder and video encoder on positive and negative pairs determined from the two modalities (e.g., audio and video) and on the direction of prediction (e.g., predicting a video representation from audio samples, predicting an audio representation from video samples, and predicting a video representation from video samples), the representation system is able to learn rich semantic information of the video data.

FIG.1illustrates a diagram of a process of training a representation system including an audio encoder and a video encoder, in accordance with one or more embodiments. As shown inFIG.1, embodiments include a training manager100. The training manager100includes a pretext classification training system106, a contrastive learning training system108, and a training module130. The training manager100employs the pretext classification training system106, contrastive learning training system108, and training module130to train a representation system150including a video encoder112and an audio encoder110. The representation system150is illustrated as being trained in various stages including a baseline representation system150-A, a pretext trained representation system150-B, and a fully trained representation system150-C.

For ease of description, the present disclosure describes training manager100the pretext classification training system106training the representation system150first, and subsequently the contrastive learning training system108training the representation system150. However, it should be appreciated that the contrastive learning training system108may train the representation system150first, and subsequently the pretext classification training system106may train the representation system150. Additionally or alternatively, the pretext classification training system106and the contrastive learning training system108may simultaneously train the representation system150(e.g., train the representation system150in parallel).

At numeral1, the training module130passes a baseline representation system150-A to the pretext classification training system106. The baseline representation system150-A includes baseline video encoder112-A and baseline audio encoder110-A. Additionally, the training module130passes training data used by the pretext classification training system106to train the baseline representation system150-A. As described herein, the training data used to train the baseline representation system150-A includes temporally manipulated video and audio signals and clips (e.g., one or more video windows or audio segments of a video or audio signal respectively).

While an encoder is described herein, it should be appreciated that any neural network or machine learning model may be trained using the training module130. A neural network is a machine-learning model that can be tuned (e.g., trained) based on training input to approximate unknown functions. In particular, a neural network can include a model of interconnected digital neurons that communicate and learn to approximate complex functions and generate outputs based on a plurality of inputs provided to the model. For instance, the neural network includes one or more machine learning algorithms. In other words, a neural network is an algorithm that implements deep learning techniques, i.e., machine learning that utilizes a set of algorithms to attempt to model high-level abstractions in data.

In some embodiments, the baseline video encoder112-A and baseline audio encoder110-A may be a video encoder and an audio encoder that have not been pretrained. In other embodiments, baseline video encoder112-A and baseline audio encoder110-A are off-the-shelf models that have been pretrained using general datasets.

The pretext classification training system106trains the baseline representation system150-A (including baseline video encoder112-A and baseline audio encoder110-A) using training data received by the training module130as described herein. Specifically,FIGS.2-5describe training the representation system150to perform temporal pretext tasks using the pretext classification training system106. As a result of the training performed by the pretext classification training system106, the pretext classification training system106fine-tunes the baseline representation system150-A. The fine-tuned baseline video encoder112-A and baseline audio encoder110-A (as part of the baseline representation system150-A) become pretext trained video encoder112-B and pretext trained audio encoder110-B (as part of the pretext trained representation system150-B).

At numeral2, the pretext classification training system106passes the pretext trained representation system150-B (including the pretext trained video encoder112-B and pretext trained audio encoder110-B) back to the training module130.

At numeral3, the training module130passes the pretext trained representation system150-B (including the pretext trained video encoder112-B and pretext trained audio encoder110-B) to the contrastive learning training system108. Additionally, the training module130passes training data (such as positive and negative pairs) used by the contrastive learning training system108to train the pretext trained representation system150-B to perform contrastive learning tasks. Training the representation system150-B using the contrastive learning training system108is described herein.

At numeral4, the contrastive learning training system108passes the fully trained representation system150-C (including the fully trained video encoder112-C and fully trained audio encoder110-C) back to the training module130. The training module130may store the weights of the fully trained video encoder112-C and fully trained audio encoder110-C for use during deployment of the representation system150.

FIG.2illustrates a multi-headed architecture of the encoder202, in accordance with one or more embodiments. A multi-headed architecture is used during training for multitask learning. Multitask learning is when a single machine learning model is trained to perform multiple tasks. A model that is trained using multitask learning includes one or more shared “backbone” layers and “heads” dedicated to perform a specific task. Each head includes the layers of a machine learning model required to perform/learn the specific task associated with that head. That is, each head may utilize a unique loss function to train the particular head to perform a task. Multitask learning improves efficiency as each head receives the same set of features (or other information) determined from the shared portion of the machine learning model (e.g., encoder202). That is, for a three-headed model, the features received by each head are computed once (e.g., by the shared backbone) instead of three-times if each head of the model was its own machine learning model. This efficient sharing is useful in cases where the multitask model learns related tasks. The multitask learning is used to learn different features (short term features and long term features) of data such that a robust representation of the data is determined. That is, the audio encoder110learns to determine a robust representation of an audio signal, and the video encoder112learns to determine a robust representation of a video signal.

As shown inFIG.1, both the audio encoder110and the video encoder112of the representation system150are trained to perform contrastive learning tasks and temporal pretext tasks using the pretext classification training system106and the contrastive learning training system108respectively. The encoder202is representative of both the audio encoder110and the video encoder112of the representation system because both the audio encoder110and the video encoder112undergo the same training. That is, the encoder202is the shared “backbone” of the multiheaded architecture.

Encoder202is trained to perform unitary intra-modal tasks such as classifying a playback speed (using speed classifier208) and classifying a playback direction (using direction classifier204). Moreover, encoder202is trained to perform pairwise intra and inter modal tasks such as temporal clip ordering using the temporal clip ordering classifier, as described inFIGS.4A and4B, and contrastive tasks using the projection multi-layer perceptron (MLP)212. The speed classifier208, the direction classifier204, the temporal clip ordering classifier, and the projection MLP212are each heads sharing the result of the shared backbone (e.g., encoder202).

The encoder202(including either the audio encoder110or the video encoder112) determines a latent space representation of an input (e.g., a video window or an audio segment). The latent space representation is a representation such as a feature vector of extracted properties/characteristics of the input. In some embodiments, the audio encoder of encoder202is a two-dimensional convolutional neural network and the video encoder of encoder202is a three-dimensional convolutional neural network. However, other suitable encoders can be used as the audio encoder and/or video encoder. The audio encoder110determines a feature vector from an audio component of video data, and the video encoder112determines a feature vector from a video component of the video data. Such features are used as input to the speed classifier208, the direction classifier204, the temporal clip ordering classifier, and the MLP212.

The pretext classification training system106trains the encoder202(e.g., the audio encoder and the video encoder) to learn the temporal features of the audio data and the video data using several temporal pre-text tasks. Pre-text tasks are tasks solved by a machine learning model (or heads of a machine learning model) to learn patterns and inferences of input data. Such learned relationships facilitate the heads to learn features of the input data. In this manner, the encoder202learns temporal features of both audio data and video data.

Specifically, the pretext classification training system106trains various classifiers to capture both short-term and longer-term features of the video data. For example, the speed classifier208and the direction classifier204are used to capture short-term audio/video features of video data. Such temporal pretext classification tasks are unitary tasks as a single video window and audio segment are classified by the video encoder and the audio encoder of the representation system respectively. The pretext classification training system106trains the speed classifier208and the direction classifier204in the audio domain and video domain respectively. The temporal clip ordering classifiers are used to capture longer-term features of video data by training the audio encoder and video encoder on clip-level tasks, where clips are audio segments or video windows of the video data. The pretext classification training system106trains such clip-level ordering in the video domain, the audio domain and the cross-modal domains.

The pretext classification training system160optimizes the weights of the encoder202based on the losses of each of the heads performing temporal pretext tasks. The losses used for training each of the heads may be any loss such as cross entropy loss, mean squared error loss, root mean squared error loss, and the like. Mathematically, the loss optimized by the pretext classification training system160is defined as Equation (1) below:

The contrastive learning training system108trains the encoder202(e.g., the audio encoder and the video encoder) to learn features of the audio data and the video data using contrastive learning tasks. Specifically, projection layers (e.g., projection MLP212) are trained by the contrastive learning training system108to learn intra modal and inter modal contrastive tasks. By training the audio encoder and video encoder on positive and negative pairs determined from the two modalities (e.g., audio and video) and on the direction of prediction (e.g., predicting a video representation from audio samples, predicting an audio representation from video samples, and predicting a video representation from video samples), the contrastive learning system108trains the encoder202to learn rich semantic information of the video data.

Contrastive learning is a mechanism of learning that utilizes self-supervised learning to minimize a distance (such as Euclidean distance) between similar samples in an embedding space and maximize a distance between dissimilar samples in the embedding space. The contrastive learning training framework used to train the projection MLP212involves one or more loss functions to push similar samples together and repel dissimilar samples away from each other. Accordingly, an input sample is compared to a similar sample (resulting in a positive pair) and a dissimilar sample (resulting in a negative pair). The contrastive learning training system108optimizes the weights of the encoder202based on the losses of each of the contrastive tasks. Mathematically, the loss optimized by the contrastive learning training system108is defined as Equation (2) below:

In Equation (2) above, the loss optimized by the contrastive learning training system108LCRLis based on two modalities (e.g., the audio domain and the video domain) and on the direction of prediction (e.g., predicting a video feature vector from audio components of video data or predicting an audio feature vector from video components of video data). Specifically, LCRLuses the video-video loss contrastive term lvv. As described herein, the video-video contrastive loss term uses the feature vector vir, sample-dependent positive pairs Pivvin the intra-modal video domain, and sample-dependent negative pairs Nivvin the intra-modal video domain. The contrastive learning loss LCRLalso uses the video-audio loss contrastive term lva. As described herein, the video-audio loss term uses the feature vector vir, sample-dependent positive pairs Pivain the inter-modal audio domain, and sample-dependent negative pairs Nivain the inter-modal audio domain. The contrastive learning loss LCRLalso uses the audio-video loss contrastive term lav. As described herein, the audio-video loss term uses the audio feature vector αir, sample-dependent positive pairs Piavin the inter-modal video domain, and sample-dependent negative pairs Niavin the inter-modal video domain.

Equation (3) below illustrates an example general form of the contrastive objective using the video-audio loss term.

As illustrated in Equation (3), d(x, y) represents the similarity between the feature representations x and y, λ represents a temperature parameter, and ϕvrepresents a predictor MLP. As illustrated, the second argument (e.g., y) is not back propagated.

As a result of both the pretext classification training system106and the contrastive learning training system108, the total loss optimized using the training module130is shown in Equation (4) below.

The training module130provides training data to the pretext classification training system106and the contrastive learning training system108. Specifically, training data is used to train each classifier head trained by the pretext classification training system106and the contrastive learning training system108. The training module130generates training data by performing one or more augmentations to data.

For example, training data provided to the classifier heads may be augmented by one or more temporal transformations. Specifically, the training module130may perform a temporal transformation to randomly temporally crop video data to generate a video component and audio component. As described herein, video data randomly cropped by the training module130results in a video window (or one or more video frames) including visual content. The size of each temporally cropped video window may be the same size or different sizes. Similarly, the audio data is cropped by the training module130, resulting in one or more audio segments including aural content. The size of each temporally cropped segment of the audio data may be the same size or different sizes. In some embodiments, each video window temporally cropped corresponds to a temporally aligned audio segment. In other words, at a given point in time, the video of the video window is accompanied by audio of the audio segment.

The training module130may also perform additional temporal transformations to the generated video components and audio components (e.g., video window and audio segment respectively). For example, the training module130may speed up or slow down the speed of a particular video window and the corresponding audio segment. In this manner, the training module130manipulates the playback speed. The training module130may be configured to manipulate the playback speed using any suitable technique. In one example implementation, the training module130speeds up the playback of the audio segment and/or the video window using temporal subsampling.

Another temporal manipulation performed by the training module130includes reversing the direction of a particular window and the corresponding audio segment. For example, the training module130may play the contents of a particular window in reverse.

The temporal manipulations (τ) applied to the video domain for a particular video window are the same as the temporal manipulations applied to the audio domain for a corresponding (e.g., temporally aligned) audio segment. That is, τr(vj), representing the temporal manipulation of a video window in the video domain, and τi(ar), representing the temporal manipulation of an audio segment in the audio domain, represent the same moment in time in both the audio domain and the video domain. In operation, the training module130applies temporal transformations to both the raw audio signals and the raw video signals (e.g. the audio/video components of video data). Specifically, the training module130subsamples the audio signal (to perform speed-related temporal manipulations) and reverses the direction of the audio signal (to perform direction-related temporal manipulations) before computing a spectrogram. Additionally or alternatively, the training module130may perform the one or more temporal manipulations in the audio spectrogram (therefore not manipulating frequency).

In contrast, additional augmentations performed by the training module130on only the video domain include randomly spatially cropping the video data. For example, the training module130may zoom in/zoom out of the video data, resizing the video data. Additionally or alternatively, geometric augmentations, such as horizontal flipping, may be performed by the training module130on the video domain. Moreover, the training module130may also perform color-jittering. In some embodiments, such augmentations are performed by the training module130when creating positive and/or negative pairs as training data for the contrastive learning training system108, as described herein. The augmentations applied to the video domain facilitate the video encoder learning invariances.

In some embodiments, the manipulations performed by the training module130are randomly sampled from a distribution of possible manipulations. For example, the training module130may randomly select a speedup class from classes such as 1×, 2×, 4×, and 8×. In other embodiments, the manipulations performed by the training module130are performed according to a sequence of manipulations.

The training module130generates training data in a self-supervised fashion. That is, the training module130performs one or more known manipulations on the audio data and/or the video data, resulting in a generated label corresponding to the performed one or more manipulations. For example, if the training module130speeds up the playback of a window and a corresponding audio segment by two, then the training module130can label the resulting sped up window and corresponding audio segment with an identifier indicating “2× speed up.”

The training module130may also generate positive/negative pairs for use as training data in the contrastive learning training system108. As described herein, when generating positive/negative pairs, the training module130may query one or more data stores210. The data store210is a database, a memory bank, a server, or an application that may be hosted by the training manager100, hosted by one or more external systems, or some combination (e.g., a cloud storage system). The data store210includes previously computed feature vectors for data such as audio segments, video windows, audio components, video components, and the like.

The training module130is configured to determine sample-dependent positive/negative pairs using features of a training sample (e.g., a feature vector of a current window and/or a feature vector of a current audio segment) and prior feature vectors retrieved from the data store210. Specifically, the training module130queries a data store for prior feature vectors of video windows of video components (e.g., a video signal of video data) and/or prior feature vectors of audio segments of audio components (e.g., an audio signal of video data). In some implementations, the training module130queries a data store210storing prior audio features (e.g., Qa). In these implementations, the training module130queries a data store210storing prior video features (e.g., Qv). In other implementations, a single data store210may include both prior audio features and prior video features.

FIG.3illustrates an example of positive pairs determined by the training module for use as training data in the contrastive learning training system, in accordance with one or more embodiments. Inter-modal (or cross-modal) positive pairs such as positive pair320may be determined by the training module130using a video window320A of the video component302and a corresponding (e.g., temporally aligned) audio segment320B of the audio component304. The training module130may also determine intra-modal positive pairs. For example, positive pair322indicates a positive pair including a first video window322A and a second video window322B of the video component302. In this manner, the training module130determines positive pairs from a current video data (including video component302and audio component304). The current video data is referred to herein as a mini-batch B.

The training module130can determine negative pairs using a data store (such as data store210ofFIG.2). Additionally, the training module130extends the set of positive samples using the data store. As described herein, the training module130determines sample-dependent positive/negative pairs using features of a training sample (e.g., a feature vector of a current video window and/or a feature vector of a current audio segment) and prior feature vectors retrieved from the data store. Specifically the training module130queries prior feature vectors of video components and/or prior feature vectors of audio components.

In particular, prior feature vectors that are “closest” to the training sample (e.g., the current feature vector of the window and/or audio segment) are determined by the training module130to be positive samples. In contrast, prior feature vectors that are a threshold number of samples away from the training sample are determined by the training module130to be “farther” feature vectors and therefore negative samples.

The training module130determines the closest prior feature vectors to the training sample by comparing the similarity of the current feature vector of the video window and/or the audio segment to a portion of (or all) the prior feature vectors stored in the data store. The training module130determines the similarity of the prior feature vectors and the current feature vector using any suitable similarity calculation. For example, the training module130may compute the cosine similarity between a prior feature vector retrieved from the data store and the current feature vector determined by the audio encoder or the video encoder.

Subsequently, the training module130sorts the prior feature vectors based on the similarity to the current feature vector. The prior feature vectors up to (and/or satisfying) a threshold are determined to be the “closest” prior feature vectors. The prior feature vectors after (and/or satisfying) the threshold are determined to be the “farther” historic feature maps. In some implementations, the training module130determines the closest prior feature vectors using a first threshold, and the training module130determines the farther prior feature vectors using a second threshold.

In some embodiments, the threshold used to distinguish “close” historic feature maps from “far” historic feature maps is a predetermined number of prior feature vectors. For example, k nearest neighbor prior feature vectors are determined to be positive samples. In a particular example, if k=4, then the first four prior feature vectors are determined to the positive samples and any prior feature vector after the fourth prior feature vector are determined to be negative samples.

Mathematically, the set of sample dependent positive pairs and sample dependent negative pairs can be represented in Equations (5)-(8) below. In Equations (5) and (6), the video feature vector viris the output of the video encoder Fv(such as encoder202, and specifically video encoder112) followed by a projection MLP ψv(such as projection MLP212) for an input with a temporal transformation τr(vi) determined by the training module130. Mathematically, vir=ψv(Fv(τr(vi). Augmentations of the video data for the video feature vector viare represented as {circumflex over (v)}is. Equations (5) and (6) represent the set of positive and negative pairs in the intra-modal video domain. In Equations (7) and (8), the audio feature vector αiris the output of the audio encoder Fa(such as encoder202, and specifically audio encoder110) followed by a projection MLP ψa(such as projection MLP212) for an input with a temporal transformation τr(ai) determined by the training module130. Mathematically, αir=ψa(Fa(τr(ai))). Equations (7) and (8) represent the set of positive and negative pairs in the inter-modal domain.

Equation (5) above represents the sample-dependent positive pairs Pivvdetermined by the training module130in the intra-modal video domain for a video feature vector vir. The set of positive pairs for a video feature vector virincludes data augmented versions of the video feature vector (such as {circumflex over (v)}is). Additionally, the set of positive pairs includes NN1:k({circumflex over (v)}is,Qv), or the set of k nearest neighbors from 1 to k of the augmented feature vector {circumflex over (v)}isdetermined from the data store Qv(illustrated as data store210). In some embodiments, the number of k nearest neighbors is set to 5. It should be appreciated that the positive pairs are not temporally aligned.

As shown in Equation (5), the k nearest neighbors of the set of positive pairs extracted from the data store Qvare weighted. In some embodiments, the training module130weighs each of the extracted positive pairs from the data store of equal importance. In other embodiments, the training module130weighs the extracted positive pairs according to a cross-view similarity. For example, the training module130weighs the extracted positive pairs through the feature space similarity to the augmented feature vector {circumflex over (v)}is. An example of weighting the set of positive pairs according to feature space similarity is represented in Equation (9) below.

Equation (6) above represents the sample-dependent negative pairs Nivvfor a video feature vector virdetermined by the training module130. The negative pairs Nivvare generated using the cross-view induced neighborhood structure such that the negative pairs are sample dependent. The negative pairs contain all of the video features (or a portion of the video features) not belonging to the video feature vector viin the current training batch B. Additionally, the negative pairs are the set of nearest neighbors from q to q+m of the augmented feature vector is determined from the data store Qv. In some embodiments,

such that the negative samples start at the nearest neighbor in the data store Qvat least a medium distance to the augmented video feature vector {circumflex over (v)}is. By determining the set of negative pairs in this sample-dependent manner (using nearest neighbors, for instance), the difficulty of the negative samples is controlled. For example, negative samples may be excluded to prevent ambiguous or confusing negative samples resulting from duplicates and/or class imbalance.

Equation (7) above represents sample-dependent positive pairs Pivaand Pivadetermined by the training module130in the inter-modal video/audio domain for a video feature vector vir. The set of positive pairs for the video feature vector virincludes temporally aligned identically transformed audio feature vector αir. Such temporal alignment is important for cross-modal contrastive learning. Additionally, the set of positive pairs includes NNk(αir,Qa), or the set of k nearest neighbors of the audio feature vector αirand the video feature vector virdetermined from the data stores Qaand Qvrespectively.

As shown in Equation (7), the k nearest neighbors of the set of positive pairs extracted from the data store Qvand Qaare weighted. In some embodiments, the training module130weighs each of the extracted positive pairs from the data store of equal importance. In other embodiments, the training module130weighs the extracted positive pairs according to a cross-view similarity. For example, the training module130weighs the extracted positive pairs through the feature space similarity to the temporally aligned sample in the other modality. For example, the set of positive pairs Pivaare weighted through the feature space similarity to the temporally aligned audio sample (e.g., audio feature vector αir), and the set of positive pairs Piavare weighted through the feature space similarity to the temporally aligned video sample (e.g., video feature vector vir). An example of weighting the set of positive pairs according to feature space similarity is represented in Equation (9) below.

Equation (8) above represents the sample-dependent negative pairs Nivaand Niavfor a video feature vector virdetermined by the training module130. The negative pairs contain all of (or a portion of) the video features and audio features not belonging to the video feature vector viand the audio feature vector αiin the current training batch B. Additionally, the negative pairs are the set of nearest neighbors from q to q+m of the audio feature vector αirand the video feature vector virdetermined from the data stores Qaand Qvrespectively. By determining the set of negative pairs in this sample-dependent manner (using nearest neighbors, for instance), the difficulty of the negative samples is controlled. For example, negative samples may be excluded to prevent ambiguous or confusing negative samples resulting from duplicates and/or class imbalance.

Equation (9) above represents the set of weights applied to the set of positive pairs extracted from the data store Qaor Qv. The weights wjapplied to each prior feature vector extracted from the data store is proportional to the similarity of the nearest neighbor njto v (either the audio feature vector αiror the video feature vector vir).

In other embodiments, the threshold used by the training module130to determine “close” historic features (and therefore positive samples) is a predetermined threshold similarity score. For example, if the threshold is 0.8, any prior feature vectors resulting in a similarity score of 0.8 and above are determined to be positive samples. In contrast, any prior feature vectors resulting in a similarity score of 0.79 and below are determined to be negative samples. In some embodiments, the threshold is determined by one or more users (e.g., administrators). In other embodiments, the threshold is dynamically determined by the training module130over time. For example, the threshold is adjusted by the training module130based on the contrastive loss error.

FIGS.4A-4Billustrate a Siamese network used in classifying temporal clip ordering, in accordance with one or more embodiments. Configuring the temporal clip ordering classifier as a Siamese network is one non-limiting way to configure pairwise learning. As described herein, the temporal clip ordering classifiers perform pairwise inter and intra modal classifications.

The classifiers use two encoders (in a Siamese fashion) and compare the similarity of features determined by each encoder using the class selector406. For example, the class selector406compares the similarity of two feature vectors using cosine similarity, Euclidean distance, a correlation of features in the feature vectors, and the like. In some implementations, the class selector406determines a temporal clip ordering classification by mapping the similarity score to a classification using one or more thresholds. In this manner, the class selector406performs a three-way classification of two temporal signals to identify whether the two signals are correctly ordered, overlapping, or wrongly ordered. As described, the temporal clip ordering classifiers classify the inputs into three classes, however the temporal clip ordering classifier may be trained to classify the inputs into other classes.

As shown, each classifier receives two inputs (e.g., time signals such as video windows and/or audio segments) such that the classifier can determine the temporal ordering. However, it should be appreciated that each classifier may receive one input of a concatenated representation of two time signals along the channel dimension.

FIG.5illustrates an example process of self-supervised learning used to train classifiers of the pretext classification training system to perform the temporal pretext tasks, in accordance with one or more embodiments. Supervised learning is a method of training a machine learning model given input-output pairs. An input-output pair is an input with an associated known output (e.g., an expected output, a labeled output, a ground truth). Because the training module130determines the outputs corresponding to the inputs, the learning is considered self-supervised.

As described herein, the training module130generates training data by applying one or more temporal manipulations to an audio component and/or a video component. In these embodiments, the input of the input-output pair is the manipulated data, and the corresponding output is the known one or more manipulations applied to the data.

Additionally or alternatively, the training module130generates training data by sampling one or more audio segments and/or video windows (referred to herein as “clips”) from an audio component and/or a video component respectively. In these embodiments, the input of the input-output pair is two clips, and the corresponding output is whether the two clips were sampled in sequence (e.g., the two clips are “correctly ordered), sampled out of sequence (e.g., the two clips are incorrectly ordered), or overlapping (e.g., one or more audio segments and/or video segments overlap between the first clip and the second clip). Sampling the clips “in sequence” refers to sampling the first clip at a first point in time and sampling the second clip at a second point in time after the first point in time. Sampling the clips “out of sequence” refers to sampling the first clip at a point in time after sampling the second clip.

The pretext classification training system106is used to train the classifiers508on input-outputs pairs. Classifiers508refer broadly to heads of encoder202. The encoder202represents both an audio encoder110and a video encoder112. Specifically, there are unitary classifiers such as a speed classifier (e.g., the speed classifier head208), and a direction classifier (e.g., the direction classifier head204), and pairwise classifiers such as temporal clip ordering classifiers (e.g., video classifier410, audio classifier402, video/audio classifier412, and audio/video classifier414as described inFIGS.4A-4B). The pretext classification training system106also trains the encoder backbone (e.g., encoder202) based on the accuracy of the classifiers508.

As a result of the self-supervised training, the encoder202and classifiers508learn how to predict known outputs (e.g., classification such as a playback speed, a playback direction, and a clip ordering) given known inputs (e.g., one or more audio segments or video windows). The unitary classifier learns intra-modal classification and the pairwise classifiers lean both intra-modal and inter-modal classification.

Specifically, if the pretext classification training system106is training a unitary classifier such as the speed classifier head (one example of classifier508) of the audio encoder (one example of encoder202), the training module130may provide, as training input502, a sped up (or slowed down) audio segment. The training module130also provides, as actual output518, a corresponding label such as the temporal manipulation applied to the audio segment indicating the amount that the audio segment was sped up (or slowed down).

Similarly, if the pretext classification training system106is training a unitary classifier such as the speed classifier head (one example of classifier508) of the video encoder (one example of encoder202), the training module130may provide, as training input502, a sped up (or slowed down) video segment. The training module130also provides, as actual output518, a corresponding label such as the temporal manipulation applied to the video segment indicating the amount that the video segment was sped up (or slowed down).

If the pretext classification training system106is training a unitary classifier such as the direction classifier head (one example of classifier508) of the audio encoder (one example encoder202), the training module130may provide, as training input502, an audio segment in a playback direction (e.g., forward or reverse). The training module130also provides a corresponding label indicating the playback direction as actual output518.

Similarly, if the pretext classification training system106is training a unitary classifier such as the direction classifier head (one example of classifier508) of the video encoder (one example encoder202), the training module130may provide, as training input502, a video segment in a playback direction (e.g., forward or reverse). The training module130also provides a corresponding label indicating the playback direction as actual output518.

If the pretext classification training system106is training pairwise classifiers (e.g., video classifier410, audio classifier402, video/audio classifier412, and audio/video classifier414as described inFIGS.4A-4B), the training module130provides two inputs to the pretext classification training system106. For example, the first input passed to the video classifier410and video/audio classifier412may be a video window. Similarly, the first input passed to the audio classifier402and audio/video classifier414may an audio segment. The second input passed to the video classifier410and audio/video classifier414may be a video window. Similarly, the second input passed to the audio classifier402and the video/audio classifier412may be an audio segment.

To train the unitary classifiers (e.g., the speed classifier208and the direction classifier204), the training module130provides a training input502to the encoder202and/or classifier508. To train the pairwise classifiers (e.g., video classifier410, audio classifier402, video/audio classifier412, and audio/video classifier414as described inFIGS.4A-4B), the training module130provides two training inputs502to the encoder202and/or classifier508. As described herein, the encoder202extracts a feature vector from the training input502. That is, audio encoder110determines an audio feature vector from an audio segment, and video encoder110determines a video feature vector from a video window.

After the classifier508(including both the unitary classifiers and the pairwise classifiers) has received the one or more training inputs502, the classifier508classifies the received training input(s).

For example, the encoder202and unitary classifier508use the training input502(e.g., the temporally manipulated signal in the audio/video domain) to predict output506by applying the current state of the classifier508to the training input502and/or feature vector determined by the encoder202. Specifically, the classifier508may use a softmax function, or a normalized exponential function, to transform real numbers into a normalized probability distribution over predicted output classes. For example, the classes of the speed classifier may include 1×, 2× speed up, 4× speed up, 8× speed up, 2× speed down, 4× speed down, 8× speed down. The classes of the direction classifier may include forward or reverse. The classifier508creates a vector of probabilities corresponding to the probability of the training input502belonging to a particular class. The vector of probabilities becomes the predicted output506.

The pairwise classifier508determines a similarity of feature vectors determined by the encoders202from the training inputs502(e.g., two intra-modal or inter-modal clips). The pairwise classifier508calculates a similarity between the feature vector associated with the first input and the feature vector associated with the second input. As described herein, the similarity may be a cosine similarity, Euclidean distance, a correlation of features in the feature vectors, and the like. For example, contrastive loss is one example technique to determine the similarity between feature vectors using distance. Feature vectors are determined to be more similar when they are close together in Euclidean space (e.g., two points have a low Euclidean distance). Similarly, feature vectors are determined to be more dissimilar when they are father apart in Euclidean space (e.g., two points have a high Euclidean distance). The similarity of the two feature vectors associated with the two training inputs502become the predicted output506

Subsequently, the predicted output506of the classifier508(including the unitary classifier and the pairwise classifier) is compared using the comparator510to the actual output518to determine an amount of error or difference from the predicted output506and the actual output518. For example, the predicted output506(e.g., a vector of probabilities determined by the unitary classifier or the similarity of the two feature vectors determined by the pairwise classifier) is compared to a one-hot encoded sparse vector indicating the actual output518, using the comparator510.

The error, represented by error signal512, is used to adjust the weights in the classifier508such that the classifier508changes (or learns). For example, the unitary classifiers learn to predict the temporal manipulations applied to the video signal based on the short-term learned features of the video signal. Similarly, the pairwise classifiers learn to predict clip ordering based on the longer-term learned features of the video signal. Additionally, the error signal512is communicated back to the encoder202such that the encoder202changes (or learns) over time to predict a more accurate feature vector to be used in each classifier head. In some implementations, the error signal512is not propagated back to the encoder202.

In one implementation, the classifier508and the encoder202are trained using the backpropagation algorithm. The backpropagation algorithm operates by propagating the error signal512through the classifier508and the encoder202. The error signal512may be calculated each iteration (e.g., each pair of training inputs502and associated actual outputs518), batch, and/or epoch and propagated through all of the algorithmic weights of the classifier508and encoder202such that the algorithmic weights are adapted based on the amount of error. The error is minimized using a loss function. Non-limiting examples of loss functions may include the square error function, the root mean square error function, and the like.

The weighting coefficients of the classifier508and encoder202are tuned to reduce the amount of error, thereby minimizing the differences between (or otherwise converging) the predicted output506with the actual output518. For example, the speed classifier outputs a speed classification that is similar to the actual speed classification, and the direction classifier outputs a direction classification that is similar to the actual direction classification. The classifier508and encoder202are trained until the error determined by the comparator510is within a certain threshold (or a threshold number of batches, epochs, or iterations have been reached).

FIG.6illustrates deployment of the trained video encoder and audio encoder, in accordance with one or more embodiments. At numeral1, input602is fed to the input manager604. Specifically, input602is data including both audio components and video components such as video data. At numeral2, the input manager604is configured to perform any one or more processing operations on the video data. For example, the input manager604may decompose the video data into unique components such as the audio component and the video component. Additionally, the input manager604may sample the video data (including the audio components and/or video components), quantize the video data, normalize the video data, and the like. The input manager604may also transform the audio component of the video data into a spectrogram representation of the audio component using any suitable technique.

At numeral3, the trained representation system150(specifically, fully trained representation system150-C inFIG.1) operates on the input602. Specifically, the video encoder112(e.g., fully trained video encoder112-C inFIG.1) and the audio encoder110(e.g., fully trained audio encoder110-C inFIG.1) generate robust feature vectors. At numeral4A, the robust video feature vector (output620A) is passed to the recognition module650at numeral5. Similarly, at numeral4B, the robust audio feature vector (output620B) is passed to the recognition module650. The feature vectors output620A-B are robust as a result of the trained video encoder112and audio encoder110.

The recognition module650receives the robust feature vectors output620A-B and performs an operation using the feature vectors. For example, the recognition module650may be any processing module such as a module configured to perform video processing tasks such as video retrieval tasks, action recognition tasks, classifying frames of a video, tagging video frames, searching video frames for objects, video fingerprinting, etc., audio processing tasks such as audio retrieval tasks, action recognition tasks, classifying audio data, tagging audio data, searching audio data for words/speakers, audio fingerprinting, etc., and the like.

Specifically, the recognition module650may be a video fingerprinting module that reduces the dimension of an input (including the output620A and620B), hashing the received input such that the input602can be uniquely identified. Such techniques may be useful in authenticating data, searching for data, and the like. The recognition module650may output a result based on the processing performed by the recognition module. For example, the recognition module650may output a result to one or more downstream processing modules. The one or more downstream processing modules may display the result determined by the recognition module650, store the result determined by the recognition module650, perform subsequent processing, or some combination.

FIG.7illustrates a schematic diagram of training system (e.g., “training system” described above) in accordance with one or more embodiments. As shown, the training manager700may include, but is not limited to, a user interface manager702, a training module704, a pretext classification training system706, a contrastive learning training system708, a representation system710, a neural network manager712, and a storage manager714. The representation system710includes the audio encoder722and the video encoder724. The storage manager714includes data store732.

As illustrated inFIG.7, the training manager700includes a user interface manager702. The user interface manager702allows a user such as an administrator to set one or more parameters of the training manager700, initiate the training manager700, exit the training manager700and the like. For example, parameters set by an administrator using the user interface manager702includes a number of training iterations, the temperature parameter (λ), and the like. The user interface manager702receive mouse movements, mouse compression/decompression (e.g., a mouse click), user interactions with a screen (e.g., haptic feedback), voice commands, keyboard entries, and the like. The user interface manager702also enables the user to view a result of the training manager700(e.g., parameters, results, statistics, and the like).

As illustrated inFIG.7, the training manager700also includes a representation system710. The representation system710includes an audio encoder722and a video encoder724. The audio encoder722determines a latent space representation (e.g., a feature vector) of an audio signal input. Similarly, the video encoder724determines a latent space representation (e.g., a feature vector) of a video signal input. As described with reference toFIG.2, the audio encoder722and video encoder724are trained using a multiheaded architecture. Each head of the multiheaded network is configured to perform a different task. Each of the tasks allow the audio encoder722and the video encoder724of the representation system to learn different patterns/relationships of audio data and video data respectively. As described herein, the audio encoder722and video encoder724are trained to perform temporal pretext classification tasks and contrastive learning tasks using the pretext classification training system706and the contrastive learning training system708respectively.

As described herein, the pretext classification training system706trains the encoders of the representation system710(e.g., audio encoder722and video encoder724) to learn temporal features of video data (including audio components and video components) using several temporal pre-text tasks. Specifically, the pretext classification training system706trains the encoders of the representation system710to classify a speed of an input, classify a direction of an input, and classify an order of a pair of inputs. Accordingly, the pretext classification training system706trains the encoders of the representation system710by training unitary classifier heads and pairwise classifier heads of a multi-headed architecture.

As described herein, the contrastive learning training system708trains the encoders of the representation system710(e.g., audio encoder722and video encoder724) to learn features of the video data (including audio components and video components) using contrastive learning tasks. Specifically, projection layers are trained by the contrastive learning training system708to learn intra modal and inter modal contrastive tasks.

As illustrated inFIG.7, the training manager700also includes a training module704. The training module704can teach, guide, tune, and/or train one or more neural networks. In some embodiments, the training module704optimizes the losses from the pretext classification training system706and the contrastive learning training system708. As the audio encoder722and video encoder724are trained (e.g., by the pretext classification training system706and the contrastive learning training system708), the training module704may receive weights736of the partially trained and/or fully trained audio encoder722and video encoder724. The weights736represent the current state of the encoders. The training module704stores such weights736such that the audio encoder722and video encoder724can use the weights736during deployment (or during an inference time). As described herein, when the audio encoder722and video encoder724are deployed, the audio encoder722and video encoder724generate representations of audio data and video data that are fed to one or more downstream processing modules (e.g., modules configured to perform video processing tasks such as video retrieval tasks, action recognition tasks, classifying frames of a video, tagging video frames, searching video frames for objects, video fingerprinting, etc., audio processing tasks such as audio retrieval tasks, action recognition tasks, classifying audio data, tagging audio data, searching audio data for words/speakers, audio fingerprinting, etc., and the like).

The training module704provides the pretext classification training system706and the contrastive learning training system708with input-output pairs during training (e.g., training data734). As described herein, the training module704generates the input-output pairs used during training such that training is performed in a self-supervised manner. Specifically, the training module704may perform one or more temporal transformations to an audio component of video data and a video component of video data.

For example, the training module704may perform a temporal transformation to randomly temporally crop video data to generate a video component and audio component. As described herein, video data randomly cropped by the training module704results in a video window (or one or more video frames) including visual content. Similarly, the audio data is cropped by the training module704, resulting in one or more audio segments including aural content. In some embodiments, each video window temporally cropped corresponds to a temporally aligned audio segment. In other words, at a given point in time, the video of the video window is accompanied by audio of the audio segment.

Additionally or alternatively, the training module704may speed up or slow down the speed of a particular video window and the corresponding audio segment. In this manner, the training module704manipulates the playback speed. Another temporal manipulation performed by the training module704includes reversing the direction of a particular window and the corresponding audio segment. For example, the training module704may play the contents of a particular window in reverse.

The training module704is also configured to generate positive/negative pairs for use as training data734. When generating positive/negative pairs, the training module704may query one or more data stores. As described herein, data stores may be local (e.g., stored by the storage manager714) and/or external. After training data734is generated by the training module704, it may be stored in the storage manager714.

As illustrated inFIG.7, the training manager700includes a neural network manager712. Neural network manager712may host a plurality of neural networks or other machine learning models, such as multi layer perceptrons including prediction MLPs (e.g., unitary classifier heads and pairwise classifier heads), projection MLPs (e.g., contrastive learning heads), and encoders (e.g., audio encoder722and video encoder724).

The neural network manager712may include an execution environment, libraries, and/or any other data needed to execute the machine learning models. In some embodiments, the neural network manager712may be associated with dedicated software and/or hardware resources to execute the machine learning models. Although depicted inFIG.7as being hosted by a single neural network manager712, in various embodiments the neural networks may be hosted in multiple neural network managers and/or as part of different components. For example, the pretext classification training system706can be hosted by a first neural network manager (or other host environment), in which the respective neural networks execute. Similarly, the contrastive learning training system708can be hosted by a first neural network manager (or other host environment), in which the respective neural networks execute.

In other embodiments, groups of machine learning models may be executed by their own neural network manager or other host environment. Additionally, or alternatively, each machine learning model (or groups of machine learning models) may be spread across multiple neural network managers depending on, e.g., the resource requirements, traffic, lag, etc.

As illustrated inFIG.7, the training manager700also includes a storage manager714. The storage manager714maintains data for the training manager700. The storage manager714can maintain data of any type, size, or kind as necessary to perform the functions of the training manager700. For example, the storage manager714, as shown inFIG.7, includes the training data734. As described herein, the training module704generates the training data734and such training data may be stored by the storage manager714. The storage manager714may also store data store732. As described herein, the data store732includes previously computed feature vectors for data such as audio segments, video windows, audio components, video components, and the like. Additionally or alternatively, the storage manager714may store addresses (e.g., memory addresses, URL addresses, MAC addresses, IP addresses, port addresses, etc.) in which to query external data stores.

Each of the components702-714of the training manager700and their corresponding elements (as shown inFIG.7) may be in communication with one another using any suitable communication technologies. It will be recognized that although components702-714and their corresponding elements are shown to be separate inFIG.7, any of components702-714and their corresponding elements may be combined into fewer components, such as into a single facility or module, divided into more components, or configured into different components as may serve a particular embodiment.

The components702-714and their corresponding elements can comprise software, hardware, or both. For example, the components702-714and their corresponding elements can comprise one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices. When executed by the one or more processors, the computer-executable instructions of the training manager700can cause a client device and/or a server device to perform the methods described herein. Alternatively, the components702-714and their corresponding elements can comprise hardware, such as a special purpose processing device to perform a certain function or group of functions. Additionally, the components702-714and their corresponding elements can comprise a combination of computer-executable instructions and hardware.

Furthermore, the components702-714of the training manager700may, for example, be implemented as one or more stand-alone applications, as one or more modules of an application, as one or more plug-ins, as one or more library functions or functions that may be called by other applications, and/or as a cloud-computing model. Thus, the components702-714of the training manager700may be implemented as a stand-alone application, such as a desktop or mobile application. Furthermore, the components702-714of the training manager700may be implemented as one or more web-based applications hosted on a remote server. Alternatively, or additionally, the components of the training manager700may be implemented in a suite of mobile device applications or “apps.” To illustrate, the components of the training manager700may be implemented as part of an application, or suite of applications, including but not limited to ADOBE CREATIVE CLOUD, ADOBE PHOTOSHOP, ADOBE ACROBAT, ADOBE ILLUSTRATOR, ADOBE LIGHTROOM and ADOBE INDESIGN. “ADOBE”, “CREATIVE CLOUD,” “PHOTOSHOP,” “ACROBAT,” “ILLUSTRATOR,” “LIGHTROOM,” and “INDESIGN” are either registered trademarks or trademarks of Adobe Inc. in the United States and/or other countries.

As shown, the training manager700can be implemented as a single system. In other embodiments, the training manager700can be implemented in whole, or in part, across multiple systems. For example, one or more functions of the training manager700can be performed by one or more servers, and one or more functions of the training manager700can be performed by one or more client devices. The one or more servers and/or one or more client devices may generate, store, receive, and transmit any type of data used by the training manager700, as described herein.

In one implementation, the one or more client devices can include or implement at least a portion of the training manager700. In other implementations, the one or more servers can include or implement at least a portion of the training manager700. For instance, the training manager700can include an application running on the one or more servers or a portion of the training manager700can be downloaded from the one or more servers. Additionally or alternatively, the training manager700can include a web hosting application that allows the client device(s) to interact with content hosted at the one or more server(s).

For example, upon a client device accessing a webpage or other web application hosted at the one or more servers, in one or more embodiments, the one or more servers can initiate the training manager700stored at the one or more servers. Specifically, the client device can generate a request (i.e., via user input) to initiate the training system. Upon receiving the request, the one or more servers can automatically perform the methods and processes described above to train a representation system (including an audio encoder and a video encoder). The one or more servers can train the representation and display updates (e.g., status, errors, etc.) to the user.

The server(s) and/or client device(s) may communicate using any communication platforms and technologies suitable for transporting data and/or communication signals, including any known communication technologies, devices, media, and protocols supportive of remote data communications, examples of which will be described in more detail below with respect toFIG.9. In some embodiments, the server(s) and/or client device(s) communicate via one or more networks. A network may include a single network or a collection of networks (such as the Internet, a corporate intranet, a virtual private network (VPN), a local area network (LAN), a wireless local network (WLAN), a cellular network, a wide area network (WAN), a metropolitan area network (MAN), or a combination of two or more such networks. The one or more networks 'M08 will be discussed in more detail below with regard toFIG.9.

The server(s) may include one or more hardware servers (e.g., hosts), each with its own computing resources (e.g., processors, memory, disk space, networking bandwidth, etc.) which may be securely divided between multiple customers (e.g. client devices), each of which may host their own applications on the server(s). The client device(s) may include one or more personal computers, laptop computers, mobile devices, mobile phones, tablets, special purpose computers, TVs, or other computing devices, including computing devices described below with regard toFIG.9.

FIGS.1-7, the corresponding text, and the examples, provide a number of different systems and devices that allows a user to train a system to generate robust representations of audio and video. In addition to the foregoing, embodiments can also be described in terms of flowcharts comprising acts and steps in a method for accomplishing a particular result. For example,FIG.8illustrates a flowchart of an exemplary method in accordance with one or more embodiments. The method described in relation toFIG.8may be performed with fewer or more steps/acts or the steps/acts may be performed in differing orders. Additionally, the steps/acts described herein may be repeated or performed in parallel with one another or in parallel with different instances of the same or similar steps/acts.

FIG.8illustrates a flowchart800of a series of acts in a method of generating audio and video representations using self-supervised learning, in accordance with one or more embodiments. In one or more embodiments, the method800is performed in a digital medium environment that includes the training manager700. The method800is intended to be illustrative of one or more methods in accordance with the present disclosure and is not intended to limit potential embodiments. Alternative embodiments can include additional, fewer, or different steps than those articulated inFIG.8.

As illustrated inFIG.8, the method800includes an act802of receiving a video signal including an audio component and a video component. While a video signal is described, it should be appreciated that any signal including audio components and video components may be received by the training system.

As illustrated inFIG.8, the method800includes an act804of training a first machine learning model to determine a representation of the audio component using a contrastive learning task and a temporal learning task. The first machine learning model may be an audio encoder. Generally, encoders determine a latent space representation of an input. The audio encoder is used to determine an audio feature vector using the audio signal. Specifically, the audio encoder is trained to generate an audio feature vector using a multiheaded architecture. Various heads of the audio encoder are trained to perform temporal pretext tasks, and various heads of the audio encoder are trained to perform contrastive learning tasks. By training the audio encoder using both temporal pretext tasks and contrastive learning, the audio encoder learns short term features, long term features, and semantic information about audio signals. In this manner, the audio encoder is able to generate a robust audio feature vector of the audio signal.

As illustrated inFIG.8, the method800includes an act806of training a second machine learning model to determine a representation of the video component using the contrastive learning task and the temporal learning task. The second machine learning model may be a video encoder. The video encoder is used to determine a video feature vector using the video signal. Specifically, the video encoder is trained to generate a video feature vector using a multiheaded architecture. Various heads of the video encoder are trained to perform temporal pretext tasks, and various heads of the video encoder are trained to perform contrastive learning tasks. By training the video encoder using both temporal pretext tasks and contrastive learning, the video encoder learns short term features, long term features, and semantic information about video signals. In this manner, the video encoder is able to generate a robust video feature vector of the video signal.

FIG.9illustrates, in block diagram form, an exemplary computing device900that may be configured to perform one or more of the processes described above. One will appreciate that one or more computing devices such as the computing device900may implement the training system. As shown byFIG.9, the computing device can comprise a processor902, memory904, one or more communication interfaces906, a storage device908, and one or more I/O devices/interfaces910. In certain embodiments, the computing device900can include fewer or more components than those shown inFIG.9. Components of computing device900shown inFIG.9will now be described in additional detail.

In particular embodiments, processor(s)902includes hardware for executing instructions, such as those making up a computer program. As an example, and not by way of limitation, to execute instructions, processor(s)902may retrieve (or fetch) the instructions from an internal register, an internal cache, memory904, or a storage device908and decode and execute them. In various embodiments, the processor(s)902may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), systems on chip (SoC), or other processor(s) or combinations of processors.

The computing device900includes memory904, which is coupled to the processor(s)902. The memory904may be used for storing data, metadata, and programs for execution by the processor(s). The memory904may include one or more of volatile and non-volatile memories, such as Random Access Memory (“RAM”), Read Only Memory (“ROM”), a solid state disk (“SSD”), Flash, Phase Change Memory (“PCM”), or other types of data storage. The memory904may be internal or distributed memory.

The computing device900can further include one or more communication interfaces906. A communication interface906can include hardware, software, or both. The communication interface906can provide one or more interfaces for communication (such as, for example, packet-based communication) between the computing device and one or more other computing devices900or one or more networks. As an example and not by way of limitation, communication interface906may include a network interface controller (NIC) or network adapter for communicating with an Ethernet or other wire-based network or a wireless NIC (WNIC) or wireless adapter for communicating with a wireless network, such as a WI-FI. The computing device900can further include a bus912. The bus912can comprise hardware, software, or both that couples components of computing device900to each other.

The computing device900includes a storage device908includes storage for storing data or instructions. As an example, and not by way of limitation, storage device908can comprise a non-transitory storage medium described above. The storage device908may include a hard disk drive (HDD), flash memory, a Universal Serial Bus (USB) drive or a combination these or other storage devices. The computing device900also includes one or more input or output (“I/O”) devices/interfaces910, which are provided to allow a user to provide input to (such as user strokes), receive output from, and otherwise transfer data to and from the computing device900. These I/O devices/interfaces910may include a mouse, keypad or a keyboard, a touch screen, camera, optical scanner, network interface, modem, other known I/O devices or a combination of such I/O devices/interfaces910. The touch screen may be activated with a stylus or a finger.

In the foregoing specification, embodiments have been described with reference to specific exemplary embodiments thereof. Various embodiments are described with reference to details discussed herein, and the accompanying drawings illustrate the various embodiments. The description above and drawings are illustrative of one or more embodiments and are not to be construed as limiting. Numerous specific details are described to provide a thorough understanding of various embodiments.