Temporal contrastive learning for semi-supervised video action recognition

A base pathway of a computerized two-pathway video action recognition model is trained using a plurality of labeled video samples. The base pathway is trained using a plurality of unlabeled video samples at a first framerate. An auxiliary pathway of the computerized two-pathway video action recognition model is trained using a plurality of the unlabeled video samples at a second framerate, the second framerate being slower than the first framerate, wherein the training of the base pathway and the training of the auxiliary pathway result in a trained computerized two-pathway video action recognition model. A candidate video is categorized using the trained computerized two-pathway video action recognition model and the categorized candidate video is stored in a computer-accessible video database system for information retrieval.

The following disclosure(s) are submitted under 35 U.S.C. 102(b)(1)(A):

BACKGROUND

The present invention relates to the electrical, electronic and computer arts, and more specifically, to video processing systems.

Video is conventionally analyzed to identify actions and objects within the video, and to categorize the video for information retrieval and other tasks. Most video action analysis systems utilize supervised learning, that is, systems that learn from labeled data. Video data sets, however, often include many unlabeled videos which are typically expensive to label. Learning to recognize actions from only a handful of labeled videos is a challenging problem due to the scarcity of tediously collected activity labels.

SUMMARY

Principles of the invention provide techniques for semi-supervised video action recognition. In one aspect, an exemplary method includes the operations of training a base pathway of a computerized two-pathway video action recognition model using a plurality of labeled video samples; training the base pathway of the computerized two-pathway video action recognition model using a plurality of unlabeled video samples at a first framerate; training an auxiliary pathway of the computerized two-pathway video action recognition model using a plurality of the unlabeled video samples at a second framerate, the second framerate being slower than the first framerate (wherein said training of said base pathway using said plurality of labeled video samples, said training of said base pathway using said plurality of unlabeled video samples at said first framerate, and said training of said auxiliary pathway using said plurality of unlabeled video samples at said second framerate, result in a trained computerized two-pathway video action recognition model); categorizing a candidate video using the trained computerized two-pathway video action recognition model; and storing the categorized candidate video in a computer-accessible video database system for information retrieval.

In one aspect, an apparatus comprises a memory and at least one processor, coupled to the memory, and operative to perform a method comprising training a base pathway of a computerized two-pathway video action recognition model using a plurality of labeled video samples; training the base pathway of the computerized two-pathway video action recognition model using a plurality of unlabeled video samples at a first framerate; training an auxiliary pathway of the computerized two-pathway video action recognition model using a plurality of the unlabeled video samples at a second framerate, the second framerate being slower than the first framerate (wherein said training of said base pathway using said plurality of labeled video samples, said training of said base pathway using said plurality of unlabeled video samples at said first framerate, and said training of said auxiliary pathway using said plurality of unlabeled video samples at said second framerate, result in a trained computerized two-pathway video action recognition model); categorizing a candidate video using the trained computerized two-pathway video action recognition model; and storing the categorized candidate video in a computer-accessible video database system for information retrieval.

In one aspect, a computer program product comprises a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising training a base pathway of a computerized two-pathway video action recognition model using a plurality of labeled video samples; training the base pathway of the computerized two-pathway video action recognition model using a plurality of unlabeled video samples at a first framerate; training an auxiliary pathway of the computerized two-pathway video action recognition model using a plurality of the unlabeled video samples at a second framerate, the second framerate being slower than the first framerate (wherein said training of said base pathway using said plurality of labeled video samples, said training of said base pathway using said plurality of unlabeled video samples at said first framerate, and said training of said auxiliary pathway using said plurality of unlabeled video samples at said second framerate, result in a trained computerized two-pathway video action recognition model); categorizing a candidate video using the trained computerized two-pathway video action recognition model; and storing the categorized candidate video in a computer-accessible video database system for information retrieval.

Techniques of the present invention can provide substantial beneficial technical effects. For example, one or more embodiments provide one or more of:utilization of rich supervisory information, in terms of ‘time’, that is present in otherwise unsupervised pool of videos;performance improvement over video extensions of sophisticated state-of-the-art, semi-supervised image recognition methods across multiple diverse benchmark datasets and network architecture;a contrastive objective between groups of videos that explores the underlying class concept that traditional Normalized Temperature-scaled Cross Entropy Loss (NT-Xent) loss among individual video instances ignores;special treatment of the time axis in unlabeled videos by processing the videos at two different speeds;a two-pathway temporal contrastive semi-supervised action recognition framework;a group-contrastive loss that couples discriminative motion representation with pace-invariance and that significantly improves semi-supervised action recognition performance;increased accuracy compared to conventional video analysis techniques;performing machine learning with limited size data sets; andimproved technological processes such as categorizing videos, retrieving videos from large databases, video recommendation, and computer vision.

DETAILED DESCRIPTION

Generally, systems and methods for analyzing videos are disclosed. In one example embodiment, temporal contrastive learning is utilized and a two-pathway temporal contrastive model is learned using unlabeled videos at different speeds (such as two different speeds) to leverage the fact that changing a video's speed does not change an action. Specifically, the similarity between encoded representations of the same video at two different speeds are maximized and a similarity between different videos played at different speeds is minimized. In this way, the rich supervisory information, in terms of ‘time,’ that is present in an otherwise unsupervised pool of videos is utilized. With this effective strategy of utilizing different video playback rates, video extensions of sophisticated state-of-the-art semi-supervised image recognition methods are outperformed across multiple diverse benchmark datasets and network architectures. Interestingly, the disclosed approach benefits from out-of-domain unlabeled videos showing generalization and robustness. The disclosed approach has been verified by performing rigorous ablations and analysis to validate the approach.

Introduction

Supervised deep learning approaches have shown remarkable progress in video action recognition. Being supervised, however, these models are critically dependent on large datasets requiring tedious human annotation efforts. Supervised methods alone may not be enough to deal with the volume of information contained in videos. Semi-supervised learning approaches use structural invariance between different views of the same data as a source of supervision for learning useful representations. In recent times, semi-supervised representation learning models have performed well, even surpassing their supervised counterparts in the case of images.

Notwithstanding their potential, semi-supervised video action recognition has received very little attention. Trivially extending the image domain approaches to videos without considering the rich temporal information may not quite bridge the performance gap between semi- and fully-supervised learning; however, in videos, another source of supervision is available: time. It is widely known that an action recognizer is good if it can recognize actions irrespective of whether the actions are performed slowly or quickly.

In one example embodiment, Temporal Contrastive Learning (TCL) for semi-supervised action recognition is introduced for use in videos where consistent features representing both slow and fast versions of the same videos are learned. Starting with a model trained with a limited amount of labeled data, a two-pathway model is generated that processes unlabeled videos at two different speeds and finds their representations. Though played at two different speeds, the videos share the same semantics. Thus, similarity between these representations is maximized. Likewise, the similarity between the representations of different videos is minimized. This is achieved, in one or more embodiments, by minimizing a modified contrastive loss between the videos with different playback rates.

While minimizing a contrastive loss helps to produce better visual representations by learning to be invariant to different views of the data, it ignores information shared among samples of the same action class as the loss treats each video individually. To this end, a new perspective of contrastive loss between neighborhoods is utilized. Neighborhoods are compact groups of unlabeled videos with high class consistency. In the absence of ground-truth labels, groups are formed by clustering videos with the same pseudo-labels and are represented by averaging the representations of the constituent videos of the group. A contrastive objective between groups formed off the two paths explores the underlying class concept that traditional contrastive loss among individual video instances does not take into account. The contrastive loss is termed considering only individual instances as the instance-contrastive loss and the same between the groups as the group-contrastive loss, respectively.

Problem Setup

In one example embodiment, only a small set of videos (Dl) has labels, but a large number of unlabeled videos (Du) are assumed to be present alongside. The set Dl{Vi, yi}i=1Niincludes Nivideos where the ithvideo and the corresponding activity label is denoted by Viand yi, respectively. For a dataset of videos with C different activities, yiis often assumed to be an element of the label set Y={1, 2, . . . , C}. Similarly, the unlabeled set Du{Ui}i=1Nuincludes Nu(>>Nl) videos without any associated labels. The unlabeled videos are used at two different frame rates (referred to as fast and slow videos herein). The fast version of the video Uiis represented as a collection of M frames, i.e., Ufi={Ff,1i, Ff,2i, . . . , Ff,Mi}. Likewise, the slow version of the same is represented as Usi={Fs,1i, Fs,2i, . . . , Fs,Ni}, where N<M. The frames can be sampled from the video, for example, following conventional techniques where a random frame is sampled uniformly from consecutive non-overlapping segments.

FIG.1is an illustration of an example Temporal Contrastive Learning (TCL) framework300, in accordance with an example embodiment. The disclosed approach utilizes a base pathway304and an auxiliary pathway308that share the same weights. The base pathway304accepts video frames324sampled at a higher rate while the auxiliary pathway308takes in frames320at a lower framerate. In one example embodiment, at first, the base neural network328is trained using limited labeled data via the base pathway304. Subsequently, both the base pathway304and the auxiliary pathway308are used for the unlabeled samples316by encouraging video representations to match in both pathways304,308in absence of labels. This is done by maximizing agreement between the outputs of the two pathways304,308for a video while minimizing the same for different videos. In addition, originally unlabeled videos316with high semantic similarity are grouped by pseudo-labels assigned to them. To exploit the high consistency and compactness of group members, the average representations of groups with the same pseudo-label in different pathways304,308are made similar while those between the varying groups are made maximally different. Two separate contrastive losses (see, sections entitled Instance-Contrastive Loss and Group-Contrastive Loss below) are used for this purpose. Given a video at test time, only the base network is used to recognize the action. The classified videos can be stored in a suitable database333and searches can be performed based on the labels applied during inferencing (e.g., queries such as find videos of people walking, find videos of cars rolling down a highway, . . . ). Results can then be returned to the person making the query. The skilled artisan will be generally familiar with database technology, training neural networks, and inferencing with neural networks, and, given the teachings herein, will be able to adapt known neural network and database technologies to implement one or more embodiments. Database333can, for example, be local or cloud-implemented database software68running on a cloud-based server, for example.

Temporal Contrastive Learning

As shown inFIG.1, the TCL framework300processes the input videos312,316in two pathways, namely, the base pathway304and the auxiliary pathway308. The fast version of the videos are processed by the base pathway304while the slow versions are processed by the auxiliary pathway308. Both pathways304,308share the same neural network backbone328(denoted by g(.)). In one example embodiment, different neural network backbones may be utilized for the pathways304,308. Different stages of training in the TCL framework300are described below.

Supervised Training Stage

The neural network backbone328is initially trained using only the small labeled data Dl(labeled samples312) by passing it through the base pathway304. Depending on whether the neural network backbone328involves 2D or 3D convolution operations, the representation (g (Vi)) of the video Viused in the TCL framework300is an average of the frame logits or the logits332from the 3D neural network backbone328respectively. The supervised cross-entropy loss (sup) is minimized on the labeled data as follows:
sup=−Σc=1c(yi)clog(g(Vi)c(1)

Equipped with an initial neural network backbone328trained with limited supervision, a goal is to learn a model that can use a large pool of unlabeled video samples316for better activity understanding. To this end, temporal co-occurrence of unlabeled activities at multiple speeds is used as a proxy task; this is enforced with a pairwise contrastive loss. Specifically, the frame sampling rate is adjusted to generate videos320,324with different speeds.

Consider a minibatch of B unlabeled video samples316. The model is then trained to match the representation g(Ufi) of the comparatively faster version of the video (Ui) with g(Usi) of the slower version; g(Ufi) and g(Usi) form the positive pair. For the rest of the B— 1 videos, g(Ufi) and g(Upk) form negative pairs, where representation of the kthvideo can come from either of the pathways304,308(i.e., p∈{f,s}). Inasmuch as different videos forming the negative pairs have different content, the representation(s) of different videos in either of the pathways304,308are pushed apart. This is achieved by employing a contrastive lossicas follows:

where

h⁡(u,v)=exp⁡(uT⁢vu2⁢v2/τ)
is the exponential of the cosine similarity measure and τ is the temperature hyperparameter. The final instance-contrastive loss is computed for all positive pairs, i.e., both (Ufi, Usi) and (Usi, Ufi) across the minibatch. The loss function encourages decreasing the similarity not only between different videos in the two pathways304,308, but also between different videos across both of them.

Directly applying contrastive loss between different video instances in the absence of class-labels does not take the high level action semantics into account. Such a strategy can inadvertently learn different representations for videos containing the same actions. In one example embodiment, contrastive loss among groups of videos with similar actions is employed, where relations within the neighborhood of different videos are explored. Specifically, each unlabeled video Uiin each of the two pathways304,308is assigned pseudo-labels that correspond to the class having the maximum activation. Let ŷfiand ŷsidenote the pseudo-labels of the video Uiin the fast and the slow pathways304,308, respectively. Videos having the same pseudo-label in a minibatch form a group in each pathway304,308and the average of the representations of constituent videos provides the representation of the group as shown below:

Rpl=∑i=1B𝕝{y^pi=l}⁢g⁡(Upi)T(3)
where ∥ is an indicator function that evaluates to 1 for the videos with a pseudo-label equal to l∈Y in each pathway p∈{f,s} is the number of such videos in the minibatch.

FIG.2illustrates an advantage of the use of group-contrastive loss over instance-contrastive loss, in accordance with an example embodiment. (InFIG.2, solid arrows represent minimal agreement and dashed arrows represent maximal agreement between the corresponding videos.) A contrastive objective between instances may try to push different instances of the same action apart (right), while forming groups of videos with the same activity class avoids such inadvertent competition (left). In the absence of true labels, such grouping is done by the predicted pseudo-labels.

Considering the high class consistency among two groups with the same label in two pathways304,308, in one or more embodiments these groups are required to give similar representations in the feature space. Thus, in the group-contrastive objective, all pairs (Rfl, Rsl) act as positive pairs while the negative pairs are the pairs (Rfl, Rpm) with p∈{f, s} and m∈Y \l such that the constituent groups are different in either of the pathways304,308. The group-contrastive loss involving these pairs is,

Similar to instance-contrastive loss, group-contrastive loss is also computed for all positive pairs—both (Rfl, Rsl) and (Rsl, Rfl) across the minibatch. Overall, the loss function for training an exemplary model involving the limited labeled data and the unlabeled data is given by:
=Lsup+γ*Lic+β*Lgc(5)
where, γ and β are weights of the instance-contrastive and group-contrastive losses respectively. The weights may be determined empirically and may be set, for example, to one.

TCL with Pretraining and Finetuning

In one example embodiment, self-supervised pretraining is used to initialize the TCL model with very minimal change in the framework. Specifically, self-supervised pretraining is employed at the beginning by considering the whole of the labeled data (labeled samples312) and the unlabeled data (unlabeled samples316) Dl∪Duas unlabeled data only and using instance-contrastive lossicto encourage consistency between representations learned in the two pathways304,308(ref. Eq. 2). These weights are then used to initialize the base pathway304and the auxiliary pathway308before the disclosed approach commences for semi-supervised learning of video representations. For effective utilization of the unlabeled data316, the base pathway204is finetuned with pseudo-labels (the skilled artisan will be familiar with pseudo-labels, wherein a network is trained in a supervised fashion with labeled and unlabeled data simultaneously; for unlabeled data, pseudo-labels, just picking up the class which has the maximum predicted probability, are used as if they were true labels) generated at the end of the contrastive learning, which greatly enhances the discriminability of the features, leading to improvement in recognition performance. It can be empirically shown that starting with the same amount of labeling, both self-supervised pretraining and finetuning with pseudo-labels (Pretraining→TCL→Finetuning) benefits more compared to the same after limited supervised training only.

Experiments

Extensive experiments were conducted to show that the disclosed TCL framework300outperforms many strong baselines on several benchmarks including one with domain shift. Comprehensive ablation experiments were also performed to verify the effectiveness of different components in detail.

Experimental Setup

Datasets

The disclosed approach was evaluated using four datasets. The first set of videos contained 81 K training videos and 12 K testing videos across 87 action classes. The second set of videos contained 119 K videos for training and 15 K videos for validation across 27 annotated classes for hand gestures. The third set of videos contained is one of the most popular large-scale benchmarks for video action recognition. It consists of 240 K videos for training and 20 K videos for validation across 400 action categories, with each video lasting 6-10 seconds. The fourth set of videos contained contains 7,860 untrimmed egocentric videos of daily indoors activities recorded from both the third and first person views. The dataset contains 68,536 temporal annotations for 157 action classes. A subset of the third person videos from the fourth set of videos was used as the labeled data while the first person videos were considered as unlabeled data to show the effectiveness of the disclosed approach under domain shift in the unlabeled data.

The disclosed approach is compared with the following baselines and existing semi-supervised approaches from the 2D image domain, extended to video data. A supervised baseline was considered where an action classifier having the same architecture as the base pathway304of the disclosed approach was trained. This is trained using a small portion of the labeled samples312assuming only a small subset of labeled samples312is available as annotated data. Second, the disclosed approach was compared with state-of-the-art semi-supervised learning approaches, including Pseudo-Label, Mean Teacher, S4L, Mix-Match, and FixMatch. The same neural network backbone328and experimental settings were used for all the baselines (including the disclosed approach) for a fair comparison.

Implementation Details

Temporal Shift Module (TSM) was used with a first conventional backbone as the base action classifier in all of the experiments. Performance of different methods was further investigated by using a second conventional backbone on the first set of videos. TSM has recently been shown to be very effective due to its hardware efficiency and lesser computational complexity. Uniformly sampled 8 and 4 frame segments from unlabeled video samples316were used as input to the base pathway304and the auxiliary pathway308, respectively, to process unlabeled video samples316in the TCL framework300. On the other hand, only 8 frame segments for labeled video samples312were used and the final performance was computed using 8 frame segments in the base pathway304for all the methods. Note that the disclosed approach is agnostic to the backbone architecture and particular values of frame rates. Following the standard practice in SSL, a certain percentage of labeled sample312was randomly chosen as a small labeled set and the labels for the remaining data were discarded to form a large unlabeled set. The disclosed approach was trained with different percentages of labeled samples312for each dataset (1%, 5% and 10%). The disclosed models were trained for 400 epochs where the model was first trained with supervised losssupusing only labeled samples312for 50 epochs. The disclosed model was then trained using the combined loss (ref. Eq. 5) for the next 300 epochs. Finally, for finetuning with pseudo-labels, the disclosed model was trained with both labeled and unlabeled videos having pseudo-label confidence more than 0.8 for 50 epochs.

During pretraining, the standard practice in self-supervised learning was followed and the disclosed model was trained using all the training videos without any labels for 200 epochs. SGD was used with a learning rate of 0.02 and a momentum value of 0.9 with cosine learning rate decay in all of the experiments. Given a minibatch of labeled samples Bl, μ×Blunlabeled samples are utilized for training. μ is set to 3 and τ is set to 0.5 in all the experiments. γ and β values were taken to be 9 and 1, respectively, unless otherwise mentioned. Random scaling and cropping were used as data augmentation during training (and random flipping for Kinetics-400 is further adopted); just 1 clip per video and a center 224λ224 crop for evaluation were used.

Experiments

Extensive experiments were performed on four standard datasets and demonstrate that TCL achieves superior performance over extended baselines of state-of-the-art image domain semi-supervised approaches.FIG.3shows a comparison of TCL with conventional techniques trained using different percentages of labeled training data, in accordance with an example embodiment. Using the same backbone network (the first conventional backbone), TCL needs only 33% and 15% of labeled data in the first set of videos and the second set of videos, respectively, to reach the performance of a conventional fully supervised approach that uses 100% labeled data. On the other hand, the two compared methods fail to reach the accuracy of the fully supervised approach with such a small amount of labeled data. Likewise, as good as 8.14% and 4.63% absolute improvement is observed in recognition performance over the next best approach using only 5% labeled data in the first set of videos and the third set of videos respectively. In a new realistic setting, it is maintained that unlabeled videos may come from a related but different domain than that of the labeled data. For instance, given a small set of labeled videos from a third person view, the disclosed approach is shown to benefit from using only first person unlabeled videos on the fourth set of videos, demonstrating the robustness to domain shift in the unlabeled set.

Large-Scale Experiments and Comparisons

Tables 1-3 ofFIG.5illustrate the performance of different methods on four datasets, in terms of average top-1 clip accuracy and standard deviation over 3 random trials.

First Set of Videos

Table 1 shows the performance comparison of both the first conventional backbone (left half) and the second conventional backbone (right half) on the first set of videos. The numbers show average Top-1 accuracy values with standard deviations over three random trials for different percentages of labeled data. TCL outperforms the video extensions of all the semi-supervised image-domain baselines for all three percentages of labeled training data. The improvement is especially prominent for the low capacity model (the first conventional backbone) and low data (only 1% and 5% data with labels) regime. Notably, the disclosed approach outperforms the first conventional technique by 1.75% while training with only 1% labeled data. The improvement is 8.14% for the case when 5% data is labeled. These improvements clearly show that the disclosed approach is able to leverage the temporal information more effectively compared to the first conventional technique that focuses on only spatial image augmentations.

FIG.4illustrates the change in classwise top-1 accuracy of TCL over the first conventional technique on the first set of videos, in accordance with an example embodiment. The vertical bars show the change in accuracy on a 5% labeled scenario, while the line shows the number of labeled videos per class (sorted). Compared to the first conventional technique, TCL improves the performance of most classes including those with less labeled data. The plot shows that an overwhelming majority of the activities experienced improvement with a decrease in performance for only 1 class out of 18 having less than 20 labeled videos per class (right-side of the figure). For a low labeled data regime (1% and 5%), a heavier model shows signs of overfitting as is shown by a slight drop in performance. On the other hand, using the second conventional backbone instead of the first conventional backbone is shown to benefit TCL if the model is fed with more labeled data. Moreover, TCL with finetuning and pretraining shows further improvement, leading to best performance in both cases.

Second Set of Videos

Table 2 shows a performance comparison on the second set of videos and the third set of videos. The numbers show the top-1 accuracy values using the first conventional backbone on both datasets. The TCL approach also surpasses the performance of existing semi-supervised approaches in Jester as shown in Table 2 (left-side). In particular, TCL achieves 10.23% absolute improvement compared to S4L (the next best) in very low labeled-data regime (1% only). Adding finetuning and self-supervised pretraining further increases this difference to 17.57%. Furthermore, TCL with pretraining and finetuning achieves a top-1 accuracy of 94.93% using 10% labeled data which is only 0.32% lower than the fully supervised baseline trained using all the labels (95.25%).

Third Set of Videos

Table 2 (right-side) summarizes the results on the third set of videos, which is one of the widely used action recognition datasets consisting of 240 K videos across 400 classes. TCL outperforms the first conventional technique by a margin of 1.31% and 4.63% on 1% and 5% scenarios, respectively, showing the superiority of the disclosed approach on large scale datasets. The top-1 accuracy achieved using TCL with finetuning and pretraining is almost two times better than the supervised approach when only 1% of the labeled data is used. The results also show that off-the-shelf extensions of sophisticated state-of-the-art semi-supervised image classification methods offer little benefit to action classification on videos.

Fourth Set of Videos

Third person videos from the fourth set of videos were used as the target while the first person videos form the additional unlabeled set. During training, labeled data is taken only from the target domain while unlabeled data is obtained from both the target and the domain-shifted videos. To modulate domain shift in unlabeled data, a new hyperparameter ρ, is introduced whose value denotes the proportion of target videos in the unlabeled set. For a fixed number of unlabeled videos |Du|, we randomly select ρ×|Du| videos from the target while the remaining (1-ρ)×|Du| are selected from the other domain. Following the standard practice in this dataset, the model was first pretrained using the fourth set of videos and three different values of ρ: 1, 0.5, 0 were used for 10% target data with labels. Table 3 shows the mean Average Precision (mAP) of the disclosed method including the supervised approach, and two conventional approaches. TCL outperforms both methods by around 1% mAP for all three p values. In the case when all the unlabeled data is from the shifted domain (ρ=0), the performance of the disclosed approach is even better than the performance of the next best approach with ρ=1, i.e., when all unlabeled data is from the target domain itself. This depicts the robustness of TCL and its ability to harness diverse domain data more efficiently in a semi-supervised setting.

Role of Pseudo-Labeling

The reliability of pseudo-labeling was tested on the second set of videos (using the first conventional backbone and 1% labeling) with 50 epoch intervals and it was observed that the pseudo-labeling accuracy gradually increases from 0% at the beginning to 65.95% at 100 epoch, and then to 93.23% at 350 epoch. This shows that, while the disclosed model may create some wrong groups at the start, it gradually improves the groups as the training proceeds, leading to a better representation by exploiting both instance and group contrastive losses.

Ablation Studies

Extensive ablation studies were performed on the first set of videos with 5% labeled data and the first conventional backbone to better understand the effect of different losses and hyperparameters in the TCL framework300. Table 4 inFIG.5illustrates the results of the ablation studies on the first set of videos. The numbers show top-1 accuracy with the first conventional backbone and 5% labeled Data.

Effect of Group Contrastive Loss

An experiment was performed by removing group contrastive loss from the TCL framework300(see, section entitled Group-Contrastive Loss) and it was observed that top-1 accuracy drops to 27.24% from 29.81% (Table 4), showing the importance of group contrastive loss in capturing high-level semantics.

Ablation on Contrastive Loss

The effectiveness of the disclosed contrastive loss was investigated by replacing it with the pseudo-label consistency loss used in the first conventional technique. It was observed that training with the disclosed contrastive loss surpasses the performance of the training with the pseudo-label consistency loss by a high margin (around 6.21% gain in the top-1 accuracy) on the first set of videos (Table 4). The disclosed approach was further compared in the absence of group-consistency (TCL w/o Group-Contrastive Loss) with a variant of the first conventional technique that uses temporal augmentation and observed that the disclosed approach still outperforms it by a margin of 2.66% (24.58% vs 27.24%) on the first set of videos (with the first conventional backbone and 5% labeling). This shows that temporal augmentation alone fails to obtain superior performance and this improvement is in fact due to the efficacy of the disclosed contrastive loss formulation over the pseudo-label loss used in the first conventional technique.

Effect of Different Frame Rate

The effect of doubling frame-rates in both pathways304,308was analyzed and it was observed that TCL (with 16 frame segments in the base pathway304and 8 frame segments in the auxiliary pathway308) improved top-1 accuracy by 1.5% on the first set of videos with the first conventional backbone and 5% labeled data (29.81% vs 31.31%).

Effect of Hyperparameters

The effect of the ratio of unlabeled data to labeled data (μ) was analyzed.FIG.6illustrates the effect of hyperparameters on the first set of videos, in accordance with an example embodiment. Varying the ratio of unlabeled data to the labeled data (μ) (right-side) and varying the instance-contrastive loss weight (γ), it was observed that setting (μ) to {3, 5, 7} (with a fixed γ=1) produces similar results on the first set of videos (FIG.6, left-side). However, as scaling μ often requires high computational resources, (μ) was set to 3 in all of the experiments to balance the efficiency and accuracy in semi-supervised action recognition. It was also found that weight of the instance-contrastive loss (γ) greatly affects the performance in semi-supervised learning as accuracy drops by more than 6% when setting γ to 3 instead of the optimal value of 9 on the first set of videos with the first conventional backbone and 5% of labeling (FIG.6, right-side).

Comparison with Self-Supervised Approaches

The disclosed methods were compared with three video self-supervised methods, namely Odd-One-Out Networks (O3N), Video Clip Order Prediction (COP), and Memory-augmented Dense Predictive Coding (MemDPC) through pretraining using a self-supervised method and then finetuning using available labels on the first set of videos (with the first conventional backbone and 5% labeled data). The disclosed approach significantly outperforms all the compared methods by a margin of 6%-10%, showing its effectiveness over self-supervised methods. Moreover, the disclosed temporal contrastive learning was replaced with a conventional method and it was observed that accuracy drops to 24.58% from 29.81%, showing the efficacy of the disclosed contrastive learning formulation over the alternate video-based self-supervised method on the first set of videos.

It will accordingly be appreciated that one or more embodiments provide a novel temporal contrastive learning frame-work for semi-supervised action recognition by maximizing the similarity between encoded representations of the same unlabeled video at two different speeds as well as minimizing the similarity between different unlabeled videos run at different speeds. In one or more embodiments, a contrastive loss is employed between different video instances (including groups of videos) with similar actions to explore high-level action semantics within the neighborhood of different videos depicting different instances of the same action. The effectiveness of one or more embodiments was demonstrated on four standard benchmark datasets, significantly outperforming several competing methods.

Given the discussion thus far, it will be appreciated that, in general terms, an exemplary method, according to an aspect of the invention, includes the operations of training a base pathway304of a computerized two-pathway video action recognition model300using a plurality of labeled video samples312; training the base pathway304of the computerized two-pathway video action recognition model300using a plurality of unlabeled video samples316at a first framerate; training an auxiliary pathway308of the computerized two-pathway video action recognition model300using a plurality of the unlabeled video samples316at a second framerate, the second framerate being slower than the first framerate; wherein said training of said base pathway304using said plurality of labeled video samples312, said training of said base pathway304using said plurality of unlabeled video samples316at said first framerate, and said training of said auxiliary pathway308using said plurality of unlabeled video samples316at said second framerate, result in a trained computerized two-pathway video action recognition model300; categorizing a candidate video using the trained computerized two-pathway video action recognition model300; and storing the categorized candidate video in a computer-accessible video database system333for information retrieval.

In one example embodiment, the two-pathway video action recognition model300comprises a temporal contrastive model.

In one example embodiment, a contrastive objective is based on a maximization of a similarity between encoded representations of a same video of the unlabeled video samples316at different framerates and a minimization of a similarity between encoded representations of different videos of the unlabeled video samples316at different speeds.

In one example embodiment, the similarity between the different videos of the unlabeled video samples316is minimized by minimizing a modified Normalized Temperature-scaled Cross Entropy Loss (NT-Xent) contrastive loss between the different videos.

In one example embodiment, the base pathway304and the auxiliary pathway308share a same set of weights.

In one example embodiment, groups of the unlabeled video samples316having a same pseudo-label in a minibatch are formed, and each group is represented with an average representation of the unlabeled video samples316within each group, wherein the contrastive objective is based on a group-contrastive loss between the groups of the unlabeled video samples316that couples discriminative motion representation with pace-invariance.

In one example embodiment, the average representation is based on:

where ∥ is an indicator function that evaluates to 1 for videos with a pseudo-label equal to l∈Y in each pathway p∈{f, s}, g(Upi) is a representation of a corresponding video, B is a count of videos in the minibatch, ŷpidenotes pseudo-labels of the video Ui, and T is a number of the videos with the pseudo-label equal to l∈Y in the minibatch.

In one example embodiment, the training operations are performed on a neural network backbone328involving at least one of two-dimensional (2D) and three-dimensional (3D) convolution operations.

In one example embodiment, a standard supervised cross-entropy loss (sup) is minimized on the labeled video samples312, the standard supervised cross-entropy loss (sup) being given by:

ℒsup=-∑c=1C(yi)c⁢log⁡(g⁡(Vi))c
where g(Vi) is a representation of a corresponding video Viand C is a count of different activities.

In one example embodiment, the two-pathway video action recognition model300is trained to match a representation g(Ufi) of a faster framerate version of a video (Ui) with a representation g(Usi) of a comparatively slower framerate version of the video (Ui).

In one example embodiment, the two-pathway video action recognition model300is trained using a loss functiongiven by:
=sup+γ*ic+β*gc
wheresupis a standard supervised cross-entropy loss,icis an instance-contrastive loss,gcis a group-contrastive loss, and γ and β are weights of the instance-contrastive and group-contrastive losses, respectively.

In one example embodiment, the instance-contrastive lossicis:

h⁡(u,v)=exp⁡(uT⁢vu2⁢v2/τ)
is an exponential of cosine similarity measure, B is a count of videos in a minibatch, τ is a temperature hyperparameter, g(Usi) is a representation of a comparatively slower framerate version of a video (Ui), g(Ufi) is a representation of a comparatively faster framerate version of the video (Ui), and (Ufi, Usi) and (Usi, Ufi) are positive pairs of the unlabeled video samples316across the minibatch.

In one example embodiment, the group-contrastive loss is:

ℒgc(Rfl,Rsl)=-log⁢h⁡(Rfl,Rsl)h⁡(Rfl,Rsl)+∑m=1C𝕝{m≠l}⁢h⁡(Rfl,Rpm)p∈{s,f}
where ∥ is an indicator function that evaluates to 1 for videos with a pseudo-label equal to l∈Y in each pathway p∈{f, s}, C is a count of different activities, Rflis an average representation of a comparatively faster framerate version of a video (Ui), Rslis an average representation of a comparatively slower framerate version of the video (Ui).

In one example embodiment, a searched video in the computer-accessible video database system333for information retrieval is identified based on a given action.

In one aspect, an apparatus comprises a memory and at least one processor, coupled to the memory, and operative to perform a method comprising training a base pathway304of a computerized two-pathway video action recognition model300using a plurality of labeled video samples312; training the base pathway304of the computerized two-pathway video action recognition model300using a plurality of unlabeled video samples316at a first framerate; training an auxiliary pathway308of the computerized two-pathway video action recognition model300using a plurality of the unlabeled video samples316at a second framerate, the second framerate being slower than the first framerate (wherein said training of said base pathway304using said plurality of labeled video samples312, said training of said base pathway304using said plurality of unlabeled video samples316at said first framerate, and said training of said auxiliary pathway308using said plurality of unlabeled video samples316at said second framerate, result in a trained computerized two-pathway video action recognition model300); categorizing a candidate video using the trained computerized two-pathway video action recognition model300; and storing the categorized candidate video in a computer-accessible video database system333for information retrieval.

In one aspect, a computer program product comprises a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising training a base pathway304of a computerized two-pathway video action recognition model300using a plurality of labeled video samples312; training the base pathway304of the computerized two-pathway video action recognition model300using a plurality of unlabeled video samples316at a first framerate; training an auxiliary pathway308of the computerized two-pathway video action recognition model300using a plurality of the unlabeled video samples316at a second framerate, the second framerate being slower than the first framerate (wherein said training of said base pathway304using said plurality of labeled video samples312, said training of said base pathway304using said plurality of unlabeled video samples316at said first framerate, and said training of said auxiliary pathway308using said plurality of unlabeled video samples316at said second framerate, result in a trained computerized two-pathway video action recognition model300); categorizing a candidate video using the trained computerized two-pathway video action recognition model300; and storing the categorized candidate video in a computer-accessible video database system333for information retrieval.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG.8, a set of functional abstraction layers provided by cloud computing environment50(FIG.7) is shown. It should be understood in advance that the components, layers, and functions shown inFIG.8are intended to be illustrative only and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

Workloads layer90provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation91; software development and lifecycle management92; virtual classroom education delivery93; data analytics processing94; transaction processing95; and a video processing component96that implements aspects of semi-supervised video action recognition and learning.

One or more embodiments of the invention, or elements thereof, can be implemented in the form of an apparatus including a memory and at least one processor that is coupled to the memory and operative to perform exemplary method steps.FIG.9depicts a computer system that may be useful in implementing one or more aspects and/or elements of the invention, also representative of a cloud computing node according to an embodiment of the present invention. Referring now toFIG.9, cloud computing node10is only one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the invention described herein. Regardless, cloud computing node10is capable of being implemented and/or performing any of the functionality set forth hereinabove.

Thus, one or more embodiments can make use of software running on a general purpose computer or workstation. With reference toFIG.9, such an implementation might employ, for example, a processor16, a memory28, and an input/output interface22to a display24and external device(s)14such as a keyboard, a pointing device, or the like. The term “processor” as used herein is intended to include any processing device, such as, for example, one that includes a CPU (central processing unit) and/or other forms of processing circuitry. Further, the term “processor” may refer to more than one individual processor. The term “memory” is intended to include memory associated with a processor or CPU, such as, for example, RAM (random access memory)30, ROM (read only memory), a fixed memory device (for example, hard drive34), a removable memory device (for example, diskette), a flash memory and the like. In addition, the phrase “input/output interface” as used herein, is intended to contemplate an interface to, for example, one or more mechanisms for inputting data to the processing unit (for example, mouse), and one or more mechanisms for providing results associated with the processing unit (for example, printer). The processor16, memory28, and input/output interface22can be interconnected, for example, via bus18as part of a data processing unit12. Suitable interconnections, for example via bus18, can also be provided to a network interface20, such as a network card, which can be provided to interface with a computer network, and to a media interface, such as a diskette or CD-ROM drive, which can be provided to interface with suitable media.

A data processing system suitable for storing and/or executing program code will include at least one processor16coupled directly or indirectly to memory elements28through a system bus18. The memory elements can include local memory employed during actual implementation of the program code, bulk storage, and cache memories32which provide temporary storage of at least some program code in order to reduce the number of times code must be retrieved from bulk storage during implementation.

Input/output or I/O devices (including but not limited to keyboards, displays, pointing devices, and the like) can be coupled to the system either directly or through intervening I/O controllers.

One or more embodiments can be at least partially implemented in the context of a cloud or virtual machine environment, although this is exemplary and non-limiting. Reference is made back toFIGS.7-8and accompanying text.

It should be noted that any of the methods described herein can include an additional step of providing a system comprising distinct software modules embodied on a computer readable storage medium; the modules can include, for example, any or all of the appropriate elements depicted in the block diagrams and/or described herein; by way of example and not limitation, any one, some or all of the modules/blocks and or sub-modules/sub-blocks described. The method steps can then be carried out using the distinct software modules and/or sub-modules of the system, as described above, executing on one or more hardware processors such as 16. Further, a computer program product can include a computer-readable storage medium with code adapted to be implemented to carry out one or more method steps described herein, including the provision of the system with the distinct software modules.

One example of user interface that could be employed in some cases is hypertext markup language (HTML) code served out by a server or the like, to a browser of a computing device of a user. The HTML is parsed by the browser on the user's computing device to create a graphical user interface (GUI).

Exemplary System and Article of Manufacture Details