Patent Publication Number: US-11640710-B2

Title: Weakly-supervised action localization by sparse temporal pooling network

Description:
PRIORITY CLAIM 
     This application_is based upon and claims the right of priority under 35 U.S.C. § 371 to International Application No. PCT/US2018/059176 filed on Nov. 5, 2018, which claims the benefit of U.S. Provisional Application Ser. No. 62/586,078 filed Nov. 14, 2017. Applicant claims priority to and the benefit of each of such applications and incorporates all such applications herein by reference in their entirety. 
    
    
     FIELD 
     The present disclosure relates generally to video action detection and localization. More particularly, the present disclosure relates to a weakly-supervised temporal action localization model for untrimmed videos. 
     BACKGROUND 
     Action recognition in videos is one of the critical problems for high-level video understanding including event detection, video summarization, and visual question answering in videos. Many researchers have been investigating the problem extensively in the last decade. The main challenge in action recognition is a lack of appropriate representation methods of videos. Contrary to the almost immediate success of convolutional neural networks (CNNs) in many visual recognition problems related to images, applying deep neural networks to video data is not straightforward due to a number of issues including the inherently complex structures of data, large file sizes, and lack of knowledge for modeling temporal information. Hence, techniques based on the representations from deep learning have not been particularly better than methods relying on hand-crafted visual features. As a result, many existing algorithms attempt to achieve the state-of-the-art performance by combining hand-crafted and learned/trained features. 
     Another issue is that most existing techniques assume trimmed videos for video-level text classification or require annotations of action intervals for temporal localization. Since an untrimmed video typically contains a large number of irrelevant frames directly related to the video level class labels, both video representation learning and action classification are likely to fail in extracting necessary information from raw videos. On the other hand, annotating a large scale dataset for action localization is prohibitively expensive and time-consuming, and thus it would be more desirable to develop competitive algorithms running without such temporal annotations. 
     SUMMARY 
     Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments. 
     One example aspect of the present disclosure is directed to a computer-implemented method for temporally localizing a target action in a video. The computer-implemented method can include inputting a video comprising a RGB stream into a sparse temporal pooling network. The sparse temporal pooling network can include a first weakly supervised temporal action localization model and a second weakly supervised temporal action localization model. The computer-implemented method can further include determining, by the sparse temporal pooling network, an optical flow stream of the video. The computer-implemented method can further include analyzing the RGB stream by the first weakly-supervised temporal action localization model to determine an RGB weighted temporal class activation map comprising a one dimensional class-specific activation map in a temporal domain. The computer-implemented method can further include analyzing the optical flow stream by the second weakly-supervised temporal action localization model to determine a flow weighted temporal class activation map comprising a one dimensional class-specific activation map in the temporal domain. The computer-implemented method can further include determining, by the sparse temporal pooling network, a temporal location of a target action in the video based at least in part on the RGB weighted temporal class activation map or the flow weighted temporal class activation map. 
     Another example aspect of the present disclosure is directed to a computer-implemented method of training a weakly supervised temporal action localization model. The computer-implemented method can include inputting an untrimmed video into the weakly supervised temporal action localization model. The computer-implemented method can further include analyzing the untrimmed video by the weakly supervised temporal action localization model to determine a predicted score for an action classification. The computer-implemented method can further include determining a loss function based at least in part on the predicted score. The loss function can include a sparsity loss and a classification loss. The computer-implemented method can further include training the weakly supervised temporal action localization model based at least in part on the loss function. 
     Another example aspect of the present disclosure is directed to a computing system. The computing system can include at least one processor, a sparse temporal pooling network, and at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The sparse temporal pooling network can include a first weakly supervised temporal action localization model and a second weakly supervised temporal action localization model. The first weakly supervised temporal action localization model can be trained to receive a video comprising a RGB stream and, in response to receipt of the RGB stream, output a RGB weighted temporal class activation map comprising a one dimensional class-specific activation map in a temporal domain. The second weakly supervised temporal action localization model can be trained to receive an optical flow stream of the video and in response to receipt of the optical flow stream, output a flow weighted temporal class activation map comprising a one dimensional class-specific activation map in a temporal domain. The operations can include obtaining the video. The operations can further include generating the optical flow stream based at least in part on the RGB stream. The operations can further include inputting the RGB stream into the first weakly supervised temporal action localization model. The operations can further include receiving, as an output of the first weakly supervised temporal action localization model, the RGB weighted temporal class activation map. The operations can further include inputting the optical flow stream into the second weakly supervised temporal action localization model. The operations can further include receiving, as an output of the second weakly supervised temporal action localization model, the flow weighted temporal class activation map. The operations can further include determining a temporal location of a target action in the video based at least in part on the RGB weighted temporal class activation map or the flow weighted temporal class activation map. 
     Additional aspects of the present disclosure are directed to various models, computer-implemented methods, computing devices, systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices. 
     These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the related principles. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended figures, in which: 
         FIG.  1    depicts a block diagram of an example sparse temporal pooling network according to example aspects of the present disclosure; 
         FIG.  2    depicts a block diagram of an example weakly supervised temporal action localization model according to example aspects of the present disclosure; 
         FIG.  3    depicts an illustration of example groundtruth actions and corresponding attention weights and temporal class activation map outputs using one or more example models according to example aspects of the present disclosure; 
         FIG.  4 A  depicts example groundtruth action instances along with prediction and a corresponding weighted temporal class activation map using one or more example models according to example aspects of the present disclosure; 
         FIG.  4 B  depicts example videos containing groundtruth action instances from different classes along with prediction a corresponding weighted temporal class activation map using one or more example models according to example aspects of the present disclosure; 
         FIG.  4 C  depicts example groundtruth action instances along with prediction and a corresponding weighted temporal class activation map using one or more example model according to example aspects of the present disclosure; 
         FIG.  5    depicts a graph showing performance of one or more example models according to example aspects of the present disclosure for different architecture choices; 
         FIG.  6    depicts a graph showing a performance of one or more example models according to example aspects of the present disclosure for different feature choices; 
         FIG.  7 A  depicts a block diagram of an example computing system that performs action classification and/or localization according to example aspects of the present disclosure; 
         FIG.  7 B  depicts a block diagram of an example computing device that performs action classification and/or localization according to example aspects of the present disclosure; 
         FIG.  7 C  depicts a block diagram of an example computing device that performs action classification and/or localization according to example aspects of the present disclosure; 
         FIG.  8    depicts a block diagram of an example action classifier/localization model according to example aspects of the present disclosure; 
         FIG.  9    depicts a flow chart diagram of an example method to perform action localization and/or classification according to example aspects of the present disclosure; 
         FIG.  10    depicts a flow chart diagram of an example method to temporally localize a target action in a video according to example aspects of the present disclosure; and 
         FIG.  11    depicts a flow chart diagram of an example method to train a weakly supervised temporal action localization model according to example aspects of the present disclosure. 
     
    
    
     Reference numerals that are repeated across plural figures are intended to identify the same features in various implementations. 
     DETAILED DESCRIPTION 
     Overview 
     Generally, example aspects of the present disclosure are directed to models, computer-implemented methods, computing devices, non-transitory, computer-readable media, systems, and methods for localizing action in video using a deep neural network. The example models, systems, methods, and devices of the present disclosure can allow for localizing actions in untrimmed videos temporally, without exploiting any temporal information of actions during training. The example models, systems, methods, and devices of the present disclosure can learn/be trained based on video-level text labels for individual videos. To this end, a network model (e.g., a deep neural network) can select a subset of frames useful for action recognition, where the loss function can measure classification error and sparsity of frame selection per video. For localization, Temporal Class Activation Mappings (T-CAMs) can be employed to generate one dimensional temporal action proposals from which target actions can be localized in a temporal domain. An overview of an example sparse temporal pooling network according to aspects of the present disclosure is illustrated in  FIG.  1   . 
     As shown in  FIG.  1   , an example sparse temporal pooling network  100  according to example aspects of the present disclosure can include two weakly supervised temporal action localization models  130 / 140 . The sparse temporal pooling network  100  can receive a two-stream input for a video  110  and perform action classification  170  and localization  160  at the same time. For localization  160 , the weighted T-CAMs  135 / 145  can be computed from the two streams  115 / 120  by the weakly supervised temporal localization models  130 / 140  and employed to generate one dimension temporal action proposals  150  from which target actions can be localized (at  160 ) in the temporal domain. 
     The systems and methods described herein may provide a number of technical effects and benefits. For example, a principled deep neural network architecture for weakly supervised action recognition and localization on untrimmed videos  110  depicted in  FIG.  1    can allow for actions to be detected with a sparse subset of frames. Further, example aspects of the present disclosure provide for a technique to compute weighted T-CAMS  135 / 145  and temporal action proposals  150  using the learned/trained network  100  for localizing target actions. Further, example aspects of the present disclosure show state-of-the-art performance on THUMOS14 and have been tested on ActivityNet1.3 for future reference. 
     Proper video datasets may be needed to train models for action detection and recognition. For example, there are various existing datasets for action recognition such as UCF101, Sports-1M, HMDB51, AVA, and Kinetics. However, such training video datasets may include only trimmed videos, where target actions appear in all frames within videos, and/or may provide bounding boxes associated with actions. Further, such video training datasets may contain background frames with annotations about which frames are relevant to target actions. 
     Action recognition aims to identify a single or multiple actions per video, which is often formulated as a simple classification problem. There has been a long history for this problem, and the algorithm based on improved dense trajectories presented outstanding performance among many algorithms before deep learning started to be used actively. Convolutional neural networks have been very successful in many computer vision problems, and have been applied to the action recognition problem as well. There are several algorithms focusing on representation learning for videos and applying the learned representations to action recognition. For example, two-stream networks and 3D convolutional neural networks (C3D) are popular solutions for video representation, and those techniques and their variations have been widely used for action recognition. Recently, a combination of two-stream network and 3D convolution, referred to as I3D, has been proposed as a generic video representation method. On the other hand, many algorithms develop technologies to learn actions based on existing representation methods. 
     Action detection and localization, however, is a slightly different problem from action recognition because it is required to extract temporal or spatio-temporal volumes containing target actions. Most algorithms for this task are based on supervised learning, and employ temporal or spatio-temporal localization annotations of actions. There are a lot of existing algorithms based on deep learning, which include, for example, structured segment network, localizing actions by learning contextual relations, multi-stage CNNs, temporal association of frame-level action detections and techniques using recurrent neural networks. To deal with action detection and localization problems, many algorithms rely on action proposals, which are an extension of object proposals in images. 
     There are only a few approaches based on weakly supervised learning, which relies on video-level labels to localize actions in the temporal space. For example, UntrimmedNets extracts proposals first to recognize and detect actions, where dense temporal attention weights are obtained and softmax functions are applied to the proposals for temporal action localization by thresholding. The use of softmax function across proposals is not effective to detect multiple instances. Hide-and-seek applies the same technique—hiding random regions to force attention learning—to weakly supervised object detection and action localization. This method works well in spatial localization but is not effective in the temporal domain. Both algorithms are motivated by recent success in weakly supervised object localization; in particular, the formulation of UntrimmedNets relies heavily on object detection. 
     Example aspects of the present disclosure are directed to a weakly supervised temporal action localization model  200  based only on video-level action labels. This is achieved by designing a deep neural network for video classification based on a sparse subset of frames and identifying time intervals relevant to target classes. 
     For example, an action can be composed of a sequence of atomic actions and can be recognized by identifying a series of keyframes corresponding to atomic actions in a video. Example models according to example aspects of the present disclosure can provide for deep neural networks to predict class labels per video using a subset of representative and unique frames to target action, which can be selected automatically from an input video. The example models according to example aspects of the present disclosure can be designed for classification but can have the capability to measure the importance of each frame in predicting classification labels. After finding the relevant classes in each video, temporal intervals corresponding to the identified actions can be estimated by computing temporal attention of individual frames, generating temporal action proposals, and aggregating relevant proposals. The present disclosure relies on video-level class labels only to perform temporal action localization and presents a principled way to extract keyframes and determine appropriate time intervals corresponding to target actions. Thus, it is possible to recognize and localize multiple actions in a single video using example aspects of the present disclosure. The overall architecture of an example weakly supervised temporal action localization model  200  according to example aspects of the present disclosure is illustrated in  FIG.  2   . In some implementations, the weakly supervised temporal action localization model  200  can be a standalone sparse temporal pooling network, such as a sparse temporal pooling network  100  shown in  FIG.  1   . 
     As shown, feature representations  215 A-N from a set of uniformly sampled video segments  205  A-N can be extracted using one or more pretrained models  210 A-N (e.g., deep neural networks such as convolutional neural networks). The attention module(s)  220 A-N can generate attention weights  230 A-N corresponding to individual representations  215 A-N, which can be employed to compute a video-level representation  245  by temporal weighted average pooling (at  240 ). The video-level representation  245  can be given to a classification module  250 / 255 , and a I 1  loss can be placed upon the attention weight vector  232  to enforce sparsity constraint. 
     To estimate a class label per video, example aspects of the present disclosure can first sample a set of video segments  205 A-N from an input video (e.g., an untrimmed video  110 ), and extract a feature representation  215 A-N from each segment  205 A-N using a model  210 A-N (e.g., convolutional neural networks). Each of these representations  215 A-N can then be fed to an attention module  220  that consists of two fully connected (“FC”) layers  222 / 226  and a Rectified Linear Unit  224  (“ReLU”) layer located between the two FC layers  222 / 226 . The output of the second FC layer  226  can be given to a sigmoid function  228  forcing the generated attention weights  230 A-N to be normalized between 0 and 1. These attention weights  230 A-N can then be used to modulate the temporal average pooling. Specifically, each of the attention weights  230 A-N can be multiplied (at  235 A-N) by its corresponding feature vector  215 A-N. These feature vectors can then be averaged (at  240 ) to create the video-level representation  245 . This representation can then be passed through a fully connected layer  250  and sigmoid layer  255  to obtain the class scores. 
     For example, x t ∈R m  can be the m dimensional feature representation  215 A-N extracted from a video segment  205 A-N centered at time t, and λ t  be the corresponding attention weight  230 A-N. The video-level representation  245 , denoted by  x , can correspond to an attention weighted temporal average pooling, which can be given by Equation (1): 
     
       
         
           
             
               
                 
                   
                     x 
                     _ 
                   
                   = 
                   
                     
                       ∑ 
                       
                         t 
                         = 
                         1 
                       
                       T 
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
                         λ 
                         t 
                       
                       ⁢ 
                       
                         x 
                         t 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     where λ=(λ 1 , . . . , λ T ) can be a vector  232  of the scalar outputs of sigmoid functions  228  to normalize the range of activations, and T can be the number of video segments  205 A-N considered together for classification. The attention weight vector λ ( 232 ) can be learned with a sparsity constraint  265  in a class agnostic way. This can be useful to identify frames relevant to any action and estimate time intervals for action candidates, as disclosed herein. 
     The loss function in the proposed model  200  can be composed of two terms, classification losses  260  and sparsity losses  265 , which can be given by Equation (2):
 
 =   class +β·   sparsity   (2)
 
     where    class  denotes a classification loss  260  computed on the video level,    sparsity  can be sparsity loss  265 , and can be a constant to control trade-off between the two terms. The classification loss  260  can be based on the standard cross-entropy loss between groundtruth and x (after passing through a few layers  250 / 255  as illustrated in  FIG.  2   ) while the sparsity loss  265  can be given by determining the l 1  norm on the attention weights ∥λ∥ 1 . A sigmoid function  228  can be applied to each attention weight λ t  ( 230 A-N), and all attention weights  230 A-N can be likely to have near 0-1 binary values by l 1  loss. Note that integrating the sparsity loss  265  is indicative that an action can be recognized with a sparse subset of keyframes in videos. 
     To identify time intervals corresponding to target actions, a sufficient number of candidates can be extracted. A one dimensional class activation mapping in the temporal domain can be derived, which can be referred to as Temporal Class Activation Mapping (T-CAM). Denote by w c (k), the k-th element in the classification parameter, w, corresponding to class c. The input to the final sigmoid layer  255  for class c can be given by Equation (3): 
     
       
         
           
             
               
                 
                   
                     
                       
                         
                           s 
                           c 
                         
                         = 
                         
                           
                             ∑ 
                             
                               k 
                               = 
                               1 
                             
                             m 
                           
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               
                                 w 
                                 c 
                               
                               ⁡ 
                               
                                 ( 
                                 k 
                                 ) 
                               
                             
                             ⁢ 
                             
                               
                                 x 
                                 _ 
                               
                               ⁡ 
                               
                                 ( 
                                 k 
                                 ) 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                         
                           
                             ∑ 
                             
                               k 
                               = 
                               1 
                             
                             m 
                           
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               
                                 w 
                                 c 
                               
                               ⁡ 
                               
                                 ( 
                                 k 
                                 ) 
                               
                             
                             ⁢ 
                             
                               
                                 ∑ 
                                 
                                   t 
                                   = 
                                   1 
                                 
                                 T 
                               
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 
                                   λ 
                                   t 
                                 
                                 ⁢ 
                                 
                                   
                                     x 
                                     t 
                                   
                                   ⁡ 
                                   
                                     ( 
                                     k 
                                     ) 
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                   
                     
                       
                         = 
                         
                           
                             ∑ 
                             
                               t 
                               = 
                               1 
                             
                             T 
                           
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           
                             
                               λ 
                               t 
                             
                             ⁢ 
                             
                               
                                 ∑ 
                                 
                                   k 
                                   = 
                                   1 
                                 
                                 m 
                               
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 
                                   
                                     w 
                                     c 
                                   
                                   ⁡ 
                                   
                                     ( 
                                     k 
                                     ) 
                                   
                                 
                                 ⁢ 
                                 
                                   
                                     x 
                                     t 
                                   
                                   ⁡ 
                                   
                                     ( 
                                     k 
                                     ) 
                                   
                                 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     T-CAM, denoted by a t =(a t   1 , a t   2 , . . . , a t   C ) T , can indicate the relevance of the representation to individual classes at time step t, where each element a t   c  for class c (c=1, . . . , C) is given by Equation (4): 
     
       
         
           
             
               
                 
                   
                     a 
                     t 
                     c 
                   
                   = 
                   
                     
                       ∑ 
                       
                         k 
                         = 
                         1 
                       
                       m 
                     
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       
                         
                           w 
                           c 
                         
                         ⁡ 
                         
                           ( 
                           k 
                           ) 
                         
                       
                       ⁢ 
                       
                         
                           x 
                           t 
                         
                         ⁡ 
                         
                           ( 
                           k 
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
           
         
       
     
       FIG.  3    illustrates examples of attention weights  310  and T-CAM outputs  320  in a video sequence  330  using the example models  100 / 200  according to example aspects of the present disclosure. As shown, the discriminative temporal regions can be highlighted by the attention weights  310  and T-CAMs  320  effectively. Note that some attention weights  310  may be large at the temporal regions that do not align with a target action, which can mean that such frames present other actions. This is expected as each attention weight  310  presents generic actionness of a segment instead of the likelihood for a specific action. 
     An I3D feature can be employed to compute frame-level representation. Using multiple streams of information such as RGB flows and optical flows has become a standard practice in action recognition and detection as it often provides a significant boost in performance. I3D is also based on a two-stream network, and we can train/learn two identical action recognition networks  130 / 140 , as illustrated in  FIG.  1   ; one is for RGB stream  115  and the other is for optical flow stream  120 . The optical flow stream  120  can be computed from the RGB stream. Note that the I3D network can be used as a feature extraction machine without any fine-tuning. The two separately trained networks  130 / 140  can be fused to localize actions in an input video  110 . 
     For example, relevant class labels  170  for an input video  110  can be identified based on video-level classification scores of the models (e.g., deep neural network) described herein. Temporal proposals  150 , e.g., one dimensional time intervals, can be generated with their class labels and confidence scores. The temporal proposals  150  can correspond to video segments that potentially enclose target actions, and can be detected using T-CAMs in the example models described herein. Using two-stream models (e.g., convolutional neural network), such as shown in  FIG.  1   , the T-CAMs can be computed using Equation (4) for both streams, which can be denoted by a t,RGB   c  and a t,FLOW   c , from which the weighted T-CAMs, denoted by ψ t,RGB   c  and ψ t,FLOW   c  can be derived, as shown in Equations 5 and 6:
 
ψ t,RGB   c =λ t,RGB ·sigmoid( a   t,RGB   c )  (5)
 
ψ t,FLOW   c =λ t,FLOW ·sigmoid( a   t,FLOW   c )  (6)
 
     Note that λ t  is an element of the sparse vector ( 232 ) and multiplying by λ t  can be interpreted as a soft selection of the values from the following sigmoid function. The weighted T-CAMs ψ t,RGB   c  and ψ t,FLOW   c  can be thresholded to segment the signals, and the temporal proposals  150  can be the one-dimensional connected components extracted from each stream. 
     Unlike the original CAM-based bounding box proposals, where only the largest bounding box is retained, all of the connected components that pass the predefined threshold can be kept. In some implementations, the weighted T-CAMS ψ t,RGB   c  and ψ t,FLOW   c  can be linearly interpolated between sampled segments before thresholding to improve the temporal resolution of the proposals  150  with minimal computation addition. Then, each proposal  150 , defined by [t start ; t end ], can be given a score for each class c, given by the weighted average T-CAM of all the frames within the proposal, as given by Equation (7): 
     
       
         
           
             
               
                 
                   
                     ∑ 
                     
                       t 
                       = 
                       
                         t 
                         start 
                       
                     
                     
                       t 
                       end 
                     
                   
                   ⁢ 
                   
                       
                   
                   ⁢ 
                   
                     
                       λ 
                       
                         t 
                         , 
                         * 
                       
                     
                     ⁢ 
                     
                       
                         
                           α 
                           · 
                           
                             a 
                             
                               t 
                               , 
                               RGB 
                             
                             c 
                           
                         
                         + 
                         
                           
                             ( 
                             
                               1 
                               - 
                               α 
                             
                             ) 
                           
                           · 
                           
                             a 
                             
                               t 
                               , 
                               FLOW 
                             
                             c 
                           
                         
                       
                       
                         
                           t 
                           end 
                         
                         - 
                         
                           t 
                           start 
                         
                         + 
                         1 
                       
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     where * ∈{RGB,FLOW} and a is a parameter to control the magnitudes of the two modality signals. This value corresponds to the temporal proposal score in each stream for class c. Finally, non-maximum suppression among temporal proposals of each class can be performed independently to remove highly overlapped detections. 
     The example models according to example aspects of the present disclosure can localize actions in untrimmed videos in the temporal domain by estimating sparse attention weights and T-CAMs for generic and specific actions, respectively. This approach is novel since it has a unique model architecture (e.g., deep neural network) with classification and sparsity losses and a completely different action localization procedure—through class-specific action proposals using T-CAMs—rather than a weakly supervised object detection technique. 
     The models, methods and systems disclosed herein were evaluated on two popular action detection benchmark datasets, THUMOS14, and ActivityNet1.3. Both datasets are untrimmed, meaning that there exist frames in videos that contain no target action, and no temporal annotations for action localization during training were exploited. In some instances, there may exist multiple actions in a single video and even in a single frame. 
     THUMOS14 dataset has 20 classes for action detection. An example model  100 / 200  according to example aspects of the present disclosure was trained using the validation set, composed of 200 untrimmed videos, where there exist many more action instances annotated with action labels and their temporal boundaries. However, note that training was performed without using the provided temporal annotations and evaluated on testing set with 213 videos. This dataset is challenging as some videos are relatively long (e.g., up to 26 minutes) and contain many action instances (e.g., up to 213 instances). The length of an action in the dataset varied significantly, from a few seconds to minutes. 
     The ActivityNet dataset is a recently introduced benchmark for action recognition and detection in untrimmed videos. ActivityNet1.3 was used for evaluation, which consists of 10,024 videos for training, 4,926 for validation, and 5,044 for testing of 200 activity classes. This dataset contains a large number of natural videos that involve various human activities into a semantic taxonomy. 
     For evaluation, the standard evaluation protocol was followed, which is based on mAP values at several different levels of intersection over union (IoU) thresholds. Both datasets provide benchmarking code for ActivityNet&#39;s Temporal Action Localization task. Accuracy on the test set of ActivityNet1.3 was obtained by submitting results to the evaluation server. 
     The two-stream I3D trained on the Kinetics dataset was employed to extract features  215  from individual video segments  205 . For the RGB stream (e.g.,  115  in  FIG.  1   ), the smallest dimension of a frame was rescaled to 256 and a central crop of size 224×224 was performed. Other suitable input sizes could similarly be used. For the optical flow stream (e.g.,  120  in  FIG.  1   ), the TV-L1 optical flow algorithm was applied and the pixel values were truncated to be in [−20, 20]. Other suitable pixel values could similarly be used. A third channel of all 0 was added to the optical flow image. The input for I3D was a stack of 16 (RGB or optical flow) frames. To save space and processing time, the video was subsampled at 10 frames per second (fps). The network was trained using Adam optimizer with the starting learning rate at 1×10 −5 . The balancing parameter between classification losses and sparsity losses, β, was set to 1×10 −5 . The training process was stopped at 150K iterations. During training, as a form of data augmentation, multiple sets of segments were randomly selected from a single video. However, during testing, a fixed number of segments spanning equal temporal distance in the video were selected. In all experiments, 400 segments for both training and testing were used at a time. The example models according to example aspects of the present disclosure were implemented in Tensor-Flow. 
     Table 1 below summarizes the results on THUMOS14 dataset for all published action localization methods in the past two years. Both fully and weakly supervised approaches are included in the table, where the difference between two kinds of approaches lie in the level of supervision, especially annotations of action intervals in temporal domain; weakly supervised methods rely only on video-level action class labels to localize action instances temporally. As shown, the example models according to example aspects of the present disclosure outperform the other two techniques based on weakly supervised learning. Even with a significant difference in the level of supervision, the models disclosed herein present competitive performance compared to many recent fully supervised approaches. 
     
       
         
           
               
               
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                   
                   
                 AP@IoU 
               
            
           
           
               
               
               
               
               
               
               
               
               
               
               
            
               
                 Supervision 
                 Method 
                 0.1 
                 0.2 
                 0.3 
                 0.4 
                 0.5 
                 0.6 
                 0.7 
                 0.8 
                 0.9 
               
               
                   
               
            
           
           
               
               
               
               
               
               
               
               
               
               
               
            
               
                 Fully 
                 Heilbron et al. 
                 — 
                 — 
                 — 
                 — 
                 13.5 
                 — 
                 — 
                 — 
                 — 
               
               
                 Supervised 
                 Richard et al. 
                 39.7 
                 35.7 
                 30.0 
                 23.2 
                 15.2 
                 — 
                 — 
                 — 
                 — 
               
               
                   
                 Shou et al. 
                 47.7 
                 43.5 
                 36.3 
                 28.7 
                 19.0 
                 10.3 
                 5.3 
                 — 
                 — 
               
               
                   
                 Yeung et al. 
                 48.9 
                 44.0 
                 36.0 
                 26.4 
                 17.1 
                 — 
                 — 
                 — 
                 — 
               
               
                   
                 Yuan et al. 
                 51.4 
                 42.6 
                 33.6 
                 26.1 
                 18.8 
                 — 
                 — 
                 — 
                 — 
               
               
                   
                 Escordia et al. 
                 — 
                 — 
                 — 
                 — 
                 13.9 
                 — 
                 — 
                 — 
                 — 
               
               
                   
                 Shou et al. 
                 — 
                 — 
                 40.1 
                 29.4 
                 23.3 
                 13.1 
                 7.9 
                 — 
                 — 
               
               
                   
                 Yuan et al. 
                 51.0 
                 45.2 
                 36.5 
                 27.8 
                 17.8 
                 — 
                 — 
                 — 
                 — 
               
               
                   
                 Xu et al. 
                 54.5 
                 51.5 
                 44.8 
                 35.6 
                 28.9 
                 — 
                 — 
                 — 
                 — 
               
               
                   
                 Zhao et al. 
                 66.0 
                 59.4 
                 51.9 
                 41.0 
                 29.8 
                 — 
                 — 
                 — 
                 — 
               
               
                   
                 Alwasssel et al. 
                 49.6 
                 44.3 
                 38.1 
                 28.4 
                 19.8 
                 — 
                 — 
                 — 
                 — 
               
               
                 Weakly 
                 Wang et al. 
                 44.4 
                 37.7 
                 28.2 
                 21.1 
                 13.7 
                 — 
                 — 
                 — 
                 — 
               
               
                 Supervised 
                 Singh &amp; Lee 
                 36.4 
                 27.8 
                 19.5 
                 12.7 
                 6.8 
                 — 
                 — 
                 — 
                 — 
               
               
                   
                 Present Disclosure 
                 52.0 
                 44.7 
                 35.5 
                 25.8 
                 16.9 
                 9.9 
                 4.3 
                 1.2 
                 0.1 
               
               
                   
               
            
           
         
       
     
     As shown in Table 1, the present disclosure is compared with other techniques tested on the THUMOS14 database. The techniques are divided into two groups based on the level of supervision, and sorted chronologically from older to new techniques. The present disclosure presents state-of-the-art performance in a weakly supervised setting, approaching performance of many fully supervised approaches. 
     The example models according to example aspects of the present disclosure were also evaluated using the validation and testing sets of ActivityNet1.3 dataset. The results of this validation set and testing set are shown below in Tables 2 and 3, respectively. 
     
       
         
           
               
               
             
               
                   
                 TABLE 2 
               
             
            
               
                   
                   
               
               
                   
                 AP@IoU 
               
            
           
           
               
               
               
               
               
            
               
                   
                 Method 
                 0.5 
                 0.75 
                 0.95 
               
               
                   
                   
               
            
           
           
               
               
               
               
               
               
            
               
                   
                 Fully 
                 Singh &amp; Cuzzonlin 
                 34.5 
                 — 
                 — 
               
               
                   
                 Supervised 
                 Wang &amp; Tao 
                 45.1 
                  4.1 
                 0.0 
               
               
                   
                   
                 Shou et al. 
                 45.3 
                 26.0 
                 0.2 
               
               
                   
                   
                 Xiong et al. 
                 39.1 
                 23.5 
                 5.5 
               
               
                   
                   
                 Montes et al. 
                 22.5 
                 — 
                 — 
               
               
                   
                   
                 Xu et al. 
                 26.8 
                 — 
                 — 
               
               
                   
                 Weakly 
                 Present Disclosure 
                 29.3 
                 16.9 
                 2.6 
               
               
                   
                 Supervised 
               
               
                   
                   
               
            
           
         
       
     
     As shown in Table 2, the present disclosure was compared on the ActivityNet 1.3 validation set to other techniques. 
     
       
         
           
               
               
               
             
               
                   
                 TABLE 3 
               
               
                   
                   
               
               
                   
                 Method 
                 mAP 
               
               
                   
                   
               
             
            
               
                   
               
            
           
           
               
               
               
               
            
               
                   
                 Fully 
                 Singh &amp; Cuzzonlin 
                 17.83 
               
               
                   
                 Supervised 
                 Wang &amp; Tao 
                 14.62 
               
               
                   
                   
                 Shou et al. 
                 26.05 
               
               
                   
                   
                 Xiong et al. 
                 17.68 
               
               
                   
                   
                 Montes et al. 
                 28.28 
               
               
                   
                 Weakly 
                 Present Disclosure 
                 20.07 
               
               
                   
                 Supervised 
               
               
                   
                   
               
            
           
         
       
     
     As shown in Table 3, the present disclosure was compared on the ActivityNet 1.3 test set to other techniques. 
     Most of available action recognition results on the ActivityNet dataset were from the ActivityNet Challenge submission, and there did not appear to be many directly comparable techniques. Moreover, the present disclosure is believed to present the first attempt to evaluate weakly supervised action localization performance on this dataset. The results are reported results as a baseline for future reference. 
     The qualitative results in both datasets are demonstrated in  FIG.  4   . As mentioned herein, videos in THUMOS14 dataset are often long and contain many action instances, including from different categories.  FIG.  4 A  shows an example with many action instances  410 A along with the groundtruth  420 A, detections (e.g., localized target action)  430 A, and the corresponding weighted T-CAM  440 A signals for the example models of the present disclosure. 
       FIG.  4 B  illustrates an example of a video containing action instances  410 B from different classes. Visually, the two involved action classes—discus throw ( 420 B- 440 B) and shotput ( 450 B- 470 B)—are similar in their appearance (green grass, person with blue shirt, standing on a gray platform) and motion patterns (circular throwing). The models according to example aspects of the present disclosure are able to not only localize the target actions but also classify the action categories successfully, despite several short term false positives. 
       FIG.  4 C  depicts an instructional video for a javelin throw (JavelinThrow) action  410 C. The example models of the present disclosure are able to detect most of the groundtruth action  420 C instances within the video; however, some false positives may be generated. Through further visual inspection, two causes for the false alarms have been observed. Many frames in  FIG.  4 C  contain the scene for a javelin throw but the respective groundtruths  420 C were often missing, which makes true detections  430 C count as false positives. The second source of false positives were related to the segments in which the instructors demonstrate javelin throwing but only parts of such actions are visible in the segments. These segments resemble a real javelin throw action in both appearance and motion. 
     Further, the contribution of several components in the example weakly supervised models and implementation variations according to example aspects of the present disclosure were investigated. All these experiments were performed on the THUMOS14 dataset. 
     The investigation evaluated whether an action can be recognized by a sparse subset of frames in a video. When the sparse temporal pooling network model according to example aspects of the present disclosure was trained, two loss terms—classification losses and sparsity losses—were employed. The baseline was the architecture without the attention module.  FIG.  5    shows comparisons between the baselines of the present disclosure and a full model. Both the sparsity loss and attention weighted pooling were observed to make substantial contributions in performance improvement. As shown in  FIG.  5   , the attention module was shown to be useful as it allows the example models according to example aspects of the present disclosure to focus on the important parts on input videos. For example, enforcing the sparsity in action recognition via sparsity loss gave a significant boost in performance. 
     As mentioned herein, the representation of each frame is based on a two-stream I3D network, which employs two sources of information; one is from RGB image and the other is from optical flow.  FIG.  6    shows the effectiveness of each modality and their combination. As a single feature, the optical flow stream offers stronger performance than the RGB stream. Similar to action recognition, the combinations of the modalities provided significant performance improvement. For example, optical flow offered stronger cues than RGB for action localization, and a combination of the two features lead to significant improvements. 
     The example models according to example aspects of the present disclosure provide a weakly supervised action localization model and techniques, which can be based on deep neural networks with classification losses and sparsity losses. The classification can be performed by evaluating a video-level representation given by a sparse weighted mean of frame-level features, where the sparse coefficients are learned with sparsity loss. For weakly supervised action localization, one dimensional action proposals can be extracted first and relevant proposals to target classes can be selected to present time intervals of actions. The example models disclosed herein are more principled than previous models. For example, the models provided herein minimized heuristic components to improve quantitative performance. The proposed approach achieved the state-of-the-art accuracy on the THUMOS14 dataset, and new results on the AcitivityNet1.3 dataset were also demonstrated. 
     Example Devices and Systems 
       FIG.  7 A  depicts a block diagram of an example computing system  700  that performs action classification and/or localization according to example aspects of the present disclosure. The system  700  includes a user computing device  702 , a server computing system  730 , and a training computing system  750  that are communicatively coupled over a network  780 . 
     The user computing device  702  can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device. 
     The user computing device  702  includes one or more processors  712  and a memory  714 . The one or more processors  712  can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory  714  can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory  714  can store data  716  and instructions  718  which are executed by the processor  712  to cause the user computing device  702  to perform operations. 
     In some implementations, the user computing device  702  can store or include one or more machine-learned models  720 . For example, the machine-learned models  720  can be or can otherwise include various machine-learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and/or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Example machine-learned models  720  are discussed with reference to  FIGS.  1 - 2   , and can include a sparse temporal pooling network, which includes two weakly supervised temporal action localization models, as disclosed herein. 
     In some implementations, the one or more machine-learned models  720  can be received from the server computing system  730  over network  780 , stored in the user computing device memory  714 , and then used or otherwise implemented by the one or more processors  712 . In some implementations, the user computing device  702  can implement multiple parallel instances of a single machine-learned model  720  (e.g., to perform parallel action classification and/or localization. 
     More particularly, the one or more machine-learned models  720  can include one or more action classification and/or localization models. For example, the one or more machine-learned models  720  can include one or more weakly supervised temporal action localization models configured to localize and/or classify actions from untrimmed videos, as described herein. In some implementations, the one or more models can include one or more two-stream CNN models, such as a two-stream sparse temporal pooling network model for a RGB stream and an optical flow stream, as described herein. 
     Additionally or alternatively, one or more machine-learned models  740  can be included in or otherwise stored and implemented by the server computing system  730  that communicates with the user computing device  702  according to a client-server relationship. For example, the machine-learned models  740  can be implemented by the server computing system  740  as a portion of a web service (e.g., a video action classification and/or localization service). Thus, one or more models  720  can be stored and implemented at the user computing device  702  and/or one or more models  740  can be stored and implemented at the server computing system  730 . 
     The user computing device  702  can also include one or more user input components  722  that receive user input. For example, the user input component  722  can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input. 
     The server computing system  730  includes one or more processors  732  and a memory  734 . The one or more processors  732  can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory  734  can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory  734  can store data  736  and instructions  738  which are executed by the processor  732  to cause the server computing system  730  to perform operations. 
     In some implementations, the server computing system  730  includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system  730  includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof. 
     As described above, the server computing system  730  can store or otherwise include one or more machine-learned models  740 . For example, the models  740  can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Example models  740  are discussed with reference to  FIGS.  1 - 2   . 
     The user computing device  702  and/or the server computing system  730  can train the models  720  and/or  740  via interaction with the training computing system  750  that is communicatively coupled over the network  780 . The training computing system  750  can be separate from the server computing system  730  or can be a portion of the server computing system  730 . 
     The training computing system  750  includes one or more processors  752  and a memory  754 . The one or more processors  752  can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory  754  can include one or more non-transitory computer-readable storage mediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory  754  can store data  756  and instructions  758  which are executed by the processor  752  to cause the training computing system  750  to perform operations. In some implementations, the training computing system  750  includes or is otherwise implemented by one or more server computing devices. 
     The training computing system  750  can include a model trainer  760  that trains the machine-learned models  720  and/or  740  stored at the user computing device  702  and/or the server computing system  730  using various training or learning techniques, such as, for example, backwards propagation of errors. In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer  760  can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained. 
     In particular, the model trainer  760  can train the machine-learned models  720  and/or  740  based on a set of training data  762 . The training data  762  can include, for example, video training datasets, such as the THUMOS14 validation set, as described herein. Other suitable training data sets can similarly be used. For example, the training data can include untrimmed videos containing a target action class that has been labelled with a video level action classification label. 
     In some implementations, if the user has provided consent, the training examples can be provided by the user computing device  702 . Thus, in such implementations, the model  720  provided to the user computing device  702  can be trained by the training computing system  750  on user-specific data received from the user computing device  702 . In some instances, this process can be referred to as personalizing the model. 
     The model trainer  760  includes computer logic utilized to provide desired functionality. The model trainer  760  can be implemented in hardware, firmware, and/or software controlling a general purpose processor. For example, in some implementations, the model trainer  760  includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer  760  includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM hard disk or optical or magnetic media. 
     The network  780  can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof and can include any number of wired or wireless links. In general, communication over the network  780  can be carried via any type of wired and/or wireless connection, using a wide variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL). 
       FIG.  7 A  illustrates one example computing system that can be used to implement the present disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing device  702  can include the model trainer  760  and the training dataset  762 . In such implementations, the models  720  can be both trained and used locally at the user computing device  702 . In some of such implementations, the user computing device  702  can implement the model trainer  760  to personalize the models  720  based on user-specific data. 
       FIG.  7 B  depicts a block diagram of an example computing device  710  that performs according to example aspects of the present disclosure. The computing device  710  can be a user computing device or a server computing device. 
     The computing device  710  includes a number of applications (e.g., applications  1  through N). Each application contains its own machine learning library and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. 
     As illustrated in  FIG.  7 B , each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application. 
       FIG.  7 C  depicts a block diagram of an example computing device  770  that performs according to example aspects of the present disclosure. The computing device  770  can be a user computing device or a server computing device. 
     The computing device  770  includes a number of applications (e.g., applications  1  through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications). 
     The central intelligence layer includes a number of machine-learned models. For example, as illustrated in  FIG.  1 C , a respective machine-learned model (e.g., a model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device  770 . 
     The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository of data for the computing device  770 . As illustrated in  FIG.  1 C , the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and/or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API). 
     Example Model Arrangements 
       FIG.  8    depicts a block diagram of an example machine-learned model  800  according to example aspects of the present disclosure. In some implementations, the machine-learned model  800  is trained to receive a set of input data  804  descriptive of a video (e.g., an untrimmed video) and, as a result of receipt of the input data  804 , provide output data  806  that localizes and/or classifies one or more actions. Thus, in some implementations, the machine-learned model  800  can include one or more weakly supervised temporal action localization model(s)  802  that are operable to temporally localize action and/or classify such action(s), as described herein. In some implementations, the machine-learned model  800  can be a two-stream sparse temporal pooling network, which can include two weakly supervised temporal action localization models  802 , as described herein. 
     Example Methods 
       FIG.  9    depicts a flow chart diagram of an example method to temporally localize a target action according to example aspects of the present disclosure. Although  FIG.  9    depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method  900  can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the present disclosure. 
     At  902 , a video can be input into a machine-learned model comprising one or more weakly supervised temporal action localization models. In some implementations, the video can include a RGB stream. In some implementations, the machine-learned model can be a sparse temporal pooling network, which can include two weakly supervised temporal action localization models. The machine-learned models can be implemented on a computing system, and the video can be data indicative of the video. For example, in some implementations, a user can use a user computing device  702  to select a video to be provided to machine-learned model  720 . In some implementations, the machine-learned model  720  can be stored on the user computing device  702 , and in other implementations, the machine-learned model  740  can be stored on a computing system  730  remote from the user computing device  702 . The data can be, for example, data indicative of an untrimmed video. 
     At  904 , the computing system can analyze the video by the one or more weakly-supervised temporal action localization models to determine one or more weighted temporal class activation maps. For example, the machine-learned model can include one or more weakly supervised temporal action localization models and/or a sparse temporal pooling network. In some implementations, the machine-learned model can be a two-stream model, wherein a first stream is a RGB stream and a second stream is an optical flow stream. The machine-learned model can be a machine-learned model configured to receive the video and output a weighted temporal class activation map. 
     At  906 , the computing system can determine a temporal location of a target action in the video based at least in part on the one or more weighted temporal class activation maps. For example, the one or more temporal class activation maps can be thresholded to determine one or more temporal proposals, as provided herein. In some implementations, a first weighted temporal class activation map can be compared to a second weighted temporal class activation map, as provided herein. In some implementations, the output can include data indicative of an action classification and/or action localization. For example, the output from the machine-learned model can temporally locate an action and/or classify one or more target present in the video. In some implementations, the output of the machine-learned model can include both a temporal localization of an action as well as an action classification. 
       FIG.  10    depicts a flow chart diagram of an example method to temporally localize a target action in a video according to example aspects of the present disclosure. The video can be, for example, an untrimmed video. Although  FIG.  10    depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method  1000  can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the present disclosure. Method  1000  can be implemented by a computing system, such as one or more computing devices, as disclosed herein. 
     At  1002 , the method  1000  can include obtaining a video comprising a RGB stream. For example, a sparse temporal pooling network comprising a first weakly supervised temporal action localization model and a second weakly supervised temporal action localization model can be implemented on a computing system. A user can provide the video to the sparse temporal pooling network, such as by inputting the video into the sparse temporal pooling network. 
     At  1004 , the method  1000  can include generating an optical flow stream based at least in part on the RGB stream. For example, the sparse temporal pooling network can be configured to generate the optical flow stream from the RGB stream by performing an optical flow algorithm on the RGB stream. 
     At  1006 , the method  1000  can include inputting the RGB stream into a first weakly supervised temporal action localization model. For example, the sparse temporal pooling network can be a two-stream network, and the RGB stream can be input into the first weakly supervised temporal action localization model. 
     At  1008 , the method  1000  can include receiving a RGB weighted temporal class activation map as an output of the first weakly supervised temporal action localization model. For example, the first weakly supervised temporal action localization model can be trained to analyze the RGB stream to determine an RGB weighted temporal class activation map. The RGB weighted temporal class activation map can comprise a one-dimensional class-specific activation map in a temporal domain. 
     At  1010 , the method  1000  can include inputting the optical flow stream into a second weakly supervised temporal action localization model. For example, the sparse temporal pooling network can be a two-stream network, and the optical flow stream can be input into the second weakly supervised temporal action localization model. 
     At  1012 , the method  1000  can include receiving a flow weighted temporal class activation map as an output of the second weakly supervised temporal action localization model. For example, the second weakly supervised temporal action localization model can be trained to analyze the optical flow stream to determine a flow weighted temporal class activation map. The flow weighted temporal class activation map can comprise a one-dimensional class-specific activation map in a temporal domain. 
     At  1014 , the method  1000  can include determining a temporal location of a target action in the video based at least in part on the RGB weighted temporal class activation map or the flow weighted temporal class activation map. For example, a sparse temporal pooling network can be configured to generate one or more class-specific temporal proposals for each of the RGB stream and the optical flow stream. Each class-specific temporal proposal can include one-dimensional connected components extracted from the RGB stream or the optical flow stream. 
     In some implementations, the one or more class-specific temporal proposals can be generated by the sparse temporal pooling network for each of the RGB stream and the optical flow stream by thresholding the RGB weighted temporal class activation map to segment the RGB stream or thresholding the flow weighted temporal class activation map to segment the optical flow stream. In some implementations, the one or more class-specific temporal proposals can be generated by linearly interpolating the RGB weighted temporal class activation map or the flow weighted temporal class activation map. For example, the weighted temporal class activation maps can be linearly interpolated between sampled segments before thresholding. 
     In some implementations, determining the temporal location of the target action in the video based at least in part on the RGB weighted temporal class activation map or the flow weighted temporal class activation map can include selecting a first class-specific temporal proposal for a particular time interval from either the RGB stream or the optical flow stream, and determining whether a second-class-specific temporal proposal for a corresponding time interval from the other stream exists. For example a first temporal proposal for a particular time interval can be selected from the RGB stream, and the sparse temporal pooling network can determine whether a second temporal proposal from the optical flow stream for a corresponding time interval exists. If not, the temporal location of the target action can be detected at the particular time interval based at least in part first class-specific temporal proposal. If a second-class-specific temporal proposal for the corresponding time interval from the other stream does exist, the sparse temporal pooling network can determine the temporal location of the target action in the video by comparing the first class-specific temporal proposal and the second-class-specific temporal proposal. 
     For example, a respective score for the target action can be determined for each of the first class-specific temporal proposal and the second-class-specific temporal proposal based at least in part on a weighted average temporal class activation map of all one-dimensional connected components within the respective class-specific temporal proposal. The temporal location of the target action can then be determined based at least in part on the respective scores. For example, the respective scores can be ranked, and the temporal proposal with the higher score can be used to determine the temporal location of the target action. In some implementations, the respective scores for the target classification can be determined based at least in part on a modality parameter. For example, the modality parameter can control the magnitude of the RGB weighted temporal class activation map and the flow weighted temporal class activation map. 
     In some implementations, determining the temporal location of the target action in the video can include performing non-maximum suppression among class-specific temporal proposals of a target classification to remove overlapped detections. 
     In some implementations, one or more relevant target action classes can first be determined by the sparse temporal pooling network by analyzing the video to determine one or more video-level classification scores. For example, the one or more video-level classification scores can determine which relevant target actions are present in the video, and target action localization can be performed on the relevant target actions. 
     In some implementations, the weakly supervised temporal action localization models of a sparse temporal pooling network can be trained using a loss function comprising a classification loss and a sparsity loss. For example, a classification loss can be determined based at least in part on a comparison of a video level classification score and a groundtruth classification. For example, multi-label cross-entropy loss between a groundtruth video-level action classification and a predicted score for an action classification can be performed. In some implementations, the sparsity loss can be determined based at least in part on determining a L1 (also referred to as l 1 ) norm of an attention weight parameter. 
       FIG.  11    depicts a flow chart diagram of an example method to train a weakly supervised temporal action localization model according to example aspects of the present disclosure. Although  FIG.  11    depicts steps performed in a particular order for purposes of illustration and discussion, the methods of the present disclosure are not limited to the particularly illustrated order or arrangement. The various steps of the method  1100  can be omitted, rearranged, combined, and/or adapted in various ways without deviating from the scope of the present disclosure. Method  1100  can be implemented by a computing system, such as one or more computing devices, as disclosed herein. 
     At  1102 , the method  1100  can include inputting an untrimmed video into a weakly supervised temporal action localization model. 
     At  1104 , the method  1100  can include analyzing the untrimmed video by the weakly supervised temporal action localization model to determine a predicted score for an action classification. 
     For example, in some implementations, analyzing the untrimmed video can include sampling a plurality of segments from the untrimmed video. In some implementations, the plurality of segments can be uniformly sized. The plurality of segments can then each be analyzed with one or more pretrained convolutional neural networks to determine a respective feature representation. 
     In some implementations, each respective feature representation can then be input into an attention module to determine a respective attention weight. For example, in some implementations, the attention module can include two fully connected layers, a rectified linear unit layer between the two fully connected layers, and a sigmoid layer. Each feature representation can be input into the attention module, and the respective attention weight can be received from the sigmoid layer. 
     In some implementations, an attention weighted temporal average pooling can be determined based at least in part on the attention weights. For example, the attention weighted temporal average pooling can be a weighted sum of the feature representations multiplied by the respective attention weights. 
     In some implementations, the predicted score for the action classification can be determined based at least in part on the attention weighted temporal average pooling. For example, the attention weighted temporal average pooling can be input into a weighting parameter fully connected layer, and the output of the weighting parameter fully connected layer can be input into a sigmoid layer. The predicted score for the action classification can then be received from the sigmoid layer. 
     At  1106 , the method  1100  can include determining a loss function comprising a sparsity loss and a classification loss based at least in part on the predicted score. For example, in some implementations, the classification loss can be determined based at least in part on a comparison of the predicted score for the action classification and a ground truth video-level action classification. For example, the classification loss can be determined using a multi-label cross-entropy loss between the groundtruth video-label and the predicted score for the action classification. In some implementations, the sparsity loss can be determined based at least in part on a L1 norm (also referred to as l 1 ) of one or more attention weights received from the attention module. 
     At  1108 , the method  1100  can include training the weakly supervised temporal action localization model based at least in part on the loss function. 
     For example, in some implementations, a weighting parameter fully connected layer of the weakly supervised temporal action localization model can be trained based at least in part on the classification loss. In some implementations, the attention module can be trained based at least in part on the sparsity loss. For example, the two fully connected layers of an attention module can be trained based at least in part on the sparsity loss. 
     ADDITIONAL DISCLOSURE 
     The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel. 
     While the present subject matter has been described in detail with respect to various specific example aspects thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such aspects. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and/or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the present disclosure cover such alterations, variations, and equivalents.