Temporal Gaussian Mixture Layer for Videos

1 INTRODUCTION . Activity videos are spatio-temporal data : they are image frames with a specific width/height ( XY ) concatenated along time axis ( T ) . Recognition from such videos requires capturing both spatial and temporal information in the videos , desirably using learned convolutional kernels . Temporal convolution is particularly beneficial in activity ‘ detection ’ tasks , which require making activity decisions at every frame given a continuous video ( Sigurdsson et al. , 2016b ; Yeung et al. , 2015 ) . Previous methods investigated using 3-D XYT convolutional filters ( Tran et al. , 2014 ; Carreira & Zisserman , 2017 ) as well as the models with 2-D XY conv . layers followed by 1-D temporal conv . ( Tran et al. , 2018 ) , pooling or attention layers ( Piergiovanni et al. , 2017 ) . Understanding complex multi-activity videos requires capturing information in long-term time intervals . Different frames contain different information , and the model needs to learn to take advantage of as many frames as possible , while abstracting them efficiently . Previous attempts of simply pooling representations over time or learning temporal conv . filters with a small number of frames ( e.g. , 16 or 64 ) was thus often insufficient to fully consider rich long-term temporal context . Simultaneously , bruteforcely increasing the temporal filter length ( to look at more frames ) results more learnable parameters , requiring more training data , which can be expensive when activities are rare . In this paper , we introduce a new convolutional layer named the Temporal Gaussian Mixture ( TGM ) layer , and present how it can be used to efficiently capture longer-term temporal information in activity videos . Our temporal Gaussian mixture layer is a temporal convolutional layer , whose filters/kernels are controlled by a set of ( temporal ) Gaussian distribution parameters . Each of our temporal Gaussian distributions specify ( temporally ) ‘ where ’ the model should look , and our Gaussian mixture layer combines them as multiple convolutional filters to be applied on top of temporallycontinuous representations . This layer allows the video representation at each time step to be constructed while focusing on different neighboring temporal regions , instead of only focusing on its local segment . It is a convolutional layer governed by a much smaller set of parameters ( i.e. , locations/variances of the Gaussians as well as their mixture weights ) that are fully differentiable . The motivation behind our temporal Gaussian mixture layer is to learn the temporal structure of an activity as a composition of temporal Gaussian regions/attentions . Such structure allows the model to obtain a compact spatio-temporal representation abstracting each ( long-term ) time interval , using multiple temporal conv . layers with far fewer parameters . It is also related to the previous temporal attention works ( Piergiovanni et al. , 2017 ) , but our model is designed to be fully convolutional to handle continuous data and it learns more compositional structures with multiple layers . We present video-CNN models using our TGM layers for activity detection in continuous videos . Our model stacks TGM layers on top of several state-of-the-art CNNs such as I3D ( Carreira & Zisserman , 2017 ) . This enables our model to capture longer-term temporal information than what we use as base CNNs , compositionally modeling temporal structure with multiple TGM layers . Our model was evaluated on multiple public datasets including MultiTHUMOS and Charades , and was able to outperform the best previous activity detection CNNs by a meaningful margin . 2 RELATED WORKS . Learning video representations for human activity recognition has been successful . CNN methods allow end-to-end learning of video features and representations optimized for the training data , performing superior to traditional works ( Aggarwal & Ryoo , 2011 ) for video understanding . Two-stream CNN models take a single RGB frame and a small number of optical flow frames as inputs to capture both motion and appearance information in videos ( Simonyan & Zisserman , 2014 ; Feichtenhofer et al. , 2016 ) . Models learning 3-D spatio-temporal ( XYT ) convolutional filters were designed and applied to many activity recognition tasks as well ( Tran et al. , 2014 ; Carreira & Zisserman , 2017 ; Tran et al. , 2017 ; Hara et al. , 2017 ) . Large scale datasets for activity detection , such as THUMOS ( Jiang et al. , 2014 ) , ActivityNet ( Heilbron et al. , 2015 ) , Kinetics ( Kay et al. , 2017 ) , and Charades ( Sigurdsson et al. , 2016b ) provided these approach the necessary training data to learn the models . Such 3-D XYT CNNs were also used to capture spatio-temporal information for activity detection ( Xu et al. , 2017 ; Shou et al. , 2016 ; 2017 ; Zhao et al. , 2017 ) . However , all these CNNs were limited to the consideration of a fixed local video segment ( e.g. , 16 frames in ( Tran et al. , 2014 ) and 64-99 frames in ( Carreira & Zisserman , 2017 ) ) when making activity decisions . Some works studied combining representations over longer-term temporal intervals ( Karpathy et al. , 2014 ; Ng et al. , 2015 ; Varol et al. , 2017 ) , but it was generally done with a temporal pooling of local representations or ( spatio- ) temporal convolutions with a bit larger fixed intervals . Recurrent neural networks ( RNNs ) have also been used to model activity transitions between frames ( Yeung et al. , 2015 ; 2016 ; Escorcia et al. , 2016 ) , but they were strictly sequential and had limitations in maintaining temporal information over a longer temporal duration , particularly for videos with multiple complex activities . Recently , CNN models using temporal attention for activity videos ( Piergiovanni et al. , 2017 ; Piergiovanni & Ryoo , 2018b ) were studied as well . However , a fully convolutional model to analyze continuous videos while efficiently representing information in long term intervals has been lacking . Our layer is different from the previous standard ( spatio- ) temporal convolutional layers in that it relies on significantly fewer parameters by forcing filter shapes to be Gaussian compositions . Our temporal layer is also different from previous Gaussian Mixture Model layers ( Variani et al. , 2015 ) in that our layer is convolutional while they are not . 3 APPROACH . In this section , we introduce a new convolutional layer named the Temporal Gaussian Mixture ( TGM ) layer , and present how it can be used for activity recognition . Our Temporal Gaussian Mixture layer is a temporal convolutional layer to be applied on top of a sequence of representations ( usually from frame-level or segment-level CNNs ) , whose filters/kernels are controlled by a set of ( temporal ) Gaussian distribution parameters . The motivation is to make each temporal Gaussian distribution specify ( temporally ) ‘ where to look ’ with respect to the activity center , and represent the activity as a collection/mixture of such temporal Gaussians convolved with video features . Our layer is fully differentiable and trainable using standard backpropagation . Our TGM layer can be interpreted as a a form of 1-D convolution where the filters are determined by a mixture of Gaussians . However , our TGM layer differs from the standard temporal convolutional layers of learning 1-D ( time ) or 2-D ( channel-by-time ) filters in the following aspects : 1 . Our temporal Gaussian mixture layer handles multiple 3-D tensors internally to preserve channels from the frame-level CNN by adding a new temporal channel axis . Its input is 3-D ( channel-by-channel-by-time ) , where one channel dimension is inherited from the frame-level CNN and this dimension size remains unchanged . 2 . Instead of learning temporal convolution filters of any arbitrary values , our filter is forced to have the form of a temporal Gaussian mixture shared across all frame-level channels . This allows the layer to rely on significantly fewer number of ( fully differentiable ) parameters , while capturing the concept of temporal structure/attention . 3.1 TEMPORAL GAUSSIAN MIXTURE LAYER . Our temporal Gaussian mixture layer takes a 3-D input with the dimensionality of Cin × D × T , where Cin is the number of input channels , D is the dimensionality of the representations from frame-level ( or segment-level ) CNNs , and T is the time . Given such input , the TGM layer convolves it with Cout number of 1 × L filters/kernels , generating a Cout ×D × T -dim representation as an output . L is the temporal length of the temporal Gaussian mixture filter . D is usually 1K or 4K and T is the number of time steps ( frames ) in each video ( i.e. , it varies per video ) . Cout is the number of different mixtures , corresponding to the number of output channels in standard convolution . Our layer is composed of a set of M Gaussians . Each Gaussian has 2 parameters : a center µ̂ and a width σ̂ . Each layer has additional hyper-parameters : L , the temporal duration and M , the number of Gaussians to learn . We force the learned center to be between −L2 and L 2 and σ to be positive : µ = ( L− 1 ) · tanh ( µ̂+ 1 ) 2 , σ2 = exp ( σ̂ ) . ( 1 ) We use the above µ and σ to construct the temporal Gaussian kernels . This acts as a strong sparsity constraint on the convolutional kernel as well as a drastic reduction of the number of learnable parameters . We construct a temporal Gaussian mixture convolutional kernel as : K̂m , l = 1 Z exp− ( l − µm ) 2 2σ2m ( 2 ) where Z is a normalization constant such that ∑L l K̂m , l = 1 , resulting in K̂ being anM×Lmatrix . Instead of making the model learn a separate set of Gaussian distributions per activity class , we take the approach of maintaining multiple Gaussian distributions shared across classes and obtain a Gaussian ‘ mixture ’ filter by learning soft-attention weights . We learn a set of soft-attention weights per output channel i , ω ∈ RCout×M . We create the soft-attention weights by applying the softmax function over the M Gaussians , enforcing each input channel weights sum to 1. ai , m = expωi , m∑ j expωi , j ( 3 ) Based on temporal Gaussian distributions K̂i and attention weights ai , m , the temporal convolution filters our TGM layer is computed as : Ki = ∑ m ai , mK̂i . ( 4 ) This provides us convolutional filters having the form of a mixture of temporal Gaussians , controlled based on 2 ·M + Cin · Cout ·M parameters ( instead of learning D2 · L parameters without any constraint , as in standard temporal convolution where C < < D ) . An overview of this process is shown in Fig . 1 . 3.1.1 SINGLE TGM LAYER - DIRECT PER-CLASS ACTIVITY MODELING . The representation we obtain by applying our base CNNs to each frame ( or local segment ) has the dimensionality of D , and stacking them along time axis provides us the representation with 1×D×T -dim . That is , in the case of using only one TGM layer to capture activity representations , our Cin is fixed to 1 and Cout is fixed to be the number of activity classes . This is the simplest case of our model , attaching one TGM layer on top of the 1×D × T representation . Our convolutional kernel , K , has a learned Gaussian mixture for each activity class . Let the video features v be a D × T matrix . Each Ki is a 2-D convolutional filter with a size of 1 × L , and convolving this with v provides us a representation S with Cout number of D × T responses since Cin is 1 in this case . This per-class representation can then be used as input to a fully-connected layer for activity classification . For i ∈ { 1 , 2 , . . . , Cout } : si = v ∗Ki , S = [ s1 , s2 , . . . , sCout ] ( 5 ) Fig . 7 in the appendix visually illustrates how each TGM filter is convolved with the input ( Fig . 7d ) , compared to the standard 1-D convolution ( Fig . 7a ) or other forms of the temporal layers ( Fig . 7b-c ) .

Summary: This paper proposes a temporal convolution layer called Temporal Gaussian Mixture (TGM) for video classification tasks. Usually, video classification models can be decomposed into a feature extraction step, in which frames are processed separately or in local groups, and a classification step where labels are assigned considering the full video features/longer range dependencies. Typical choices for the classification step are Recurrent Neural Networks (RNNs) and variations, pooling layers and 1D temporal convolutions. TGMs are 1D temporal convolutions with weights defined by taking samples from a mixture of Gaussians. This allows to define large convolutional kernels with a reduced number of parameters -the mean and std of the gaussians- at the cost of having reduced expressiveness. The authors experiment replacing 1D temporal convolutions with TGMs in standard video classifications models and datasets and report state-of-the-art results.