Title: TEMPORAL GAUSSIAN MIXTURE LAYER FOR VIDEOS Anonymous

Abstract: We introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer and present how it can be used to efficiently capture longer-term temporal information in continuous activity videos. The TGM layer is a temporal convolutional layer governed by a much smaller set of parameters (e.g., location/variance of Gaussians) that are fully differentiable. We present our fully convolutional video models with multiple TGM layers for activity detection. The experiments on multiple datasets including Charades and MultiTHUMOS confirm the effectiveness of TGM layers, outperforming the state-of-the-arts.

1. INTRODUCTION:

Activity videos are spatio-temporal data: they are image frames with a specific width/height (XY) concatenated along time axis (T). Recognition from such videos requires capturing both spatial and temporal information in the videos, desirably using learned convolutional kernels. Temporal convolution is particularly beneficial in activity 'detection' tasks, which require making activity decisions at every frame given a continuous video (Sigurdsson et al., 2016b; Yeung et al., 2015) . Previous methods investigated using 3-D XYT convolutional filters (Tran et al., 2014; Carreira & Zisserman, 2017) as well as the models with 2-D XY conv. layers followed by 1-D temporal conv. (Tran et al., 2018) , pooling or attention layers (Piergiovanni et al., 2017) .

Understanding complex multi-activity videos requires capturing information in long-term time intervals. Different frames contain different information, and the model needs to learn to take advantage of as many frames as possible, while abstracting them efficiently. Previous attempts of simply pooling representations over time or learning temporal conv. filters with a small number of frames (e.g., 16 or 64) was thus often insufficient to fully consider rich long-term temporal context. Simultaneously, bruteforcely increasing the temporal filter length (to look at more frames) results more learnable parameters, requiring more training data, which can be expensive when activities are rare.

In this paper, we introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer, and present how it can be used to efficiently capture longer-term temporal information in activity videos. Our temporal Gaussian mixture layer is a temporal convolutional layer, whose filters/kernels are controlled by a set of (temporal) Gaussian distribution parameters. Each of our temporal Gaussian distributions specify (temporally) 'where' the model should look, and our Gaussian mixture layer combines them as multiple convolutional filters to be applied on top of temporallycontinuous representations. This layer allows the video representation at each time step to be constructed while focusing on different neighboring temporal regions, instead of only focusing on its local segment. It is a convolutional layer governed by a much smaller set of parameters (i.e., locations/variances of the Gaussians as well as their mixture weights) that are fully differentiable.

The motivation behind our temporal Gaussian mixture layer is to learn the temporal structure of an activity as a composition of temporal Gaussian regions/attentions. Such structure allows the model to obtain a compact spatio-temporal representation abstracting each (long-term) time interval, using multiple temporal conv. layers with far fewer parameters. It is also related to the previous temporal attention works (Piergiovanni et al., 2017) , but our model is designed to be fully convolutional to handle continuous data and it learns more compositional structures with multiple layers.

We present video-CNN models using our TGM layers for activity detection in continuous videos. Our model stacks TGM layers on top of several state-of-the-art CNNs such as I3D (Carreira & Zisserman, 2017) . This enables our model to capture longer-term temporal information than what we use as base CNNs, compositionally modeling temporal structure with multiple TGM layers. Our model was evaluated on multiple public datasets including MultiTHUMOS and Charades, and was able to outperform the best previous activity detection CNNs by a meaningful margin.

2. RELATED WORKS:

Learning video representations for human activity recognition has been successful. CNN methods allow end-to-end learning of video features and representations optimized for the training data, performing superior to traditional works (Aggarwal & Ryoo, 2011) for video understanding. Two-stream CNN models take a single RGB frame and a small number of optical flow frames as inputs to capture both motion and appearance information in videos (Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016) . Models learning 3-D spatio-temporal (XYT) convolutional filters were designed and applied to many activity recognition tasks as well (Tran et al., 2014; Carreira & Zisserman, 2017; Tran et al., 2017; Hara et al., 2017) . Large scale datasets for activity detection, such as THUMOS (Jiang et al., 2014 ), ActivityNet (Heilbron et al., 2015) , Kinetics (Kay et al., 2017), and Charades (Sigurdsson et al., 2016b) provided these approach the necessary training data to learn the models. Such 3-D XYT CNNs were also used to capture spatio-temporal information for activity detection (Xu et al., 2017; Shou et al., 2016; 2017; Zhao et al., 2017) . However, all these CNNs were limited to the consideration of a fixed local video segment (e.g., 16 frames in (Tran et al., 2014) and 64-99 frames in (Carreira & Zisserman, 2017)) when making activity decisions. Some works studied combining representations over longer-term temporal intervals (Karpathy et al., 2014; Ng et al., 2015; Varol et al., 2017) , but it was generally done with a temporal pooling of local representations or (spatio-)temporal convolutions with a bit larger fixed intervals. Recurrent neural networks (RNNs) have also been used to model activity transitions between frames (Yeung et al., 2015; 2016; Escorcia et al., 2016) , but they were strictly sequential and had limitations in maintaining temporal information over a longer temporal duration, particularly for videos with multiple complex activities. Recently, CNN models using temporal attention for activity videos (Piergiovanni et al., 2017; Piergiovanni & Ryoo, 2018b) were studied as well. However, a fully convolutional model to analyze continuous videos while efficiently representing information in long term intervals has been lacking.

Our layer is different from the previous standard (spatio-)temporal convolutional layers in that it relies on significantly fewer parameters by forcing filter shapes to be Gaussian compositions. Our temporal layer is also different from previous Gaussian Mixture Model layers (Variani et al., 2015) in that our layer is convolutional while they are not.

3. APPROACH:

In this section, we introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer, and present how it can be used for activity recognition. Our Temporal Gaussian Mixture layer is a temporal convolutional layer to be applied on top of a sequence of representations (usually from frame-level or segment-level CNNs), whose filters/kernels are controlled by a set of (temporal) Gaussian distribution parameters. The motivation is to make each temporal Gaussian distribution specify (temporally) 'where to look' with respect to the activity center, and represent the activity as a collection/mixture of such temporal Gaussians convolved with video features. Our layer is fully differentiable and trainable using standard backpropagation.

Our TGM layer can be interpreted as a a form of 1-D convolution where the filters are determined by a mixture of Gaussians. However, our TGM layer differs from the standard temporal convolutional layers of learning 1-D (time) or 2-D (channel-by-time) filters in the following aspects: 1. Our temporal Gaussian mixture layer handles multiple 3-D tensors internally to preserve channels from the frame-level CNN by adding a new temporal channel axis. Its input is 3-D (channel-by-channel-by-time), where one channel dimension is inherited from the frame-level CNN and this dimension size remains unchanged. 2. Instead of learning temporal convolution filters of any arbitrary values, our filter is forced to have the form of a temporal Gaussian mixture shared across all frame-level channels. This allows the layer to rely on significantly fewer number of (fully differentiable) parameters, while capturing the concept of temporal structure/attention. Our layer is composed of a set of M Gaussians. Each Gaussian has 2 parameters: a center μ and a width σ. Each layer has additional hyper-parameters: L, the temporal duration and M , the number of Gaussians to learn. We force the learned center to be between -L 2 and L 2 and σ to be positive:

EQUATION

We use the above µ and σ to construct the temporal Gaussian kernels. This acts as a strong sparsity constraint on the convolutional kernel as well as a drastic reduction of the number of learnable parameters. We construct a temporal Gaussian mixture convolutional kernel as:

EQUATION

)

where Z is a normalization constant such that L l Km,l = 1, resulting in K being an M ×L matrix. Instead of making the model learn a separate set of Gaussian distributions per activity class, we take the approach of maintaining multiple Gaussian distributions shared across classes and obtain a Gaussian 'mixture' filter by learning soft-attention weights. We learn a set of soft-attention weights per output channel i, ω ∈ R Cout×M . We create the soft-attention weights by applying the softmax function over the M Gaussians, enforcing each input channel weights sum to 1.

EQUATION

Based on temporal Gaussian distributions Ki and attention weights a i,m , the temporal convolution filters our TGM layer is computed as: 

EQUATION

1 × D × T -dim.

That is, in the case of using only one TGM layer to capture activity representations, our C in is fixed to 1 and C out is fixed to be the number of activity classes. This is the simplest case of our model, attaching one TGM layer on top of the 1 × D × T representation.

Our convolutional kernel, K, has a learned Gaussian mixture for each activity class. Let the video features v be a D × T matrix. Each K i is a 2-D convolutional filter with a size of 1 × L, and convolving this with v provides us a representation S with C out number of D × T responses since C in is 1 in this case. This per-class representation can then be used as input to a fully-connected layer for activity classification. For i ∈ {1, 2, . . . , C out }:

EQUATION

Fig. 7 in the appendix visually illustrates how each TGM filter is convolved with the input (Fig. 7d ), compared to the standard 1-D convolution (Fig. 7a ) or other forms of the temporal layers (Fig. 7b-c ).

3.1.2. MULTIPLE TGM LAYERS -GROUPED CONVOLUTION:

We generalize the above formulation to allow the TGM layers to be sequentially applied. The idea is to enable our model to capture more complex, nonlinear temporal structure by having multiple levels of temporal layers. In this case, the input for each layer is

C in × D × T dimensional (instead of 1 × D × T )

, where the input channels are the number of output channels from the previous layer.

Our kernels at each layer, K i , are parameterized and learned as before.

By using grouped convolution with the number of groups set to C in , we can efficiently separate the input into per-channel values and convolve each of them with the designated K i kernel, as shown in Fig. 2 . That is, we learn a filter K i per channel by setting

EQUATION

Here, f is a C in × D × T tensor, where D is the dimensionality of the feature and T is the number of frames. The result of the per-channel convolution, s i , is a D × T representation. We concatenate these representations along the channel axis, resulting in S, a C out × D × T representation. As this convolution results in the same output shape, we can stack these layers. Each layer is able to capture increasing temporal resolution, allowing the model to capture levels of abstractions.

3.1.3. MULTIPLE TGM LAYERS -CHANNEL COMBINATION:

In the above subsection, we introduced an approach of stacking multiple TGM layers to model a hierarchical composition of temporal representations. However, in the grouped convolution case, each output channel of the layer is solely dependent on its corresponding input channel. That is, each kernel only considers information from a single output channel of the previous layer.

Therefore, we further generalize our TGM layer so that the layer combines representations from multiple input channels for each output channel while using the learned temporal kernels. We learn a set of convolutional kernels K ∈ R Cout×Cin×L (i.e., we learn

C out • C in Gaussian mixtures). Given f which is the C in × D × T representation, for each output channel i ∈ [1, C out ] and each input channel j ∈ [1, C in ]

pair, we convolve the associated filters with the input.

EQUATION

where each G i,j is a D × T -dim representation.

We then learn a 1x1 convolution followed by a ReLU activation function for each i ∈ [1, C out ], which we call w i , that maps from C in channels to 1 channel. The 1x1 convolution learns to combine the channels from the previous layer. By design, the TGM kernel is positive and sums to 1. Adding the unconstrained 1x1 convolution adds non-linearity (using the ReLU activation function) to our layer and only adds C out • C in parameters.

EQUATION

We then stack the s i representations along the channel axis to produce S, the C out × D × T -dim representation. This process is illustrated in Fig. 3 . This method generalizes our approach to allow the layer to take input of C in × D × T and produce output of C out × D × T . These layers can easily be stacked to learn a hierarchical representation.

3.2. VIDEO CNN MODELS WITH TGM LAYERS:

Our goal is to do activity detection which we define as making a per-frame (or per-segment) classification. Given a video, at each time step t, we want to make the model decide which activity the frame corresponds to (including no-activity). As a baseline, we train a fully-connected layer that classifies each per-frame D-dimensional vector, v t . As multiple activities can occur at the same time, or no activities at all, we treat this as a mutli-label classification task. We minimize binary cross entropy:

L(v) = t,c z t,c log(p(c|v t )) + (1 -z t,c ) log(1 -p(c|v t )) (9)

where z t,c is the ground truth label, 1 if activity c is occurring at time t and p(c|v t ) is the output of our model for class c at time t. Fig. 4 shows an example CNN. 

4. EXPERIMENTS:



4.1. IMPLEMENTATION AND BASELINES:

Implementation We used I3D (Carreira & Zisserman, 2017) and the two-stream version of In-ceptionV3 (Szegedy et al., 2016) pretrained on Imagenet and Kinetics as our base per-frame CNNs. Our default L setting used for the TGM layers as well as the other baselines was as follows: when using I3D segment features (collected at 3fps), the 1 layer models used L = 15 and the 3 layer models used L = 5. When using InceptionV3 frame feature (collected at 8fps), the 1 layer models used L = 30 and the 3 layer models used L = 10. These layers were attached on top of the base CNN, as described in Subsection 3.2. Please check the appendix for implementation and training details and results on other datasets.

Baselines In order to confirm the advantages of our TGM layers, particularly against previous temporal models, we implemented several baselines. The first is (i) a standard per-frame classifier in which the prediction at each time-step only depends on a single feature vector with no contextual temporal information. We also used (ii) LSTMs on top of per-frame representations, which were popularly used to capture temporal information (Donahue et al., 2015) . We train a bi-directional LSTM with 512 hidden units to make per-frame predictions. We also tried (iii) the fixed pyramid temporal max-pooling of level 3 (Ryoo et al., 2015) . Finally, we compare our model against (iv) the model with standard temporal convolutional layers (i.e., 1-D convolution with a D × L kernel) on top of per-frame representations. This is similar to the temporal conv. used in (Tran et al., 2018) . Temporal lengths (i.e., L) of the 1-D conv. filters and the pooling windows were set to be identical to the TGM filters. That is, they capture the same temporal duration as TGMs. In all our experiments, we follow the standard evaluation setting of computing per-frame mean average precision (mAP) and report those values. We also compare to different versions of the TGM layer, (v) with a learned mixture of random temporal filters and (vi) with a learned mixture of fixed Gaussians.

In addition, we also tried the approach of combining our TGM layers with the recent super-event representations (Piergiovanni & Ryoo, 2018b) . We concatenated the learned super-event representation with our representations from TGM layers.

4.2. MULTITHUMOS:

Dataset MultiTHUMOS (Yeung et al., 2015) is an extended version of the THUMOS (Jiang et al., 2014) dataset that densely annotates the continuous videos. The dataset consists of 65 different classes, compared to 20 in THUMOS, and contains on average 10.5 activities per video and 1.5 labels per frame and up to 25 activity instances in each video. This is in contrast to many other activity detection dataset such as ActivityNet (Heilbron et al., 2015) , which only has on average ∼1 activity per video. MultiTHUMOS consists of YouTube videos of various sport activities such as basketball games, volleyball games, weight lifting, and track and field.

We followed the standard MultiTHUMOS evaluation setting of measuring mAP based on per-frame annotations. There are 1010 validation videos and 1574 test videos. We used these continuous validation videos for the training of our models. We did not need to take advantage of the separate training set with segmented videos; even without them, we outperformed the state-of-the-arts.

Results:

We compared baselines as well as multiple different versions of our architectures, shown in Table 1 . The model with our TGM layers consistently outperformed baseline I3D (or Incep-tionV3) while using the same per-segment representations. Learning 3 TGM layers further improved the performances. On the other hand, we found that stacking multiple standard temporal convolutional layers does not improve performance, often performing worse than the baseline. While a single standard temporal conv. layer improves over the baseline, having multiple of them significantly increases the number of parameters to learn (Table 2 ) and we suspect that this was causing the overfitting with the limited amount of samples in the dataset. In Table 3 , we compare the results of using a LSTM or temporal conv. with a similar number of parameters. This was done by making their temporal conv. filters to share values across multiple channels. These models result in nearly random performance, as they were not designed to cope with a small number of parameters. We also show results with a mixture of random (fixed) temporal filters and with a mixture of fixed Gaussians. These results confirm that (i) modeling the temporal structure as a learned Gaussian mixture is beneficial and that (ii) further learning the Gaussian distribution parameters is important.

Table 1 : Comparison of various architectures on MultiTHUMOS using both I3D per-segment and InceptionV3 per-frame features. We found that TGM layers with 1x1 convolution channel combination performed the best. Results are in mAP %. Note that we use the same filter length for "Temporal Conv" and "TGM" models, as described in Section 4.1. Learning multiple TGM layers with channel combination outperforms the grouped convolution version of TGM and all the baselines. We also experimented with a version using soft-attention weights to combine the TGM layer channels, in addition to our method (Fig. 3 ) of using 1x1 convolution followed by a ReLU (to gain non-linearity). We found that the 1x1 convolution performed better. We tested various number of Gaussian mixtures (i.e., output channels) and found that using 80 for the first and second layer and using 65 (i.e., number of classes) for the final layer performs best.

Table 4 compares our model using TGM layers with multiple previous state-of-the-art approaches and baselines such as LSTM. Our approach meaningfully outperforms all previous approaches. Importantly, we are comparing our approach with different methods of capturing temporal information such as LSTMs and fixed temporal pyramid pooling while making them use the exactly same per-frame representations. We found that while all these methods capture some temporal information, the TGM layers provide the best performance. Further, combining the super-event representation (Piergiovanni & Ryoo, 2018b) with our TGM feature also benefited detection, confirming that our TGMs and super-events capture different aspects of the activity videos. In Fig. 5 , we show an example of the various models predictions on a basketball video. We outperform the previous state-of-the-art performance (mAP) by 10% (36.4 vs. 46.4).

4.3. CHARADES:

Dataset Charades (Sigurdsson et al., 2016b) is a large scale dataset with 9848 videos across 157 activity classes. These videos were recorded in home environments of the participants based on provided scripts. Each video contains on an average of 6.8 activity instances, and there are often complex activities co-occurring. The activities were mainly performed at home. For example, some activity classes are 'preparing a meal', 'eating', 'sitting', 'cleaning', etc.

In our experiments, we follow the original Charades detection setting (i.e., Charades v1 localize evaluation), which is the setting used in many previous approaches (Sigurdsson et al., 2016a Table 4 : Performances of the state-of-the-art methods and our approach on MultiTHUMOS. Our approach meaningfully outperforms all previous results. mAP Two-stream (Yeung et al., 2015) 27.6 Two-stream + LSTM (Yeung et al., 2015) 28.1 Multi-LSTM (Yeung et al., 2015) 29.6 Predictive-corrective (Dave et al., 2017) 29.7 I3D baseline 29.7 I3D + LSTM 29.9 I3D + temporal pyramid 31.2 I3D + super-events (Piergiovanni & Ryoo, 2018b) 36.4 I3D + our TGMs 44.3 I3D + super-events (Piergiovanni & Ryoo, 2018b ) + our TGMs 46.4 et al., 2017; Piergiovanni & Ryoo, 2018b) . This is the original setting more challenging than the Charades Challenge 2017 setting (whose evaluation server was no longer approving new account access), in the aspect that it uses less amount of training videos.

We compare our results with the state-of-the-arts in Table 5 . To our knowledge, our method is obtaining the best known performance in the original localization setting of the Charades dataset. Notably, it is performing better than I3D that obtained the best competition performance, while using the same feature. Our method also outperforms standard temporal convolution, LSTMs, and fixed pyramid pooling, as well as the use of latent super-events. When setting L = 30 and using 3 TGM layers, our model is able to capture around 800 frames (about ±15 seconds from each frame) of temporal information, significantly more than previous works (e.g., I3D only captures ±2 seconds).

Table 5 : Per-frame mAP on Charades, evaluated with the 'Charades v1 localize' setting. I3D models are two-stream, using both RGB and optical flow inputs. mAP Predictive-corrective (Dave et al., 2017) 8.9 Two-stream (Sigurdsson et al., 2016a) 8.94 Two-stream+LSTM (Sigurdsson et al., 2016a) 9.6 R-C3D (Xu et al., 2017) 12.7 Sigurdsson et al. (Sigurdsson et al., 2016a) 12.8 I3D baseline 17.2 I3D + 3 temporal conv. layers (L = 5) 17.5 I3D + 3 temporal conv. layers (L = 30)

12.5 I3D + LSTM 18.1 I3D + fixed temporal pyramid 18.2 I3D + super-events (Piergiovanni & Ryoo, 2018b) 

A IMPLEMENTATION DETAILS:

As our base per-segment CNN, we use the I3D (Carreira & Zisserman, 2017) network pretrained on the ImageNet and Kinetics (Kay et al., 2017) datasets. I3D obtained state-of-the-art results on segmented video tasks, and this allows us to obtain reliable v t . We also use two-stream version of InceptionV3 (Szegedy et al., 2016) pretrained on Imagenet and Kinetics as our base per-frame CNN, and compared them. We chose InceptionV3 as it is deeper than previous two-stream CNNs such as (Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016) . We extracted frames from the videos at 25 fps, computed TVL1 (Zach et al., 2007) optical flow, clipped to [-20, 20] . For InceptionV3, we computed features for every 3 frames (8 fps). For I3D, every frame was used as the input. I3D has a temporal stride of 8, resulting in 3 features per second (3 fps).

We implemented our TGM layers as well as other baseline layers in PyTorch. Our default setting was as follows: for 3-layer models, we set L = 10 for frame-based features (i.e., InceptionV3) and L = 5 for segment-based features (i.e., I3D), as each segment already contains some temporal information. For 1-layer models, we set L = 30 for frame-based features and L = 15 for segmentbased features. We set M = 16 and C out = 80 and C out = 65 for the last TGM layer. We found these values to work well on a held out portion of the training set of MultiTHUMOS. In all models, we used one fully-connected layer at the end to make the per-frame or per-segment classification.

We trained our models using the Adam (Kingma & Ba, 2014) optimizer with the learning rate set to 0.01. We decayed the learning rate by a factor of 10 after every 10 training epochs. We trained our models for 50 epochs. We plan to make all our source code and trained models publicly available once the paper is published.

B HYPERPARAMETER EXPERIMENTS:

We conducted a set of experiments to compare the effects of the temporal duration, L, number of Gaussians, M , and the number of output channels, C out . For these experiments, we only used the one-stream version of I3D with RGB inputs.

Effect of L::

In Table 6 , we compare different values of L. For these experiments, we use M = 16 and C out = 16. We find that the 3-layer model with L = 5 performs the best. With I3D features, this allows the model to capture up to 8 seconds of information. The average activity in MultiTHUMOS is 3.3 seconds long and the maximum is 14.7 seconds long, and with this setting, the model is able to capture enough temporal context to perform well. Larger values of L capture too much temporal information, but due to the Gaussian structure, it does not drastically harm performance. Figure 6 shows that even with longer kernels, the Gaussians learn to focus mostly on the center of the interval and capture the rough duration of the activities. Thus, having too long intervals does not drastically harm performance, which is in contrast to the standard 1-D convolution. Note that for Charades, the temporal kernels are learned to capture much longer temporal duration, as the average activity in charades is 12.8 seconds and larger values of L perform better.

Figure 6 illustrates examples of the learned TGM kernels of various lengths. The figure shows that the kernels focus on short temporal intervals on MultiTHUMOS even if we make the filters longer, as the activities are an average of 3.3 seconds long. On Charades, the TGM kernels learn to capture much longer intervals, as the activities are an average of 12.8 seconds long. We believe that this suggests TGMs are learning to capture information from the important necessary intervals.

In Table 6 , we also report the results of using a standard 1-D conv. layer with different L values.

The number of parameters in our TGM layer is independent of L, however, with the standard 1-D conv. layer, the number of parameters increases as L increases. We find that increasing L with 1-D convolution helps for small values of L, but for L > 15, the performance drastically drops, while TGM layers only show a small decrease.

Effect of M : In Table 7 , we compare different values of M . For these experiments, we set L = 15 and C out = 16. We find that M = 16 performs best, suggesting that smaller values of M restrict the possible temporal kernels too much. We also observe that larger values of M performs slightly worse than M = 16 (but not much), likely because they introduce more parameters than needed. When M and L have similar values, it allows the model to learn a sufficient number of Gaussians and create a diverse range of temporal kernels. When M is larger than L, it results in learning a kernel similar to standard 1-D convolution.

Effect of C out : In Table 8 , we compare different values of C out . For these experiments, L = 15, we used 1-layer and M = 16. We find that C out performs best when set to 16 or larger on these datasets. Larger values of C out seem to capture redundant information, as it does not lower performance.

MultiTHUMOS L=15 MultiTHUMOS L=30:

Charades L=30 Charades L=15

Figure 6 : Illustration of several learned TGM kernels. On MultiTHUMOS, it learns to focus on shorter intervals to capture shorter events. On Charades, the Gaussians have a larger σ value, resulting in filters that attend to longer temporal durations. 

C COMPARISON OF DIFFERENT LAYER FORMS:

To confirm the various aspects of our design, we conducted experiments comparing different types of temporal convolution. In Fig. 7a we illustrate the standard 1-D convolution, taking D × T input and producing a C × T output, where D is the number of input channels and C is the number of output channels. In Fig. 7b , we illustrate the method of applying a Gaussian mixture kernel as 1-D convolution. Here, the Gaussian mixture kernel is shared by all D input channels and we learn a C number of such kernels. In Fig. 7c , we illustrate the approach of applying a Gaussian mixture kernel as 1-D convolution while learning D different Gaussian mixtures. This is very similar to the standard 1-D convolution, except that the filter values are constrained to have the shape of Gaussian mixtures.

Fig. 8 illustrates one more baseline. This is similar to our full TGM layer with the channelcombination described Fig. 3 . However, in this baseline, instead of learning Gaussian mixtures, we learn C in • C out number of 1 × L kernels. The kernel values are left unconstrained. While the TGM layer has 2

• M + C in • C out • M + C in • C out parameters, this layer has L • C in • C out • M + C in • C out ,

which is more than the TGM layer.

In Table 9 , we compare the results of the various above-mentioned layers on MultiTHUMOS using RGB I3D features. We find that the Fig. 7b method performs poorly, while the Fig. 7c method slightly outperforms the standard 1-D convolution. The Fig. 8 method is slightly better than the standard 1-D convolution, but performs worse than Fig. 7c . However, none of these layers perform as well as our TGM layer, confirming that both the design of learning Gaussian mixtures and maintaining temporal channel axis are important for activity detection.

Table 9 : Comparison of the different forms of temporal convolution on MultiTHUMOS using RGB I3D features. We set L = 15 and used 1 layer models for these experiments.

MultiTHUMOS:

Standard 1-D Convolution (Fig. 7a ) 32.5 The layer described in Fig. 7b 28.6 The layer described in Fig. 7c 33.2 The layer described in Fig. 8 32.8 Our TGM Layer 36.1 The MLB-YouTube dataset (Piergiovanni & Ryoo, 2018a ) consists of 20 baseball games from the 2017 MLB post-season available on YouTube. This dataset consists of over 42 hours of video. For these experiments, we used the continuous video setting which have 2,126 1-2 minute long clips. Each clip is densely annotated with the baseball activities that occur. There are 8 activity classes: pitch, strike, ball, swing, hit, foul, hit by pitch, and bunt. Examples of some of these classes are shown in Fig. 9 . Each continuous clip contains on average of 7.2 activities, giving a total of over 15,000 activity instances in the dataset.

(a) (b) (c) (d) (e)

What makes this dataset challenging is that the variation between classes is very small. In Activi-tyNet (Heilbron et al., 2015) , for example, the difference between swimming and brushing hair is drastic. The background, motion, and even size of the person in the video is different. However, in broadcast baseball videos, the difference between a ball and a strike, or a swing and a bunt, are small. All actions are recorded from the same camera angle as we can confirm from Fig. 9 .

D.1.2 RESULTS:

In Table 10 , we compare various approaches on this dataset. Our TGM layers improve over the baseline by ∼6% (40.1 vs. 34.2). Additionally, we compare to methods using the super-event representation (Piergiovanni & Ryoo, 2018b) , which previously achieved state-of-the-art performance on several activity detection datasets. On this dataset, our approach outperforms the super-event representation, and further the concatenation of our TGM representation with such super-event representation performs best by a significant margin (∼13% compared to the baseline). This suggests that TGMs and super-event capture different temporal information and are both useful to the detection task.

We further find that using multiple, standard temporal convolution layers leads to worse performance, likely due to overfitting from the large number of parameters. While using multiple TGM layers improves performance, confirming that the Gaussian structure and sparsity constraint benefits model learning. (Piergiovanni & Ryoo, 2018b) 9.8 I3D + 1 TGMs 11.2 I3D + 3 TGMs 14.5 I3D + 3 TGMs + super-events 14.9 D.2 AVA D.2.1 DATASET AVA (Gu et al., 2017 ) is a large-scale video dataset containing of 80 atomic action classes in 57k video clips. These clips are drawn from movies. Existing datasets, such as Charades, have very specific actions that depend on objects, such as holding a cup vs. holding a picture. In AVA, the actions are intentionally generic, such as sit, stand, hold, carry, etc. Further, the AVA dataset is annotated with both spatial and temporal locations of activities. Since we are interested in temporal activity detection, we follow the setting of Piergiovanni & Ryoo (2018b) and label each frame with the occurring activities while ignoring the spatial location. We evaluate performance following the same method as MultiTHUMOS, Charades and MLB-YouTube by measuring per-frame mAP.

D.2.2 RESULTS:

In Table 11 , we present the results of our model. We again find that temporal convolution and LSTMs provide some benefit over the baseline, but TGM layers further improve performance. Again, combining the TGM, which captures local temporal structure, with super-events which capture global temporal structure, provides the best performance by ∼ 7.4%.

:



Figures and Tables:

Figure 1: Example illustrating how our Temporal Gaussian Mixture layer is computed. Multiple (M ) temporal Gaussian distributions are learned, and they are combined with the learned soft attention weights to form the C temporal convolution filters. L is the temporal length of the filter.

Figure 2: Illustration of a TGM layer with grouped convolution. This layer learns a set of C Gaussian mixtures that are convolved with the input channels.

Figure 3: Illustration of a TGM layer with channel combination. The kernels are applied to each input channel, C in , and a 1x1 convolution is applied to combine the C in input channels for each output channel, C out .

Figure 4: An overview of an example video CNN model with two TGM layers. It is able to handle videos with any length, because of its fully convolutional design.

Figure 5: Illustration of the temporal regions classified as various basketball activities from a basketball game video in MultiTHUMOS. Our TGM layers greatly improve performance.

(L = 5) + super-events 21.8 I3D + 3 TGMs (L = 30) + super-events 22.3 Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

Figure 7: (a-c) Different forms of 1-D temporal convolutions which take a D ×T input and produces a C × T output based on C number of D × L kernels: (a) the standard 1-D convolution, (b) Gaussian mixtures for 1-D convolution while sharing Gaussian mixtures across input channels, and (c) using D different Gaussian mixtures for 1-D convolution. (d) Our TGM layer in its simplest form (i.e., 1-layer case) applying the 1 × L temporal kernel in a 2-D convolutional fashion, maintaining both time and feature axis.

Figure8: A temporal convolutional layer with channel combination similar to Fig.3. The difference is that this layer does not learn Gaussian mixtures, but unconstrained 1-D temporal kernels.

Figure 9: Examples of several of the activities in the MLB-YouTube dataset: (a) Pitch, (b) Hit, (c) Bunt, (d) Hit by pitch, (e) No activity. This shows the difficulty of this dataset, as the difference between hit and bunt, swing and no swing are very small.

; Xu

Table content: <table><tr><td>Ground Truth</td><td>Baseline</td><td>Super-Events</td><td>Temporal Conv</td><td>TMG</td><td>TMG + Super</td></tr><tr><td>Guard</td><td/><td/><td/><td/><td/></tr><tr><td>Shot</td><td/><td/><td/><td/><td/></tr><tr><td>Block</td><td/><td/><td/><td/><td/></tr><tr><td>Dribble</td><td/><td/><td/><td/><td/></tr><tr><td>Dunk</td><td/><td/><td/><td/><td/></tr></table>

Effect of L on MultiTHUMOS and Charades using only RGB I3D features. Note that the 3 TGM layer models have larger temporal resolution than the 1 TGM layer models for the same values of L. We also compare to using standard one-layer 1-D conv layer with different values of L.

Table content: <table><tr><td/><td/><td>MultiTHUMOS</td><td/><td/><td>Charades</td><td/></tr><tr><td/><td colspan="6">1 Layer 3 Layers 1-D Conv 1 Layer 3 Layers 1-D Conv</td></tr><tr><td>I3D Baseline</td><td>22.3</td><td>-</td><td>-</td><td>15.3</td><td>-</td><td>-</td></tr><tr><td>L = 3</td><td>30.2</td><td>31.7</td><td>26.6</td><td>15.5</td><td>16.1</td><td>15.5</td></tr><tr><td>L = 5</td><td>32.5</td><td>37.2</td><td>28.3</td><td>15.7</td><td>17.8</td><td>16.3</td></tr><tr><td>L = 10</td><td>34.5</td><td>35.4</td><td>31.7</td><td>16.1</td><td>18.2</td><td>16.6</td></tr><tr><td>L = 15</td><td>36.1</td><td>34.1</td><td>32.5</td><td>17.5</td><td>18.6</td><td>16.8</td></tr><tr><td>L = 30</td><td>32.5</td><td>33.9</td><td>26.5</td><td>18.1</td><td>18.9</td><td>12.1</td></tr><tr><td>L = 50</td><td>32.1</td><td>33.7</td><td>15.4</td><td>18.3</td><td>18.8</td><td>6.7</td></tr></table>

Comparison of various values of M on MultiTHUMOS and Charades using RGB I3D features. For these experiments, 1 layer was used with L = 15 and C out = 16.

Table content: <table><tr><td/><td colspan="2">MultiTHUMOS Charades</td></tr><tr><td>M = 2</td><td>27.8</td><td>15.5</td></tr><tr><td>M = 4</td><td>33.1</td><td>16.2</td></tr><tr><td>M = 8</td><td>34.8</td><td>17.5</td></tr><tr><td>M = 16</td><td>36.1</td><td>17.5</td></tr><tr><td>M = 32</td><td>35.7</td><td>17.1</td></tr><tr><td>M = 64</td><td>35.8</td><td>17.3</td></tr></table>

Comparison of values of C out on Mul-tiTHUMOS and Charades using RGB I3D features. For these experiments, 1 layer was used with L = 15 and M = 16.

Table content: <table><tr><td/><td colspan="2">MultiTHUMOS Charades</td></tr><tr><td>C out = 1</td><td>33.5</td><td>16.2</td></tr><tr><td>C out = 4</td><td>34.2</td><td>17.4</td></tr><tr><td>C out = 8</td><td>35.5</td><td>17.5</td></tr><tr><td>C out = 16</td><td>36.1</td><td>17.5</td></tr><tr><td>C out = 32</td><td>36.0</td><td>17.2</td></tr><tr><td>C out = 64</td><td>36.1</td><td>17.4</td></tr><tr><td>C out = 80</td><td>36.1</td><td>17.5</td></tr></table>

Result mAP on the MLB-YouTube dataset using InceptionV3 and I3D to obtain features. Our TGM layers significantly outperform the baseline models.

Table content: <table><tr><td>Model</td><td colspan="3">Spatial Temporal Two-stream</td></tr><tr><td>Random</td><td>13.4</td><td>13.4</td><td>13.4</td></tr><tr><td>InceptionV3</td><td>31.2</td><td>31.8</td><td>31.9</td></tr><tr><td>InceptionV3 + LSTM</td><td>32.1</td><td>33.5</td><td>34.1</td></tr><tr><td>InceptionV3 + 1 temporal conv</td><td>32.8</td><td>34.4</td><td>35.2</td></tr><tr><td>InceptionV3 + 3 temporal conv</td><td>28.4</td><td>29.8</td><td>30.1</td></tr><tr><td>InceptionV3 + super-events</td><td>31.5</td><td>36.2</td><td>39.6</td></tr><tr><td>InceptionV3 + 1 TGM</td><td>32.4</td><td>36.3</td><td>37.4</td></tr><tr><td>InceptionV3 + 3 TGM</td><td>33.2</td><td>38.2</td><td>38.2</td></tr><tr><td>InceptionV3 + 3 TGM+super-events</td><td>34.6</td><td>42.4</td><td>42.9</td></tr><tr><td>I3D</td><td>33.8</td><td>35.1</td><td>34.2</td></tr><tr><td>I3D + LSTM</td><td>36.2</td><td>37.3</td><td>39.4</td></tr><tr><td>I3D + 1 temporal conv</td><td>37.3</td><td>38.6</td><td>39.9</td></tr><tr><td>I3D + 3 temporal conv</td><td>32.4</td><td>34.6</td><td>35.6</td></tr><tr><td>I3D + super-events</td><td>38.7</td><td>38.6</td><td>39.1</td></tr><tr><td>I3D + 1 TGM</td><td>35.5</td><td>37.5</td><td>38.5</td></tr><tr><td>I3D + 3 TGM</td><td>36.5</td><td>38.4</td><td>40.1</td></tr><tr><td>I3D + 3 TGM+super-events</td><td>39.4</td><td>46.0</td><td>47.1</td></tr></table>

Results on AVA dataset with the temporal annotation-only setting (i.e., frame classification without using bounding box training labels).

Table content: <table><tr><td/><td>mAP</td></tr><tr><td>Random</td><td>2.65</td></tr><tr><td>I3D baseline</td><td>7.5</td></tr><tr><td>I3D + 3 temporal conv. layers</td><td>7.9</td></tr><tr><td>I3D + LSTM</td><td>7.8</td></tr><tr><td>I3D + super-events</td><td/></tr></table>

Peer review (iclr2019):
 Summary: This paper proposes a temporal convolution layer called Temporal Gaussian Mixture (TGM) for video classification tasks. Usually, video classification models can be decomposed into a feature extraction step, in which frames are processed separately or in local groups, and a classification step where labels are assigned considering the full video features/longer range dependencies. Typical choices for the classification step are Recurrent Neural Networks (RNNs) and variations, pooling layers and 1D temporal convolutions. TGMs are 1D temporal convolutions with weights defined by taking samples from a mixture of Gaussians. This allows to define large convolutional kernels with a reduced number of parameters -the mean and std of the gaussians- at the cost of having reduced expressiveness. The authors experiment replacing 1D temporal convolutions with TGMs in standard video classifications models and datasets and report state-of-the-art results. Strengths: [+] The idea of defining weights as samples from a Mixture of Gaussians is interesting. It allows to define convolutional kernels that scale with the number of channels (equivalent to the number of Gaussians) instead of the number of channels and receptive field. Weaknesses: [-] The explanation of the TGM layers is very confusing in its current state. Certain aspects of TGMs that would help understand them are not clearly stated in the text. For example, it is not clear that 1) TGMs are actually restricted 1D convolutions as the kernel is expressed through a mixture of gaussians, 2) In a TGM the number of mixtures corresponds to the number of output channels in a standard 1D convolution and 3) despite TGMs are 1D convolutions, they are applied differently by keeping the number of frame features (output channels of a CNN) constant through them. Section 3 does not explain any of these points clearly and induces confusion. [-] The comparisons in the experimental section are unfair for the baselines and do not justify the advantages of TGMs. TGMs are 1D convolutions but they are applied to a different axis - 1D convolutions take as input channels the frame features. Instead, TGMs add a dummy axis, keep the number of frame features constant through them, and finally use a 1x1 convolution to map the resulting tensor to the desired number of classes. There is no reason why TGMs can’t be used as standard 1D convolutions taking as the first input channels the frame features - in other words, consider as many input mixture components as frame features for the first TGM. This is a crucial difference because the number of parameters of TGMs and the baselines is greatly affected by it. The implicit argument in the paper is that TGMs perform better than standard 1D convolutions because they have less parameters. However, in the experiments they compare to 1D convolutions that have smaller temporal ranges – TGMs have a temporal kernel of 30 timesteps or 10 per layer (in a stack of 3 TGMs), whereas the 1D convolutions either have 5 or 10 timestep kernels according to section 4. Thus, it is impossible to know whether TGMs work better because 1) in these experiments they have longer temporal range – we would need a comparison to 1D convolutions applied in the same way and with the same temporal range to clarify it, 2) because they are applied differently – we would need a comparison when TGMs are used the same way as 1D convolutions, 3) because the reduced number of parameters leads to less overfitting – we would need to see training metrics and see the generalization gap when compared to equivalent 1D convolutions or 4) because the reduced number of parameters eases the optimization process. To sum up, it*s true that TGMs would have a reduced number of parameters compared to the equivalent 1D convolutions with the same temporal range, but how does that translate to better results and, in this case, why is the comparison made with non-equivalent 1D convolutions? What would happen if the authors compared to 1D convolutions used in the same way as TGMs? [-] To claim SOTA, there are comparisons missing to recently published methods. In particular, [1] and [2] are well-known models that report metrics for the Charades dataset also used in this paper. Rating: At this moment I believe the TGM layer, while a good idea with potential, is not sufficiently well explained in the paper and is not properly compared to other baselines. Thus, I encourage the authors to perform the following changes: -Rewrite section 3, better comparing to 1D convolutions and justifying why TGMs are not used in the same manner but instead using a dummy axis and a linear/1x1 conv layer at the end and what are the repercussions of it. -Show a fair comparison to 1D convolutions as explained above. -If claiming SOTA results, compare to [1] or [2] which use different methods to the one proposed in these paper but outperform the baselines used in the experiments. [1] Wang, Xiaolong, et al. "Non-local neural networks." CVPR 2018 [2] Zhou, Bolei et al. "Temporal Relational Reasoning in Videos" ECCV 2018 ---------------------- Update: In light of the authors* rebuttal I have updated my rating from 5 to 6.