Year: 2018

Abstract:
We introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer and present how it can be used to efficiently capture longer-term temporal information in continuous activity videos. The TGM layer is a temporal convolutional layer governed by a much smaller set of parameters (e.g., location/variance of Gaussians) that are fully differentiable. We present our fully convolutional video models with multiple TGM layers for activity detection. The experiments on multiple datasets including Charades and MultiTHUMOS confirm the effectiveness of TGM layers, outperforming the state-of-the-arts.

heading: 1 INTRODUCTION
text: Activity videos are spatio-temporal data: they are image frames with a specific width/height (XY) concatenated along time axis (T). Recognition from such videos requires capturing both spatial and temporal information in the videos, desirably using learned convolutional kernels. Temporal convolution is particularly beneficial in activity ‘detection’ tasks, which require making activity decisions at every frame given a continuous video (Sigurdsson et al., 2016b; Yeung et al., 2015). Previous methods investigated using 3-D XYT convolutional filters (Tran et al., 2014; Carreira & Zisserman, 2017) as well as the models with 2-D XY conv. layers followed by 1-D temporal conv. (Tran et al., 2018), pooling or attention layers (Piergiovanni et al., 2017).
Understanding complex multi-activity videos requires capturing information in long-term time intervals. Different frames contain different information, and the model needs to learn to take advantage of as many frames as possible, while abstracting them efficiently. Previous attempts of simply pooling representations over time or learning temporal conv. filters with a small number of frames (e.g., 16 or 64) was thus often insufficient to fully consider rich long-term temporal context. Simultaneously, bruteforcely increasing the temporal filter length (to look at more frames) results more learnable parameters, requiring more training data, which can be expensive when activities are rare.
In this paper, we introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer, and present how it can be used to efficiently capture longer-term temporal information in activity videos. Our temporal Gaussian mixture layer is a temporal convolutional layer, whose filters/kernels are controlled by a set of (temporal) Gaussian distribution parameters. Each of our temporal Gaussian distributions specify (temporally) ‘where’ the model should look, and our Gaussian mixture layer combines them as multiple convolutional filters to be applied on top of temporallycontinuous representations. This layer allows the video representation at each time step to be constructed while focusing on different neighboring temporal regions, instead of only focusing on its local segment. It is a convolutional layer governed by a much smaller set of parameters (i.e., locations/variances of the Gaussians as well as their mixture weights) that are fully differentiable.
The motivation behind our temporal Gaussian mixture layer is to learn the temporal structure of an activity as a composition of temporal Gaussian regions/attentions. Such structure allows the model to obtain a compact spatio-temporal representation abstracting each (long-term) time interval, using multiple temporal conv. layers with far fewer parameters. It is also related to the previous temporal attention works (Piergiovanni et al., 2017), but our model is designed to be fully convolutional to handle continuous data and it learns more compositional structures with multiple layers.
We present video-CNN models using our TGM layers for activity detection in continuous videos. Our model stacks TGM layers on top of several state-of-the-art CNNs such as I3D (Carreira & Zisserman, 2017). This enables our model to capture longer-term temporal information than what we use as base CNNs, compositionally modeling temporal structure with multiple TGM layers. Our
model was evaluated on multiple public datasets including MultiTHUMOS and Charades, and was able to outperform the best previous activity detection CNNs by a meaningful margin.

heading: 2 RELATED WORKS
text: Learning video representations for human activity recognition has been successful. CNN methods allow end-to-end learning of video features and representations optimized for the training data, performing superior to traditional works (Aggarwal & Ryoo, 2011) for video understanding.
Two-stream CNN models take a single RGB frame and a small number of optical flow frames as inputs to capture both motion and appearance information in videos (Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016). Models learning 3-D spatio-temporal (XYT) convolutional filters were designed and applied to many activity recognition tasks as well (Tran et al., 2014; Carreira & Zisserman, 2017; Tran et al., 2017; Hara et al., 2017). Large scale datasets for activity detection, such as THUMOS (Jiang et al., 2014), ActivityNet (Heilbron et al., 2015), Kinetics (Kay et al., 2017), and Charades (Sigurdsson et al., 2016b) provided these approach the necessary training data to learn the models. Such 3-D XYT CNNs were also used to capture spatio-temporal information for activity detection (Xu et al., 2017; Shou et al., 2016; 2017; Zhao et al., 2017). However, all these CNNs were limited to the consideration of a fixed local video segment (e.g., 16 frames in (Tran et al., 2014) and 64-99 frames in (Carreira & Zisserman, 2017)) when making activity decisions.
Some works studied combining representations over longer-term temporal intervals (Karpathy et al., 2014; Ng et al., 2015; Varol et al., 2017), but it was generally done with a temporal pooling of local representations or (spatio-)temporal convolutions with a bit larger fixed intervals. Recurrent neural networks (RNNs) have also been used to model activity transitions between frames (Yeung et al., 2015; 2016; Escorcia et al., 2016), but they were strictly sequential and had limitations in maintaining temporal information over a longer temporal duration, particularly for videos with multiple complex activities. Recently, CNN models using temporal attention for activity videos (Piergiovanni et al., 2017; Piergiovanni & Ryoo, 2018b) were studied as well. However, a fully convolutional model to analyze continuous videos while efficiently representing information in long term intervals has been lacking.
Our layer is different from the previous standard (spatio-)temporal convolutional layers in that it relies on significantly fewer parameters by forcing filter shapes to be Gaussian compositions. Our temporal layer is also different from previous Gaussian Mixture Model layers (Variani et al., 2015) in that our layer is convolutional while they are not.

heading: 3 APPROACH
text: In this section, we introduce a new convolutional layer named the Temporal Gaussian Mixture (TGM) layer, and present how it can be used for activity recognition. Our Temporal Gaussian Mixture layer is a temporal convolutional layer to be applied on top of a sequence of representations (usually from frame-level or segment-level CNNs), whose filters/kernels are controlled by a set of (temporal) Gaussian distribution parameters. The motivation is to make each temporal Gaussian distribution specify (temporally) ‘where to look’ with respect to the activity center, and represent the activity as a collection/mixture of such temporal Gaussians convolved with video features. Our layer is fully differentiable and trainable using standard backpropagation.
Our TGM layer can be interpreted as a a form of 1-D convolution where the filters are determined by a mixture of Gaussians. However, our TGM layer differs from the standard temporal convolutional layers of learning 1-D (time) or 2-D (channel-by-time) filters in the following aspects:
1. Our temporal Gaussian mixture layer handles multiple 3-D tensors internally to preserve channels from the frame-level CNN by adding a new temporal channel axis. Its input is 3-D (channel-by-channel-by-time), where one channel dimension is inherited from the frame-level CNN and this dimension size remains unchanged.
2. Instead of learning temporal convolution filters of any arbitrary values, our filter is forced to have the form of a temporal Gaussian mixture shared across all frame-level channels. This allows the layer to rely on significantly fewer number of (fully differentiable) parameters, while capturing the concept of temporal structure/attention.

heading: 3.1 TEMPORAL GAUSSIAN MIXTURE LAYER
text: Our temporal Gaussian mixture layer takes a 3-D input with the dimensionality of Cin × D × T , where Cin is the number of input channels, D is the dimensionality of the representations from frame-level (or segment-level) CNNs, and T is the time. Given such input, the TGM layer convolves it with Cout number of 1 × L filters/kernels, generating a Cout ×D × T -dim representation as an output. L is the temporal length of the temporal Gaussian mixture filter. D is usually 1K or 4K and T is the number of time steps (frames) in each video (i.e., it varies per video). Cout is the number of different mixtures, corresponding to the number of output channels in standard convolution.
Our layer is composed of a set of M Gaussians. Each Gaussian has 2 parameters: a center µ̂ and a width σ̂. Each layer has additional hyper-parameters: L, the temporal duration and M , the number of Gaussians to learn. We force the learned center to be between −L2 and L 2 and σ to be positive:
µ = (L− 1) · tanh (µ̂+ 1) 2 , σ2 = exp (σ̂). (1)
We use the above µ and σ to construct the temporal Gaussian kernels. This acts as a strong sparsity constraint on the convolutional kernel as well as a drastic reduction of the number of learnable parameters. We construct a temporal Gaussian mixture convolutional kernel as:
K̂m,l = 1 Z exp− (l − µm) 2 2σ2m (2)
where Z is a normalization constant such that ∑L
l K̂m,l = 1, resulting in K̂ being anM×Lmatrix. Instead of making the model learn a separate set of Gaussian distributions per activity class, we take the approach of maintaining multiple Gaussian distributions shared across classes and obtain a Gaussian ‘mixture’ filter by learning soft-attention weights. We learn a set of soft-attention weights per output channel i, ω ∈ RCout×M . We create the soft-attention weights by applying the softmax function over the M Gaussians, enforcing each input channel weights sum to 1.
ai,m = expωi,m∑ j expωi,j
(3)
Based on temporal Gaussian distributions K̂i and attention weights ai,m, the temporal convolution filters our TGM layer is computed as:
Ki = ∑ m ai,mK̂i. (4)
This provides us convolutional filters having the form of a mixture of temporal Gaussians, controlled based on 2 ·M + Cin · Cout ·M parameters (instead of learning D2 · L parameters without any constraint, as in standard temporal convolution where C << D). An overview of this process is shown in Fig. 1.

heading: 3.1.1 SINGLE TGM LAYER - DIRECT PER-CLASS ACTIVITY MODELING
text: The representation we obtain by applying our base CNNs to each frame (or local segment) has the dimensionality of D, and stacking them along time axis provides us the representation with
1×D×T -dim. That is, in the case of using only one TGM layer to capture activity representations, our Cin is fixed to 1 and Cout is fixed to be the number of activity classes. This is the simplest case of our model, attaching one TGM layer on top of the 1×D × T representation. Our convolutional kernel, K, has a learned Gaussian mixture for each activity class. Let the video features v be a D × T matrix. Each Ki is a 2-D convolutional filter with a size of 1 × L, and convolving this with v provides us a representation S with Cout number of D × T responses since Cin is 1 in this case. This per-class representation can then be used as input to a fully-connected layer for activity classification. For i ∈ {1, 2, . . . , Cout}:
si = v ∗Ki, S = [s1, s2, . . . , sCout ] (5)
Fig. 7 in the appendix visually illustrates how each TGM filter is convolved with the input (Fig. 7d), compared to the standard 1-D convolution (Fig. 7a) or other forms of the temporal layers (Fig. 7b-c).

heading: 3.1.2 MULTIPLE TGM LAYERS - GROUPED CONVOLUTION
text: We generalize the above formulation to allow the TGM layers to be sequentially applied. The idea is to enable our model to capture more complex, nonlinear temporal structure by having multiple levels of temporal layers. In this case, the input for each layer is Cin ×D× T dimensional (instead of 1×D× T ), where the input channels are the number of output channels from the previous layer. Our kernels at each layer, Ki, are parameterized and learned as before.
By using grouped convolution with the number of groups set to Cin, we can efficiently separate the input into per-channel values and convolve each of them with the designated Ki kernel, as shown in Fig. 2. That is, we learn a filter Ki per channel by setting Cin = Cout. For i ∈ [1, Cout],
si = fi ∗Ki, S = [s1, s2, . . . sCout ] (6)
Here, f is a Cin ×D × T tensor, where D is the dimensionality of the feature and T is the number of frames. The result of the per-channel convolution, si, is a D × T representation. We concatenate these representations along the channel axis, resulting in S, a Cout ×D× T representation. As this convolution results in the same output shape, we can stack these layers. Each layer is able to capture increasing temporal resolution, allowing the model to capture levels of abstractions.

heading: 3.1.3 MULTIPLE TGM LAYERS - CHANNEL COMBINATION
text: In the above subsection, we introduced an approach of stacking multiple TGM layers to model a hierarchical composition of temporal representations. However, in the grouped convolution case, each output channel of the layer is solely dependent on its corresponding input channel. That is, each kernel only considers information from a single output channel of the previous layer.
Therefore, we further generalize our TGM layer so that the layer combines representations from multiple input channels for each output channel while using the learned temporal kernels. We learn a set of convolutional kernels K ∈ RCout×Cin×L (i.e., we learn Cout · Cin Gaussian mixtures). Given f which is the Cin × D × T representation, for each output channel i ∈ [1, Cout] and each input channel j ∈ [1, Cin] pair, we convolve the associated filters with the input.
Gi,j = (fj ∗Ki,j) (7)
where each Gi,j is a D × T -dim representation. We then learn a 1x1 convolution followed by a ReLU activation function for each i ∈ [1, Cout], which we call wi, that maps from Cin channels to 1 channel. The 1x1 convolution learns to combine
the channels from the previous layer. By design, the TGM kernel is positive and sums to 1. Adding the unconstrained 1x1 convolution adds non-linearity (using the ReLU activation function) to our layer and only adds Cout · Cin parameters.
si = Gi ∗ wi = (fj ∗Ki,j) ∗ wi, S = [s1, s2 . . . , sCout ] (8)
We then stack the si representations along the channel axis to produce S, the Cout × D × T -dim representation. This process is illustrated in Fig. 3. This method generalizes our approach to allow the layer to take input of Cin×D×T and produce output of Cout×D×T . These layers can easily be stacked to learn a hierarchical representation.

heading: 3.2 VIDEO CNN MODELS WITH TGM LAYERS
text: Our goal is to do activity detection which we define as making a per-frame (or per-segment) classification. Given a video, at each time step t, we want to make the model decide which activity the frame corresponds to (including no-activity). As a baseline, we train a fully-connected layer that classifies each per-frame D-dimensional vector, vt. As multiple activities can occur at the same time, or no activities at all, we treat this as a mutli-label classification task. We minimize binary cross entropy:
L(v) = ∑ t,c zt,c log(p(c|vt)) + (1− zt,c) log(1− p(c|vt)) (9)
where zt,c is the ground truth label, 1 if activity c is occurring at time t and p(c|vt) is the output of our model for class c at time t. Fig. 4 shows an example CNN.

heading: 4 EXPERIMENTS

heading: 4.1 IMPLEMENTATION AND BASELINES
text: Implementation We used I3D (Carreira & Zisserman, 2017) and the two-stream version of InceptionV3 (Szegedy et al., 2016) pretrained on Imagenet and Kinetics as our base per-frame CNNs. Our default L setting used for the TGM layers as well as the other baselines was as follows: when using I3D segment features (collected at 3fps), the 1 layer models used L = 15 and the 3 layer models used L = 5. When using InceptionV3 frame feature (collected at 8fps), the 1 layer models used L = 30 and the 3 layer models used L = 10. These layers were attached on top of the base CNN, as described in Subsection 3.2. Please check the appendix for implementation and training details and results on other datasets.
Baselines In order to confirm the advantages of our TGM layers, particularly against previous temporal models, we implemented several baselines. The first is (i) a standard per-frame classifier in which the prediction at each time-step only depends on a single feature vector with no contextual temporal information. We also used (ii) LSTMs on top of per-frame representations, which were popularly used to capture temporal information (Donahue et al., 2015). We train a bi-directional LSTM with 512 hidden units to make per-frame predictions. We also tried (iii) the fixed pyramid temporal max-pooling of level 3 (Ryoo et al., 2015). Finally, we compare our model against (iv) the model with standard temporal convolutional layers (i.e., 1-D convolution with a D × L kernel) on top of per-frame representations. This is similar to the temporal conv. used in (Tran et al., 2018). Temporal lengths (i.e., L) of the 1-D conv. filters and the pooling windows were set to be identical to the TGM filters. That is, they capture the same temporal duration as TGMs. In all our experiments, we follow the standard evaluation setting of computing per-frame mean average precision (mAP) and report those values. We also compare to different versions of the TGM layer, (v) with a learned mixture of random temporal filters and (vi) with a learned mixture of fixed Gaussians.
In addition, we also tried the approach of combining our TGM layers with the recent super-event representations (Piergiovanni & Ryoo, 2018b). We concatenated the learned super-event representation with our representations from TGM layers.

heading: 4.2 MULTITHUMOS
text: Dataset MultiTHUMOS (Yeung et al., 2015) is an extended version of the THUMOS (Jiang et al., 2014) dataset that densely annotates the continuous videos. The dataset consists of 65 different classes, compared to 20 in THUMOS, and contains on average 10.5 activities per video and 1.5 labels per frame and up to 25 activity instances in each video. This is in contrast to many other activity detection dataset such as ActivityNet (Heilbron et al., 2015), which only has on average ∼1 activity per video. MultiTHUMOS consists of YouTube videos of various sport activities such as basketball games, volleyball games, weight lifting, and track and field.
We followed the standard MultiTHUMOS evaluation setting of measuring mAP based on per-frame annotations. There are 1010 validation videos and 1574 test videos. We used these continuous validation videos for the training of our models. We did not need to take advantage of the separate training set with segmented videos; even without them, we outperformed the state-of-the-arts.
Results We compared baselines as well as multiple different versions of our architectures, shown in Table 1. The model with our TGM layers consistently outperformed baseline I3D (or InceptionV3) while using the same per-segment representations. Learning 3 TGM layers further improved the performances. On the other hand, we found that stacking multiple standard temporal convolutional layers does not improve performance, often performing worse than the baseline. While a single standard temporal conv. layer improves over the baseline, having multiple of them significantly increases the number of parameters to learn (Table 2) and we suspect that this was causing the overfitting with the limited amount of samples in the dataset. In Table 3, we compare the results of using a LSTM or temporal conv. with a similar number of parameters. This was done by making their temporal conv. filters to share values across multiple channels. These models result in nearly random performance, as they were not designed to cope with a small number of parameters. We also show results with a mixture of random (fixed) temporal filters and with a mixture of fixed Gaussians. These results confirm that (i) modeling the temporal structure as a learned Gaussian mixture is beneficial and that (ii) further learning the Gaussian distribution parameters is important.

heading: 1 TGM 35.1 37.8 40.5 16.3 17.5 18.0

heading: 3 TGM 36.4 42.3 43.5 17.5 18.3 19.2
text: Learning multiple TGM layers with channel combination outperforms the grouped convolution version of TGM and all the baselines. We also experimented with a version using soft-attention weights to combine the TGM layer channels, in addition to our method (Fig. 3) of using 1x1 convolution followed by a ReLU (to gain non-linearity). We found that the 1x1 convolution performed better. We tested various number of Gaussian mixtures (i.e., output channels) and found that using 80 for the first and second layer and using 65 (i.e., number of classes) for the final layer performs best.
Table 4 compares our model using TGM layers with multiple previous state-of-the-art approaches and baselines such as LSTM. Our approach meaningfully outperforms all previous approaches. Importantly, we are comparing our approach with different methods of capturing temporal information such as LSTMs and fixed temporal pyramid pooling while making them use the exactly same per-frame representations. We found that while all these methods capture some temporal information, the TGM layers provide the best performance. Further, combining the super-event representation (Piergiovanni & Ryoo, 2018b) with our TGM feature also benefited detection, confirming that our TGMs and super-events capture different aspects of the activity videos. In Fig. 5, we show an example of the various models predictions on a basketball video. We outperform the previous state-of-the-art performance (mAP) by 10% (36.4 vs. 46.4).

heading: 4.3 CHARADES
text: Dataset Charades (Sigurdsson et al., 2016b) is a large scale dataset with 9848 videos across 157 activity classes. These videos were recorded in home environments of the participants based on provided scripts. Each video contains on an average of 6.8 activity instances, and there are often complex activities co-occurring. The activities were mainly performed at home. For example, some activity classes are ‘preparing a meal’, ‘eating’, ‘sitting’, ‘cleaning’, etc.
In our experiments, we follow the original Charades detection setting (i.e., Charades v1 localize evaluation), which is the setting used in many previous approaches (Sigurdsson et al., 2016a; Xu
et al., 2017; Piergiovanni & Ryoo, 2018b). This is the original setting more challenging than the Charades Challenge 2017 setting (whose evaluation server was no longer approving new account access), in the aspect that it uses less amount of training videos.
Results We compare our results with the state-of-the-arts in Table 5. To our knowledge, our method is obtaining the best known performance in the original localization setting of the Charades dataset. Notably, it is performing better than I3D that obtained the best competition performance, while using the same feature. Our method also outperforms standard temporal convolution, LSTMs, and fixed pyramid pooling, as well as the use of latent super-events. When setting L = 30 and using 3 TGM layers, our model is able to capture around 800 frames (about ±15 seconds from each frame) of temporal information, significantly more than previous works (e.g., I3D only captures ±2 seconds).

heading: 5 CONCLUSIONS
text: We newly introduced the Temporal Gaussian Mixture (TGM) layer and demonstrated its effectiveness for multi-activity detection in continuous videos. Our layer is fully differentiable and trainable using standard backpropagation, designed to learn temporal structure. We were able to confirm that our layer performs superior to state-of-the-art methods on activity detection datasets including MultiTHUMOS and Charades, obtaining the best known performance. We also tested our approach with two more public video datasets, MLB-YouTube (Piergiovanni & Ryoo, 2018a) and AVA (Gu et al., 2017), and confirmed its advantage over the previous works in Appendix.

heading: B HYPERPARAMETER EXPERIMENTS
text: We conducted a set of experiments to compare the effects of the temporal duration, L, number of Gaussians, M , and the number of output channels, Cout. For these experiments, we only used the one-stream version of I3D with RGB inputs.
Effect of L: In Table 6, we compare different values of L. For these experiments, we use M = 16 andCout = 16. We find that the 3-layer model with L = 5 performs the best. With I3D features, this allows the model to capture up to 8 seconds of information. The average activity in MultiTHUMOS is 3.3 seconds long and the maximum is 14.7 seconds long, and with this setting, the model is able to capture enough temporal context to perform well. Larger values of L capture too much temporal information, but due to the Gaussian structure, it does not drastically harm performance. Figure 6 shows that even with longer kernels, the Gaussians learn to focus mostly on the center of the interval and capture the rough duration of the activities. Thus, having too long intervals does not drastically harm performance, which is in contrast to the standard 1-D convolution. Note that for Charades, the temporal kernels are learned to capture much longer temporal duration, as the average activity in charades is 12.8 seconds and larger values of L perform better.
Figure 6 illustrates examples of the learned TGM kernels of various lengths. The figure shows that the kernels focus on short temporal intervals on MultiTHUMOS even if we make the filters longer, as the activities are an average of 3.3 seconds long. On Charades, the TGM kernels learn to capture much longer intervals, as the activities are an average of 12.8 seconds long. We believe that this suggests TGMs are learning to capture information from the important necessary intervals.
In Table 6, we also report the results of using a standard 1-D conv. layer with different L values. The number of parameters in our TGM layer is independent of L, however, with the standard 1-D conv. layer, the number of parameters increases as L increases. We find that increasing L with 1-D convolution helps for small values of L, but for L > 15, the performance drastically drops, while TGM layers only show a small decrease.
Effect ofM : In Table 7, we compare different values ofM . For these experiments, we set L = 15 and Cout = 16. We find that M = 16 performs best, suggesting that smaller values of M restrict the possible temporal kernels too much. We also observe that larger values of M performs slightly worse than M = 16 (but not much), likely because they introduce more parameters than needed. When M and L have similar values, it allows the model to learn a sufficient number of Gaussians
and create a diverse range of temporal kernels. When M is larger than L, it results in learning a kernel similar to standard 1-D convolution.
Effect of Cout: In Table 8, we compare different values of Cout. For these experiments, L = 15, we used 1-layer and M = 16. We find that Cout performs best when set to 16 or larger on these datasets. Larger values of Cout seem to capture redundant information, as it does not lower performance.

heading: C COMPARISON OF DIFFERENT LAYER FORMS
text: To confirm the various aspects of our design, we conducted experiments comparing different types of temporal convolution. In Fig. 7a we illustrate the standard 1-D convolution, taking D × T input and producing a C × T output, where D is the number of input channels and C is the number of output channels. In Fig. 7b, we illustrate the method of applying a Gaussian mixture kernel as 1-D convolution. Here, the Gaussian mixture kernel is shared by all D input channels and we learn a C number of such kernels. In Fig. 7c, we illustrate the approach of applying a Gaussian mixture kernel as 1-D convolution while learning D different Gaussian mixtures. This is very similar to the standard 1-D convolution, except that the filter values are constrained to have the shape of Gaussian mixtures.
Fig. 8 illustrates one more baseline. This is similar to our full TGM layer with the channelcombination described Fig. 3. However, in this baseline, instead of learning Gaussian mixtures, we learn Cin ·Cout number of 1×L kernels. The kernel values are left unconstrained. While the TGM layer has 2 ·M+Cin ·Cout ·M+Cin ·Cout parameters, this layer has L ·Cin ·Cout ·M+Cin ·Cout, which is more than the TGM layer.
In Table 9, we compare the results of the various above-mentioned layers on MultiTHUMOS using RGB I3D features. We find that the Fig. 7b method performs poorly, while the Fig. 7c method slightly outperforms the standard 1-D convolution. The Fig. 8 method is slightly better than the standard 1-D convolution, but performs worse than Fig. 7c. However, none of these layers perform as well as our TGM layer, confirming that both the design of learning Gaussian mixtures and maintaining temporal channel axis are important for activity detection.

heading: D EXPERIMENTS ON ADDITIONAL DATASETS
text: D.1 MLB-YOUTUBE DATASET
D.1.1 DATASET
The MLB-YouTube dataset (Piergiovanni & Ryoo, 2018a) consists of 20 baseball games from the 2017 MLB post-season available on YouTube. This dataset consists of over 42 hours of video. For these experiments, we used the continuous video setting which have 2,126 1-2 minute long clips. Each clip is densely annotated with the baseball activities that occur. There are 8 activity classes: pitch, strike, ball, swing, hit, foul, hit by pitch, and bunt. Examples of some of these classes are shown in Fig. 9. Each continuous clip contains on average of 7.2 activities, giving a total of over 15,000 activity instances in the dataset.
What makes this dataset challenging is that the variation between classes is very small. In ActivityNet (Heilbron et al., 2015), for example, the difference between swimming and brushing hair is drastic. The background, motion, and even size of the person in the video is different. However, in broadcast baseball videos, the difference between a ball and a strike, or a swing and a bunt, are small. All actions are recorded from the same camera angle as we can confirm from Fig. 9.
D.1.2 RESULTS
In Table 10, we compare various approaches on this dataset. Our TGM layers improve over the baseline by ∼6% (40.1 vs. 34.2). Additionally, we compare to methods using the super-event representation (Piergiovanni & Ryoo, 2018b), which previously achieved state-of-the-art performance on several activity detection datasets. On this dataset, our approach outperforms the super-event representation, and further the concatenation of our TGM representation with such super-event representation performs best by a significant margin (∼13% compared to the baseline). This suggests that TGMs and super-event capture different temporal information and are both useful to the detection task.
We further find that using multiple, standard temporal convolution layers leads to worse performance, likely due to overfitting from the large number of parameters. While using multiple TGM layers improves performance, confirming that the Gaussian structure and sparsity constraint benefits model learning.
AVA (Gu et al., 2017) is a large-scale video dataset containing of 80 atomic action classes in 57k video clips. These clips are drawn from movies. Existing datasets, such as Charades, have very specific actions that depend on objects, such as holding a cup vs. holding a picture. In AVA, the actions are intentionally generic, such as sit, stand, hold, carry, etc. Further, the AVA dataset is annotated with both spatial and temporal locations of activities. Since we are interested in temporal activity detection, we follow the setting of Piergiovanni & Ryoo (2018b) and label each frame with the occurring activities while ignoring the spatial location. We evaluate performance following the same method as MultiTHUMOS, Charades and MLB-YouTube by measuring per-frame mAP.
D.2.2 RESULTS
In Table 11, we present the results of our model. We again find that temporal convolution and LSTMs provide some benefit over the baseline, but TGM layers further improve performance. Again, combining the TGM, which captures local temporal structure, with super-events which capture global temporal structure, provides the best performance by ∼ 7.4%.

Review summaries:
This paper proposes a temporal convolution layer called Temporal Gaussian Mixture (TGM) for video classification tasks. Usually, video classification models can be decomposed into a feature extraction step, in which frames are processed separately or in local groups, and a classification step where labels are assigned considering the full video features/longer range dependencies. Typical choices for the classification step are Recurrent Neural Networks (RNNs) and variations, pooling layers and 1D temporal convolutions. TGMs are 1D temporal convolutions with weights defined by taking samples from a mixture of Gaussians. This allows to define large convolutional kernels with a reduced number of parameters -the mean and std of the gaussians- at the cost of having reduced expressiveness. The authors experiment replacing 1D temporal convolutions with TGMs in standard video classifications models and datasets and report state-of-the-art results.