Supplemental material : Temporal Residual Networks for Dynamic Scene Recognition

Slow feature analysis (SFA) approaches analyze temporal data to extract features that vary most slowly over time, taking those to be most indicative of the stable properties of the input [32]. The approach has been applied to dynamic scene recognition [28] by extracting features from filter responses that are reputed to model primate V1 cortical operations, as they result from local maxima of spatially oriented, multiscale Gabor filters [25]. The slowest varying features among those are identified by taking their temporal derivatives and subsequently are encoded via soft assignment with respect to a dictionary built with unsupervised sampling. Following encoding, the features are pooled into a feature vector via application of max-pooling to the entire video in spatial pyramid regions [18]. Bags of spacetime energies (BoSE) [11] is the penultimate version of spatiotemporal energy approaches applied to dynamic scenes [8, 10]. The approach extracted dense measurements of spatiotemporal energy across a range of scales and orientations as well as CIE-LUV colour measurements. Here, the spatiotemporal features are augmented with dense SIFT measurements [19] to more finely capture spatial orientation. The descriptors are encoded by Improved Fisher Vector (IFV) [21, 22] encoding with a visual word dictionary represented by a Gaussian Mixture Model (diagonal covariance) with 64 centres. We average the frame-level BoSE encodings over a video which simplifies the the temporal slice-based SVM prediction of the original BoSE system [11]. The simplification is employed for equality in comparison to other baselines which also train a single one-vs-rest SVMs for video classification. Trajectory features (IDT) have been investigated with respect to a variety of video understanding tasks, e.g., [20, 23, 24, 30]. Curiously, it appears that they have not previously been applied to dynamic scene recognition. Recently, however, they have provided the basis for a number of outstanding approaches to action recognition as instantiated in improved Dense Trajectories (IDTs) [31]; therefore, it is of interest to evaluate their performance on scene recognition, as follows. Trajectory features are extracted across stabilized video sequences by concatenating a series of optical flow vectors for densely extracted interest points. Feature descriptors are aggregated across each trajectory in terms of trajectory shape [30], HOG [5], HOF [17] and MBH [6] measurements. Following extraction, the features are encoded using (improved) Fisher Vectors (FVs) [22] with dictionary represented by a Gaussian Mixture model (diagonal covariance) having 256 centres. Before training the GMM, all features are augmented with their normalized (x, y) image coordinates as an efficient way to capture location information. Details of extraction of the trajectories, their descriptors and encoding are exactly as in their original application to action recognition [31]. All DT and IDT parameters are used as in [30, 31] and their publicly available code is used to extract the descriptors. Spatial convolutional network (S-CNN) features [3] are generated from the last convolutional layer of a VGG16 network [27]. The model is pre-trained on ImageNet [7]. It has been shown that the features from such pretrained CNNs are transferable to many other vision domains [2, 3, 9, 13]. This approach derives its features from the last convolutional layer of a VGG-16, which uses features from the last conv-layer of VGG-16. The resulting 512dimensional features are encoded using (improved) Fisher Vectors (FVs) [22] with dictionary represented by a Gaussian Mixture model (diagonal covariance) having 64 centres. Features from a single video are extracted with a stride of 16 frames. Before encoding the features are augmented with their normalized (x, y) image coordinates, as with the above IDT approach. Temporal convolutional network (T-CNN) uses a stack of 10 optical flow frames as input, with optical flow extracted by a standard algorithm [1] and is first pre-trained on the UCF101 action recognition dataset [16]. The final model is a CNN-M-2048 network [2]. (In our preliminary evaluation with this implementation, a recognition accuracy of 82.6% on UCF101 (split 1) was achieved, which compares favourably to the 81.2% reported originally [26]. ) The same IFV encoding procedure as used for the spatial CNN