Title: DISTRIBUTED NON-PARAMETRIC DEEP AND WIDE NETWORKS

Abstract: In recent work, it was shown that combining multi-kernel based support vector machines (SVMs) can lead to near state-of-the-art performance on an action recognition dataset (HMDB-51 dataset). In the present work, we show that combining distributed Gaussian Processes with multi-stream deep convolutional neural networks (CNN) alleviate the need to augment a neural network with handcrafted features. In contrast to prior work, we treat each deep neural convolutional network as an expert wherein the individual predictions (and their respective uncertainties) are combined into a Product of Experts (PoE) framework.

INTRODUCTION:

Recognizing actions in a video stream requires the aggregation of temporal as well as spatial features (as in object classification). These video streams, unlike still images, have short and long temporal correlations, attributes that single frame convolutional neural networks fail to discover. Therefore, the first hurdle to reach human-level performance is designing feature extractors that can learn this latent temporal structure. Nonetheless, there has been much progress in devising novel neural network architecture since the work of Karpathy et al. (2014) . Another problem is the large compute, storage and memory requirement for analysing moderately sized video snippets. One requires a relatively larger computing resource to train ultra deep neural networks that can learn the subtleties in temporal correlations, given varying lighting, camera angles, pose, etc. It is also difficult to utilise standard image augmentation (like random rotations, shears, flips, etc.) techniques on a video stream. Additionally, features for a video stream (unlike static images) evolve with a dynamics across several orders of time-scales.

Nonetheless, the action recognition problem has reached sufficient maturity using the two-stream deep convolutional neural networks (CNN) framework (Simonyan & Zisserman, 2014) . Such a framework utilises a deep convolutional neural network (CNN) to extract static RGB (Red-Green-Blue) features as well as motion cues by deconstructing the optic-flow of a given video clip. Notably, there has been plenty of work in utilising a variety of network architectures for factorising the RGB and optical-flow based features. For example, an inception network (Szegedy et al., 2016) uses 1 × 1 convolutions in its inception block to estimate cross-channel corrections, which is then followed by the estimation of cross-spatial and cross-channel correlations. A residual network (ResNet), on the other hand, learns residuals on the inputs instead of learning unreferenced functions (He et al., 2016) . While such frameworks have proven useful for many action recognition datasets (UCF101, UCF50, etc.), they are yet to show promise where videos have varying signal-to-noise ratio, viewing angles, etc.

We improve upon existing technology by combining Inception networks and ResNets using a Gaussian Process classifier that is further combined in a product-of-expert (PoE) framework to yield, to the best of our knowledge, a state-of-the-art performance on the HMDB51 data-set (Kuehne et al., 2013) . Under a Bayesian setting, our framework provide not only mean predictions, but also the uncertainty associated with each prediction. Notably, our work forwards the following contributions:

• We introduce a framework that allow for independent multi-stream deep neural networks, enabling horizontal scalability • Ability to classify video snippets that have heterogeneity regarding camera angle, video quality, pose, etc.

• Combine deep convolutional neural networks with non-parametric Bayesian models, wherein there is a possibility to train them using less amount of data

• Demonstrate the utility of model averaging that takes uncertainty around mean predictions into account

METHODS:

In this section, we describe the dataset, the network architectures and the nonparametric Bayesian setup that we utilise in our four-stream CNN pillar network for activity recognition. We refer the readers to the original network architectures in Wang et al. (2016) and Ma et al. (2017) for further technical details. Utilising classification methodologies like AdaBoost, gradient boosting, random forests, etc. provide us with accuracies in the range of 5-55% for this dataset, for either the RGB or the optic-flow based features.

DATASET:

The HMDB51 dataset (Kuehne et al., 2013) is an action classification dataset that comprises of 6,766 video clips which have been divided into 51 action classes. Although a much larger UCF-sports dataset exists with 101 action classes (Soomro et al., 2012) , the HMDB51 has proven to be more challenging. This is because each video has been filmed using a variety of viewpoints, occlusions, camera motions, video quality, etc. anointing the challenges of video-based prediction problems.

The second motivation behind using such a dataset lies in the fact that HMDB51 has storage and compute requirement that is fulfilled by a modern workstation with GPUs -alleviating deployment on expensive cloud-based compute resources.

All experiments were done on Intel Xeon E5-2687W 3 GHz 128 GB workstation with two 12GB nVIDIA TITAN Xp GPUs. As in the original evaluation scheme, we report accuracy as an average over the three training/testing splits.

INCEPTION LAYERS FOR RGB AND FLOW EXTRACTION:

We use the inception layer architecture described in Wang et al. (2016) . Each video is divided into N segments, and a short sub-segment is randomly selected from each segment so that a preliminary prediction can be produced from each snippet. This is later combined to form a video-level prediction. An Inception with Batch Normalisation network (Ioffe & Szegedy, 2015) is utilised for both the spatial and the optic-flow stream. The feature size of each inception network is fixed at 1024. For further details on network pre-training, construction, etc. please refer to Wang et al. (2016) .

RESIDUAL LAYERS FOR RGB AND FLOW EXTRACTION:

We utilise the network architecture proposed in Ma et al. (2017) where the authors leverage recurrent networks and convolutions over temporally constructed feature matrices as shown in Fig. 1 . In our instantiation, we truncate the network to yield 2048 features, which is different from Ma et al. (2017) where these features feed into an LSTM (Long Short Term Memory) network. The spatial stream network takes in RGB images as input with a ResNet-101 (He et al., 2016) as a feature extractor; this ResNet-101 spatial-stream ConvNet has been pre-trained on the ImageNet dataset. The temporal stream stacks ten optical flow images using the pre-training protocol suggested in Wang et al. (2016) .

The feature size of each ResNet network is fixed at 2048. For further details on network pre-training, construction, etc. please refer to Ma et al. (2017) .

NON-PARAMETRIC BAYESIAN CLASSIFICATION:

Gaussian Processes (GP) emerged out of filtering theory (Wiener, 1949) in non-parametric Bayesian statistics via work done in geostatistics (Matheron, 1973) . Put simply, GPs are collection of random variables that have a joint Gaussian distribution, Obervation:

EQUATION

where, k ( x, x| θ) is the kernel function parameterized by θ; φ is the parameter of the observation model; f (x) is the latent function evaluated at x i.e., the features. y denotes the class of the input features and {φ, θ} ∈ χ denote the set of hyper-parameters.

For multi-class problem with a non-Gaussian likelihood (softmax; p (

y i | f (x i )) = exp(f y i i ) 51 c=1 exp(f c i )

), the conditional posterior is approximated via the Laplace approximation (Williams & Barber, 1998) i.e., a second order Taylor expansion of log p ( f | y, θ, φ)around the mode f as,

EQUATION

D is the (input,output) tuple. After the Laplace approximations, the approximate posterior distribution becomes,

EQUATION

Finally, we can evaulate the approximate conditional predictive density of ỹi ,

p ( ỹi | D, θ, φ) ≈ p ỹi | fi , φ q fi D, θ, φ d fi (4) PRODUCT OF EXPERTS

For each of the neural network, we subdivide the training set into K = 7 sub-sets so that K different GPs could be trained, giving us 28 GPs for the 4 deep networks (2 Inception networks and 2 ResNets) that we have trained in the first part of our training. We assume that each of the 7 GPs are independent, such that the marginal likelihood in our product of expert (PoE) becomes, What we have done is to reduce the computational expenditure from O(n 3 ) to O(n 3 k ). Notice that unlike GPs with inducing inputs or variational parameters such a distributed GP does not require optimisation of additional parameters. Finally, a product-of-GP-experts is instantiated that predicts the function f * at the test point x * as,

p (y |X, χ ) ≈ K k=1 p k y (k) X (k) , χ p k y (k) X (k) , χ = - 1 2 y (k) K (k) θ + σ 2 φ I -1 y (k) - 1 2 log K (k) θ + σ 2 φ I (5)

EQUATION

RESULTS:

We used 3570 videos from HMDB51 as the training data-set; this was further split into seven subsets, each with 510 videos. We select ten videos randomly chosen from each category, and each sub-set is non-overlapping. Based on seven sub-sets, seven GPs are trained on different features (RGB and Flow) from different Networks (TSN-Inception Wang et al. (2016) and ResNet-LSTM Ma et al. (2017) ). In total, twenty-eight GPs are generated. The features for both the RGB and the optical flow were extracted from the last connected layer with 1024 dimension for the Inception network and 2028 for the ResNet network. The fusion is then performed both vertically (seven sub-sets) and horizontally (four networks). The accuracies of individual GPs and different fusion combinations (PoE) on split-1 are shown in Table 1 . Fusion-1 represents the results from the fusion of seven GPs for each feature; Fusion-2 show the fusion result of RGB and Flow using different Fusion-all shows the result by fusion of all the 28 GPs. The average result for three splits is displayed in Table 2 . We also demonstrate the improvement when hand-crafted features like iDT are combined with dCNNs to yield an 1.4% improvement.

DISCUSSION:

Here, we make two contributions -(a) we build on recently proposed pillar networks (Sengupta & Qian, 2017) and combine deep convolutional neural networks with non-parametric Bayesian models, wherein they have the possibility of being trained with less amount of data and (b) demonstrate the utility of model averaging that takes uncertainty around mean predictions into account. Combining different methodologies allow us to supersede the current state-of-the-art in video classification especially, action recognition.

We utilised the HMDB-51 dataset instead of UCF101 as the former has proven to be difficult for deep networks due to the heterogeneity of image quality, camera angles, etc. As is well-known, videos contain extensive long-range temporal structure; using different networks (2 ResNets and 2 Inception networks) to capture the subtleties of this temporal structure is an absolute requirement.

Since each network implements a different non-linear transformation, one can utilise them to learn very deep yet different features. Utilising the distributed-GP architecture then enables us to parcel-late the feature tensors into computable chunks (by being distributed) of input for a Gaussian Process classifier. Such an architectural choice, therefore, enables us to scale horizontally by plugging in a variety of networks as per requirement. While we have used this architecture for video based classification, there is a wide range of problems where we can apply this methodology -from speech processing (with different networks) to natural-language-processing (NLP).

Ultra deep convolutional networks have been influential for a variety of problems, from image classification to natural language processing (NLP). Recently, there has been work on combining the Inception network with that of a Residual network such that the resulting network builds on the advantages offered by either network in isolation (Szegedy et al., 2017) . In future, it would be useful to see how different are the features when they are extracted from Inception module, ResNet module or a combination of both. Not only this, a wide variety of hand-crafted features can also be augmented as inputs to the distributed GPs; our initial experiments using the iDT features show that this is indeed the case (a 1.4% improvement). Input data can also be augmented using RGB difference or optic flow warps, as had been done in Wang et al. (2016) .

Also The Bayesian product-of-GPs would suffer from a problem were we to increase the number of experts. This is because the precision of the experts adds up which leads to overconfident predictions, especially in the absence of data. In unpublished work, we have utilised generalised Product of Experts (gPoE) (Cao & Fleet, 2014) and Bayesian Committee Machine (BCM) (Tresp, 2000) to increase the fidelity of our predictions. These would be reported in a subsequent publication along with results from a robust Bayesian Committee Machine (rBCM) which includes the product-of-GPs and the BCM as special cases (Deisenroth & Ng, 2015) .

For inference, we have limited our experiments to the Laplace approximation inference under a distributed GP framework. An alternative inference methodology for multi-class classification include (stochastic) expectation propagation (Riihimäki et al.; Villacampa-Calvo & Hernández-Lobato, 2017) or variational approximations (Hensman et al., 2015) . Free-energy minimization is attractive simply due to lower computational overhead. Indeed, it comes with its problems such as underestimation of the variability of the posterior density, inability to describe multi-modal densities and the inaccuracy due to the presence of multiple equilibrium points. All being said, some of these problems are also shared by state-of-the-art MCMC samplers for dynamical systems. Due to the flexibility of utilising GPUs, both methods (variational inference and EP) can prove to be computationally efficient, especially for streaming data. Thus, there is a scope of future work where one can apply these inference methodologies and compare it with vanilla Laplace approximations, as utilised here.

:



Figures and Tables:

Figure1: The distributed non-parametric deep and wide network framework: Each pillar represents either a single ultra-deep neural network or other feature tensors that can be learnt automatically from the input data. For action recognition, we factorize the static (RGB), and dynamic (optic flow) features using a ResNet and an Inception Network. Using the features of the last layer, we train seven Gaussian Processes for each of the network, which is combined under a Product-of-Experts (PoE) formalism. This hierarchy is then fused again to give us a prediction of action types.

Accuracy scores for the HMDB51 data-set

Table content: <table><tr><td>Methods</td><td colspan="2">Accuracy [%] Reference</td></tr><tr><td>Two-stream</td><td>59.4</td><td>Simonyan &amp; Zisserman (2014)</td></tr><tr><td>Rank Pooling (ALL)+ HRP (CNN)</td><td>65</td><td>Fernando &amp; Gould (2017)</td></tr><tr><td>Convolutional Two-stream</td><td>65.4</td><td>Feichtenhofer et al. (2016)</td></tr><tr><td>Pillar Networks + soft-max + cross-entropy</td><td>67</td><td>Sengupta &amp; Qian (2017)</td></tr><tr><td>Temporal-Inception</td><td>67.5</td><td>Ma et al. (2017)</td></tr><tr><td>TS-LSTM</td><td>69</td><td>Ma et al. (2017)</td></tr><tr><td>ST-multiplier network</td><td>68.9</td><td>Feichtenhofer et al. (2017)</td></tr><tr><td>ST-ResNet + iDT</td><td>70.3</td><td>Ma et al. (2017)</td></tr><tr><td>Temporal Segment Network (2/3/7 modalities)</td><td>68.5/69.4/71</td><td>Wang et al. (2016)</td></tr><tr><td>ST-multiplier network + iDT</td><td>72.2</td><td>Feichtenhofer et al. (2017)</td></tr><tr><td>Pillar Networks + SVM-MKL</td><td>72.8</td><td>Sengupta &amp; Qian (2017)</td></tr><tr><td>Pillar Networks + iDT + SVM-MKL</td><td>73.0</td><td>Sengupta &amp; Qian (2017)</td></tr><tr><td>Pillar Networks + MIFS + SVM-MKL</td><td>73.3</td><td>Sengupta &amp; Qian (2017)</td></tr><tr><td>Deep Convolutional Networks + GP-PoE</td><td>73.6</td><td>this paper</td></tr><tr><td colspan="2">Deep Convolutional Networks + iDT + GP-PoE 75.0</td><td>this paper</td></tr><tr><td>networks;</td><td/><td/></tr></table>

, the second stage of training, i.e., the GP classifiers work with far fewer examples than what a deep learning network requires. It would be useful to see how our framework performs on immensely large datasets such as the Youtube-8m data-set (Abu-El-Haija et al., 2016). Additionally, recently published Kinetics human action video dataset from DeepMind(Kay et al., 2017) is equally attractive, as pre-training, our framework on this dataset before fine-grained training on HMDB-51 will invariably increase the accuracy of the current network architecture.

Table content: <table/>

Peer review (iclr2018):
 This paper, although titled "Distributed Non-Parametric Deep and Wide Networks", is mostly about fusion of existing models for action recognition on the HMDB51 dataset. The fusion is performed with Gaussian Processes, where each of the i=1,..,4 inspected models (TSN-Inception RGB, TSN-Inception Flow, ResNet-LSTM RGB, ResNet-LSTM Flow) returns a (.mu_i, .sigma_i), which are then combined in a product of experts formulation, optimized w.r.t. maximum likelihood. At its current form this paper is unfit for submission. First, the novelty of the paper is not clear. It is stated that a framework is introduced for independent deep neural networks. However, this framework, the Gaussian Processes, already exists. Also, it is stated that the method can classify video snippets that have heterogeneity regarding camera angle, video quality, pose, etc. This is something characterizes also all other methods that report similar results on the same dataset. The third claim is that deep networks are combined with non-parameteric Bayesian models. That is a good claim, which is also shared between papers at http://bayesiandeeplearning.org/. The last claim is that model averaging taking into account uncertainty is shown to be useful. That is not true, the only result are the final accuracies per GP model, there is no experiment that directly reports any results regarding uncertainty and its contribution to the final accuracy. Second, it is not clear that the proposed method is the one responsible for the reported improvements in the experiments. Currently, the training set is split into 7 sets, and each set is used to train 4 models, totalling 28 GP experts. It is unclear what new is learned by the 7 GP expert models for the 7 splits. Why is this better than training a single model on the whole dataset? Also, why is difference bigger between ResNet Fusion-1 and Resnet SVM-SingleKernel? Third, the method reports results only on a single dataset, HMDB51, which is also rather small. Deriving conclusions from results on a single dataset is suboptimal. Other datasets that can be considered are (mini) Kinetics or Charades. Forth, the paper does not have the structure of a scientific publication. It rather looks like an unofficial technical report. There is no related work. The methodology section reads more like a tutorial of existing methods. And the discussion section is larger than any other section in the paper. All in all, there might be some interesting ideas in the paper, specifically how to integrate GPs with deep nets. However, at the current stage the submission is not ready for publication.