Title: PARAMETRIZING FULLY CONVOLUTIONAL NETS WITH A SINGLE HIGH-ORDER TENSOR

Abstract: Recent findings indicate that over-parametrization, while crucial to the success of deep learning, also introduces large amounts of redundancy. Tensor methods have the potential to parametrize over-complete representations in a compact manner by leveraging this redundancy. In this paper, we propose fully parametrizing Convolutional Neural Networks (CNNs) with a single, low-rank tensor. Previous works on network tensorization haved focused on parametrizing individual layers (convolutional or fully connected) only, and perform the tensorization layer-by-layer disjointly. In contrast, we propose to jointly capture the full structure of a CNN by parametrizing it with a single, high-order tensor, the modes of which represent each of the architectural design parameters of the CNN (e.g. number of convolutional blocks, depth, number of stacks, input features, etc). This parametrization allows to regularize the whole network and drastically reduce the number of parameters by imposing a low-rank structure on that tensor. Further, our network is end-to-end trainable from scratch, which has been shown to be challenging in prior work. We study the case of networks with rich structure, namely Fully Convolutional CNNs, which we propose to parametrize them with a single 8-dimensional tensor. We show that our approach can achieve superior performance with small compression rates, and attain high compression rates with negligible drop in accuracy for the challenging task of human pose estimation.

1. INTRODUCTION:

State-of-the-art performance is currently attained with Convolutional Neural Networks for a wide range of challenging tasks including recognition (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014; He et al., 2016) , detection (Ren et al., 2015) , semantic segmentation (Long et al., 2015; He et al., 2017) and human pose estimation (Newell et al., 2016) outperforming prior works by a large margin. There is evidence that a key feature behind the success of these methods is overparameterization which helps find good local minima (Du & Lee, 2018; Soltanolkotabi et al., 2018) . However, at the same time, over-parameterization leads to a great amount of redundancy, and from a statistical perspective, it might hinder generalization because it excessively increases the number of network parameters. Furthermore, models with excessive number of parameters have increased storage and computation requirements rendering them problematic for deployment on devices with limited resources. This paper focuses on a novel way of leveraging the redundancy in the parameters of CNNs by tensorizating the whole network.

There is a significant amount of recent work on using tensors to reduce the redundancy and improve the efficiency of CNNs; mostly focusing on reparametrizing individual layers. For example, Tai et al. (2015) ; Kim et al. (2016) treat a convolutional layer as a 4D tensor and then computes a lowrank decomposition of the 4D convolution kernel tensor into a sum of a small number of low rank tensors. Similarly, Novikov et al. (2015) proposed tensorizing the fully-connected layers. The bulk of these methods focus on tensorizing individual layers only, and perform the tensorization layer-bylayer disjointly, usally by applying tensor decomposition to the pretrained weights and fine-tuning to compensate for performance loss. For example Tai et al. (2015) tensorize the second convolutional layer of AlexNet Krizhevsky et al. (2012) .

Our paper primarily differs from prior work by the use of a single tensor to parametrize the whole CNN (as opposed to using different tensors to parametrize the individual layers). In particular, we propose to parametrize the network with a single high-order tensor each dimension of which represents a different architectural design parameter of the CNN. For the case of Fully Convolutional Networks (FCNs) with an encoder-decoder structure considered herein (see Fig. 1 , each dimension of the 8-dimensional tensor represents a different architectural design parameter of the network such as the number of (stacked) FCNs used, the depth of each network, the number of input and output features for each convolutional block and the spatial dimensions of each of the convolutional kernels. By modelling the whole CNN with a single tensor, our approach allows to learn correlations between the different tensor dimensions and hence fully capturing the structure of the network. Moreover, this parametrization allows to regularize the whole network and drastically reduce the number of parameters by imposing a low-rank structure on that tensor. Owing to these properties, our framework is much more general, and flexible compared to prior work offering increased accuracy and high compression rates. In summary the contributions of this work are:

• We are the first to propose using a single high-order tensor for whole network tensorization, and applying it for capturing the rich structure of Fully Convolutional Networks. Our approach allows for a wide spectrum of network decompositions and compression rates which can be chosen and optimized for a particular application. Further, our network can be trained end-to-end -not only by fine-tuning, by also from scratch, which has been shown to be challenging in prior work. • We show that, for a wide spectrum of compression rates (both high and low), our method preserves high accuracy. Compared to prior work based on tensorizing individual convolutional layers, our method consistently achieves higher accuracy, especially for the case of high compression rates. In addition, we show that, for lower compression rates, our method outperforms the original uncompressed network. • We illustrate the favorable properties of our method by performing a large number of experiments and ablation studies for the challenging computer vision task of human pose estimation. The experiments sheds light on several interesting properties of our method including the effect of varying the rank for each mode of the tensor, as well as the decomposition method used. Further, we validate some of our conclusions by conducting experiments for a different dense prediction task, namely facial part segmentation.

1.1. RELATED WORK:

Tensor methods offer a natural extension of traditional algebraic methods to higher orders. For instance, Tucker decomposition can be seen as a generalization of PCA to higher dimensions (Kolda & Bader, 2009) . Tensor decompositions have large reaching applications, including for learning a wide range of probabilistic latent-variable models (Anandkumar et al., 2014) . Tensor methods have recently been applied to deep learning, for instance to provide a theoretical analysis of deep neural nets (Cohen et al., 2015) . New layers were also proposed, leveraging tensor methods. Kossaifi et al. (2017) propose tensor contraction layers to reduce the dimensionality of activation tensors while preserving their multilinear structure. Tensor regression layers (Kossaifi et al., 2018a) express outputs through a low-rank multilinear mapping from a high-order activation tensor to an output tensor of arbitrary order.

A lot of the existing work has been dedicated to leveraging tensor decompositions in order to reparametrizing existing layers, either to speed up computation or to reduce the number of parameters. Separable convolutions, for instance, can be obtained from existing ones by applying CP decomposition to their kernel. The authors in Lebedev et al. (2014) propose such parametrization of individual convolutional layers using CP decomposition with the goal of speeding them up. Specifically, each of the 4D tensors parametrizing the convolutional layers of a pretrained network are decomposed into a sum of rank-1 tensors using CP decomposition. The resulting factors are used to replace each existing convolutions with a series of 4 convolutional layers with smaller kernels. The network is then fine-tuned to restore the performance. Kim et al. (2016) propose a similar approach but use Tucker decomposition instead of CP to decompose the convolutional layers of a pretrained network, before fine-tuning to restore the performance. Specifically, Tucker decomposition is applied to each convolutional kernel of a pretrained network, on two of the modes (input and output channel modes). The resulting network is fine-tuned to compensate for the drop in performance induced by the compression. Astrid & Lee (2017) also reparametrize the layers of deep convolutional neural networks using CP decomposition, but optimize this using the tensor power method. The network is then iteratively fine-tuned to restore performance. Similarly, Tai et al. (2015) propose to use tensor decomposition to remove redundancy in convolutional layers and express these as the composition of two convolutional layers with less parameters. Each 2D filter is approximated by a sum of rank-1 matrices. Thanks to this restricted setting, a closed-form solution can be readily obtained with SVD. This is done for each convolutional layer with a kernel of size larger than 1 × 1. Layers other than convolutional can be also parametrized. For instance, Novikov et al. (2015) use the Tensor-Train (TT) format to impose low-rank tensor structure on the weights of the fully-connected layers. Tensorization of generative adversarial networks (Cao & Zhao, 2017) and sequence models (Yu et al., 2017) have also been proposed.

Tensor decomposition has also been used for weight sharing between layers and multi-task learning. Chen et al. (2017) proposes the so-called collective residual units (CRU), which are obtained by stacking the convolutional kernels of several residual units sharing the same size along the last mode (corresponding to the number of output channels) to obtain a 4 th order tensor f size (width (= 3), height (= 3), input channels, L× output channels). Generalized block term decomposition (shortly put, a sum of Tucker decompositions), is used to decompose the last two modes of the resulting tensors along these dimensions. These CRUs are stacked to form the final network. The sharing is done for 6 residual units only, and each of the CRUs is parametrized individually. Yang & Hospedales (2016) leverage tensor decomposition for multi-task learning, to allow for weight sharing between the fully-connected and convolutional layers of two or more deep neural networks.

However, to the best of our knowledge, our work is the first that proposes an end-to-end trainable architecture, fully parametrized by a single, high order, low-rank tensor.

Human pose estimation. CNN-based methods have recently produced results of remarkable accuracy for the task of human pose estimation, outperforming traditional methods by large margin (Toshev & Szegedy, 2014; Tompson et al., 2014; Pfister et al., 2015; Bulat & Tzimiropoulos, 2016; Newell et al., 2016; Wei et al., 2016) . Arguably, one of the most widely used architectures for this task is the stacked Hourglass (HG) network proposed by Newell et al. (2016) . An HG is an encoder-decoder network with skip connections between the encoder and the decoder, suitable for making predictions at a pixel level in a fully convolutional manner. Newell et al. (2016) use a stack of eight of these networks to achieve state-of-the-art performance on the MPII dataset (Andriluka et al., 2014) . In this work, we choose tensorizing the HG network primarily because of its rich structure which makes it suitable to model it with a high-order tensor. We note that the aim of this work is not to produce state-of-the-art results for the task of human pose estimation but to show the benefits of modelling a state-of-the-art architecture with a single high-order tensor.

2. METHODOLOGY:

In this section we first introduce some mathematical background in subsection 2.1, before introducing our method in subsection 2.2.

2.1. MATHEMATICAL BACKGROUND:

Notation: We denote v, vectors (1 st order tensors), M matrices (2 nd order tensors) and X tensors of order 3 or greater. We denote element

(i 0 , i 1 , • • • , i N ) of a tensor as Xi0,i1,••• ,i N or X (i 0 , i 1 , • • • , i N ).

A colon is used to denote all elements of a mode e.g. the mode-1 fibers of X are denoted as X (: 2016), containing 2 convolutional layers with 3 × 3 kernels followed by BatchNorm and ReLU. For all of our experiments, unless explicitly stated otherwise, we used a stack of 4 encoder-decoder sub-networks.

, i 2 , i 3 , • • • , i N ). Id is the identity matrix. Finally, for any i, j ∈ N, [i . . j] denotes the set of integers {i, i + 1, • • • , j -1, j}. Mode-n unfolding of a tensor X ∈ R I0×I1×•••×I N , is a matrix X [n] ∈ R In,I M , with M = N k=0, k =n I k , defined by the mapping from element (i 0 , i 1 , • • • , i N ) to (i n , j), with j = N k=0, k =n i k × N m=k+1, m =n I m . Tensor vectorization of a tensor X ∈ R I0×I1×•••×I N , is a vector vec( X ) of size (I 0 × • • • × I N ) defined by the mapping from element (i 0 , i 1 , • • • , i N ) of X to element j of vec( X ), with j = N k=0 i k × N m=k+1 I m .

Mode-n product: for a tensor X ∈ R I0×I1×•••×I N and a matrix M ∈ R R×In , the n-mode product of a tensor is a tensor of size

(I 0 × • • • × I n-1 × R × I n+1 × • • • × I N )

and can be expressed using the unfolding of X and the classical dot product as

X × n M = M X[n] ∈ R I0×•••×In-1×R×In+1×•••×I N

Tensor diagrams: given the high order of the tensor manipulated, it is impractical to represent them graphically. We instead use tensor diagrams, where tensors are represented by circles (vertices), and their order by edges from originating from this circle. Tensor contraction over two modes is then represented by simply linking together the two edges corresponding to these two modes. Figure 2 shows the Tucker decomposition of the 8 th order weight tensor with tensor diagrams.

2.2. FULLY-TENSORIZED ARCHITECTURE:

In this section, we describe how to tensorize the stacked hourglass architecture of Newell et al. (2016) . The hourglass has a number of design parameters namely the number of (stacked) HGs, the depth of each HG, the three signal pathways of each HG (skip, downsample and upsample), the number of convolutional blocks, the number of input and output features for each convolutional block and finally the spatial dimensions of each of the convolutional kernels. To enable the tensorization of the whole network, we used a modified HG architecture in which we replaced all the residual modules with the basic block introduced by He et al. ( 2016), maintaining the same number of input and output channels throughout the network. We made the encoder and the decoder symmetric, with 4 residual modules each. Figure 1 illustrates the modified HG architecture.

From the network described above, we derive the proposed Tensorized-Network (T-Net) as follows: all weights are parametrized by a single 8 th order tensor W ∈ R I0×I1×•••×I7 , the modes of which correspond to the number of HGs (I 0 = #hg), the depth of each HG (I 1 = hg depth ), the three signal pathways (I 2 = hg subnet ), the number of convolutional blocks (I 3 = #blocks), the number of input features (I 4 = f in ), the number of output features (I 5 = f out ), and finally the height (I 6 = h) and width (I 7 = w) of each of the convolutional kernels.

Tucker form: Given the weight tensor W ∈ R I0×I1×•••×I7 , we can express it as a rank-

(R 0 , • • • , R 7 ) Tucker tensor, composed of a low rank core G ∈ R R0×R1×•••×R7 along with pro- jection factors U (0) , • • • , U (7) , with U (k) ∈ R R k ,I k , k ∈ (0, • • • , 7

). This allows us to write the weight tensor in a decomposed form as: 7) See also Fig. 2 for a tensor diagram of the tucker form of the weight tensor.

W = G × 0 U (0) × 1 U (2) × • • • × 7 U (

In this work, we either initialize both the factors and the core randomly, when training from scratch, or apply Tucker decomposition to the full weight tensor for fine-tuning.

Matrix-Product-State (MPS) form also known as tensor-train (Oseledets, 2011) , the Matrix-Product-State expresses a tensor as a series of third order tensors (the cores) and allows for especially large space-savings. In our case, given the weight tensor

W ∈ R I0×I1×•••×I7 , we can decompose it into a rank R 0 , R 1 , • • • R 8 -MPS as a series of third order cores G0 ∈ R R0,I0,R1 , G0 ∈ R R1,I1,R2 , • • • , G0 ∈ R R7,I7,R8 . The boundary conditions dictate R 0 = R 8 = 1. In term of individ- ual elements, we can write, for i 0 ∈ [0 . . I 0 ], i 1 ∈ [0 . . I 1 ], • • • , i 7 ∈ [0 . . I 7 ]: W(i 0 , i 1 , • • • , i 7 ) = G0 [i 0 ] 1×R0 × G1 [i 1 ] R0×R1 × • • • × G7 [i 7 ] R6×1

Parameter analysis: Considering a Tucker rank-R 0 , R 1 , • • • , R 7 of the weight tensor parametrizing the whole network, the corresponding number of parameters is:

N T -N et = 7 k=0 R k + 7 k=0 R k × I k .

Compressing each of the N conv convolutional layer separately (Kim et al., 2016) , with a rank R 4 and R 5 for the number of input and output features, respectively, we obtain a number of parameters of:

N conv × (R 4 × R 5 × I 6 × I 7 + R 4 × I 4 + R 5 I 5 ), with N conv = 4 k=0 I k .

In comparison, our model, with the same ranks R 4 and R 5 imposed on the number of features, would only have

N conv × (R 4 × R 5 × 3 × 3) + R 4 × I 4 + R 5 I 5 parameters.

In other words, our model has

4 k=0 I k -1 (R 4 × I 4 + R 5 I 5

) parameters less than a corresponding layer-wise decomposition.

Speeding up the convolutions: convolutions expressed in CP or Tucker format, can be replaced by a series of convolutions with smaller kernels (Lebedev et al., 2014; Kim et al., 2016) , thus allowing for large computational speedups. This also applies to our model as, given a weight tensor W ∈ R I0×I1×•••×I7 , we can write, using the Tucker form, 7) . Each element W(i 0 , i 1 , i 2 , i 3 , :, :, :) corresponds to one of the convolutional kernels. Let's denote K = W(i 0 , i 1 , i 2 , i 3 , :, :, :) one such kernel. By leaving the last two dimensions untouched (U (6) = U (7) = Id) and re-arranging the terms, we can write: ,I5,I6,I7) . This gives us an effective way of approximating each convolution by three, smaller convolutions (Kim et al., 2016) .

W = G × 0 U (0) × 1 U (2) × • • • × 7 U (

K(s, t, j, k) = R4 r4=0 R5 r5=0 C(r 4 , r 5 , j, k)U (4) (s, r 3 )U (5) (s, r 4 ) with C = G × 0 U (0) × 1 U (2) × • • • × 3 (i 0 , i 1 , i 2 , i 3 , :, :, :, :) ∈ R (I4

3. EXPERIMENTAL SETUP:

The bulk of our experiments are conducted for the task of human pose estimation. We also validate some of our conclusions by conducting experiments for a different dense prediction task, namely facial part segmentation.

Human Pose Estimation. Following Tompson et al. (2014) , throughout this work, we conduct experiments using the standard train and validation split of one of the most challenging single pose human pose estimation datasets, namely MPII (Andriluka et al., 2014) . The dataset contains 22,000 images for training and another 3,000 for validation.

Semantic Facial Part Segmentation. We constructed the facial part segmentation dataset similarly to (Bulat & Tzimiropoulos, 2017) : for training, we used the 300W training dataset (more than 3,000 images) and for testing the whole 300W competition test set (600 images) Sagonas et al. (2013) .

Implementation details We used a stacked HG architecture with the following architectural parameters: #hg = 4, hg depth = 4, hg subnet =3, #blocks = 2, f in =128, f out =128, and h = w = 3. This resulted in a 8 th -order tensor of size 4 × 4 × 3 × 2 × 128 × 128 × 3 × 3. The network is composed of a total of 15, 830, 976 parameters (1, 675, 200 for the base convolutions and 14, 555, 776 for the weight tensor parametrizing the convolutional blocks).

For the uncompressed baseline network, we reduce the number of its parameters by simple decreasing the number of channels in each residual block, varying from 128 to 64. By doing so, (as opposed to reducing the number of stacks), we maintain all the architectural advantages offered by the stacked HG architecture and ensure a fair comparison with the tensorized network.

Training All models were trained for 110 epochs using RMSprop (Tieleman & Hinton, 2012) . The learning rate was varied from from 2.5e -4 to 1e -6 using a Multi-Step fixed scheduler. During training, we randomly augment the data using: rotation(-25 • to 25 • for human pose and -40 • to 40 • for face parsing), scale distortion (0.75 to 1.25), horizontal flipping and color jittering.

All experiments were run on a single NVIDIA TITAN V GPU. All networks were implemented using PyTorch (Paszke et al., 2017) , and TensorLy (Kossaifi et al., 2018b) was used for all tensor operations.

Metrics For the human pose estimation experiments, we report accuracy using the PCKh (Andriluka et al., 2014) . For facial part segmentation, we report segmentation accuracy using the mean accuracy and mIO metrics (Long et al., 2015) . Finally, we measure the compression of the number of parameters using the compression ratio = uncompressed compressed , defined as the total number of parameters of the uncompressed network, divided by the number of parameters of the compressed network.

4. RESULTS:

This section offers an in-depth analysis of the performance and accuracy of the proposed T-Net. Our main results are that the proposed approach: i) Outperforms the layer-wise decomposition of Kim et al. (2016) , which is the most closely related work to our method; ii) Outperforms the uncompressed, original network for low compression rates; iii) Offers consistent compression ratios across arbitrary dimensions. Finally, we further validate some of these results for the task of semantic facial part segmentation.

All results reported are obtained by fine-tuning our networks in an end-to-end manner from a pretrained uncompressed original network. We were able to reach the same level of accuracy when training from scratch, though this required training for more iterations. In contrast, we found that when trained from scratch, the layer-wise method of Kim et al. (2016) reaches sub-par performance, as also reported in their paper.

In order to better understand the compressibility of each mode of the weight tensor, we first investigate the redundancy of each of the modes of the tensor by compressing only one of the modes at a time. (Kim et al., 2016) Tucker-(96, 96, 3, 3) 1.32x 86.85% (Kim et al., 2016) Tucker-(64, 64, 3, 3) 2.33x 86.39% (Kim et al., 2016) Tucker-(32, 32, 3, 3) 4.73x 85.33% (Kim et al., 2016) Tucker-(16, 16, 3, 3) 6.91x 83.74%

T-Net [Ours]:

Tucker -(4, 3, 3, 2, 110, 110, 3, 3) 1.66x

87.45% T-Net [Ours]:

Tucker -(4, 4, 2, 2, 110, 110, 3, 3) 1.82x

87.36% T-Net [Ours]:

Tucker- (3, 3, 3, 2, 110, 110, 2, 2) 3.67x

87.09% T-Net [Ours]:

Tucker -(3, 2, 3, 2, 96, 96, 3, 3) 3.38x

86.73% T-Net [Ours]:

Tucker -(3, 3, 2, 2, 80, 80, 3, 3 one of the modes, corresponding respectively to the number of HGs (#hg), the depth of each HG (hg depth ), the three pathways of each HG (hg subnet ), the number of convolutional blocks (#blocks) and finally,the number of input features (f in ), output features (f out ), height (h) and the width (w) of each of the convolutional kernels. These results are shown along with the performance of the original uncompressed network. We observe that by taking advantage of the redundancy at the network-level (as opposed to (Lebedev et al., 2014; Kim et al., 2016) which compress individual layers), the proposed approach is able to efficiently compress across arbitrary dimensions, for large compression ratios while maintaining similar or even in some cases, higher accuracy than that of the original uncompressed network. Based on the insights gained from the previous experiment, we select promising configurations and compress over multiple dimensions simultaneously. The results can be seen in Table 2 along with (a) the performance of the original network "compressed" by just reducing the number of feature dimensions and (b) the layer-wise decomposition method of Kim et al. (2016) . We firstly observe that by just reducing the number of feature dimensions in the original network, a significant drop in performance can be noticed. Secondly, our method consistently outperforms (Kim et al., 2016) across the whole spectrum of compression ratios. This can be seen by comparing the accuracy provided for any compression ratio for (Kim et al., 2016) with the accuracy of the closest but higher compression ratio for our method (for example, compare 2.33x for (Kim et al., 2016) with 3.67x for our method). It can be seen that our method always achieves higher accuracy even the compression ratio is also higher. In addition, contrary to (Kim et al., 2016) which does not seem to work well when the size of the convolutional kernel is compressed from 3 × 3 to 2 × 2, our method is able to compress that dimension while maintaining similar level of performance. In the same table, we also report the performance of a variant of our method, using an MPS decomposition on the weights rather than a Tucker one. This result shows that our method works effectively with other decomposition methods as well. However we focused mainly on Tucker as it is the most flexible compression method, allowing us to control the rank of each mode of the weight tensor.

Finally, we select two of our best performing models and retrain them on semantic facial part segmentation. Our method offers significant compression ratios (up to 6x) with virtually no loss in accuracy (see Table 3 ). These results further confirm that our method is task-independent. Comparison between our T-Net and a network with the same architecture and the same number of features as the compressed one. Our approach is capable of retaining a high accuracy even at high compression rates (up to 7x).

Method:



5. CONCLUSIONS:

We proposed a novel method to jointly capture the full structure of a new Fully Convolutional Network by parametrizing it with a single, high-order tensor. The modes of this tensor represent each of the architectural design parameters of the network (e.g. number of convolutional blocks, depth, number of stacks, input features, etc). This parametrization allows to jointly regularize the whole network and drastically reduce the number of parameters by imposing a low-rank structure on that tensor. Further, our network can be trained in an end-to-end manner not only during finetuning but also from scratch which has been shown to be challenging in prior work. We showed that our approach can achieve superior performance with small compression rates, and attain high compression rates with negligible drop in accuracy, on both the challenging task of human pose estimation and face segmentation.

:



Figures and Tables:

Figure 1: Overall network architecture. Each block in the fully convolutional network is a basicblock module (blue insert), as introduced by He et al. (2016), containing 2 convolutional layers with 3 × 3 kernels followed by BatchNorm and ReLU. For all of our experiments, unless explicitly stated otherwise, we used a stack of 4 encoder-decoder sub-networks.

Figure 2: Tensor diagram of the Tucker form of the weight tensor W.

Figure 3: Tensor diagram of the MPS/TTrain form of the weight tensor W. Note the train-like shape from which the method takes its name, as well as the boundary conditions (R 0 = R 7 = 1).

Figure 4: Qualitative results produced by our method on MPII.

Figure 5: Qualitative results produced by our method on the facial part segmentation task.

Table1shows the accuracy (PCKh) as well as the compression ratio obtained by compressing Human pose estimation task. Study of the redundancy of each of the modes of the parameter tensor. We compress one dimension at a time by reducing its corresponding rank in the Tucker-tensor. Reported accuracy is in terms of PCKh.

Table content: <table><tr><td/><td/><td/><td>Tucker-rank</td><td/><td/><td colspan="2">Accuracy Compression</td></tr><tr><td colspan="5">#hg hg depth hg subnet #blocks f in</td><td>f out h w</td><td>(PCKh)</td><td>ratio</td></tr><tr><td/><td/><td/><td>Original</td><td/><td/><td>86.99%</td><td>1.0x</td></tr><tr><td>3</td><td>4</td><td>3</td><td>2</td><td colspan="2">128 128 3 3</td><td>87.42%</td><td>1.28x</td></tr><tr><td>2</td><td>4</td><td>3</td><td>2</td><td colspan="2">128 128 3 3</td><td>86.95%</td><td>1.82x</td></tr><tr><td>1</td><td>4</td><td>3</td><td>2</td><td colspan="2">128 128 3 3</td><td>86.05%</td><td>3.03x</td></tr><tr><td>4</td><td>3</td><td>3</td><td>2</td><td colspan="2">128 128 3 3</td><td>87.71%</td><td>1.28x</td></tr><tr><td>4</td><td>2</td><td>3</td><td>2</td><td colspan="2">128 128 3 3</td><td>87.59%</td><td>1.82x</td></tr><tr><td>4</td><td>1</td><td>3</td><td>2</td><td colspan="2">128 128 3 3</td><td>86.89%</td><td>3.03x</td></tr><tr><td>4</td><td>4</td><td>2</td><td>2</td><td colspan="2">128 128 3 3</td><td>87.53%</td><td>1.43x</td></tr><tr><td>4</td><td>4</td><td>1</td><td>2</td><td colspan="2">128 128 3 3</td><td>86.19%</td><td>2.50x</td></tr><tr><td>4</td><td>4</td><td>3</td><td>1</td><td colspan="2">128 128 3 3</td><td>82.59%</td><td>1.82x</td></tr><tr><td>4</td><td>4</td><td>3</td><td>2</td><td>96</td><td>96 3 3</td><td>87.43%</td><td>1.64x</td></tr><tr><td>4</td><td>4</td><td>3</td><td>2</td><td>64</td><td>64 3 3</td><td>86.13%</td><td>3.03x</td></tr><tr><td>4</td><td>4</td><td>3</td><td>2</td><td>32</td><td>32 3 3</td><td>83.10%</td><td>6.25x</td></tr><tr><td>4</td><td>4</td><td>3</td><td>2</td><td colspan="2">128 128 2 2</td><td>87.30%</td><td>1.98x</td></tr><tr><td/><td>Method</td><td/><td colspan="2">Parameters</td><td colspan="3">Compression ratio Accuracy</td></tr><tr><td/><td>Original</td><td/><td colspan="2">full, f in =f out =128</td><td/><td>1x</td><td>86.99%</td></tr><tr><td/><td>Original</td><td/><td colspan="2">f in =f out =112</td><td/><td>1.32x</td><td>86.87%</td></tr><tr><td/><td>Original</td><td/><td colspan="2">f in =f out =92</td><td/><td>1.97x</td><td>85.92%</td></tr><tr><td/><td>Original</td><td/><td colspan="2">f in =f out =64</td><td/><td>3.99x</td><td>84.54%</td></tr><tr><td colspan="2">(Kim et al., 2016)</td><td/><td colspan="3">Tucker-(128, 128, 2, 2)</td><td>1.42x</td><td>84.86%</td></tr></table>

Human pose estimation task. Comparison between our T-Net and (a) the original network "compressed" by reducing the number of features and (b) the layer-wise decomposition method ofKim et al. (2016). Reported accuracy is in terms of PCKh.

Table content: <table><tr><td>)</td><td>4.20x</td><td>86.27%</td></tr></table>

Segmentation task.

Table content: <table><tr><td/><td>Parameters</td><td>Compression ratio</td><td>mIO</td><td>mAcc</td></tr><tr><td>Uncompressed</td><td>full, f in =f out =128</td><td>1x</td><td colspan="2">76.02% 97.31%</td></tr><tr><td>Tucker</td><td>rank-(3, 2, 3, 2, 96, 96, 3, 3)</td><td>3.38x</td><td colspan="2">76.01% 97.29%</td></tr><tr><td>Tucker</td><td>rank-(2, 2, 2, 2, 64, 64, 3, 3)</td><td>6.94x</td><td colspan="2">75.57% 97.01%</td></tr></table>

Peer review (iclr2019):
 In this paper, the authors propose to organize all parameters of a multilayer neural network in a higher order tensor (8 dimension) and use low-rank tensor decomposition models (Tucker and TT/MPS formats) to compress it. They applied this approach to a very specific architecture, the stacked hourglass architecture of Newell et all (2016) full convolutional neural network and used in the specific problem of human pose estimation. The idea of the paper is simple and technically sounded but unfortunately, I found it not well presented with many important technical details missing. Also, there are not any theoretical justification for the method to work, the results are restricted to a very specific network architecture and only one application, which limits their generalization to a broader class of problems. Below, I provide the main comments about the paper: Major issues: - The paper claims that, for lower compression rates, the proposed method outperforms the uncompressed version of the network. What is the justification for this behavior? Is it related to a regularization effect? How this effect depends on the number of available samples? I think the authors should provide some theoretical insights about this behavior or, at least, a deeper experimental study. - The authors do not give details on how the tensorized network is optimized/trained. They state that “All models were trained for 110 epochs using RMSprop (Tieleman & Hinton, 2012). The learning rate was varied from from 2.5e-4 to 1e-6 using a Multi-Step fixed scheduler.” But It is not clear if a back-propagation strategy can be still used after tensorization and what are the updating rules for the parameters in the tensor decomposition model. I think this missing information must be included in the paper, at least as an appendix. - The title of section 2.2, “Fully-tensorized architecture” suggests that all design parameters of the network are compressed through a tensor decomposition, however, in the implementation details section, they state that total number of parameters is 15,830,976 where 14,555,776 are in the weights tensor and 1,675,200 (based convolutions) are not considered as part of the tensor decomposition model. It is not clear, why this particular tensorization approach is used. - They compared the results of the compressed approach against an uncompressed architecture but reducing the number of parameters by choosing 64 channels of residual blocks instead of 128 as used in the compressed version. I think this difference in number of parameters prevents from obtaining a fair comparison of the results. Why the authors have chosen fewer parameters in the uncompressed version? If it is because there is a memory size limitation, I would suggest to use 64 channels in the compressed version too, so the results can be fairly compared. Minor comments: - Tensor notation is a bit strange in this paper. Tensors are usually denoted by capital calligraphic letters or underlined bold capital letters but in this work the authors use calligraphic with tilde, which makes the notation a little bit overcrowded.