Title: ADAPTIVE MIXTURE OF LOW-RANK FACTORIZATIONS FOR COMPACT NEURAL MODELING

Abstract: Modern deep neural networks have a large amount of weights, which make them difficult to deploy on computation constrained devices such as mobile phones. One common approach to reduce the model size and computational cost is to use lowrank factorization to approximate a weight matrix. However, performing standard low-rank factorization with a small rank can hurt the model expressiveness and significantly decrease the performance. In this work, we propose to use a mixture of multiple low-rank factorizations to model a large weight matrix, and the mixture coefficients are computed dynamically depending on its input. We demonstrate the effectiveness of the proposed approach on both language modeling and image classification tasks. Experiments show that our method not only improves the computation efficiency but also maintains (sometimes outperforms) its accuracy compared with the full-rank counterparts.

1. INTRODUCTION:

Modern neural networks usually contain millions of parameters (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014) , and they are difficult to be deployed on mobile devices with limited computation resources. To solve this problem, model compression techniques are proposed in recent years. For example, (Wu et al., 2018; Li et al., 2016; Han et al., 2015a) try to limit the weights (and activations) of neural networks to lower precisions by quantization. This can save the model size by 4 to 16 times. While quantization can reduce the number of bits per weight, it cannot reduce the number of weights. To reduce this redundancy, (Han et al., 2015b; Ullrich et al., 2017; Louizos et al., 2017) propose pruning the weight matrices, leading to sparse neural networks that require less computation. However, sparse neural networks often require specialized ASIC or FPGA to accelerate (Han et al., 2016; 2017) .

Alternatively, low-rank factorization is a popular way of reducing the matrix size. It has been extensively explored in the literature (Lu et al., 2016; Nakkiran et al., 2015; Jaderberg et al., 2014; Yu et al., 2017) . Mathematically, a large weight matrix W ∈ R m×n is factorized to two small rank-d matrices U ∈ R m×d , V ∈ R n×d with W = U V T . Since both U and V are dense, no sparsity support is required from specialized hardware. It naturally fits the general-purpose, off-the-shelf CPUs and GPUs.

To significantly reduce the model size and computation, the rank d in the low-rank factorization needs to be small. However, a small rank can limit the expressiveness of the model (Yang et al., 2018) and lead to worse performance. To understand the limitations, given a n-dim feature vector h, we observe that V T h, as in U (V T h), is a linear projection from a high-dimensional space (n dims) to a low-dimensional space (d dims) . This can lead to a significant loss of information. The conflict between the rank d and the model expressiveness prevents us from obtaining a both compact and accurate model.

To address the dilemma, we propose to increase the expressiveness by learning an adaptive, inputdependent factorization, rather than performing a fixed factorization of a weight matrix. To do so, we use a mixture of multiple low-rank factorizations. The mixing weights are computed based on the input. This creates an adaptive linear projection from a high-dimensional space to a low-dimensional space. Compared to the conventional low-rank factorization, the proposed approach can significantly improve its performance while only introducing a small additional cost. c) are distributions of 2D data in the 1D projected space. The linear projection to lower dimension leads to significant information loss (results in 83% classification accuracy), while our proposed approach learns to adaptively avoid this (achieving 97% classification accuracy). (c) is the distribution of projection through a random matrix followed by tanh.

To demonstrate the effectiveness of adaptive low-rank factorization, we experiment with both recurrent and convolutional neural networks on language modeling and image classification tasks. Experimental results on both tasks show that our method consistently improves upon conventional low-rank factorization. On the Penn Tree Bank dataset, we achieved 40% reduction in FLOPs, and 1.7 better perplexity than the full rank baseline LSTM for language modeling. On ImageNet dataset, we use 48% less computation, 12% less parameters, but achieve 2.5% better Top-1 accuracy than MobileNet-V1 (Howard et al., 2017) . Compared to MobileNet-V2 which utilizes a standard low-rank bottleneck structure, our proposed method achieves 1.95% better Top-1 accuracy with less than 1% extra FLOPs, which is significant given that MobileNet-V2 is already very compact.

2. LOW-RANK FACTORIZATION AND THE LINEAR BOTTLENECK:

A common linear transformation between two spaces can be represented by a linear function F : R n → R m , F(h; W ) = W h where W ∈ R m×n and h ∈ R n . To reduce the size and computation of this linear transformation, a low-rank factorization of W , i.e. W = U V , can be applied, where U ∈ R m×d , V ∈ R n×d , and d < min(m, n). With this factorization, we can compute the transformation with h = U V h. This reduces computations from O(mn) to O((m + n)d). In the context of neural networks where W represents a weight matrix for a layer, both U and V can be learned using gradient-based algorithms.

From the model compression perspective, we want to minimize the rank d, since that relates to smaller model size and computation1 . However, the expressiveness of the transformation is limited by the rank d. By applying the factorized transformation h = U V h in the reverse order, i.e. h = U (V h), we observe that the first transformation for h is to project it from a high-dimensional to a low-dimensional space since d < n. The latent feature distribution in high dimensional space may be either high dimensional, or lie on a non-linear manifold. In either case, projecting it into the low-dimensional space can lead to significant information loss if d is small. This is less appealing for preserving information for latter layers.

To demonstrate the expressiveness issue, we conduct a toy classification task in 2D spaces. We first generate a 2D dataset with XNOR labels (two diagonal blocks are labeled with the same class) 1a. A non-linear classifier with one hidden layer is trained to predict the output probability by P (y|x) = softmax(W 2 σ(W 1 x)) where W 1 ∈ R 2×2 , and we factorize W 1 using two 2 × 1 matrices, i.e. W 1 = U V T . Since the rank d = 1, the 2D data points are first projected into 1D space and then projected into class probabilities. The visualization of one-dim space is shown in Figure 1b . We observe a significant amount of previously separated data points are now overlapped, and the class information is lost. We attribute this loss of information to the linear bottleneck.

This limitation cannot be solved by adding non-linear activation at the bottleneck (after the linear transformation), since the information is already lost before the application of the non-linearity. Even worse, adding lossy non-linearity to a low-dimensional manifold will further negatively impact the network's performance, as pointed out in (Howard et al., 2017) . Therefore, a new approach to boost the expressiveness of the linear bottleneck without much overhead is in demand.

3. ADAPTIVE MIXTURE OF LOW-RANK FACTORIZATIONS:

To overcome the linear bottleneck in the low-rank factorization approach presented above, we propose to use an unnormalized learned mixture of low-rank factorizations whose mixing weights are computed adaptively based on the input. More specifically, denoting the input by h and the number of mixture components by K, we decompose a large weight matrix by

EQUATION

where π(•) : R n → R K is the function which maps each input to its mixture coefficients, and

U (k) ∈ R m×d/K , V (k) ∈ R n×d/K

. For example, π can be a small neural network. This introduces a small amount of extra parameters and computation. We will later discuss the details of efficient ways to implement the mixture function π.

If π k , k = 1, ..., K, is chosen to be constant (input independent), it can be absorbed into either U (k) or V (k) . Thus, the proposed method reduces to the low-rank factorization. This is evidenced by rewriting 1) , ..., V (K) ] . In other words, the conventional low-rank factorization can be considered as a special case of our method.

EQUATION

Adaptive mixing weights π(h). The mixing weights can encode important information that we can use to increase the expressiveness of the projected low-dimensional space. Under our framework, the generation of the mixing weights π(h) is flexible. A straight-forward approach is to use a non-linear transformation of the input to the weight matrix. For example, π(h) = σ(P h), where σ is a non-linear transformation, such as sigmoid or hyperbolic tangent function, and P ∈ R K×n is an extra weight matrix. This adds some extra parameters and computation to the model since the linear projection that we construct is R n → R K . To further reduce the parameter and computation in the mixing weights π, we propose the following strategies.

Pooling before projection. We do not require the whole input to compute the mixture function π. Instead, we can apply pooling to the input h before projection. For example, a global average pooling can be applied if the input is a 3D tensor (for images); for a 1D vector, we can segment the vector and average each segmentations. By applying pooling, we can both save the computation and better capture the global information.

Random projection. To reduce the number of parameters in the linear projection of h, we can use a random matrix P random in place of a fully adjustable P , i.e. π(h) = σ(P random h). Although we cannot control freely the information captured by a random matrix, hopefully the linear projection induced by U and V can adaptively learn features that are complementary to the mixture weights. Note that we can simply save a seed to recover the random matrix, but it still requires the same amount of memory and computation as the fully adjustable linear projection of h.

Increased expressiveness. Due to the use of data-dependent mixing weights for multiple lowrank factorizations, we expect the expressiveness of the model to increase. To demonstrate this intuition, in the toy example of Figure 1 , the distribution of adopting our approach that computes the mixing weights from a random matrix P and tanh non-linearity is shown in Figure 1c . Compared to conventional linear bottleneck in Figure 1b , we see a better separation among data between the two classes, leading to improved classification accuracy (97% vs. 83%). This is not surprising since the mixing weights encode certain class distribution in 2D space and augments the linear projection. The original overlapped projection is now better separated.

More precisely, the adaptive mixing weights introduce a non-linear transformation into the highto-low-dimensional projection that can be more expressive. Since each W (h) is a data-dependent low-rank matrix, there is no constant linear weight independent to the input (even a full-rank matrix) that can mimic the transformation W (h) induced by our proposed method. It is worth noting that generating the whole weight matrices can be very expensive. Our method can be seen as a swift 

z k = π k (h)((V (k) ) T h) and then h = k U (k) z k

, where z can be treated as the middle layer. Techniques like pooling can be applied to compute π so it does not induce much extra parameters and costs.

approach to generate the weights by adaptively adjusting mixing weights for the linear bottleneck. It assigns weights into groups and dynamically controls them at the group level, as demonstrated in Figure 2 .

To efficiently compute the whole linear transformation K k=1 π k (h)U (k) V (k) h, we use the reverse order, i.e. first computing the linear projection into low-dimensional space with mixing weights, i.e. z k = π k (h)((V (k) ) T h), and then map to a higher dimensional space, i.e. h = k U (k) z k . This reduces the FLOPs and also avoids to store different weight matrices W (h) for different training examples in a mini-batch. An illustration of the computation framework is presented in Figure 3 . Compared to original low-rank factorization, extra parameters and computation cost are from the mixing weights. They can be very small with techniques like pooling aforementioned. We also introduce a way in appendix to avoid breaking a bulk matrix multiplication to segments.

4. EXPERIMENTS:

In this section, we first showcase the linear bottleneck in MNIST with multi-layer perceptron (MLP), and demonstrate how our proposed method improves upon the regular low-rank factorization. Then we conduct extensive experiments on both recurrent neural networks for language modeling and convolutional neural networks for image recognition on ImageNet.

4.1. ADAPTIVE LOW-RANK FACTORIZATION FOR MLP:

In this experiment, we construct a MLP for digit recognition using MNIST dataset. We use a simple one-layer MLP of 300 hidden units (whose input and output sizes are 784 and 10, respectively), and it can be written as P (y|x) = softmax(W 2 σ(W 1 x)). We factorize W 1 ∈ R 784×300 with a rank-d matrix. We use d = 2 in this case to better expose the issue of linear bottleneck and better visualize the latent data distribution. We also set the number of mixture K = 2. To compute mixing weights, we first reduce x ∈ R 784 to R 28 with a segment-based mean pooling, so that the extra parameters is of dimension 28 × 2 and computations only accounts for a small amount (< 1% of the overall parameters and FLOPs).

The accuracy of non-adaptive low-rank factorization is only 73%, but the adaptive version is 82.6%, a significant boost. We visualize data distributions in the 2D feature space for non-adaptive and adaptive low-rank factorizations in Figure 4 (we also present additional figures with TSNE (Maaten & Hinton, 2008) to enhance the visualization). We can see that with the adaptive mixing weights, data points of different classes are better separated in the projected low-dimensional space.

4.2. COMPRESSING RECURRENT NEURAL NETWORKS FOR LANGUAGE MODELING:

Recurrent neural networks (RNNs) are widely used in language modeling, machine translation and sequence modeling in general. In RNNs, we need to compute the transition of hidden states, e.g.,

h t = σ(W h h t-1 + W x x t + b)

at each time step. The transition weight matrices can be very large and very suitable for low-rank factorizations (Lu et al., 2016) .

In our experiment, we adopt the same Long Short Term Memory (LSTM) models and follow the settings from a previous state-of-the-art model (Zaremba et al., 2014) for language modeling, and use Penn Tree Bank (PTB) as well as Text8 datasets. More specifically, we use the medium sized model introduced in (Zaremba et al., 2014) , which consists of two layers LSTM with 650 hidden units. Dropouts of 0.5 are added between different layers.

The performance of a language model is commonly measured with perplexity, which is basically the exponential of average negative likelihood, and a smaller number is more desirable. By default, we directly factorize a concatenated joint weight matrix in LSTM, and make comparisons using different rank-d, measured by the ratio to the averaged input size n and output size m, i.e. 2d/(m + n). we set the number of mixtures to the rank, i.e. K = d, since in a good compression the rank d is expected to be small. More details are presented in the supplementary materials. We use the sigmoid activation for computing the mixing weights.

Our main baseline is the regular low-rank factorization, and we test three variants of the proposed model, each with different ways of computing mixing weights, namely (1) MIX-ALL-PJ: direct linear projection of the input vector h, (2) MIX-POOL-PJ: linear projection after segment-based mean pooling of the input vector h, and (3) MIX-RND-PJ: use a random projection for the input vector h. Among these adaptive projection methods, MIX-ALL-PJ has a large amount of extra parameters, MIX-POOL-PJ has a small amount of extra parameters, and MIX-RND-PJ has no extra parameters. We compute the FLOPs of a single time-step of applying LSTM (excluding the output softmax layer), and the perplexity associated to different settings.

The results are shown in Figure 5 . Firstly, with adaptive mixtures, the low-rank factorization model achieved 40% reduction in FLOPs, and even surpassed the performance of the full matrix baseline by decreasing the perplexity by 1.7 points in Penn Tree Bank. Secondly, as we decrease the rank and reduces the FLOPs, we observe the performance degradation, which is as expected. However, the use of adaptive mixtures can significantly improve the performance compared with regular, non-adaptive low-rank factorization (e.g. in Text8 data set, we reduce the FLOPs by 70% while maintaining the same perplexity). Thirdly, we see that different ways of generating the mixing weights have impacts on the performance, and the trade-off between performance and FLOPs/model size. We find that using pooling before projection can be a good choice for computing the mixing weights π. It not only reduces the computation and parameter size, but can better capture the global information and achieve better accuracy. We further explore the effects of the number of mixtures used in our method by using different ratio of mixtures to the low-rank dimensionality d. The results are shown in Figure 6 . We find that using more mixtures generally leads to better results, although the performance start to plateau when the number of mixtures is large enough. However, to obtain a larger compression rate and speedup, the rank-d we use in the low-rank factorization can be already small, thus the extras of using different number of mixtures may not differ too much.

4.3. COMPRESSING CONVOLUTIONAL NEURAL NETWORKS FOR IMAGE RECOGNITION:

We further demonstrate the effectiveness of the proposed approach on compressing CNN models. We chose to use modern compact CNN models as the baseline (which are harder to compress), rather than using the bulky CNN models (which is easier to compress). Recently, a major advance in designing compact CNNs architecture is so called depth-wise separable convolutions (Chollet, 2016; Howard et al., 2017; Sandler et al., 2018) . Compared to a standard convolutional kernel that computes the transformation (R H×W ×C → R H×W ×C )2 , a depth-wise separable convolution includes a depth-wise convolution (R H×W ×1 → R H×W ×1 ) and a point-wise convolution (R C → R C ) that are shared for spatial locations (pixels). It can greatly speed up the inference and reduce model size as well. This type of convolutional operations have been proved very effective and establish a new standard for compact CNNs design. In such a model design, the depth-wise convolution only accounts for 3% of the overall FLOPs, while the point-wise convolution takes up 95% of the FLOPs (Howard et al., 2017) . To demonstrate that our method can be well combined with the state-of-the-art CNNs consisting of depth-wise separable convolutions, we compare the regular and the proposed adaptive low-rank factorizations to decompose the point-wise convolutional weight matrix (W ∈ R C×C ). Different from RNNs/LSTM where the input vector h is a vector, h in CNNs is a 3D feature map composed of width, height and channels. Since the feature map can have a large spatial size, we do not use direct projection of h to compute π(h), instead we use a global mean pooling to reduce the height and width to 1, i.e. h pool = ij h ijk /z where i, j sum over all values, in width and height, averaged by the size z. By default, we set the number of mixture K to the rank d, since d has already been quite small observed from Figure 6 . Furthermore, we use the sigmoid activation for computing the mixing weights.

We experiment on both CIFAR-10 ( Krizhevsky & Hinton, 2009) and large-scale ImageNet (Deng et al., 2009) datasets. Specifically, we apply the regular and adaptive low-rank factorization on pointwise convolutional kernel W ∈ R C→C in MobileNet (Howard et al., 2017) . For CIFAR experiments, we follow the setting in (Zoph et al., 2017) to build a smaller MobileNet-CIFAR3 model containing 0.32M parameters and 44.5M FLOPs. For the initial 300 epochs, we train the network with learning rate 0.1, and halve it for every 25 epochs. For ImageNet experiments, we realize that applying the low-rank factorization for the pointwise convolutional kernel in MobileNet, and adding skip connection, we obtain a network architecture that is the same as MobileNet V2 (Sandler et al., 2018) . Therefore, we regard MobileNet V2 as the regular low-rank factorization of the original MobileNet model, and we apply the proposed adaptive low-rank factorization by directly computing mixing weights for the bottleneck layer (illustrated in Figure 3 ). To make a fair comparison, we follow the same experimental protocol as MobileNet V2 model, including the strategy of learning rate, training epochs, weight decays, etc. (More details can be found in the supplementary materials.)

As a result, Table 1 shows the performance comparison on CIFAR-10 between the regular low-rank factorization and our adaptive mixture method with different rank d's. Our method consistently outperforms the conventional one under different compression ratios. The performance gains are more significant on large compression ratios (or small FLOPs), which demonstrates that our adaptive mixture low-rank factorizations are efficient even for large compression ratios.

For ImageNet, Table 2 shows the comparison of different state-of-art compact convolutional models. We observed that compared to the regular low-rank factorization of MobileNet model (i.e. MobileNet V2), the proposed method achieves significantly better results (2.5% and 2% for two different Low-rank MobileNet settings, respectively), while only adding negligible extra FLOPs (less than 1%). 

5. RELATED WORK:

Our work is mostly related to model compression techniques to improve the efficiency of large neural networks. It has been shown that parameters of many modern neural networks are largely redundant (Han et al., 2015a; Ullrich et al., 2017) . To reduce the redundancy, various pruning techniques are widely explored (Han et al., 2015b; Ullrich et al., 2017; Louizos et al., 2017) . It turns out that more than 90% of connections in a large weight matrix can be pruned without significant loss of information (Han et al., 2015b) . This also results in many sparse matrices that require less computation theoretically, however, in practice, specialized hardware (ASIC or FPGA) is required to speed up sparse computations (Han et al., 2016; 2017) .

The low-rank factorization of large weight matrices is also a commonly used compression technique (Lu et al., 2016; Nakkiran et al., 2015; Jaderberg et al., 2014; Yu et al., 2017) . It does not have the sparse computation issue as in most pruning-base methods. However, the use of a small rank conflicts with the expressiveness as well as the performance of neural networks. Other efforts to improve the efficiency include designing more efficient convolutional operators (Chollet, 2016; Howard et al., 2017; Sandler et al., 2018) and apply quantization on neural network weights (Wu et al., 2018; Li et al., 2016; Achterhold et al., 2018) . It is worth noting that the quantization technique is orthogonal to our work and may be integrated together.

The weight matrix in our method is adaptive according to its input. This is also related to dynamic weight generations (Ha et al., 2016; Jia et al., 2016; Hu et al., 2017) . The dynamically generated weights can be flexible, but the computational cost for generating the weights during both training and inference can be expensive. In our method, we add the adaptiveness using unnormalized mixture of low-rank factorizations, thus our method can take advantage of dynamic weights while reducing the computation cost.

The use of mixing weights of multiple low-rank factorizations also resembles the mixture of experts (Jacobs et al., 1991; Shazeer et al., 2017; Yang et al., 2018) . While both methods use adaptive weights, ours utilizes an unnormalized mixture, and found that normalized mixture can lead to even worse performance. We suspect that unnormalized mixture can better boost the expressiveness for small ranks by more actively using more than one "low-rank factorizations".

6. CONCLUSIONS:

In this paper, we propose a generic adaptive mixture of low-rank matrix factorization framework, which dynamically incorporate low-rank factorizations with data-dependent weighting based on the input. Our experimental results show that the proposed adaptive mixture method can significantly improve the performance of low-rank factorization on both recurrent and convolutional neural networks. Our method not only keeps the efficiency of low-rank factorization, but also is comparable to (and often outperforms) the accuracy of their full-rank counterparts.

A MORE DETAILS AND ANALYSIS ON THE PROPOSED MODEL:



Increased expressiveness:

The following proposition demonstrates that the increased expressiveness of the proposed adaptive mixtures: there is no constant weight matrix, even full-rank ones, can mimic the data dependent dynamic weights.

Proposition 1. Given a set of low-rank factorization matrices {U (k) } and

EQUATION

h i = k π k (h i )W (k) h i and H = (h 1 , h 2 , • • • , h N ) ,

such that for some non-zero , and any P ∈ R m×n , H -HP F > .

Proof sketch. We can prove it by contradiction. Assuming that H -HP F = 0, then there will be a constant matrix W such that W = k π k (h i )W (k) , which cannot be true since we can freely choose function for computing π k .

B MORE DETAILS AND EXPERIMENTAL RESULTS FOR COMPRESSING:

RECURRENT NEURAL NETWORKS

B.1 LONG-SHORT TERM MEMORY AND LANGUAGE MODELING:

Language model is a fundamental task in natural language processing. Its goal is to accurately compute the likelihood of a sequence of words, and this can be formulated as follows.

P (w 1•••T ) = P (h 0 )P (w 1 |h 0 )P (w 2 |w 1 , h 1 ) • • • P (w t |w T -1 , h T )

where h t is used to summarizes the context information between the t-th word. It is very common to use a recurrent neural network to model the sequence (Zaremba et al., 2014) .

Long-short term memory (LSTM) (Hochreiter & Schmidhuber, 1997 ) is a widely used variant of vanilla recurrent neural networks, where it is shown better to deal with gradient vanishing problem.

The LSTM can be summarized as follows.

f t = σ g (W f x t + U f h t-1 + b f ) i t = σ g (W i x t + U i h t-1 + b i ) o t = σ g (W o x t + U o h t-1 + b o ) j t = σ c (W c x t + U c h t-1 + b c ) c t = f t • c t-1 + i t • j t h t = o t • σ h (c t )

To simply our implementation, we merge different weight matrices in LSTM above, i.e.

(W f , U f , W i , U i , W o , U o , W c , U c

), into a single matrix below.

Q = W f W i W o W c U f U i U o U c

We also concatenate the t-th hidden state h t and the t-th input vector x t into a single vector (x T t , h T ) T , so that (f T t , i T t , o T t , j T t ) T = Q(x T t , h T t ) T . With this re-organization of the computation, we can directly factorize large weight matrix Q.

As for the FLOPs, we only count the multiplications and adds of the LSTM layers (excluding the output softmax layer). The non-linear activations are not included. The major extra non-linear activations are in the computation of mixing weights, so the extra cost is expected to be small.

For the MIX-POOL-PJ method, we use segmentation-based pooling for h in computing π. We make sure that the pooling reduce h to the same dimension as the rank d in our experiments.

B.2 MORE EXPERIMENTAL RESULTS:

In the main paper, we present the trade-off curves between performance/perplexity and computational FLOPS in a LSTM cell. Figure 8 shows similar trade-off curves between performance/perplexity and the number of parameters. While the results are fairly similar to the FLOPS curves, one of the major differences is for the random projection since it has the same FLOPS but no (trainable) parameters compared to the regular projection of input vector h. For storage, we can simply save a random seed instead of the whole transformation weight matrix. We find that random projection actually work reasonably well considering it does not containing trainable parameters. As mentioned in main paper, we apply the proposed approach for the already compact models with depth-wise separable convolutions (Chollet, 2016; Howard et al., 2017; Sandler et al., 2018) . That is, we use (adaptive) low-rank factorization to decompose the pointwise convolutional kernel in MobileNet model, and this process can be illustrated in Figure 9 .

C.2 THE NETWORK STRUCTURE FOR MOBILENET-CIFAR:

As mentioned in the main paper, we build upon the state-of-the-art compact model, namely MobileNet (Howard et al., 2017) . Since the original MobileNet is design for ImageNet and thus too large for images in CIFAR-10, with size of 32 × 32 × 3, we construct a smaller MobileNet-CIFAR architecture.

The detailed structure of MobileNet-CIFAR is provided in Table 3 . A mobile cell is a depth-wise separable convolution, including a depth-wise 3 × 3 convolution and a point-wise 1 × 1 convolution. We conduct low-rank factorization on the weight of point-wise convolutions.

C.3 CONNECTION BETWEEN MOBILENET V1 AND V2 VIA LOW-RANK FACTORIZATION:

In our ImageNet experiments, we directly apply the low-rank factorization on the pointwise convolutional kernel. We realize that applying the low-rank factorization for the pointwise convolutional kernel in MobileNet, and adding skip connection, we obtained a network architecture that is the same as MobileNet V2. (See Figure 10 for details.) Following (Sandler et al., 2018) , no non-linearity other than Batch Normalization is inserted in the bottleneck. The detailed architectures are shown in Table 4 and 5 .

D EFFICIENT IMPLEMENTATION OF ADAPTIVE LOW-RANK FACTORIZATIONS:

Here we introduce a computation technique to avoid splitting a bulk of computation to segments. The default computation with reverse order introduced above can be written as W (h)h = K k=1 π k (h)U (k) V (k) h , which could introduce 2K segmented vector matrix multiplications, and possibly lead to more overheads. However, we could rewrite the same computation as follows.

W (h)h = U (π(h) (V T h)):

Where indicates element-wise multiplication. Since low-rank dimension is usually small, so we (by default) set K to the low-rank dimension d, in which case we have U ∈ R m×K and V ∈ R n×K . With this formulation, we do not need to explicit break the computation of regular low-rank of W h = U V T h into K branches/mixtures. Rather, we just use non-liear π(h) to re-weight the bottleneck vector (i.e. V T h).

E TRANSLATING FLOPS INTO WALL-CLOCK TIME:

The methods compared in this work are mainly evaluated via accuracy versus FLOPs, as the actual inference time depends on implementation, and also include runtime by other factors (such as final softmax layer in RNN language modeling). In order to access how the FLOPs translate into Wall-Clock Time in a more concentrated environment where we could compare regular low-rank factorization and the proposed counterpart, i.e. adaptive low-rank factorizations.

In this experiment, we measured the actual inference time for (1) linear transformation operator W h, (2) its low-rank factorization U V T h and (3) the proposed K k=1 π k (h)U (k) V (k) h. In our experiment, we used pooling before projection to compute π k (h), and the number of mixtures was the same as the low-rank dimension d to enable fast computing. We set both the dimensions of W to 1024, namely m = n = 1024. The actual inference time was measured with NVIDIA Jetson TX2 device to simulate edge deployment. Timing was averaged from 500 runs after 200 warm-up runs. The results are shown in Table 6 .

:

The compression ratio, as well as the ratio of FLOPS, is mn/((m + n)d). A smaller d is better.

(H, W, C) stands for the height, width and channel of a receptive field in CNNs, respectively.

The original MobileNet is built for ImageNet, while CIFAR image has a smaller size 32 × 32 × 3.

Figures and Tables:

Figure 1: A toy classification problem with a rank-1 factorization of the weight matrices. (b) and (c) are distributions of 2D data in the 1D projected space. The linear projection to lower dimension leads to significant information loss (results in 83% classification accuracy), while our proposed approach learns to adaptively avoid this (achieving 97% classification accuracy). (c) is the distribution of projection through a random matrix followed by tanh.

Figure 2: Illustration of different types linear projection weights V colored by responses to a particular input h: (a) A data-independent non-adaptive weight matrix, (b) fully adaptive weight matrix which can be very expensive, (c) the proposed adaptive mixture approach.

Figure 3: (a) regular factorization and (b) adaptive mixture of low-rank factorizations. First computez k = π k (h)((V (k) ) T h) and then h = k U (k) z k, where z can be treated as the middle layer. Techniques like pooling can be applied to compute π so it does not induce much extra parameters and costs.

Figure 4: Visualization of low-rank projected 2D space. Non-adaptive versus adaptive low-rank, in both original 2-d space and TSNE enhanced 2D space. With adaptive mixtures, we observe better separation among data points of different classes, closely positioning of the data of the same class. (Best view in color.)

Figure5: FLOPs vs. perplexity. The horizontal line is the full LSTM's baseline accuracy. We also compare variants of the proposed approaches with regular low-rank factorization, indicated by different colors and markers. Lower perplexity is better.

Figure 7: Visualization of the mixtures from the last bottleneck layer in our MobileNet-CIFAR (1/4 size). Each row is averaged from one of the 10 classes. The mixtures show a clear class-discriminative pattern. (Best viewed in color.)

Figure 9: Illustration of low-rank decomposition in depth-wise separable convolutions. No nonlinearity is added in the bottleneck.

Figure 8: Number of parameters vs perplexity. The horizontal line is for the full matrix baseline, and different color/shape indicating different methods: including the non-adaptive and adaptive low-rank factorizations, with three ways of generating the adaptive maxing weights. The smaller perplexity is more desirable.

Figure 10: The connection between MobileNet V1 and V2. MobileNet-V2 can be regarded as the low-rank factorized version of V1, where residual connection is added in the bottleneck.

Performance of MobileNet-CIFAR on CIFAR-10 dataset with different rand-d, as a ratio to input channel size ( d n ). Our adaptive mixture method provides consistent performance gain with negligible FLOPs increase.

Table content: <table><tr><td/><td>Original</td><td>1/4</td><td>1/4 ours</td><td>1/8</td><td colspan="3">1/8 ours 1/16 1/16 ours</td></tr><tr><td>Accuracy (%)</td><td>93.04</td><td>92.92</td><td>93.01</td><td>92.67</td><td>92.9</td><td>91.92</td><td>92.37</td></tr><tr><td>FLOPs (M)</td><td>44.5</td><td>27.35</td><td>27.37</td><td>18.96</td><td>18.98</td><td>14.88</td><td>14.90</td></tr><tr><td>Param. (M)</td><td>0.32</td><td>0.194</td><td>0.214</td><td>0.13</td><td>0.147</td><td>0.098</td><td>0.115</td></tr></table>

Performance for different networks on ImageNet. With negligible FLOPs increase, adaptive low-rank factorizations outperforms regular ones.

Table content: <table><tr><td>Network</td><td colspan="3">Top 1 Params MACs</td></tr><tr><td>ShuffleNet (1.5)</td><td>69.0</td><td>2.9M</td><td>292M</td></tr><tr><td>ShuffleNet (x2)</td><td>70.9</td><td>4.4M</td><td>524M</td></tr><tr><td>MobileNet</td><td>70.6</td><td>4.2M</td><td>575M</td></tr><tr><td>Low-rank MobileNet (0.75)</td><td>68.8</td><td>2.6M</td><td>209M</td></tr><tr><td>Adaptive Low-rank MobileNet (0.75)</td><td>70.5</td><td>2.8M</td><td>209M</td></tr><tr><td>Low-rank MobileNet</td><td>71.7</td><td>3.4M</td><td>300M</td></tr><tr><td>Adaptive Low-rank MobileNet</td><td>73.1</td><td>3.7M</td><td>300M</td></tr></table>

MobileNet-CIFAR: Each line describes a sequence of 1 or more identical (modulo stride) layers, repeated for n times. All layers in the same sequence have the same number c of output channels. The first layer of each sequence has a stride s and all others use stride 1. A mobile cell consists of a depth-wise convolution and a point-wise 1 × 1 convolution. All spatial convolutions use 3 × 3 kernels.

Table content: <table><tr><td>Input</td><td>Operator</td><td>c</td><td>n s</td></tr><tr><td>32 2 × 3</td><td>conv2d</td><td>32</td><td>1 1</td></tr><tr><td>32 2 × 32</td><td>mobile cell</td><td>64</td><td>3 1</td></tr><tr><td>32 2 × 64</td><td>mobile cell</td><td colspan="2">128 4 2</td></tr><tr><td>16 2 × 128</td><td>mobile cell</td><td colspan="2">256 4 2</td></tr><tr><td>8 2 × 256</td><td>mean pool 8 × 8</td><td>-</td><td>1 -</td></tr><tr><td>1 × 1 × 256</td><td>conv2d 1 × 1</td><td>10</td><td>--</td></tr></table>

MobileNet-ImageNet: Each line describes a sequence of 1 or more identical (modulo stride) layers, repeated for n times. All layers in the same sequence have the same number c of output channels. The first layer of each sequence has a stride s and all others use stride 1. A mobile cell consists of a depth-wise convolution and a point-wise 1 × 1 convolution. All spatial convolutions use 3 × 3 kernels.

Table content: <table><tr><td>Input</td><td>Operator</td><td>c</td><td>n s</td></tr><tr><td>224 2 × 3</td><td>conv2d</td><td>32</td><td>1 2</td></tr><tr><td>112 2 × 32</td><td>mobile cell</td><td>64</td><td>1 1</td></tr><tr><td>112 2 × 64</td><td>mobile cell</td><td>128</td><td>2 2</td></tr><tr><td>56 2 × 128</td><td>mobile cell</td><td>256</td><td>2 2</td></tr><tr><td>28 2 × 256</td><td>mobile cell</td><td>512</td><td>6 2</td></tr><tr><td>14 2 × 512</td><td>mobile cell</td><td colspan="2">1024 2 2</td></tr><tr><td colspan="2">7 2 × 1024 mean pool 7 × 7</td><td>-</td><td>1 -</td></tr><tr><td>1 2 × 1024</td><td>conv2d 1 × 1</td><td colspan="2">1000 --</td></tr></table>

MobileNet-ImageNet-LR: Each line describes a sequence of 1 or more identical (modulo stride) layers, repeated n times. All layers in the same sequence have the same number c of output channels. The first layer of each sequence has a stride s and all others use stride 1. All spatial convolutions use 3 × 3 kernels. The expansion factor t is always applied to the input size. Therefore, our method can be directly applied to the original MobileNetV2 to replace the standard low-rank factorization, by computing the adaptive weights dynamically. We evaluated our models with width multiplier 0.75 and 1.0 with input size 224 × 224. The training is conducted under the same protocol as in the original paper.

Table content: <table><tr><td>Input</td><td>Operator</td><td>t</td><td>c</td><td>n s</td></tr><tr><td>224 2 × 3</td><td>conv2d</td><td>-</td><td>32</td><td>1 2</td></tr><tr><td>112 2 × 32</td><td>bottleneck cell</td><td>1</td><td>16</td><td>1 1</td></tr><tr><td>112 2 × 16</td><td>bottleneck cell</td><td>6</td><td>24</td><td>2 2</td></tr><tr><td>56 2 × 24</td><td>bottleneck cell</td><td>6</td><td>32</td><td>3 2</td></tr><tr><td>28 2 × 32</td><td>bottleneck cell</td><td>6</td><td>64</td><td>4 2</td></tr><tr><td>28 2 × 64</td><td>bottleneck cell</td><td>6</td><td>96</td><td>3 1</td></tr><tr><td>14 2 × 96</td><td>bottleneck cell</td><td colspan="2">6 160</td><td>3 2</td></tr><tr><td>7 2 × 160</td><td>bottleneck cell</td><td colspan="2">6 320</td><td>1 1</td></tr><tr><td>7 2 × 320</td><td>conv2d 1 × 1</td><td colspan="3">-1280 1 1</td></tr><tr><td colspan="3">7 2 × 1280 mean pool 7 × 7 -</td><td>1</td><td>-</td></tr><tr><td>1 2 × k</td><td>conv2d 1 × 1</td><td colspan="3">-1000 --</td></tr></table>

Peer review (iclr2019):
 The paper proposes an input-dependent low rank approximations of weight matrices in neural nets with a goal of accelerating inference. Instead of decomposing a matrix (a layer) to a low rank space like previous work did, the paper proposes to use a linear combination/mixture of multiple low rank spaces. The linear combination is dynamic and data dependent, making it some sense of non-linear combination. The paper is interesting, however, I doubt its significance at three aspects: (1) computation efficiency: the primary motivation of this paper is to accelerate inference stage; however, it might not be wise to break computation in a single low-rank space to segments in multiple low-rank spaces. In the original low-rank approximation, only two matrix-vector multiplications are needed, but this paper increases it to 2*K plus some additional overheads. Although the theoretical FLOPs can be cut, but modern hardware runs much faster when a whole bulk of data are computed all together. Because of this, the primary motivation in this paper wasn*t successfully supported by wall-clock time; (2) low-rank approximation: low-rank approximation only makes sense when matrices are known and redundant, otherwise, no approximation target exists (i.e., what matrix is it approximating?). Because of this, low-rank neural nets [1][2] start from trained models, approximate it and fine-tune it, while this method trains from scratch without an approximation target. Although, we can fit the method to approximate trained matrices, then decomposing a matrix to a mixture of low-rank spaces is equivalent to decomposing to one single low-rank space (the only difference is the combination is data dependent). Therefore, I view this paper more in a research line of designing compact neural nets, which brings me to a concern in (3). (3) efficient architecture design: essentially, the paper proposes a class of compact neural nets, at each layer of which there are K "low-rank" branches with a gating mechanism to select those branches. However, branching and gating is not new [3][4]. I like the results in Table 2, but, to show the efficiency, it should have been compared with more SOTA compact models like CondenseNet. Clarity: How FLOPs reduction are exactly calculated? I am not convinced by FLOPs reduction in the LSTM experiments, since in LSTM (especially in language modeling), FLOPs in the output layer are large because of a large vocabulary size (650x10000 in the experiments). However, output layer is not explicitly mentioned in the paper. Improvement: (1) Accuracy improvement in Table 1 is not statistically significant, but used more parameters. For example, an improvement of 93.01% over 92.92% is within an effect of training noise; (2) It is a little hacking to conclude that a random matrix P_random has a small storage size because we can storage a seed for recovery. When we deploy models across platforms, we cannot guarantee they use the same random generator and has a consistent implementation; (3) sparse gating of low-rank branches may make this method more computation efficient. [1] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems (NIPS). 2014. [2] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference (BMVC), 2014. [3] Szegedy, Christian, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. "Going deeper with convolutions." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9. 2015. [4] Huang, Gao, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. "Multi-scale dense networks for resource efficient image classification." arXiv preprint arXiv:1703.09844 (2017).