Year: 2018

Abstract:
Modern deep neural networks have a large amount of weights, which make them difficult to deploy on computation constrained devices such as mobile phones. One common approach to reduce the model size and computational cost is to use lowrank factorization to approximate a weight matrix. However, performing standard low-rank factorization with a small rank can hurt the model expressiveness and significantly decrease the performance. In this work, we propose to use a mixture of multiple low-rank factorizations to model a large weight matrix, and the mixture coefficients are computed dynamically depending on its input. We demonstrate the effectiveness of the proposed approach on both language modeling and image classification tasks. Experiments show that our method not only improves the computation efficiency but also maintains (sometimes outperforms) its accuracy compared with the full-rank counterparts.

heading: 1 INTRODUCTION
text: Modern neural networks usually contain millions of parameters (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014), and they are difficult to be deployed on mobile devices with limited computation resources. To solve this problem, model compression techniques are proposed in recent years. For example, (Wu et al., 2018; Li et al., 2016; Han et al., 2015a) try to limit the weights (and activations) of neural networks to lower precisions by quantization. This can save the model size by 4 to 16 times. While quantization can reduce the number of bits per weight, it cannot reduce the number of weights. To reduce this redundancy, (Han et al., 2015b; Ullrich et al., 2017; Louizos et al., 2017) propose pruning the weight matrices, leading to sparse neural networks that require less computation. However, sparse neural networks often require specialized ASIC or FPGA to accelerate (Han et al., 2016; 2017).
Alternatively, low-rank factorization is a popular way of reducing the matrix size. It has been extensively explored in the literature (Lu et al., 2016; Nakkiran et al., 2015; Jaderberg et al., 2014; Yu et al., 2017). Mathematically, a large weight matrix W ∈ Rm×n is factorized to two small rank-d matrices U ∈ Rm×d, V ∈ Rn×d with W = UV T . Since both U and V are dense, no sparsity support is required from specialized hardware. It naturally fits the general-purpose, off-the-shelf CPUs and GPUs.
To significantly reduce the model size and computation, the rank d in the low-rank factorization needs to be small. However, a small rank can limit the expressiveness of the model (Yang et al., 2018) and lead to worse performance. To understand the limitations, given a n-dim feature vector h, we observe that V Th, as in U(V Th), is a linear projection from a high-dimensional space (n dims) to a low-dimensional space (d dims). This can lead to a significant loss of information. The conflict between the rank d and the model expressiveness prevents us from obtaining a both compact and accurate model.
To address the dilemma, we propose to increase the expressiveness by learning an adaptive, inputdependent factorization, rather than performing a fixed factorization of a weight matrix. To do so, we use a mixture of multiple low-rank factorizations. The mixing weights are computed based on the input. This creates an adaptive linear projection from a high-dimensional space to a low-dimensional space. Compared to the conventional low-rank factorization, the proposed approach can significantly improve its performance while only introducing a small additional cost.
To demonstrate the effectiveness of adaptive low-rank factorization, we experiment with both recurrent and convolutional neural networks on language modeling and image classification tasks. Experimental results on both tasks show that our method consistently improves upon conventional low-rank factorization. On the Penn Tree Bank dataset, we achieved 40% reduction in FLOPs, and 1.7 better perplexity than the full rank baseline LSTM for language modeling. On ImageNet dataset, we use 48% less computation, 12% less parameters, but achieve 2.5% better Top-1 accuracy than MobileNetV1 (Howard et al., 2017). Compared to MobileNet-V2 which utilizes a standard low-rank bottleneck structure, our proposed method achieves 1.95% better Top-1 accuracy with less than 1% extra FLOPs, which is significant given that MobileNet-V2 is already very compact.

heading: 2 LOW-RANK FACTORIZATION AND THE LINEAR BOTTLENECK
text: A common linear transformation between two spaces can be represented by a linear function F : Rn → Rm, F(h;W ) = Wh where W ∈ Rm×n and h ∈ Rn. To reduce the size and computation of this linear transformation, a low-rank factorization of W , i.e. W = UV >, can be applied, where U ∈ Rm×d, V ∈ Rn×d, and d < min(m,n). With this factorization, we can compute the transformation with h′ = UV >h. This reduces computations from O(mn) to O((m+ n)d). In the context of neural networks where W represents a weight matrix for a layer, both U and V can be learned using gradient-based algorithms.
From the model compression perspective, we want to minimize the rank d, since that relates to smaller model size and computation 1. However, the expressiveness of the transformation is limited by the rank d. By applying the factorized transformation h′ = UV >h in the reverse order, i.e. h′ = U(V >h), we observe that the first transformation for h is to project it from a high-dimensional to a low-dimensional space since d < n. The latent feature distribution in high dimensional space may be either high dimensional, or lie on a non-linear manifold. In either case, projecting it into the low-dimensional space can lead to significant information loss if d is small. This is less appealing for preserving information for latter layers.
To demonstrate the expressiveness issue, we conduct a toy classification task in 2D spaces. We first generate a 2D dataset with XNOR labels (two diagonal blocks are labeled with the same class) 1a. A non-linear classifier with one hidden layer is trained to predict the output probability by P (y|x) = softmax(W2σ(W1x)) where W1 ∈ R2×2, and we factorize W1 using two 2× 1 matrices, i.e. W1 = UV T . Since the rank d = 1, the 2D data points are first projected into 1D space and then projected into class probabilities. The visualization of one-dim space is shown in Figure 1b. We observe a significant amount of previously separated data points are now overlapped, and the class information is lost. We attribute this loss of information to the linear bottleneck.
This limitation cannot be solved by adding non-linear activation at the bottleneck (after the linear transformation), since the information is already lost before the application of the non-linearity. Even worse, adding lossy non-linearity to a low-dimensional manifold will further negatively impact the
1The compression ratio, as well as the ratio of FLOPS, is mn/((m+ n)d). A smaller d is better.
network’s performance, as pointed out in (Howard et al., 2017). Therefore, a new approach to boost the expressiveness of the linear bottleneck without much overhead is in demand.

heading: 3 ADAPTIVE MIXTURE OF LOW-RANK FACTORIZATIONS
text: To overcome the linear bottleneck in the low-rank factorization approach presented above, we propose to use an unnormalized learned mixture of low-rank factorizations whose mixing weights are computed adaptively based on the input. More specifically, denoting the input by h and the number of mixture components by K, we decompose a large weight matrix by
W (h) = K∑ k=1 πk(h)U (k) ( V (k) )> , (1)
where π(·) : Rn → RK is the function which maps each input to its mixture coefficients, and U (k) ∈ Rm×d/K , V (k) ∈ Rn×d/K . For example, π can be a small neural network. This introduces a small amount of extra parameters and computation. We will later discuss the details of efficient ways to implement the mixture function π.
If πk, k = 1, ...,K, is chosen to be constant (input independent), it can be absorbed into either U (k) or V (k). Thus, the proposed method reduces to the low-rank factorization. This is evidenced by rewriting W as W = [π1U (1), ..., πKU (K)][V (1), ..., V (K)]>. In other words, the conventional low-rank factorization can be considered as a special case of our method.
Adaptive mixing weights π(h). The mixing weights can encode important information that we can use to increase the expressiveness of the projected low-dimensional space. Under our framework, the generation of the mixing weights π(h) is flexible. A straight-forward approach is to use a non-linear transformation of the input to the weight matrix. For example, π(h) = σ(Ph), where σ is a non-linear transformation, such as sigmoid or hyperbolic tangent function, and P ∈ RK×n is an extra weight matrix. This adds some extra parameters and computation to the model since the linear projection that we construct is Rn → RK . To further reduce the parameter and computation in the mixing weights π, we propose the following strategies.
Pooling before projection. We do not require the whole input to compute the mixture function π. Instead, we can apply pooling to the input h before projection. For example, a global average pooling can be applied if the input is a 3D tensor (for images); for a 1D vector, we can segment the vector and average each segmentations. By applying pooling, we can both save the computation and better capture the global information.
Random projection. To reduce the number of parameters in the linear projection of h, we can use a random matrix Prandom in place of a fully adjustable P , i.e. π(h) = σ(Prandomh). Although we cannot control freely the information captured by a random matrix, hopefully the linear projection induced by U and V can adaptively learn features that are complementary to the mixture weights. Note that we can simply save a seed to recover the random matrix, but it still requires the same amount of memory and computation as the fully adjustable linear projection of h.
Increased expressiveness. Due to the use of data-dependent mixing weights for multiple lowrank factorizations, we expect the expressiveness of the model to increase. To demonstrate this intuition, in the toy example of Figure 1, the distribution of adopting our approach that computes the mixing weights from a random matrix P and tanh non-linearity is shown in Figure 1c. Compared to conventional linear bottleneck in Figure 1b, we see a better separation among data between the two classes, leading to improved classification accuracy (97% vs. 83%). This is not surprising since the mixing weights encode certain class distribution in 2D space and augments the linear projection. The original overlapped projection is now better separated.
More precisely, the adaptive mixing weights introduce a non-linear transformation into the highto-low-dimensional projection that can be more expressive. Since each W (h) is a data-dependent low-rank matrix, there is no constant linear weight independent to the input (even a full-rank matrix) that can mimic the transformation W (h) induced by our proposed method. It is worth noting that generating the whole weight matrices can be very expensive. Our method can be seen as a swift
approach to generate the weights by adaptively adjusting mixing weights for the linear bottleneck. It assigns weights into groups and dynamically controls them at the group level, as demonstrated in Figure 2.
To efficiently compute the whole linear transformation ∑K k=1 πk(h)U (k) ( V (k) )> h, we use the reverse order, i.e. first computing the linear projection into low-dimensional space with mixing weights, i.e. zk = πk(h)((V (k))Th), and then map to a higher dimensional space, i.e. h′ =∑
k U (k)zk. This reduces the FLOPs and also avoids to store different weight matrices W (h) for different training examples in a mini-batch. An illustration of the computation framework is presented in Figure 3. Compared to original low-rank factorization, extra parameters and computation cost are from the mixing weights. They can be very small with techniques like pooling aforementioned. We also introduce a way in appendix to avoid breaking a bulk matrix multiplication to segments.

heading: 4 EXPERIMENTS
text: In this section, we first showcase the linear bottleneck in MNIST with multi-layer perceptron (MLP), and demonstrate how our proposed method improves upon the regular low-rank factorization. Then we conduct extensive experiments on both recurrent neural networks for language modeling and convolutional neural networks for image recognition on ImageNet.

heading: 4.1 ADAPTIVE LOW-RANK FACTORIZATION FOR MLP
text: In this experiment, we construct a MLP for digit recognition using MNIST dataset. We use a simple one-layer MLP of 300 hidden units (whose input and output sizes are 784 and 10, respectively), and it can be written as P (y|x) = softmax(W2σ(W1x)). We factorize W1 ∈ R784×300 with a rank-d matrix. We use d = 2 in this case to better expose the issue of linear bottleneck and better visualize the latent data distribution. We also set the number of mixture K = 2. To compute mixing weights, we first reduce x ∈ R784 to R28 with a segment-based mean pooling, so that the extra parameters is of dimension 28 × 2 and computations only accounts for a small amount (< 1% of the overall parameters and FLOPs).
The accuracy of non-adaptive low-rank factorization is only 73%, but the adaptive version is 82.6%, a significant boost. We visualize data distributions in the 2D feature space for non-adaptive and adaptive low-rank factorizations in Figure 4 (we also present additional figures with TSNE (Maaten
& Hinton, 2008) to enhance the visualization). We can see that with the adaptive mixing weights, data points of different classes are better separated in the projected low-dimensional space.

heading: 4.2 COMPRESSING RECURRENT NEURAL NETWORKS FOR LANGUAGE MODELING
text: Recurrent neural networks (RNNs) are widely used in language modeling, machine translation and sequence modeling in general. In RNNs, we need to compute the transition of hidden states, e.g., ht = σ(Whht−1 +Wxxt + b) at each time step. The transition weight matrices can be very large and very suitable for low-rank factorizations (Lu et al., 2016).
In our experiment, we adopt the same Long Short Term Memory (LSTM) models and follow the settings from a previous state-of-the-art model (Zaremba et al., 2014) for language modeling, and use Penn Tree Bank (PTB) as well as Text8 datasets. More specifically, we use the medium sized model introduced in (Zaremba et al., 2014), which consists of two layers LSTM with 650 hidden units. Dropouts of 0.5 are added between different layers.
The performance of a language model is commonly measured with perplexity, which is basically the exponential of average negative likelihood, and a smaller number is more desirable. By default, we directly factorize a concatenated joint weight matrix in LSTM, and make comparisons using different rank-d, measured by the ratio to the averaged input size n and output size m, i.e. 2d/(m+n). we set the number of mixtures to the rank, i.e. K = d, since in a good compression the rank d is expected to be small. More details are presented in the supplementary materials. We use the sigmoid activation for computing the mixing weights.
Our main baseline is the regular low-rank factorization, and we test three variants of the proposed model, each with different ways of computing mixing weights, namely (1) MIX-ALL-PJ: direct linear projection of the input vector h, (2) MIX-POOL-PJ: linear projection after segment-based mean pooling of the input vector h, and (3) MIX-RND-PJ: use a random projection for the input vector h. Among these adaptive projection methods, MIX-ALL-PJ has a large amount of extra parameters, MIX-POOL-PJ has a small amount of extra parameters, and MIX-RND-PJ has no extra parameters. We compute the FLOPs of a single time-step of applying LSTM (excluding the output softmax layer), and the perplexity associated to different settings.
The results are shown in Figure 5. Firstly, with adaptive mixtures, the low-rank factorization model achieved 40% reduction in FLOPs, and even surpassed the performance of the full matrix baseline by decreasing the perplexity by 1.7 points in Penn Tree Bank. Secondly, as we decrease the rank and reduces the FLOPs, we observe the performance degradation, which is as expected. However, the use of adaptive mixtures can significantly improve the performance compared with regular, non-adaptive low-rank factorization (e.g. in Text8 data set, we reduce the FLOPs by 70% while maintaining the same perplexity). Thirdly, we see that different ways of generating the mixing weights have impacts on the performance, and the trade-off between performance and FLOPs/model size. We find that using pooling before projection can be a good choice for computing the mixing weights π. It not only reduces the computation and parameter size, but can better capture the global information and achieve better accuracy.
We further explore the effects of the number of mixtures used in our method by using different ratio of mixtures to the low-rank dimensionality d. The results are shown in Figure 6. We find that using more mixtures generally leads to better results, although the performance start to plateau when the number of mixtures is large enough. However, to obtain a larger compression rate and speedup, the rank-d we use in the low-rank factorization can be already small, thus the extras of using different number of mixtures may not differ too much.

heading: 4.3 COMPRESSING CONVOLUTIONAL NEURAL NETWORKS FOR IMAGE RECOGNITION
text: We further demonstrate the effectiveness of the proposed approach on compressing CNN models. We chose to use modern compact CNN models as the baseline (which are harder to compress), rather than using the bulky CNN models (which is easier to compress). Recently, a major advance in designing compact CNNs architecture is so called depth-wise separable convolutions (Chollet, 2016; Howard et al., 2017; Sandler et al., 2018). Compared to a standard convolutional kernel that computes the transformation (RH×W×C → RH×W×C′) 2, a depth-wise separable convolution includes a depth-wise convolution (RH×W×1 → RH×W×1) and a point-wise convolution (RC → RC′) that are shared for spatial locations (pixels). It can greatly speed up the inference and reduce model size as well. This type of convolutional operations have been proved very effective and establish a new standard for compact CNNs design. In such a model design, the depth-wise convolution only accounts for 3% of the overall FLOPs, while the point-wise convolution takes up 95% of the FLOPs (Howard et al., 2017). To demonstrate that our method can be well combined with the state-of-the-art CNNs consisting of depth-wise separable convolutions, we compare the regular and the proposed adaptive low-rank factorizations to decompose the point-wise convolutional weight matrix (W ∈ RC×C′ ).
2(H,W,C) stands for the height, width and channel of a receptive field in CNNs, respectively.
Different from RNNs/LSTM where the input vector h is a vector, h in CNNs is a 3D feature map composed of width, height and channels. Since the feature map can have a large spatial size, we do not use direct projection of h to compute π(h), instead we use a global mean pooling to reduce the height and width to 1, i.e. hpool = ∑ ij hijk/z where i, j sum over all values, in width and height, averaged by the size z. By default, we set the number of mixture K to the rank d, since d has already been quite small observed from Figure 6. Furthermore, we use the sigmoid activation for computing the mixing weights.
We experiment on both CIFAR-10 (Krizhevsky & Hinton, 2009) and large-scale ImageNet (Deng et al., 2009) datasets. Specifically, we apply the regular and adaptive low-rank factorization on pointwise convolutional kernel W ∈ RC→C′ in MobileNet (Howard et al., 2017). For CIFAR experiments, we follow the setting in (Zoph et al., 2017) to build a smaller MobileNet-CIFAR 3 model containing 0.32M parameters and 44.5M FLOPs. For the initial 300 epochs, we train the network with learning rate 0.1, and halve it for every 25 epochs. For ImageNet experiments, we realize that applying the low-rank factorization for the pointwise convolutional kernel in MobileNet, and adding skip connection, we obtain a network architecture that is the same as MobileNet V2 (Sandler et al., 2018). Therefore, we regard MobileNet V2 as the regular low-rank factorization of the original MobileNet model, and we apply the proposed adaptive low-rank factorization by directly computing mixing weights for the bottleneck layer (illustrated in Figure 3). To make a fair comparison, we follow the same experimental protocol as MobileNet V2 model, including the strategy of learning rate, training epochs, weight decays, etc. (More details can be found in the supplementary materials.)
As a result, Table 1 shows the performance comparison on CIFAR-10 between the regular low-rank factorization and our adaptive mixture method with different rank d’s. Our method consistently outperforms the conventional one under different compression ratios. The performance gains are more significant on large compression ratios (or small FLOPs), which demonstrates that our adaptive mixture low-rank factorizations are efficient even for large compression ratios.
For ImageNet, Table 2 shows the comparison of different state-of-art compact convolutional models. We observed that compared to the regular low-rank factorization of MobileNet model (i.e. MobileNet V2), the proposed method achieves significantly better results (2.5% and 2% for two different Low-rank MobileNet settings, respectively), while only adding negligible extra FLOPs (less than 1%).
3The original MobileNet is built for ImageNet, while CIFAR image has a smaller size 32× 32× 3.
Visualization of Mixture. To see whether the mixtures π generated for each sample provides class-discriminative information, we visualize the values of mixtures in CIFAR experiments. We record all the mixing weights on CIFAR-10 validation set and average them for each classes. The results are shown in Fig 7, we find the distribution of the mixtures are different for each classes. It shows that the adaptive mixture is able to capture class-discriminative features.

heading: 5 RELATED WORK
text: Our work is mostly related to model compression techniques to improve the efficiency of large neural networks. It has been shown that parameters of many modern neural networks are largely redundant (Han et al., 2015a; Ullrich et al., 2017). To reduce the redundancy, various pruning techniques are widely explored (Han et al., 2015b; Ullrich et al., 2017; Louizos et al., 2017). It turns out that more than 90% of connections in a large weight matrix can be pruned without significant loss of information (Han et al., 2015b). This also results in many sparse matrices that require less computation theoretically, however, in practice, specialized hardware (ASIC or FPGA) is required to speed up sparse computations (Han et al., 2016; 2017).
The low-rank factorization of large weight matrices is also a commonly used compression technique (Lu et al., 2016; Nakkiran et al., 2015; Jaderberg et al., 2014; Yu et al., 2017). It does not have the sparse computation issue as in most pruning-base methods. However, the use of a small rank conflicts with the expressiveness as well as the performance of neural networks. Other efforts to improve the efficiency include designing more efficient convolutional operators (Chollet, 2016; Howard et al., 2017; Sandler et al., 2018) and apply quantization on neural network weights (Wu et al., 2018; Li et al., 2016; Achterhold et al., 2018). It is worth noting that the quantization technique is orthogonal to our work and may be integrated together.
The weight matrix in our method is adaptive according to its input. This is also related to dynamic weight generations (Ha et al., 2016; Jia et al., 2016; Hu et al., 2017). The dynamically generated weights can be flexible, but the computational cost for generating the weights during both training and inference can be expensive. In our method, we add the adaptiveness using unnormalized mixture of low-rank factorizations, thus our method can take advantage of dynamic weights while reducing the computation cost.
The use of mixing weights of multiple low-rank factorizations also resembles the mixture of experts (Jacobs et al., 1991; Shazeer et al., 2017; Yang et al., 2018). While both methods use adaptive weights, ours utilizes an unnormalized mixture, and found that normalized mixture can lead to even worse performance. We suspect that unnormalized mixture can better boost the expressiveness for small ranks by more actively using more than one “low-rank factorizations”.

heading: 6 CONCLUSIONS
text: In this paper, we propose a generic adaptive mixture of low-rank matrix factorization framework, which dynamically incorporate low-rank factorizations with data-dependent weighting based on the input. Our experimental results show that the proposed adaptive mixture method can significantly improve the performance of low-rank factorization on both recurrent and convolutional neural networks. Our method not only keeps the efficiency of low-rank factorization, but also is comparable to (and often outperforms) the accuracy of their full-rank counterparts.

heading: A MORE DETAILS AND ANALYSIS ON THE PROPOSED MODEL
text: Increased expressiveness The following proposition demonstrates that the increased expressiveness of the proposed adaptive mixtures: there is no constant weight matrix, even full-rank ones, can mimic the data dependent dynamic weights.
Proposition 1. Given a set of low-rank factorization matrices {U (k)} and {V (k)} (U ∈ Rm×d, V ∈ Rn×d), W (k) = U (k)(V (k))>, and input data H = (h1, h2, · · · , hN )> ∈ RN×m. There exists a data-dependent mixing weight function π(·) : Rn → RK , where π(h) = (π1(h), π2(h), · · · , πK(h)) and
h′i = (∑ k πk(hi)W (k) ) hi and H ′ = (h′1, h ′ 2, · · · , h′N )>,
such that for some non-zero , and any P ∈ Rm×n, ‖H ′ −HP‖F > .
Proof sketch. We can prove it by contradiction. Assuming that ‖H ′ −HP‖F = 0, then there will be a constant matrix W such that W = ∑ k πk(hi)W
(k), which cannot be true since we can freely choose function for computing πk.

heading: B MORE DETAILS AND EXPERIMENTAL RESULTS FOR COMPRESSING RECURRENT NEURAL NETWORKS
text: B.1 LONG-SHORT TERM MEMORY AND LANGUAGE MODELING
Language model is a fundamental task in natural language processing. Its goal is to accurately compute the likelihood of a sequence of words, and this can be formulated as follows.
P (w1···T ) = P (h0)P (w1|h0)P (w2|w1, h1) · · ·P (wt|wT−1, hT )
where ht is used to summarizes the context information between the t-th word. It is very common to use a recurrent neural network to model the sequence (Zaremba et al., 2014).
Long-short term memory (LSTM) (Hochreiter & Schmidhuber, 1997) is a widely used variant of vanilla recurrent neural networks, where it is shown better to deal with gradient vanishing problem. The LSTM can be summarized as follows.
ft = σg(Wfxt + Ufht−1 + bf )
it = σg(Wixt + Uiht−1 + bi)
ot = σg(Woxt + Uoht−1 + bo)
jt = σc(Wcxt + Ucht−1 + bc)
ct = ft ◦ ct−1 + it ◦ jt ht = ot ◦ σh(ct)
To simply our implementation, we merge different weight matrices in LSTM above, i.e. (Wf , Uf ,Wi, Ui,Wo, Uo,Wc, Uc), into a single matrix below.
Q = [ Wf Wi Wo Wc Uf Ui Uo Uc ]
We also concatenate the t-th hidden state ht and the t-th input vector xt into a single vector (xTt , h T )T , so that (fTt , i T t , o T t , j T t ) T = Q(xTt , h T t )
T . With this re-organization of the computation, we can directly factorize large weight matrix Q.
As for the FLOPs, we only count the multiplications and adds of the LSTM layers (excluding the output softmax layer). The non-linear activations are not included. The major extra non-linear activations are in the computation of mixing weights, so the extra cost is expected to be small.
For the MIX-POOL-PJ method, we use segmentation-based pooling for h in computing π. We make sure that the pooling reduce h to the same dimension as the rank d in our experiments.
B.2 MORE EXPERIMENTAL RESULTS
In the main paper, we present the trade-off curves between performance/perplexity and computational FLOPS in a LSTM cell. Figure 8 shows similar trade-off curves between performance/perplexity and the number of parameters. While the results are fairly similar to the FLOPS curves, one of the major differences is for the random projection since it has the same FLOPS but no (trainable) parameters compared to the regular projection of input vector h. For storage, we can simply save a random seed instead of the whole transformation weight matrix. We find that random projection actually work reasonably well considering it does not containing trainable parameters.

heading: C MORE DETAILS AND EXPERIMENTAL RESULTS FOR COMPRESSING CONVOLUTIONAL NEURAL NETWORKS
text: C.1 LOW-RANK FACTORIZATION FOR POINTWISE CONVOLUTIONAL KERNEL
As mentioned in main paper, we apply the proposed approach for the already compact models with depth-wise separable convolutions (Chollet, 2016; Howard et al., 2017; Sandler et al., 2018). That is, we use (adaptive) low-rank factorization to decompose the pointwise convolutional kernel in MobileNet model, and this process can be illustrated in Figure 9.
C.2 THE NETWORK STRUCTURE FOR MOBILENET-CIFAR
As mentioned in the main paper, we build upon the state-of-the-art compact model, namely MobileNet (Howard et al., 2017). Since the original MobileNet is design for ImageNet and thus too large for images in CIFAR-10, with size of 32×32×3, we construct a smaller MobileNet-CIFAR architecture. The detailed structure of MobileNet-CIFAR is provided in Table 3. A mobile cell is a depth-wise separable convolution, including a depth-wise 3× 3 convolution and a point-wise 1× 1 convolution. We conduct low-rank factorization on the weight of point-wise convolutions.
C.3 CONNECTION BETWEEN MOBILENET V1 AND V2 VIA LOW-RANK FACTORIZATION
In our ImageNet experiments, we directly apply the low-rank factorization on the pointwise convolutional kernel. We realize that applying the low-rank factorization for the pointwise convolutional kernel in MobileNet, and adding skip connection, we obtained a network architecture that is the same as MobileNet V2. (See Figure 10 for details.)
Therefore, our method can be directly applied to the original MobileNetV2 to replace the standard low-rank factorization, by computing the adaptive weights dynamically. We evaluated our models
with width multiplier 0.75 and 1.0 with input size 224× 224. The training is conducted under the same protocol as in the original paper.
Following (Sandler et al., 2018), no non-linearity other than Batch Normalization is inserted in the bottleneck. The detailed architectures are shown in Table 4 and 5.

heading: D EFFICIENT IMPLEMENTATION OF ADAPTIVE LOW-RANK FACTORIZATIONS
text: Here we introduce a computation technique to avoid splitting a bulk of computation to segments. The default computation with reverse order introduced above can be written as W (h)h =∑K k=1 πk(h)U (k) (( V (k) )> h ) , which could introduce 2K segmented vector matrix multiplications, and possibly lead to more overheads. However, we could rewrite the same computation as follows.
W (h)h = U(π(h) (V Th))
Where indicates element-wise multiplication. Since low-rank dimension is usually small, so we (by default) set K to the low-rank dimension d, in which case we have U ∈ Rm×K and V ∈ Rn×K . With this formulation, we do not need to explicit break the computation of regular low-rank of Wh = UV Th into K branches/mixtures. Rather, we just use non-liear π(h) to re-weight the bottleneck vector (i.e. V Th).

heading: E TRANSLATING FLOPS INTO WALL-CLOCK TIME
text: The methods compared in this work are mainly evaluated via accuracy versus FLOPs, as the actual inference time depends on implementation, and also include runtime by other factors (such as final softmax layer in RNN language modeling). In order to access how the FLOPs translate into Wall-Clock Time in a more concentrated environment where we could compare regular low-rank factorization and the proposed counterpart, i.e. adaptive low-rank factorizations.
In this experiment, we measured the actual inference time for (1) linear transformation operator Wh, (2) its low-rank factorization UV Th and (3) the proposed ∑K k=1 πk(h)U (k) ( V (k) )> h. In our experiment, we used pooling before projection to compute πk(h), and the number of mixtures was the same as the low-rank dimension d to enable fast computing. We set both the dimensions of W to 1024, namely m = n = 1024. The actual inference time was measured with NVIDIA Jetson TX2 device to simulate edge deployment. Timing was averaged from 500 runs after 200 warm-up runs. The results are shown in Table 6.
We observe that the proposed method provides similar speedup as regular low-rank factorization, especially when the bottleneck layer becomes smaller (which is the case we care more about) and the overhead of computation for mixing weights lessen. With similar speedup, the proposed adaptive lowrank factorization provides better theoretical expressiveness and accuracy/perplexity improvement in practice.

Review summaries:
The paper proposes an input-dependent low rank approximations of weight matrices in neural nets with a goal of accelerating inference. Instead of decomposing a matrix (a layer) to a low rank space like previous work did, the paper proposes to use a linear combination/mixture of multiple low rank spaces. The linear combination is dynamic and data dependent, making it some sense of non-linear combination. The paper is interesting, however,