Deep Learning meets Projective Clustering

1 INTRODUCTION AND MOTIVATION . Deep Learning revolutionized Machine Learning by improving the accuracy by dozens of percents for fundamental tasks in Natural Language Processing ( NLP ) through learning representations of a natural language via a deep neural network ( Mikolov et al. , 2013 ; Radford et al. , 2018 ; Le and Mikolov , 2014 ; Peters et al. , 2018 ; Radford et al. , 2019 ) . Lately , it was shown that there is no need to train those networks from scratch each time we receive a new task/data , but to fine-tune a full pre-trained model on the specific task ( Dai and Le , 2015 ; Radford et al. , 2018 ; Devlin et al. , 2019 ) . However , in many cases , those networks are extremely large compared to classical machine learning models . For example , both BERT ( Devlin et al. , 2019 ) and XLNet ( Yang et al. , 2019 ) have more than 110 million parameters , and RoBERTa ( Liu et al. , 2019b ) consists of more than 125 million parameters . Such large networks have two main drawbacks : ( i ) they use too much storage , e.g . memory or disk space , which may be infeasible for small IoT devices , smartphones , or when a personalized network is needed for each user/object/task , and ( ii ) classification may take too much time , especially for real-time applications such as NLP tasks : speech recognition , translation or speech-to-text . Compressed Networks . To this end , many papers suggested different techniques to compress large NLP networks , e.g. , by low-rank factorization ( Wang et al. , 2019 ; Lan et al. , 2019 ) , prun- ∗equal contribution ing ( McCarley , 2019 ; Michel et al. , 2019 ; Fan et al. , 2019 ; Guo et al. , 2019 ; Gordon et al. , 2020 ) , quantization ( Zafrir et al. , 2019 ; Shen et al. , 2020 ) , weight sharing ( Lan et al. , 2019 ) , and knowledge distillation ( Sanh et al. , 2019 ; Tang et al. , 2019 ; Mukherjee and Awadallah , 2019 ; Liu et al. , 2019a ; Sun et al. , 2019 ; Jiao et al. , 2019 ) ; see more example papers and a comparison table in Gordon ( 2019 ) for compressing the BERT model . There is no consensus on which approach should be used in what contexts . However , in the context of compressing the embedding layer , the most common approach is low-rank factorization as in Lan et al . ( 2019 ) , and it may be combined with other techniques such as quantization and pruning . In this work , we suggest a novel low-rank factorization technique for compressing the embedding layer of a given model . This is motivated by the fact that in many networks , the embedding layer accounts for 20 % − 40 % of the network size . Our approach - MESSI : Multiple ( parallel ) Estimated SVDs for Smaller Intralayers - achieves a better accuracy for the same compression rate compared to the known standard matrix factorization . To present it , we first describe an embedding layer , the known technique for compressing it , and the geometric assumptions underlying this technique . Then , we give our approach followed by geometric intuition , and detailed explanation about the motivation and the architecture changes . Finally , we report our experimental results that demonstrate the strong performance of our technique . Embedding Layer . The embedding layer aims to represent each word from a vocabulary by a real-valued vector that reflects the word ’ s semantic and syntactic information that can be extracted from the language . One can think of the embedding layer as a simple matrix multiplication as follows . The layer receives a standard vector x ∈ Rn ( a row of the identity matrix , exactly one nonzero entry , usually called one-hot vector ) that represents a word in the vocabulary , it multiplies x by a matrix AT ∈ Rd×n to obtain the corresponding d-dimensional word embedding vector y = ATx , which is the row in A that corresponds to the non-zero entry of x . The embedding layer has n input neurons , and the output has d neurons . The nd edges between the input and output neurons define the matrix A ∈ Rn×d . Here , the entry in the ith row and jth column of A is the weight of the edge between the ith input neuron to the jth output neuron ; see Figure . 1 . Compressing by Matrix Factorization . A common approach for compressing an embedding layer is to compute the j-rank approximation Aj ∈ Rn×d of the corresponding matrix A via SVD ( Singular Value Decomposition ; see e.g. , Lan et al . ( 2019 ) ; Yu et al . ( 2017 ) and Acharya et al . ( 2019 ) ) , factor Aj into two smaller matrices U ∈ Rn×j and V ∈ Rj×d ( i.e . Aj = UV ) , and replace the original embedding layer that corresponds to A by a pair of layers that correspond to U and V . The number of parameters is then reduced to j ( n + d ) . Moreover , computing the output takes O ( j ( n+ d ) ) time , compared to the O ( nd ) time for computing ATx . As above , we continue to use Aj to refer to a rank-j approximation of a matrix A . Fine tuning . The layers that correspond to the matrices U and V above are sometimes used only as initial seeds for a training process that is called fine tuning . Here , the training data is fed into the network , and the error is measured with respect to the final classification . Hence , the structure of the data remains the same but the edges are updated in each iteration to give a better accuracy . Observe that typically , the SVD takes the form Aj = UDṼ , where the columns of U ∈ Rn×j are orthogonal , the rows of Ṽ ∈ Rj×d are orthogonal , and D ∈ Rj×j is a diagonal matrix . In this paper and in others , we say that Aj = UV where V = DṼ . Furthermore , the orthogonalization is used only to obtain a low rank approximation Aj = UV using SVD . After that , this property is not kept in the network during the training process ( when applying the fine-tuning ) . Geometric intuition . The embedding layer can be encoded into a matrix A ∈ Rn×d as explained above . Hence , each of the n rows of A corresponds to a point ( vector ) in Rd , and the j-rank approximation Aj ∈ Rn×d represents the projection on the j-dimensional subspace that minimizes the sum of squared distances ( “ errors ” ) to the points . Projecting these points onto any j-dimensional subspace of Rd would allow us to encode every point only via its j-coordinates on this subspace , and store only nj entries instead of the original nd entries of A . This is the matrix U ∈ Rn×j , where each row encodes the corresponding row in A by its j-coordinates on this subspace . The subspace itself can be represented by its basis of j d-dimensional vectors ( jd entries ) , which is the column space of a matrix V T ∈ Rd×j . Figure 2 illustrates the small pair of layers that corresponds to U and V , those layers are a compression for the original big layer that corresponds to A . However , our goal is not only to compress the network or matrix , but also to approximate the original matrix operator A . To this end , among all the possible j-subspaces of Rd , we may be interested in the j-subspace that minimizes the sum of squared distances to the points , i.e. , the sum of squared projected errors . This subspace can be computed easily via SVD . The corresponding projections of the rows of A on this subspace are the rows of the j-rank matrix Aj . The hidden or statistical assumption in this model is that the rows of the matrix A ( that represents the embedding layer ) were actually generated by adding i.i.d . Gaussian noise to each point in a set of n points on a j-dimensional subspace , that is spanned by what are called latent variables or factors . Given only the resulting matrix A , the j-subspace that maximizes the likelihood ( probability ) of generating the original points is spanned by the j largest singular vectors of A . Why a single distribution ? Even if we accept the assumption of Gaussian noise , e.g . due to simplicity of computations or the law of large numbers , it is not intuitively clear why we should assume that the rows of A were sampled from a single distribution . Natural questions that arise are : ( i ) Can we get smaller and/or more accurate models in real-world networks by assuming multiple instead of a single generating distribution ( i.e . multiple subspaces ) ? ( ii ) Can we efficiently compute the corresponding factorizations and represent them as part of a network ? 2 OUR CONTRIBUTION . We answer the above open questions by suggesting the following contributions . In short , the answers are : ( i ) In all the real-world networks that we tested , it is almost always better to assume k ≥ 2 distributions rather than a single one that generated the data . It is better in the sense that the resulting accuracy of the network is better compared to k = 1 ( SVD ) for the same compression rate . ( ii ) While approximating the global minimum is Max-SNP-Hard , our experiments show that we can efficiently compute many local minima and take the smallest one . We then explain how to encode the result back into the network . This is by suggesting a new embedding layer architecture that we call MESSI ( Multiple ( parallel ) Estimated SVDs for Smaller Intralayers ) ; see Figure 3 . Extensive experimental results show significant improvement . Computational Geometry meets Deep Learning . Our technique also constructs the matrix A ∈ Rn×d from a given embedding layer . However , inspired by the geometric intuition from the previous section , we suggest to approximate the n rows of A by clustering them to k ≥ 2 subspaces instead of one . More precisely , given an integer k ≥ 1 we aim to compute a set of k subspaces in Rd , each of dimension j , that will minimize the sum over every squared distance of every point ( row in A ) to its nearest subspace . This can be considered as a combination of j-rank or j-subspace approximation , as defined above , and k-means clustering . In the k-means clustering problem we wish to approximate n points by k center points that minimizes the sum over squared distance between every point to its nearest center . In our case , the k centers points are replaced by k subspaces , each of dimension j . In computational geometry , this type of problem is called projective clustering ( see Figure 4 ) , and its used in many tasks in the fields of Machine Learning and Computer Vision ( Feng et al. , 2011 ; Xu et al. , 2005 ; Liu et al. , 2012 ; Trittenbach and Böhm , 2019 ) , From Embedding layer to Embedding layers . The result of the above technique is a set of k matrices A1j , · · · , Akj , each of rank j and dimension ni × d where the ith matrix corresponds to the cluster of ni points that were projected on the ith j-dimensional subspace . Each of those matrices can be factored into two smaller matrices ( due to its low rank ) , i.e. , for every i ∈ { 1 , · · · , k } , we have Aij = U iV i , where U i ∈ Rni×j , and V i ∈ Rj×d . To plug these matrices as part of the final network instead of the embedded layer , we suggest to encode these matrices via k parallel sub-layers as described in what follows and illustrated in Figure 3 . Our pipeline : MESSI . We construct our new architecture as follows . We use A to refer to the n× d matrix from the embedding layer we seek to compress . The input to our pipeline is the matrix A , positive integers j and k , and ( for the final step ) parameters for the fine-tuning . 1 . Treating the n rows of A as n points in Rd , compute an approximate ( k , j ) -projective clustering . The result is k subspaces in Rd , each of dimension j , that minimize the sum of squared distances from each point ( row in A ) to its closest subspace . For the approximation , we compute a local minimum for this problem using the Expectation-Maximization ( EM ) method ( Dempster et al. , 1977 ) . 2 . Partition the rows of A into k different subsets according to their nearest subspace from the previous step . The result is submatrices A1 , . . . , Ak where Ai is a ni × d matrix and n1 + . . .+ nk = n. 3 . For each matrix Ai where 1 ≤ i ≤ k , factor it to two smaller matrices U i ( of dimensions ni × j ) and V i ( of dimensions j × d ) such that U iV i is the rank-j approximation of Ai . 4 . In the full network , replace the original fully-connected embedding layer by 2 layers . The first layer is a parallelization of k separate fully-connected layers , where for every i ∈ { 1 , · · · , k } the ith parallel layer consists of the matrix U i , i.e. , it has ni input neurons and j output neurons . Here , each row of A is mapped appropriately . The second layer is by combining the matrices V 1 , · · ·V k. Each of the k output vectors from the previous layer u1 , . . . , uk are combined as V 1u1 + . . .+ V kuk ; see Figure 3 for an illustration . 5 . Fine-tune the network . The result is a compressed embedding layer . Every matrix U i has nij parameters , and the matrix V i has jd parameters . Therefore the compressed embedding layer consists of nj + kjd parameters , in comparison to the uncompressed layer of nd parameters . Practical Solution . The projective clustering problem is known to be Max-SNP-hard even for d = 2 and j = 2 , for any approximation factor that is independent of n. Instead , we suggest to use an algorithm that provably converges to a local minimum via the Expectation-Maximization ( EM ) method ( Dempster et al. , 1977 ) , which is a generalization of the well known Lloyd algorithm ( Lloyd , 1982 ) . The resulting clusters and factorizations are used to determine the new architecture and its initial weights ; see Figure 3 for more details . We run on instances of AWS Amazon EC2 cloud , and detail our results in the next section . Open code and networks . Complete open code to reproduce the resulting networks is provided . We expect it to be useful for future research , and give the following few examples .

Summary: This work proposes a new approach, based on projective clustering, for compressing the embedding layers of DNNs for natural language modeling tasks. The authors show that the trade-off between compression and model accuracy can be improved by considering a set of k subspaces rather than just a single subspace. Methods for compressing DNNs is an active area of research and this paper presents a promising approach to do so as well as interesting results.