Image Dataset Compression Based on Matrix Product States

1 INTRODUCTION . Large-scale datasets consisting of millions of samples are becoming the norm to obtain state-of-the-art machine learning models in several fields including speech enhancement and recognition ( Sun et al. , 2020 ; SainathTN et al. , 2013 ) , computer vision ( Russakovsky et al. , 2015 ) and natural language processing ( Devlin et al. , 2019 ) . At such a scale , the resources needed to store datasets and train neural networks become very large , and training machine learning models on it requires the use of specialized equipment and infrastructure . Therefore , it is the critical problems in machine learning that effectively reduce the size of the dataset as well as maintaining the model performance . An intuitive way is data selection , also known as core-set construction method , i.e. , identifying the most representative training samples , which aims to improve the data efficiency of machine learning techniques . ( Agarwal et al. , 2004 ; Chen , 2009 ; Feldman et al. , 2020 ) , mainly focusing on clustering problems . Another dataset compression method is dataset distillation ( Wang et al. , 2018 ; Zhao et al. , 2021 ) , which can learn a small set of informative images from large training data and improve the weakness of data selection methods . However , these studies mainly adopt the existing sample reduction techniques to dataset compression , which may not be intrinsically appropriate for architecture of the image dataset . For example , most dataset compression methods need to adopt the whole image information , although only a small proportion of information ( i.e. , locality information ) will significantly influence the performance during training . Meanwhile , the performance of models trained the offline compressed datasets on downstream tasks would be restricted . In this paper , we introduce a novel matrix product states ( MPS ) technique ( Fannes et al. , 1992 ) from quantum many-body physics for compressing image dataset . The MPS is an algorithm that factorizes a matrix into a sequential product of local tensors ( i.e. , a multi-way array ) . Here , we call the `` locality of pixel dependencies '' in image with short-range correlation and the `` global dependencies '' in image with long-range correlation . An important merit of the MPS decomposition is that it establishes the structural of classical pixel correlation with quantum entanglement entropy ( Srednicki , 1993 ) in the dataset : the larger the entanglement entropy is , the less short-range correlation is , and vice versa . The information that has larger classical correlation is mainly short-range correlation in the image dataset ( Krizhevsky et al. , 2017 ) . The dataset distillation can Such a property motivates us to think about whether such an MPS can be applied to derive a better dataset compression approach . We can compress the dataset by filtering long-range correlation information in task-agnostic scenarios and use the dataset distillation to supplement the information in task-specific scenarios . To this end , we propose a MPS-based Dataset compression approach , called MPSD , to compress the image dataset , which not only enables deep neural networks to obtain similar performance as on the original dataset but also can be used for different models as well as different types of tasks . We have made two import technical contributions for image dataset compression based on MPS . First , we introduce a new task-agnostic dataset compression procedure that efficiently extracting short-range correlation among pixels . We formulate this goal as the problem of minimizing the difference between multiple low-rank tensors with constraints and the original data samples . We present both theoretical discussion and experimental verification for the effectiveness of this dataset compression strategy . Second , we propose a new task-specific module for information supplementation , tailored for machine learning model . Since different downstream tasks have different information for image datasets , the offline dataset compression does not contain task-specific information . We propose a module based on dataset distillation to make the compressed datasets adaptable to different tasks . To our knowledge , it is the first time that MPS is applied to the image dataset compression , which is well suited for model training and dataset storage . We construct experiments to evaluate the effectiveness of the proposed compression approach for CIFAR , FashinMNSIT , and ImageNet , respectively . Extensive experiments have demonstrated the effectiveness of the proposed approach in dataset compression , especially obtained better model performance ( 3.19 % on average ) than similar methods for the same compression rate . In the rest of the paper , we first review the related work in Section 2 . Then we present MPS decomposition and theoretical analysis about quantum entanglement with classical correlation in Section 3 . Section 4 introduce our proposed MPS-based dataset compression approach . We report experimental results in Section 5 and conclude the paper in Section 6 . We will release code and pre-processed data to reproduce our experiments . 2 RELATED WORK . We review the related works in three aspects . Core-set Construction The core-set construction technique of selecting the valid knowledge through an illuminating or a priori approach ( Toneva et al. , 2019 ; Castro et al. , 2018 ; Aljundi et al. , 2019 ; Sener & Savarese , 2018 ) , either by giving illuminating knowledge about the task or by finding representative samples . The core-set construction define representative criterion in the first ( e.g. , compactness ( Rebuffi et al. , 2017 ; Castro et al. , 2018 ) , forgetfulness ( Toneva et al. , 2019 ) , diversity ( Sener & Savarese , 2018 ; Aljundi et al. , 2019 ) ) , then select representative samples from original dataset based on the criterion , finally use the selected small dataset to train the machine learning model for a downstream task . In contrast , our approach does not require the presence of a representative sample and is a more general approach . Knowledge Distillation Knowledge distillation is a technique of transferring knowledge from a collection of models into a single model ( Hinton et al. , 2015 ; Buciluǎ et al. , 2006 ; Ba & Caruana , 2014 ; Romero et al. , 2015 ) . While network distillation aims to distill the knowledge of multiple networks into a single model , dataset distillation models network parameters as a function of synthetic training data and learn their synthetic data by minimizing the training loss on the original training data and the synthetic training data ( Wang et al. , 2018 ) . We use the idea of knowledge distillation to complement the learning of task-relevant information under different tasks . In other words , our goal is to capture the portion of information in the dataset sample that is valid for training deep neural networks , and to perform a `` selection '' of information in the dataset sample . Tensor-based Matrix Representation . Tensor-based method of matrices is a technique that allows representing dataset samples in the tensor form such that quantum entanglement corresponds to classical correlations between different coarse-grained textures ( Latorre , 2005 ) . Another application is the compression of neural networks . Matrix product operators have been used to compress linear layers of deep neural networks ( Gao et al. , 2020 ) . The idea of reshaping weights of fully connected layers into high-dimensional tensors and encoding them in Tensor Train format was introduced by Novikov et al . ( 2015 ) . In contrast , we represent a dataset sample jointly with multiple low-rank tensors , each low-rank tensor describing the difference in information between the previous other tensors and the original graph ( i.e. , residual information ) . Our work is highly built on these studies , while we have a new perspective by designing the dataset compression algorithm which enables extracted short-range correlation in the image . It is the first time that MPS is applied to image dataset compression , and we make contributions for a novel approach to dataset compression . 3 PRELIMINARY . In this paper , scalars are denoted by lowercase letters ( e.g. , a ) , matrices are denoted by boldface capital letters ( e.g. , M ) , and high-order ( order three or higher ) tensors are denoted by boldface Euler script letters ( e.g. , T ) . A 3-order tensor Ti1 , i2 , i3 can be considered as a ( potentially multi-dimensional ) array with 3 indices { i1 , i2 , i3 } . 3.1 MATRIX PRODUCT STATE . Originating from quantum many-body physics , matrix product states ( MPS ) is a standard algorithm to factorize a matrix into a sequential product of multiple local tensors ( i.e. , a multi-way array ) ( Latorre , 2005 ; Perez-Garcia et al. , 2007 ) . This MPS decomposition establishes the structure of classical pixel correlation with quantum entanglement entropy . Formally , given a matrix M ∈ RI×J , its MPS decomposition into a product of n local tensors can be represented as : MPS ( M ) = n∏ k=1 T ( k ) [ dk−1 , jk , dk ] , dk = min ( k∏ m=1 jm , n∏ m=k+1 jm ) , ( 1 ) where the T ( k ) [ dk−1 , jk , dk ] is a 3-order tensor with size dk−1 × jk × dk in which ∏n k=1 jk = I × J and d0 = dn = 1 . We use the concept of bond to connect two adjacent tensors ( Fannes et al. , 1992 ) . The bond dimension dk is defined by : we can see from Equation ( 1 ) that the dk is large in the middle and small on both sides . We present a detailed algorithm for MPS decomposition in Algorithm 1 . It is usually take an odd number of local tensors with MPS . Algorithm 1 MPS decomposition for a matrix . Require : matrix M , the number of local tensors n Ensure : : MPS tensor list { T ( k ) } nk=1 1 : for k = 1→ n− 1 do 2 : M [ I , J ] −→M [ dk−1 × jk , −1 ] 3 : UλV > = SVD ( M ) 4 : U [ dk−1 × jk , dk ] −→ U [ dk−1 , jk , dk ] 5 : T ( k ) : = U 6 : M : = λV > 7 : end for 8 : T ( n ) : = M 9 : Normalization 10 : return { T ( k ) } nk=1 3.2 MPS-BASED LOW-RANK APPROXIMATION .. With the MPS decomposition describe in Equation ( 1 ) , we can exactly decompose a matrix by MPS into the form of a series of products of local tensors and multiply these tensors together to completely reconstruct the original matrix M. We can truncate the k-th bond dimension dk ( see Equation ( 1 ) ) of local tensors to d ′ k for low-rank approximation ( dk > d ′ k ) . Different values for { dk } nk=1 can be set to control the filtering ability of long-range correlation . Definition 1 . ( Local truncation error ) . Let { λj } dkj=1 are the singular values of M [ j1 , . . . , jk , jk+1 , . . . , jn ] . We define the truncation error induced by the k-th bond dimension dk local truncation error k , which can be efficiently computed as k = ∑dk j=dk−d ′ k λj . After defining the local truncation error in Definition 1 , we can derive the upper exact bound of the truncation error of the MPS decomposition by iteration . Theorem 1 . ( Truncation error for MPS-based approximation ) . Let k denoted the local truncation error of k-th bond dimension . The upper exact bound of the truncation error with MPS decomposition can be caclulated by : ||M−MPS ( M ) ||F ≤ √√√√n−1∑ k=1 2k . ( 2 ) The proof can be found in the supplementary materials . Suppose that we have truncated the dimensions of local tensors from { dk } nk=1 to { d ′ k } nk=1 , the compression ratio can be computed by ρ = ∑n k=1 d ′ k−1jkd ′ k∏n k=1 jk . The smaller is the compression ratio , the fewer parameters are kept in MPS representation .

Summary: This paper presented an image compression technique based on matrix product states (MPS) decomposition. After compressing each image, knowledge distillation is applied to compensate the information loss for some specific task. The authors verify the effectiveness of the proposed method on several datasets and neural networks.