Year: 2021

Abstract:
Large-scale datasets have produced impressive advances in machine learning. However, storing datasets and training neural network models on large datasets have become increasingly expensive. In this paper, we present an effective dataset compression approach based on the matrix product states (MPS) from quantum many-body physics. It can decompose an original image into a sequential product of tensors which effectively retain short-range correlation information in the data for training deep neural networks from scratch. Based on the MPS structure, we propose a new dataset compression method that compresses datasets by filtering long-range correlation information in task-agnostic scenarios and uses dataset distillation to supplement the information in task-specific scenarios. Our approach boosts the model performance by information supplementation, and meanwhile maximizes useful information for the downstream task. Extensive experiments have demonstrated the effectiveness of the proposed approach in dataset compression, especially obtained better model performance (3.19% on average) than state-of-theart methods for the same compression rate.

heading: 1 INTRODUCTION
text: Large-scale datasets consisting of millions of samples are becoming the norm to obtain state-of-the-art machine learning models in several fields including speech enhancement and recognition (Sun et al., 2020; SainathTN et al., 2013), computer vision (Russakovsky et al., 2015) and natural language processing (Devlin et al., 2019). At such a scale, the resources needed to store datasets and train neural networks become very large, and training machine learning models on it requires the use of specialized equipment and infrastructure. Therefore, it is the critical problems in machine learning that effectively reduce the size of the dataset as well as maintaining the model performance.
An intuitive way is data selection, also known as core-set construction method, i.e., identifying the most representative training samples, which aims to improve the data efficiency of machine learning techniques. (Agarwal et al., 2004; Chen, 2009; Feldman et al., 2020), mainly focusing on clustering problems. Another dataset compression method is dataset distillation (Wang et al., 2018; Zhao et al., 2021), which can learn a small set of informative images from large training data and improve the weakness of data selection methods. However, these studies mainly adopt the existing sample reduction techniques to dataset compression, which may not be intrinsically appropriate for architecture of the image dataset. For example, most dataset compression methods need to adopt the whole image information, although only a small proportion of information (i.e., locality information) will significantly influence the performance during training. Meanwhile, the performance of models trained the offline compressed datasets on downstream tasks would be restricted.
In this paper, we introduce a novel matrix product states (MPS) technique (Fannes et al., 1992) from quantum many-body physics for compressing image dataset. The MPS is an algorithm that factorizes a matrix into a sequential product of local tensors (i.e., a multi-way array). Here, we call the "locality of pixel dependencies" in image with short-range correlation and the "global dependencies" in image with long-range correlation. An important merit of the MPS decomposition is that it establishes the structural of classical pixel correlation with quantum entanglement entropy (Srednicki, 1993) in the dataset: the larger the entanglement entropy is, the less short-range correlation is, and vice versa. The information that has larger classical correlation is mainly short-range correlation in the image dataset (Krizhevsky et al., 2017). The dataset distillation can Such a property motivates us to think
about whether such an MPS can be applied to derive a better dataset compression approach. We can compress the dataset by filtering long-range correlation information in task-agnostic scenarios and use the dataset distillation to supplement the information in task-specific scenarios.
To this end, we propose a MPS-based Dataset compression approach, called MPSD, to compress the image dataset, which not only enables deep neural networks to obtain similar performance as on the original dataset but also can be used for different models as well as different types of tasks. We have made two import technical contributions for image dataset compression based on MPS. First, we introduce a new task-agnostic dataset compression procedure that efficiently extracting short-range correlation among pixels. We formulate this goal as the problem of minimizing the difference between multiple low-rank tensors with constraints and the original data samples. We present both theoretical discussion and experimental verification for the effectiveness of this dataset compression strategy. Second, we propose a new task-specific module for information supplementation, tailored for machine learning model. Since different downstream tasks have different information for image datasets, the offline dataset compression does not contain task-specific information. We propose a module based on dataset distillation to make the compressed datasets adaptable to different tasks.
To our knowledge, it is the first time that MPS is applied to the image dataset compression, which is well suited for model training and dataset storage. We construct experiments to evaluate the effectiveness of the proposed compression approach for CIFAR, FashinMNSIT, and ImageNet, respectively. Extensive experiments have demonstrated the effectiveness of the proposed approach in dataset compression, especially obtained better model performance (3.19% on average) than similar methods for the same compression rate.
In the rest of the paper, we first review the related work in Section 2. Then we present MPS decomposition and theoretical analysis about quantum entanglement with classical correlation in Section 3. Section 4 introduce our proposed MPS-based dataset compression approach. We report experimental results in Section 5 and conclude the paper in Section 6. We will release code and pre-processed data to reproduce our experiments.

heading: 2 RELATED WORK
text: We review the related works in three aspects.
Core-set Construction The core-set construction technique of selecting the valid knowledge through an illuminating or a priori approach (Toneva et al., 2019; Castro et al., 2018; Aljundi et al., 2019; Sener & Savarese, 2018), either by giving illuminating knowledge about the task or by finding representative samples. The core-set construction define representative criterion in the first (e.g., compactness (Rebuffi et al., 2017; Castro et al., 2018), forgetfulness (Toneva et al., 2019), diversity (Sener & Savarese, 2018; Aljundi et al., 2019)), then select representative samples from original dataset based on the criterion, finally use the selected small dataset to train the machine learning model for a downstream task. In contrast, our approach does not require the presence of a representative sample and is a more general approach.
Knowledge Distillation Knowledge distillation is a technique of transferring knowledge from a collection of models into a single model (Hinton et al., 2015; Buciluǎ et al., 2006; Ba & Caruana, 2014; Romero et al., 2015). While network distillation aims to distill the knowledge of multiple networks into a single model, dataset distillation models network parameters as a function of synthetic training data and learn their synthetic data by minimizing the training loss on the original training data and the synthetic training data (Wang et al., 2018). We use the idea of knowledge distillation to complement the learning of task-relevant information under different tasks. In other words, our goal is to capture the portion of information in the dataset sample that is valid for training deep neural networks, and to perform a "selection" of information in the dataset sample.
Tensor-based Matrix Representation. Tensor-based method of matrices is a technique that allows representing dataset samples in the tensor form such that quantum entanglement corresponds to classical correlations between different coarse-grained textures (Latorre, 2005). Another application is the compression of neural networks. Matrix product operators have been used to compress linear layers of deep neural networks (Gao et al., 2020). The idea of reshaping weights of fully connected
layers into high-dimensional tensors and encoding them in Tensor Train format was introduced by Novikov et al. (2015). In contrast, we represent a dataset sample jointly with multiple low-rank tensors, each low-rank tensor describing the difference in information between the previous other tensors and the original graph (i.e., residual information).
Our work is highly built on these studies, while we have a new perspective by designing the dataset compression algorithm which enables extracted short-range correlation in the image. It is the first time that MPS is applied to image dataset compression, and we make contributions for a novel approach to dataset compression.

heading: 3 PRELIMINARY
text: In this paper, scalars are denoted by lowercase letters (e.g., a), matrices are denoted by boldface capital letters (e.g., M), and high-order (order three or higher) tensors are denoted by boldface Euler script letters (e.g., T ). A 3-order tensor Ti1,i2,i3 can be considered as a (potentially multi-dimensional) array with 3 indices {i1, i2, i3}.

heading: 3.1 MATRIX PRODUCT STATE
text: Originating from quantum many-body physics, matrix product states (MPS) is a standard algorithm to factorize a matrix into a sequential product of multiple local tensors (i.e., a multi-way array) (Latorre, 2005; Perez-Garcia et al., 2007). This MPS decomposition establishes the structure of classical pixel correlation with quantum entanglement entropy. Formally, given a matrix M ∈ RI×J , its MPS decomposition into a product of n local tensors can be represented as:
MPS (M) = n∏ k=1 T(k)[dk−1, jk,dk], dk = min ( k∏ m=1 jm, n∏ m=k+1 jm ) , (1)
where the T(k)[dk−1, jk, dk] is a 3-order tensor with size dk−1 × jk × dk in which ∏n
k=1 jk = I × J and d0 = dn = 1. We use the concept of bond to connect two adjacent tensors (Fannes et al., 1992). The bond dimension dk is defined by: we can see from Equation (1) that the dk is large in the middle and small on both sides. We present a detailed algorithm for MPS decomposition in Algorithm 1. It is usually take an odd number of local tensors with MPS.
Algorithm 1 MPS decomposition for a matrix. Require: matrix M, the number of local tensors n Ensure: : MPS tensor list {T(k)}nk=1
1: for k = 1→ n− 1 do 2: M[I, J ] −→M[dk−1 × jk,−1] 3: UλV> = SVD (M) 4: U[dk−1 × jk, dk] −→ U [dk−1, jk, dk] 5: T (k) := U 6: M := λV> 7: end for 8: T (n) := M 9: Normalization
10: return {T(k)}nk=1

heading: 3.2 MPS-BASED LOW-RANK APPROXIMATION.
text: With the MPS decomposition describe in Equation (1), we can exactly decompose a matrix by MPS into the form of a series of products of local tensors and multiply these tensors together to completely reconstruct the original matrix M. We can truncate the k-th bond dimension dk (see Equation (1)) of local tensors to d ′
k for low-rank approximation (dk > d ′
k). Different values for {dk}nk=1 can be set to control the filtering ability of long-range correlation.
Definition 1. (Local truncation error). Let {λj}dkj=1 are the singular values of M [j1, . . . , jk, jk+1, . . . , jn]. We define the truncation error induced by the k-th bond dimension dk local truncation error k, which can be efficiently computed as k = ∑dk j=dk−d ′ k λj .
After defining the local truncation error in Definition 1, we can derive the upper exact bound of the truncation error of the MPS decomposition by iteration. Theorem 1. (Truncation error for MPS-based approximation). Let k denoted the local truncation error of k-th bond dimension. The upper exact bound of the truncation error with MPS decomposition can be caclulated by:
||M−MPS(M)||F ≤ √√√√n−1∑ k=1 2k. (2)
The proof can be found in the supplementary materials. Suppose that we have truncated the dimensions of local tensors from {dk}nk=1 to {d ′ k}nk=1, the compression ratio can be computed by
ρ = ∑n k=1 d ′ k−1jkd ′ k∏n
k=1 jk . The smaller is the compression ratio, the fewer parameters are kept in MPS
representation.

heading: 3.3 QUANTUM ENTANGLEMENT AND CLASSICAL CORRELATIONS
text: Since the MPS representation can be constructed by operating a series of successive Schmidt decompositions (Vidal, 2004). The entanglement entropy is suitable as a metric to measure correlation information contained in bonds of MPS, which is analogous to entropy in information theory but replaces probabilities with normalized singular values created by SVD. Following (Calabrese & Cardy, 2004), the entanglement entropy with the k-th bond can be calculated by:
Ek = − dk∑ j=1 vj lnvj , k = 1, 2, . . . , n− 1 (3)
where {vj}dkj=1 denote the normalized SVD eigenvalues of orignal matrix M[j1 . . . jk, jk+1 . . . jn]. From Equation (3), we note that the larger the entanglement entropy is, the less short-range correlation is, and vice versa.

heading: 4 APPROACH
text: The short-range correlation information of image datasets is very important for training step. MPS decomposition can effectively filter the short-range correlation information from the image dataset. Hence, it would be natural to apply an MPS-based approximation to compress the image matrices in the dataset by truncating the bond dimensions of MPS. In particular, we propose two main improvements for MPS-based dataset compression, which can efficiently compress the image dataset and effectively complementary task-specific information.

heading: 4.1 TASK-AGNOSTIC DATASET COMPRESSION
text: Suppose we are given a large dataset consisting of |S| training samples S = {(Si)}|S|i=1 where Si ∈ S ⊂ Rd, S is a d-dimensional input space. We denotes the MPS(Si) as the truncated tensor set with MPS decomposition on Si. Similar with Image Compression with Entanglement (ICE), which was proposed to use MPS for truncated compression after performing a discrete cosine Fourier transform of Si (Latorre, 2005). We use the MPS decomposition of images to filter long-range correlation information to achieve dataset compression. However, the dataset after this method for compression has a significant information loss since the truncated dimension {d′k}nk=1 decreases (this is discussed in Section 3.2.).
To address this problem, inspired by He et al. (2016), we propose to insert residual information (i.e., the difference between Si and MPS(Si)) to the MPS representation, which is defined by:
R(Si) = Si −MPS(Si). (4)
Then, we explicitly let R(Si) approximate the residual information. The original image dataset sample Si can be computed as follows:
MPSTotal(Si) = MPS(Si) +MPS(R(Si)). (5)
In particular, when dk = d ′
k in MPS(Si), the value of R(Si) is 0 and MPS(Si) is strictly equal to the original matrix Si (the upper bound of the error is 0 according to Equation (2)). Algorithm 2 presents a complete procedure for our approach.
Algorithm 2 Task-agnostic Dataset Compression Procedure. Require: Image training dataset with N samples (S). Ensure: : Compressed training dataset.
1: for i = 1→ N do 2: Perform MPS decomposition: MPS(Si) = ∏n k=1 T(k)[dk−1, jk,dk] 3: Compress MPS tensors by trucating {dk}nk=1 −→ {d ′
k}nk=1 4: Computing residual information: R(Si) = Si −MPS(Si) 5: Perform MPS decomposition: MPS(R(Si)) = ∏n k=1R(k)[d (r) k−1, jk,d (r) k ] 6: Compress MPS tensors by trucating {d(r)k }nk=1 −→ {d ′(r) k }nk=1 7: Computing Compressed sample: MPSTotal(Si) = MPS(Si) +MPS(R(Si)) 8: end for 9: return Compressed dataset
As a result, we estimate Si with multiple MPS. Because each MPS has limited parameters, the sum of their parameters is still considerably smaller than the original Si, allowing for the dataset compression effect. This idea of using multiple MPS to approximate the original data is inspired by the idea of residuals, and the new MPS state is a description of the discrepancy information between the original data sample Si and the truncated representation MPS(Si). We show empirically (Table 1) that our proposed approach can improve the model performance significantly than ICE (Latorre, 2005) for the same compression rate.

heading: 4.2 TASK-SPECIFIC INFORMATION SUPPLEMENTATION
text: Task-independent dataset compression is the extraction of short-range correlation information from the dataset. Considering the different network structures and task types, it is necessary to add further information from the learning procedure. Hence we initialize trainable matrix S̃i as implicit bias between MPSTotal(Si) and real dataset sample Si. Then we introduce the knowledge distillation loss function to learn S̃i.
Knowledge distillation is used to distill the knowledge from a large training dataset into a small one (Wang et al., 2018). They synthesize data matrix as training data to approximate models trained on the original data. Inspired by data distillation, we initialize trainable matrix S̃i with zeros as implicit bias so that adding S̃i would not hurt the model performance at the first step of training. Similarly, in the context of information supplementation, the synthetic dataset is calculated by Si ∗ = MPSTotal(S̃i) + S̃i. Then the synthetic dataset Si∗ is trained to mimic the behaviors of the real dataset Si with the model fixed. Formally, This training process can be modeled as minimizing the following objective function:
LKD = ∑ S∈S L(f(MPS(Si) +MPS(R(Si)) + S̃i), f(Si)), (6)
where L(·) is a loss function that evaluates the difference between outputs of real and synthetic datasets. Finally, our approach provides a more principle way of information supplementation. By updating implicit bias S̃i, the synthetic dataset sample Si∗ can better adapt to a specific task or network architecture, and thus achieve better performance.

heading: 4.3 THE OVERALL PROCEDURE
text: Our approach can compress is general. In other words, it can work with existing dataset compression methods to further obtain better compression performance. Here, we choose CIFAR 1 and FashionMNIST 2 as representative image datasets and use our algorithm for these datasets.
The procedure can be simply summarized as follows. First, we perform MPS decomposition separately for the samples in the image dataset. Each sample matrix will be decomposed into a series of local tensors as decrypt in Equation (1). Next, we truncate the connection bond to filtering long-range correlation in the image according to the compression requirements. Then, task-agnostic dataset compression is performed to supplement the difference information between the original image and the truncated MPS image. Finally, a task-specific information supplementation strategy based on knowledge distillation is executed on the task-specific model training. In this way, we expect the dataset to be effectively compressed. In particular, this task-agnostic compressed dataset can be easily generalized to different models as well as to different tasks.

heading: 4.4 DISCUSSION
text: Inspired by Latorre (2005), the image can be represented in the form of matrix product states such that quantum entanglement corresponds to the classical correlation between different coarse-grained textures. The truncation of MPS corresponds to the compression of the original image. In the classification and detection problems, the information that helps the model is mainly short-range correlation information (Krizhevsky et al., 2017). While long-range correlations are noisy for machine learning models and they are not helpful for convergence of machine model training (Gao et al., 2020). With the help of MPS representation of the dataset samples, short-range and long-range correlation information can be effectively decoupling. Our goal is to achieve compression of the dataset by filtering long-range correlation in image. We empirically demonstrate the greater ability of multiple MPS to filter long-range correlation information from images.
In mathematics, MPS-based approximation can be considered as a special low-rank approximation method. We compare it with other low-rank approximation methods, including SVD (Henry & Hofrichter, 1992), CP decomposition (Hitchcock, 1927), and Tucker decomposition (Tucker, 1966). We present the results of these tensor based low-rank approximation methods in Section 5.1 (Table 2).
1Available at https://www.cs.toronto.edu/~kriz/cifar.html 2Available at https://www.worldlink.com.cn/en/osdir/fashion-mnist.html
In practice, we do not need to strictly follow the original image size. Instead, it is easy to pad additional zero entries to enlarge matrix rows or columns, so that we can obtain different MPS decomposition results. Another note is that the MPS-based approach can work with other compression methods: it can compress matrices condensed by previous methods even more.

heading: 5 EXPERIMENTS
text: In this section, we first set up the experiments, and then report the results and analysis. Furthermore, the effectiveness of our approach is further demonstrated on other tasks.
Datasets. We first evaluate classification performance with compressed images on four three standard benchmark datasets: CIFAR10, CIFAR100 and FashionMNIST. In particular, the FashionMNIST dataset has 60,000 training and 10,000 testing images of 10 classes, while CIFAR10 and CIFAR100 both have 50,000 training and 10,000 testing images from 10 and 100 object categories, respectively. In all experiments, we use the standard train/test splits of the datasets and finally report the accuracy of the testing dataset.
Baselines. Our baseline methods include:
• ICE (Latorre, 2005): It first transforms images into MPS representation after performing a discrete cosine Fourier transform and truncate the dimensions only once for compression.
• Core-set Selection: It reduces the large dataset into a small equally informative portion of data, including Random and K-Center. In Random, the training samples are randomly selected as the core-set. K-Center (Wolf, 2011) picks multiple center points such that the largest distance between a data point and its nearest center is minimized.
• Tensor Based Methods: They compress the dataset by applying low-rank approximation to each image, including Tucker decomposition and CP decomposition.
Implementations. Following Latorre (2005), we first represent the image with MPS format (i.e., a product of n local tensors) and then apply task-agnostic dataset compression as well as task-specific information supplementation to reduce total dataset size. To ensure a fair comparison, we adopt the same architecture, ResNet18 (He et al., 2016), , for different dataset compression methods. For simplicity and generality, we use default model hyper-parameters of ResNet18 and a consistent augmentation strategy for all datasets. For example, the learning rate, the minibatch size, and training epochs are set to 0.1, 128 and 180, respectively. Finally, we report results on testing datasets with the best model on evaluation datasets.

heading: 5.1 EXPERIMENTAL RESULTS
text: Comparison to Image Compression. As shown in Table 1, our approach is very competitive in the three image classification benchmark, and it outperforms the image compression method in all
tasks. Looking at CIFAR100, compared with ICE, our approach improves 6.23% in terms of test performance at the same compression ratio. By zooming in on a specific dataset, the improvements over CIFAR100 are larger than the other tasks. Note that compared to CIFAR10 and FashionMNIST, CIFAR100 is more challenging, as recognizing 10 times more categories with 110 fewer images per class in CIFAR100. The MPS dataset seems to work better with few shot tasks, which enhances the data efficiency of the training dataset.
Comparison to Core-set Methods. To demonstrate the strength of image compression with MPS over the core-set selection, we do experiments on CIFAR10 and use Random and K-Center for comparison. Table 1 summarizes the results. Overall, our approach achieves competitive results over coreset selection methods, especially for the K-Center method. This is achieved due to the completeness that the dataset compression method has and its intrinsic characteristic of effectively preserving short-range correlation for each image. To compare the quality of truncated MPS representation for CIFAR10 and understand them intuitively, we visualize images from five categories with different dimensions d ′
k in Figure 3. We observe that it is impossible to see the difference before and after compression if ρ is larger
than 36%. Compared to losing some images by the core-set selection, filtering long-range correlation information by our approach can minimize the damage to the dataset.
Comparison to Tensor Based Methods. As discussed in Section 4.4, we use other tensor-based methods (i.e., SVD (Henry & Hofrichter, 1992), CP (Hitchcock, 1927)) for comparison to demonstrate the effectiveness of preserving short-range correlations with MPS. From the Tabel 1, we observe that SVD and CP decomposition failed to preserve useful information for model performance especially when the compression ratio is less than 40%. While our approach can still have competitive accuracy over CIFAR10 when very limited information is preserved.
Comparison to Different Models In general, our approach can be applied to any kind of network architecture. We have evaluated its performance with ResNet18. Now, We continue to test our approach using another two standard deep network architectures: VGG-16 (Simonyan & Zisserman, 2015) and MobileNetV2 (Sandler et al., 2018). These models are famous pre-trained models that showed state-of-the-art accuracy for several challenging recognition tasks on ImageNet and competitions. Table 2 presents the comparison of the testing accuracy with three network architectures. As we can see, the MPS dataset can cooperate with different kinds of network architectures.

heading: 5.2 EVALUATION ON MORE TASKS
text: As introduced in Section 4, our approach contains task-agnostic dataset compression and task-specific information supplementation. Due to task-agnostic compression, MPS representation can be applied in other computer vision tasks (i.e., Pedestrian Detection, Visual Question Answering and Large-scale Image Classification).
Pedestrian Detection First, we apply our approach to an pedestrian detection scenario where the goal is to accurately locate pedestrians in an image. We build our model on Mask R-CNN (He et al., 2017) method and fine-tune a pre-trained Mask R-CNN model in the Penn-Fudan Database (Wang et al., 2007) for Pedestrian Detection and Segmentation task. The dataset contains 170 images with 345 instances of pedestrians and we decompose the original images with MPS and use a trainable matrix to supplement information. The desired outcome is to obtain a high mean of average precision over all the classes. Finally, we report a COCO-style mAP score after 10 epochs of training, and the result is shown in Table 3. The result indicates that MPS dataset can achieve competitive model performance while reducing 25% parameters of the dataset.
Visual Question Answering The Visual Question Answering task typically uses paired images and text to bridge vision and language respectively. Current approaches to this heavily rely on image feature extraction processes. Here we explore the use of our approach on VQAv2. The VQAv2 task asks for answers given pairs of an image and a question in natural language. Test-dev score is calculated by comparing the inferred answer to the 10 ground-truth answers. Our goal is to verify that short-range correlation in images can be used to efficiently model the cross-modal interaction between image-text pairs. To this end, we replace the image with MPS representation in the image-text pairs. Following Kim et al. (2021), we use a pre-trained ViLT model and fine-tune the model on the MPS dataset. Finally, from Tabel 3 we observe that our approach achieves comparable testing performance, and meanwhile significantly decreases the size of the dataset.
Large-scale Image Classification Pre-trained deep learning models (ResNet, VGG) learned on large-scale datasets have shown their effectiveness over conventional methods. Instead of training a model from scratch, one can fine-tune a pre-trained model to solve some specific task. To demonstrate the effectiveness of short-range correlation on transfer learning, we apply our approach to the ImageNet dataset. To this end, we observe that the total dataset size is significantly reduced due to the compression on each image in the dataset. Furthermore, we evaluate the performance of the pre-training ResNet18 model on both the original ImageNet and the MPS compressed dataset. We observe that the MPS dataset achieves comparable accuracy to the original ImageNet dataset. This result shows that the MPS dataset with short-range correlation can support large-scale pre-training.

heading: 6 CONCLUSION
text: We propose a dataset compression approach based on MPS and distillation. With MPS decomposition, it is able to efficiently reorganize and decouple short-range and long-range correlation information in local tensors. The MPS can be used to correspond the classical correlations to quantum entanglement, and the short-range and long-range correlations to small and large magnitude of entanglement entropy, respectively. We empirically found that the short-range correlation in images is important for training. Inspired by this, we design a novel dataset compression approach that achieves effective compression of dataset by filtering long-range correlation features from images in task-agnostic scenario, while using distillation to complement task-relevant information. Extensive experiments have demonstrated the effectiveness of our approach, especially in that the compressed dataset using the MPS decomposition can be directly applied to a variety of different neural network tasks. To the best of our knowledge, this is the first application of MPS for dataset compression. In future work, we will consider exploring more decomposition structures for MPS.

heading: A TENSOR AND MATRIX PRODUCT STATES
text: As introduced in Cichocki et al. (2009), the concept of tensor is specified as:
Definition1 (Tensor). Let D1, D2..., DN ∈ N denote index upper bounds. A tensor T ∈ RD1,...,Dn of order N is an N -way array where elements Td1,d2,...,dn are indexed by dn ∈ {1, 2, ..., Dn} for 1 ≤ n ≤ N
Definition2 (Matrix product state). We can reshape a matrix to high order tensor, denote as:
Mx×y = Mi1i2...in . (7)
Here, the one-dimensional coordinate x and y means the input and output signal, respectively. So that x× y with dimension NxNy is reshaped into a coordinate in a n-dimensional space, labelled by (i1i2 · · · in). Hence, there is a one-to-one mapping between original matrix and (i1i2 · · · in). If Ik are the dimensions of ik, then:
n∏ k=1 Ik = NxNy. (8)
The MPS representation of M is obtained by factorizing it into a product of n local tensors.
Mi1···in,j1···jn = T (1)[i1, d1, j1] · · · T (n)[in, dn, jn], (9)
where T (k)[ik, dk, jk] is a 3-order tensor. The Dk is the virtual basis dimension on the bond linking T (k) and T (k+1) with D0 = Dn = 1. Dk−1 ×Dk matrix with Dk the virtual basis dimension on the bond linking T (k) and T (k+1) with D0 = Dn = 1.

heading: B THEOREM
text: Theorem 1. Suppose that the tensor W(k) of matrix W that is satisfy
W = W(k) +E(k), D(W(k)) = dk,
where ||E(k)||2F = 2k, k = 1, ..., d− 1. (10)
Then MPS (W) with the k-th bond dimension dk upper bound of truncation error satisfy:
||W −MPS (W)||F ≤ √√√√d−1∑ k=1 2k (11)
Proof. The proof is by induction. For n = 2 the statement follows from the properites of the SVD. Consider an arbitrary n > 2. Then the first unfolding W(1) is decomposed as
W(1) = U1λ1V1 +E (1) = U1B (1) +E(1) (12)
where U1 is of size r1 × i1 × j1 and ||E(1)||2F = 21. The matrix B1 is naturally associated with a (n− 1)-dimensional tensor B(1) with elements B(1)(α, i2, j2, ..., in, jn), which will be decomposed
further. This means that B1 will be approximated by some other matrix B̂1. From the properties of the SVD it follows that UT1 E (1) = 0, and thus
||W − B(1)||2F = ||W1 −U1B̂1||2F = ||W1 −U1(B̂1 +B1 −B1)||2F = ||W1 −U1B1||2F + ||U1(B̂1 −B1)||2F (13)
and since U1 has orthonormal columns,
||W − B(1)||2F ≤ 21 + ||B1 − B̂1||2F . (14)
and thus it is not difficult to see from the orthonormality of columns of U1 that the distance of the k-th unfolding (k = 2, ..., dk − 1) of the (d− 1)-dimensional tensor B(1) to the dk-th rank matrix cannot be larger then k. Proceeding by induction, we have
||B1 − B̂1||2F ≤ d−1∑ k=2 2k, (15)
combine with Eq. equation 14, this complets the proof.

heading: C EXPERIMENT
text: C.1 ADDITIONAL DETAILS OF MPS DATASET
In this paper, the MPS is proposed for compressing CIFAR10, CIFAR100 and FashionMNIST, respectively. In order to show that the process of incorporating several MPS structures into different dataset. We introduce MPS decomposition in different image shape:
We design different MPS structures based on the image size of the dataset. Accordingly, the compression ratio is calculated by corresponding truncated dimension.

Review summaries:
While Large-scale data is beneficial for machine learning, it causes high storage and training consumption. Stressing this problem, this paper proposes to use the tensor decomposition method for compressing datasets. Filtering long-range correlation information in task-agnostic scenarios via MPS structure, this method achieves a good compression ratio while preserving the model performance.

This work proposes matrix product state (MPS) tensor networks for compression of image data. The authors propose to approximate input image matrices (order-2 tensors) with a tensor train comprising order-2 and order-3 tensors and claim possible task agnostic compression of the datasets. Experiments are reported on CIFAR, FashionMNIST for two different compression levels and the MPS compression is shown to fare better than other (simple) baselines. Several important details are unclear/missing in the manuscript making it difficult to fully understand/review this work. 

This paper presented an image compression technique based on matrix product states (MPS) decomposition. After compressing each image, knowledge distillation is applied to compensate the information loss for some specific task. The authors verify the effectiveness of the proposed method on several datasets and neural networks.