# Least Square Support Tensor Regression Machine Based on Submatrix of the Tensor

Authors: Tuo Shu; Zhi-Xia Yang
Journal: Mathematical Problems in Engineering
 (2017)

---

## Abstract

For tensor regression problem, a novel method, called least square support tensor regression machine based on submatrix of a tensor (LS-STRM-SMT), is proposed. LS-STRM-SMT is a method which can be applied to deal with tensor regression problem more efficiently. First, we develop least square support matrix regression machine (LS-SMRM) and propose a fixed point algorithm to solve it. And then LS-STRM-SMT for tensor data is proposed. Inspired by the relation between photochrome and the gray pictures, we reformulate the tensor sample training set and form the new model (LS-STRM-SMT) for tensor regression problem. With the introduction of projection matrices and another fixed point algorithm, we turn the LS-STRM-SMT model into several related LS-SMRM models which are solved by the algorithm for LS-SMRM. Since the fixed point algorithm is used twice while solving the LS-STRM-SMT problem, we call the algorithm dual fixed point algorithm (DFPA). Our method (LS-STRM-SMT) has been compared with several typical support tensor regression machines (STRMs). From theoretical point of view, our algorithm has less parameters and its computational complexity should be lower, especially when the rank of submatrixK is small. The numerical experiments indicate that our algorithm has a better performance.


---

## Body

## 1. Introduction

As we all know, in the past decades, matrices or more generally multiway arrays (tensors) types of data have an increasing number of applications. For example, all raster images are essentially digital readings of a grid of sensors and matrix analysis is widely applied in image processing, for example, photo realistic images of faces [1], palms [2], and medical images [3]. In web search, a large amount of tensors that stand for images [4] can be found easily. Therefore, tensor data analysis [5], particularly regression problem [6, 7], has become one of the most important topics for face recognition [8], palmprint recognition [9], and so on.Tensor types of data have greatly drawn the attention of people. Recently, several tensor learning for regression approaches [10, 11] appears, but the majority of them dealing with tensor regression problems work on vector spaces that are derived by stacking the original tensor elements in a more or less arbitrary order. This vectorization of data causes many new problems. First, the structural information is destroyed. Second, the vectorization of a tensor may bring an extremely high dimensionality vector which may lead to high computational complexity, overfitting, and large memory requirement. The rest of the methods mainly take advantage of the decomposition of a matrix [12] or tensor [6], which can reduce the high computational complexity as well as high dimensionality at the expense of slight decline of accuracy, but the structural information is destroyed totally. So a more reasonable method which can reserve the underlying structural information and avoid overfitting, high dimensionality, and high computational complexity is needed.Considering the fact that a colorful photograph can be expressed as a third-order tensor, of which each frontal slice is indeed a gray image that contains almost all the information of the colorful photograph, we can take advantage of this by introducing submatrix of a tensor and abstract vector space when solving tensor learning for regression problems. That means, each tensor data sample can be regarded as an abstract vector [13] whose elements are submatrix types of features. Gathering together the same feature information of different tensor data sample, we can construct new submatrix training sets and the same number of related training models, from which we can get an equal amount of weight submatrix. And then, the weight tensor is obtained. Besides, we improve the fixed point algorithm [14] via some projection matrices [15], including a series of left projection matrices and a right projection matrix. The improved algorithm is called dual fixed point algorithm (DFPA). The projection matrices can not only join the training models up but also reduce computational complexity and large memory requirement. That is to say, we turn the LS-STRM-SMT problems into solving a battery of least square support matrix machine (LS-SMRM) by fixing the projection matrices and then solve the LS-SMRMs problems with fixed point algorithm. The numerical experiments indicate that our method and algorithm have a better performance.The paper is organized as follows: in Section2, notations and preliminaries are introduced, such as definitions related to tensors and notation that will be used. In Section 3, we propose our (LS-SMRM) for matrix regression problems and the fixed point algorithm for them. In Section 4, we propose the LS-STRM-SMT models and develop the DFPA to solve them. Computational comparisons on both UCI data sets and artificial data are done in Section 5 and conclusions in Section 6.


## 2. Notations and Preliminaries

Here, we will give a brief description of the notations that will be used in the later sections. More specifically, boldface capital letters, for example,A, boldface lowercase letters, for example, a, and lowercase letters, for example, a, are used to denoted matrices, vectors, and scalars, respectively. Tensors regarded as multidimensional arrays will be denoted by Euler script calligraphic letters, for example, X∈RI1×I2⋯×IM, where M denotes the order of the tensor. Inspired by the fact that ith element of a vector x∈Rn is denoted by xi,i=1,2,…,n, the elements of an M-order tensor X will be denoted by xi1i2⋯iM,il=1,2,…,Il,l=1,2,…,M.For anM-order tensor X∈RI1×I2⋯×IM, the d-mode matricization also known as unfolding or flattening is denoted by(1)Xd=matdX∈RId×I1I2⋯Id-1Id+1⋯IM,d=1,2…,M.It is quite clear that we can reorder the elements of the tensor into a matrix in such a way. On the contrary, define a mapping function(2)·d:RId×I1I2⋯Id-1Id+1⋯IM⟶RI1×I2⋯×IM,to recover a tensor from its unfolding matrix. Particularly, when M=3, we have(3)·1X1=X,X1∈RI1×I2I3,·2X2=X,X2∈RI2×I1I3,·3X3=X,X3∈RI3×I1I2.The inner product of the two same size tensors X, Y∈RI1×I2⋯×IM is defined as(4)X,Y=∑i1=1I1​⋯∑iM=1IMxi1i2⋯inyi1i2⋯in.The Frobenius norm of a tensor is thus defined as (5)XFro=X,X.And it can be shown that (6)XFro=XdFro=trXdXdT,d=1,2,…,M.The Contrast Pyramid, referred to as CP, decomposition factorizes an M order tensor X∈RI1×I2⋯×IM into a linear combination of R rank-one tensors, written as(7)X≈∑r=1Rur1∘ur2⋯∘urM≜U1,U2,…,UM,where the operator ∘ is the outer product of vectors and the factor matrix Uk=[u1k,u2k,…,uRk]∈RIk×R,k=1,2,…,M, of the size Ik×R,k=1,2,…,M. For the convenience, the mentioned tensor is the third-order tensor in the following content if there is no special instruction.


## 3. Least Square Support Matrix Regression Machine (LS-SMRM)

In the section, we propose least square support matrix regression machine, shorten as LS-SMRM, for the regression problem with matrix input.Give a training set(8)T=X1,y1,X2,y2,…,XL,yL,where Xi∈RI1×I2 is the input and yi∈R is the output, i=1,2,…,L. Our task is to find a predictor(9)y=W,X+b,where W denotes the weight matrix and b is the bias. For a new input matrix, we can predict its output through the above-mentioned predictor.In order to get predictor (9), we develop the following optimization problem:(10)minW,b,η12WFro2+C2∑i=1Lηi2s.t.yi-W,Xi+b=ηi,i=1,2,…,L,where C>0 is penalty parameter.According to the CP decomposition (7), the matrices V and U can be found to make(11)W≈∑r=1Kvr∘ur≜V,U,where V=(v1,v2,…,vK),U=(u1,u2,…,uK),vi∈RI1,ui∈RI2,i=1,2,…,K, and K=2,3,…,min{I1,I2}. Then, optimization problem (10) can be turned into as follows:(12)minV,U,η,b12VUTFro2+C2∑i=1Lηi2,s.t.yi-∑j=1KvjTXiuj+b=ηi,i=1,2,…,L.The fixed point algorithm is applied to solve optimization problem (12). When U=(u1,u2,…,uK) is fixed, we need to compute a set of vj for j=1,2,…,K and b. Firstly, denote(13)f^i=Xiu1Xiu2⋮XiuKI1K×1,v^=v1v2⋮vKI1K×1D=UTU⊗II1×I1,and the optimization problems (12) are equivalent to(14)minv^,η,b12v^TDv^+C2∑i=1Lηi2s.t.yi-v^Tf^i+b=ηi,i=1,2,…,L.And let(15)v=D1/2v^,fi=D-1/2f^i;optimization problems (14) are reformulated as(16)minv,η,b12vTv+C2∑i=1Lηi2,s.t.yi-vTfi+b=ηi,i=1,2,…,L.The Lagrange function of optimization problems (16) can be expressed as(17)L=12vTv+C2∑i=1Lηi2+∑i=1Lαiyi-vTfi-b-ηi,where α=(α1,α2,…,αL)T is Lagrangian multiplier vector. Then, the KKT system of (17) is(18)∂L∂v=v-∑i=1Lαifi=0⟹v=∑i=1Lαifi,(19)∂L∂b=-∑i=1Lαi=0⟹∑i=1Lαi=0,(20)∂L∂ηi=C∑i=1Lηi-∑i=1Lαi=0⟹ηi=1Cαi.Rewrite (18), (19), and (20) into the form of equation; we can get(21)HF10HTbα=y0,where(22)y=y1,y2,…,yLT,H=ones1,LT,F1=f1,f1+1Cf1,f2⋯f1,fLf2,f1f2,f2+1C⋯f2,fL⋮⋮⋮fL,f1fL,f2⋯fL,fL+1C.b and α can be got by solving linear system (21). Then, v is obtained according to (18) and the right projection matrix U is accessed through the relation among v, (13) and (15). In summary, when we fix vi,i=1,2,…,K, the solution of optimization problems (12) can be computed by solving linear system (21) directly.Similarly, whenV=(v1,v2,…,vK) is fixed, we can also change the formulation of our algorithm in optimization problems (12) and derive its optimal U and b by solving another linear system. That is to say, we need to compute a set of uj for j=1,2,…,K and b. Firstly, denote(23)g^i=XiTv1XiTv2⋮XiTvKI2K×1,u^=u1u2⋮uKI2K×1,Q=VTV⊗II2×I2;optimization problems (12) is equivalent to(24)minu^,η,b12u^TQu^+C2∑i=1Lξi2s.t.yi-u^Tg^i+b=ξi,i=1,2,…,L.And let(25)u=Q1/2u^,gi=Q-1/2g^i;the optimization problems (24) are reformulated as(26)minv,η,b12uTu+C2∑i=1Lξi2,s.t.yi-uTgi+b=ξi,i=1,2,…,L.The Lagrange function of optimization problems (26) can be expressed as(27)G=12uTu+C2∑i=1Lηi2+∑i=1Lβiyi-uTgi-b-ξi,where β=(β1,β2,…,βL)T is Lagrangian multiplier vector. Then, the KKT system of (27) is(28)∂G∂u=u-∑i=1Lβigi=0⟹u=∑i=1Lβigi,(29)∂G∂b=-∑i=1Lβi=0⟹∑i=1Lβi=0,(30)∂G∂ξi=C∑i=1Lξi-∑i=1Lβi=0⟹ξi=1Cβi.Rewrite (28), (29), and (30) into the form of equation; we can get(31)HF20HTbβ=y0,where(32)y=y1,y2,…,yLT,H=ones1,LT,F2=g1,g1+1Cg1,g2⋯g1,gLg2,g1g2,g2+1C⋯g2,gL⋮⋮⋮gL,g1gL,g2⋯gL,gL+1C.Repeating the iterative operation until convergence, the weight matrixW is obtained by (11), and the predictor is (33)yi=W,Xi+b=VUT,Xi+b=∑j=1KvjTXiuj+b.According to the above description, we can summarize the following algorithm.Algorithm 1 (LS-SMRM).
(
1) Training Process
Input. Training set (8).
Output. Left and right projection matrices V,U, and bias b. (a)
InitializeU=(u1,u2,…,uK),ui∈RI1,i=1,2,…,K,K=2,3,…,min⁡{I1,I2}.(b)
Formulatefi,i=1,2,…,L. by (13).(c)
Alternatively updateV and U until convergence.
(1)
UpdateV and b.
(1.1)
Getα and b by solving linear system (21).(1.2)
Getv from (19).(1.3)
Getv^ from (15).(1.4)
Getv from (13).(2)
UpdateU and b.
(2.1)
Getβ and b by solving linear system (31).(2.2)
Getu from (28).(2.3)
Getu^ from (25).(2.4)
Getu from (23).
(
2) Testing Process
Input. Testing point X, left and right projection matrices V,U, and bias b.
Output. The value of testing set: y(34)y=W,X+b=VUT,X+b=∑j=1KvjTXuj+b.


## 4. LS-STRM Based on Submatrix of the Tensor (LS-STRM-SMT)

In this section, we propose least square support tensor regression machine, shorten as LS-STRM, for the regression problem with the tensor input. In fact, with the introduction of submatrix of the tensor, the LS-STRM problem is turned intoI3 LS-SMRM problems which are independent. However, the right project matrices should be made equal to fit the practical situation. That is to say, we need to solve I3 LS-SMRM problems with the same right projection matrix. To show the effectiveness of the proposed algorithm, we will provide some deep analyses in the section.


### 4.1. Formulation

Give a training set(35)T=X1,y1,X2,y2,…,XL,yL,where Xi∈RI1×I2×I3 is the input and yi∈R is the output, i=1,2,…,L. Our task is to find a predictor(36)y=W,X+b,where W denotes the weight tensor and b is the bias. For a new input tensor, we can predict its output through predictor (36). For convenience, we set b=0.Definition 2.
For anM-order tensor X∈RI1×I2⋯×IM, the submatrix of the tensor is defined as(37)mat1X:,:,…,:,k,k=1,2,…,IM.Particularly, when M=3, the submatrices of a third-order tensor are indeed the frontal slices.However, we do not construct the model for training set (35) directly. We transform the training set (35) into I3 training sets similar to training set (8) by the introduction of the submatrix of the tensor and then construct I3 regression problems. That means, for the training set (35), every input tensor Xi∈RI1×I2×I3,i=1,2,…,L, can be regarded as an abstract vector(38)Xi1,Xi2,…,XiI3,i=1,2,…,L,where Xi(j),j=1,2,…,I3, is the jth frontal slice of the tensor Xi. Next, we take the jth frontal slice of each tensor Xi,i=1,2,…,L, out and construct the following training set:(39)Tj=Xij,Yij,i=1,2,…,L,where Yi(j)∈R denotes the jth element of Yi, Yi=(yi(1),yi(2),…,yi(I3))T is a vector obey normal distribution with mean yi and variance σ.According to training set (39), I3 optimization problems are constructed as follows:(40)minWj12WjF2+Cj2∑i=1Lηij2s.t.yij-trWjTXij=ηij,j=1,2,…,I3,where Wj=W(:,:,j)=VjUjT,Xi(j)=Xi(:,:,j), j=1,2,…,I3,i=1,2,…,L. So we can get W(1)=(W1,W2,…,WI3), and then the weight tensor can be obtained by (41)W=·1W1.It is clear that the I3 model is independent, but this is contrary to the truth. In order to reflect the relation among them, we set (42)U1=U2=⋯=UI3=U,so that the models can fit the practical situation better. The I3 models can be expressed together as follows:(43)minU,W12WF2+Cj2∑i=1L∑j=1I3ηij2s.t.yij-trWjTXij=ηij,i=1,2,…,L,j=1,2,…,I3,where Wj=W(:,:,j)=[VjUT], Xi(j)=Xi(:,:,j),Cj=C,i=1,2,…,L,j=1,2,…,I3. That means what we need to solve are optimization problems (43).


### 4.2. Dual Fixed Point Algorithm (DFPA) for LS-STRM-SMT

FixingV1,V2,…,Vj-1,Vj+1,…,VI3,U and optimizing Vj, optimization problems (43) can be reformulated as follows:(44)minVj12VjUTF2+C2∑i=1Lηij2s.t.yij-∑k=1KvkjTXijuk=ηij,i=1,2,…,L,where Vj=(v1j,v2j,…,vKj),U=(u1,u2,…,uK).That is to say, a series of problems similar to optimization problems (44) which are indeed LS-SMRMs rather than optimization problems (43) are needed to be solved.FixingVj,j=1,2,…,I3 and optimizing U. Similarly, when Vj,j=1,2,…,I3, is fixed, optimization problems (43) can be reformulated as follows:(45)minU12PUTF2+C2∑i=1Lηi2(46)s.t.yi-∑k=1KpkTmat1Xqk=ηi,i=1,2,…,L,where(47)P=p1,p2,…,pK=V1V2⋮VkI1I3×K,Q=q1,q2,…,qK=W1W2⋮WkI1I3×K. It is clear that U can be obtained by solving optimization problems (45)-(46) with Algorithm 1. This will lead Algorithm 3.Algorithm 3 (LS-STRM-SMT).
(
1) Training Process
Input. Training set (35).
Output.
V
1
,
V
2
,
…
,
V
I
3, and U. (a)
Construct theI3 number of new submatrix training sets (39).(b)
Give a positive integerK(K=2,3,…,min⁡{I1,I2}).(c)
InitializeV1,V2,…,Vi-1,Vi+1,…,VI3 and U,Vj∈RI1×K,j∈{1,2,…,i-1,i+1,…,I3},U∈RI2×K.(d)
Alternatively updateV1,V2,…,VI3 and U until convergence.
(1)
UpdateVi,i=1,2,…,I3, by solving optimization problems (44) with Algorithm 1.(2)
UpdateU by solving optimization problems (45)-(46) with Algorithm 1.
(
2) Testing Process
Input. Testing point X, V1,V2,…,Vi,Vi+1,…,VI3, and U.
Output. The value of testing point: y(48)y=W,X=∑j=1I3W:,:,j,X:,:,j=∑j=1I3VjUT,X:,:,j.


### 4.3. Extension

For a more general tensor, that meansM>3. We can also take advantage of the submatrix of the tensor to turn the tensor learning for regression problems into IM LS-SMRM problems that can be solved by Algorithm 1. The details can be shown as follows.Give a training set(49)T=X1,y1,X2,y2,…,XL,yL,where Xi∈RI1×I2×⋯×IM,(M>3) is the input and yi∈R is the output, i=1,2,…,L. Our task is to find a predictor(50)y=W,X+b,where W∈RI1×I2×⋯×IM,(M>3) denotes the weight tensor, b is the bias. For a new input tensor, we can predict its output through predictor (50). The new training set is constructed through (1) as follows:(51)Tj=mat1X:,:,…,:,i,Yij,i=1,2,…,L,j=1,2,…,IM,where mat1(X(:,:,…,:,i))∈RI1×(I2I3⋯IM) denotes the jth submatrix of the ith sample, Yi(j) denotes the jth element of Yi which is a L-dimension vector obey normal distribution with mean yi and variance σ. Then we can get a IM-dimension abstract vector whose elements are matrices of the size I1×(I2I3⋯IM-1). According to mapping function (2), the matrices can be reformulated as equal number of (M-1)-order tensors. And then the abstract vector with (M-1)-order tensor elements is indeed a M-order tensor that we need to get.


## 4.1. Formulation

Give a training set(35)T=X1,y1,X2,y2,…,XL,yL,where Xi∈RI1×I2×I3 is the input and yi∈R is the output, i=1,2,…,L. Our task is to find a predictor(36)y=W,X+b,where W denotes the weight tensor and b is the bias. For a new input tensor, we can predict its output through predictor (36). For convenience, we set b=0.Definition 2.
For anM-order tensor X∈RI1×I2⋯×IM, the submatrix of the tensor is defined as(37)mat1X:,:,…,:,k,k=1,2,…,IM.Particularly, when M=3, the submatrices of a third-order tensor are indeed the frontal slices.However, we do not construct the model for training set (35) directly. We transform the training set (35) into I3 training sets similar to training set (8) by the introduction of the submatrix of the tensor and then construct I3 regression problems. That means, for the training set (35), every input tensor Xi∈RI1×I2×I3,i=1,2,…,L, can be regarded as an abstract vector(38)Xi1,Xi2,…,XiI3,i=1,2,…,L,where Xi(j),j=1,2,…,I3, is the jth frontal slice of the tensor Xi. Next, we take the jth frontal slice of each tensor Xi,i=1,2,…,L, out and construct the following training set:(39)Tj=Xij,Yij,i=1,2,…,L,where Yi(j)∈R denotes the jth element of Yi, Yi=(yi(1),yi(2),…,yi(I3))T is a vector obey normal distribution with mean yi and variance σ.According to training set (39), I3 optimization problems are constructed as follows:(40)minWj12WjF2+Cj2∑i=1Lηij2s.t.yij-trWjTXij=ηij,j=1,2,…,I3,where Wj=W(:,:,j)=VjUjT,Xi(j)=Xi(:,:,j), j=1,2,…,I3,i=1,2,…,L. So we can get W(1)=(W1,W2,…,WI3), and then the weight tensor can be obtained by (41)W=·1W1.It is clear that the I3 model is independent, but this is contrary to the truth. In order to reflect the relation among them, we set (42)U1=U2=⋯=UI3=U,so that the models can fit the practical situation better. The I3 models can be expressed together as follows:(43)minU,W12WF2+Cj2∑i=1L∑j=1I3ηij2s.t.yij-trWjTXij=ηij,i=1,2,…,L,j=1,2,…,I3,where Wj=W(:,:,j)=[VjUT], Xi(j)=Xi(:,:,j),Cj=C,i=1,2,…,L,j=1,2,…,I3. That means what we need to solve are optimization problems (43).


## 4.2. Dual Fixed Point Algorithm (DFPA) for LS-STRM-SMT

FixingV1,V2,…,Vj-1,Vj+1,…,VI3,U and optimizing Vj, optimization problems (43) can be reformulated as follows:(44)minVj12VjUTF2+C2∑i=1Lηij2s.t.yij-∑k=1KvkjTXijuk=ηij,i=1,2,…,L,where Vj=(v1j,v2j,…,vKj),U=(u1,u2,…,uK).That is to say, a series of problems similar to optimization problems (44) which are indeed LS-SMRMs rather than optimization problems (43) are needed to be solved.FixingVj,j=1,2,…,I3 and optimizing U. Similarly, when Vj,j=1,2,…,I3, is fixed, optimization problems (43) can be reformulated as follows:(45)minU12PUTF2+C2∑i=1Lηi2(46)s.t.yi-∑k=1KpkTmat1Xqk=ηi,i=1,2,…,L,where(47)P=p1,p2,…,pK=V1V2⋮VkI1I3×K,Q=q1,q2,…,qK=W1W2⋮WkI1I3×K. It is clear that U can be obtained by solving optimization problems (45)-(46) with Algorithm 1. This will lead Algorithm 3.Algorithm 3 (LS-STRM-SMT).
(
1) Training Process
Input. Training set (35).
Output.
V
1
,
V
2
,
…
,
V
I
3, and U. (a)
Construct theI3 number of new submatrix training sets (39).(b)
Give a positive integerK(K=2,3,…,min⁡{I1,I2}).(c)
InitializeV1,V2,…,Vi-1,Vi+1,…,VI3 and U,Vj∈RI1×K,j∈{1,2,…,i-1,i+1,…,I3},U∈RI2×K.(d)
Alternatively updateV1,V2,…,VI3 and U until convergence.
(1)
UpdateVi,i=1,2,…,I3, by solving optimization problems (44) with Algorithm 1.(2)
UpdateU by solving optimization problems (45)-(46) with Algorithm 1.
(
2) Testing Process
Input. Testing point X, V1,V2,…,Vi,Vi+1,…,VI3, and U.
Output. The value of testing point: y(48)y=W,X=∑j=1I3W:,:,j,X:,:,j=∑j=1I3VjUT,X:,:,j.


## 4.3. Extension

For a more general tensor, that meansM>3. We can also take advantage of the submatrix of the tensor to turn the tensor learning for regression problems into IM LS-SMRM problems that can be solved by Algorithm 1. The details can be shown as follows.Give a training set(49)T=X1,y1,X2,y2,…,XL,yL,where Xi∈RI1×I2×⋯×IM,(M>3) is the input and yi∈R is the output, i=1,2,…,L. Our task is to find a predictor(50)y=W,X+b,where W∈RI1×I2×⋯×IM,(M>3) denotes the weight tensor, b is the bias. For a new input tensor, we can predict its output through predictor (50). The new training set is constructed through (1) as follows:(51)Tj=mat1X:,:,…,:,i,Yij,i=1,2,…,L,j=1,2,…,IM,where mat1(X(:,:,…,:,i))∈RI1×(I2I3⋯IM) denotes the jth submatrix of the ith sample, Yi(j) denotes the jth element of Yi which is a L-dimension vector obey normal distribution with mean yi and variance σ. Then we can get a IM-dimension abstract vector whose elements are matrices of the size I1×(I2I3⋯IM-1). According to mapping function (2), the matrices can be reformulated as equal number of (M-1)-order tensors. And then the abstract vector with (M-1)-order tensor elements is indeed a M-order tensor that we need to get.


## 5. Numerical Experiment

In the following numerical experiments, we use four groups of vector data from the UCI database and an artificial tensor data for evaluation of our algorithm. The data are Slump, Ticdata2000, ConcreteData, and BlogData from the UCI database. The artificial data is given by the function “rand” in Matlab. We reformulate the vector data into matrices or tensors by rearranging the order of vector’s elements. The detailed statistical characters are listed in Table1.Table 1
Characters of different data sets.
Data
#trainings
#attributes
Artificial data
50
31
Slump
103
9
Ticdata2000
200
85
ConcreteData
100
9
BlogData
300
280Since our method is solved in a fixed point algorithm or in other words an alternative way, the final solution has close relationship with initialization. We would like to introduce two different kinds of initialization strategies in this part. The first initialization is random. We generate some random elements to form the matrices with corresponding sizes. In the following experiments, this kind of initialization is used without specification. The second initialization is an empirical method. That means we can setVi=[IK×K,0K×(Ii-K)],i=1,2,…,M.When solving the LS-STRM-SMT, penalty parameterC is selected from the set {2-8,2-7,…,27,28}. Another important parameter that may influence the result is K. Experiment performances show that the mean square error (MSE) which we used to measure whether the solutions to the LS-STRM-SMT problems are good or not decrease as the growth of the K value. That means the larger K is, the smaller MSEs are. But, this phenomenon is not very obvious and when K increases the computational complexity will rise quickly. We will give the MSE of different K for two data sets. And in the other experiments, we simply set K=2. Besides, the parameter C can be obtained by fivefold cross validation. In this section, we compute the tensor learning regression problems for third-order data with our LS-STRM-SMT algorithm and compare it with some other regression method, three problems are talked about. First, how does the MSE for different data change with the increasing of K? Second, compare our algorithm with support vector regression (SVR) [16], least square support vector regression (LSSVR) [17], regularized matrix regression RMR, [18], and lasso regression [19, 20]. The last problem which is being talked about is the convergence of our algorithm.The information of the data used is listed in Table1. Tables 2 and 3 show the influence of different K for artificial data and Ticdata2000, respectively, and the comparison of our algorithm and others are in Table 4. From Tables 2 and 3, we know that, with the increasing of K, the MSE tends to reduced gradually. But that is not very obvious and the largest K does not always means the best performance. In Figure 2, (a) and (b) show the relation between K and MSE more intuitively. From (a) and (b) in Figure 3 which is the average MSE of different K for artificial data and Ticdata2000, respectively. We can easily come to the conclusion that when K become large, the change of MSE is not that big, so we can set K=2 to reduce computation complexity.Table 2
MSE of differentK for Ticdata2000.
Size
K
=
2
K
=
3
K
=
4
K
=
5
K
=
6
3
×
4
×
7
0.0079(±2.097e-05)
0.0064(±8.997e-05)
0.0042(±5.627e-05)
0.0061(±4.244e-05)
0.0037 (±1.173e-05)
3
×
7
×
4
0.0070(±2.225e-06)
0.0055 (±3.918e-06)
0.0073(±9.876e-06)
0.0082(±6.824e-06)
0.0055 (±9.405e-07)
4
×
3
×
7
0.0074(±6.088e-06)
0.0061(±1.237e-05)
0.0049 (±7.482e-06)
0.0075(±1.843e-05)
0.0079(±4.441e-06)
4
×
7
×
3
0.0076(±4.419e-06)
0.0085(±4.998e-06)
0.0066(±3.798e-06)
0.0060(±7.888e-06)
0.0059 (±1.367e-05)
7
×
3
×
4
0.0076(±8.519e-06)
0.0077(±2.120e-05)
0.0065(±1.374e-06)
0.0056(±6.836e-06)
0.0051 (±1.365e-05)
7
×
4
×
3
0.0059 (±9.144e-06)
0.0090(±1.603e-06)
0.0078(±2.004e-05)
0.0078(±6.602e-06)
0.0083(±1.071e-05)Table 3
MSE of differentK for artificial data.
Size
K
=
2
K
=
3
K
=
4
K
=
5
K
=
6
2
×
3
×
5
0.2302(±0.0151)
0.1722 (±0.0025)
0.2106(±0.0106)
0.2432(±0.0011)
0.2640(±0.0057)
2
×
5
×
3
0.2270(±0.0011)
0.2143(±0.0026)
0.1975(±0.0110)
0.1540 (±0.0019)
0.1857(±0.0084)
3
×
2
×
5
0.1787(±0.0071)
0.2177(±0.0053)
0.1519 (±0.0038)
0.1607(±0.0056)
0.1865(±0.0029)
3
×
5
×
2
0.1601 (±0.0017)
0.1685(±0.0046)
0.1734(±0.0006)
0.1886(±0.0242)
0.1880(±0.0029)
5
×
2
×
3
0.2301(±0.0091)
0.2104(±0.0032)
0.1884(±0.0055)
0.1902(±0.0096)
0.1839 (±0.0067)
5
×
3
×
2
0.1999(±0.0039)
0.2033(±0.0135)
0.1976(±0.0052)
0.1413(±0.0002)
0.1114 (±0.0023)Table 4
MSE of different algorithm for different data set.
Data
SVR
LS-SVRM
RMR
LASSO
LS-STRM-SMT
Artificial data
1.8015 (±0.3047)
0.3347 (±0.0148)
0.7574  (±0.1778)
0.2190 (±0.0016)
0.2043 (±0.0063)
Slump
27.2552 (±119.9840)
0.4482 (±0.0052)
0.1970  (±0.0014)
0.3126 (±0.0057)
0.1931 (±0.0011)
Ticdata2000
0.1149 (±3.923e-04)
0.00742 (±1.193e-04)
0.1270  (±0.0027)
0.0557 (±0.720e-05)
0.0072 (±8.561e − 05)
ConcreteData
0.2114 (±0.0024)
0.1972 (±0.0080)
0.2026  (±0.0041)
0.1823 (±0.0010)
0.1459  (±0.0115)
BlogData
2.9480 (±0.6374)
0.5119 (±0.0010)
8.9211  (±0.1616)
0.5212 (±0.0077)
0.2899  (±0.0041)In Table4, the variance is represented by the number in parentheses while numbers out of the parentheses denote the MSE of different algorithm for different data set. Since the vector sample can be reformulated as matrix (tensor) of different size, numbers with italic font mean the average value of different data size. The numbers which are bold denote the best performance. It is clear that our LS-STRM-SMT always has a better performance which is obvious in Figure 4. More details are included in Tables 5, 6, and 7.Table 5
RMR for Slump and ConcreteData of different data size, respectively.
Slump
MSE (variance)
ConcreteData
MSE (variance)
2
×
4
0.1986 (±0.0011)
2
×
4
0.2058 (±0.0048)
4
×
2
0.1954 (±0.0016)
4
×
2
0.1823 (±0.0010)Table 6
RMR for artificial data, Ticdata2000, and BlogData of different data size, respectively.
Artificial data
MSE (variance)
Ticdata2000
MSE (variance)
BlogData
MSE (variance)
2
×
15
0.7001 (±0.1825)
2
×
42
0.1325 (±0.0011)
4
×
70
9.0560 (±0.1809)
3
×
10
0.7624 (±0.0505)
3
×
28
0.1314 (±0.0121)
7
×
40
10.2741 (±0.1544)
5
×
6
0.7056 (±0.2212)
4
×
21
0.1488 (±0.0019)
10
×
28
9.7221 (±0.1097)
6
×
5
0.6659 (±0.0577)
6
×
14
0.0942 (±0.0008)
28
×
10
7.7979 (±0.1603)
10
×
3
0.9848 (±0.4923)
7
×
12
0.1279 (±0.0007)
40
×
7
8.9549 (±0.1950)
15
×
2
0.7258 (±0.0626)
12
×
7
0.1071 (±0.0016)
70
×
4
7.7215 (±0.1695)
—
—
14
×
6
0.1364 (±0.0011)
—
—
—
—
21
×
4
0.1348 (±0.0052)
—
—
—
—
28
×
3
0.1264 (±0.0031)
—
—
—
—
42
×
2
0.1304 (±0.0006)
—
—Table 7
LS-STRM-SMT for BlogData of different data size.
BlogData
MSE (variance)
4
×
7
×
10
0.2518 (±0.0037)
4
×
10
×
7
0.3244 (±0.0068)
7
×
4
×
10
0.3045 (±0.0058)
7
×
10
×
4
0.2872 (±0.0032)
10
×
4
×
7
0.2387 (±0.0037)
10
×
7
×
4
0.3325 (±0.0013)In order to show the convergence behavior, we computerize the objective value of our method for artificial data and Ticdata2000. After 20 times iterations, the objective function values are shown in Figure1. There are mainly two observations from the result. First, the objective values of our algorithm are extremely small, which implies that our algorithm performs good to some degree. Second, with the increasing of iteration, our algorithm tends to be a certain number which guarantee the convergence of our algorithm.Figure 1
Convergence of LS-STRM-BST.
(a)
Convergence of LS-STRM-BST for artificial data
(b)
Convergence of LS-STRM-BST for Ticdata2000Figure 2
MSE of differentK for artificial data and Ticdata2000.
(a)
MSE of differentK for artificial data
(b)
MSE of differentK for Ticdata2000Figure 3
MSE of differentK for artificial data and Ticdata2000.
(a)
MSE of differentK for artificial data
(b)
MSE of differentK for Ticdata2000Figure 4
Comparison of our algorithm and the other algorithm that performs best.
(a)
Line graph according to Tables2, 3, and 4
(b)
Bar graph according to Tables2, 3, and 4In Figure1, the abscissa and the ordinate represent the iterations and the objective value, respectively, and the curve with different color and mark in the same figure stands for the relationship between objective value and iterations for the same data of different data size.In Figure2, the abscissa and the ordinate represent the parameter K and the MSE, respectively, and the line graph with different color and mark in the same fig stands for the relationship between the parameter K and the MSE for the same data of different data size.Figure3 show the relationship between average MSE and K for the artificial data and Ticdata2000, respectively.Figure4 is plotted based on Tables 2, 3, and 4. In (a) the full line stands for the performance of our algorithm while the imaginary line stands for the best performance of other algorithm. In (b) the bar graph in blue represents the performance of our algorithm while the bar graph in brown represents the best performance of other algorithm. From Figure 4, we can easily come to the conclusion that our method has a better performance.


## 6. Conclusion

In this paper, we propose a novel method for tensor learning regression problems, called least square support tensor regression machine based on submatrix of the tensor (LS-STRM-SMT), which is inspired by the idea of multiple rank multilinear SVM for matrix data classification (MRMLSVM). And we give our dual fixed point algorithm (DFPA) to solve the problems. However, LS-STRM-SMT is not a straightforward extension of MRMLSVM. On one hand, MRMLSVM is a method for matrix data classification while LS-STRM-SMT is used to solve tensor data regression problems; on the other hand, the LS-STRM-SMT and MRMLSVM are connected by the introducing of submatrix of the tensor which can be applied to reformulate the LS-STRM-SMT problem as a series of LS-SMRM problems that can be figure out in a way similar to MRMLSVM. What is more, LS-STRM-SMT differs from traditional approach which destroys the structural information of the tensor totally; it can reserve the structure tensor information of the tensor to some extent which may be the reason why our method has a better performance than some other proposed algorithms. Besides, a small parameterK can decrease the dimensions of the unknown quantity which can avoid overfitting to some degree. In addition, from a practical point of view, it has also been shown by the preliminary numerical experiments that the performance of our LS-STRM-SMT is better.



---

*Source: 3818949-2017-11-09.xml*