# The Simplified Expression of Machine Learning and Multivariate Statistical Analysis Based on the Centering Matrix

Authors: Zhen Xue; Liangliang Zhang
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

In machine learning (ML) algorithms and multivariate statistical analysis (MVA) problems, it is usually necessary to center (zero centered or mean subtraction) the original data. The centering matrix plays an important role in this process. The full consideration and use of its properties may contribute to the speed or stability improvement of some related ML algorithms. Therefore, in this paper, we discussed in detail the properties of the centering matrix, proved some previously known properties, and deduced some new properties. The involved properties mainly consisted of the centering, quadratic form, spectral decomposition, null space, projection, exchangeability, Kronecker square, and so on. Based on this, we explored the simplified expression role of the centering matrix in the principal component analysis (PCA) and regression analysis theory. The results show that the sum of deviation squares which is widely used in regression theory and variance analysis can be expressed by the quadratic form with the centering matrix as the kernel matrix. The ML algorithms introducing the centering matrix can greatly simplify the learning process and improve predictive ability.


---

## Body

## 1. Introduction

Among the regression problems of machine learning (ML) and during the course of training the neural networks, we usually need to center (zero centered or mean subtraction) the original data. The centering matrix can achieve this goal. It is a symmetric idempotent matrix and has been successfully applied in various fields, such as transfer learning [1, 2], feature learning [3], extreme learning [4], ensemble learning [5–9], dictionary learning [10, 11], and multivariate statistical analysis (MVA) [12–16]. The practical applications involve a wide range of aspects (e.g., data-driven fault diagnosis and prognosis [3, 17, 18], sentiment analysis [7, 8], web page classification [9], information retrieval [4, 19, 20], image denoising [10], and signal processing of electronics [11], theoretical chemistry and graph theory [21], and rank data analysis [22]). For instance, Chen S. Z. et al.[4] proposed a flexible ranking extreme learning machine (ELM) method based on matrix-centering transformation to replace the traditional graph Laplacian matrix-based methods. In [19, 20], T. Pahikkala et al. presented two kinds of ranking algorithm for information retrieval through introducing the centering matrix to construct block diagonal matrix as the weighted matrix and finding the linear solution of Rank Reweighted Least Squares (RankRLS). The application of the centering matrix in the optimization problem of transfer learning is discussed in [1, 2]. Long M. S. et al.[2] established an optimization problem and then proposed a feature transformation by adopting the centering matrix. In a recent study, the covariance matrices of the source and target features were expressed by the centering matrix [3]. Wang Q. and Zhang L. investigated the online updating of the generalized inverse of the centered matrix of the data matrix in [5, 6]. In [23], Liu L. P. et al. pointed out the least square solution to linear discriminant analysis (LDA) [24] is the multiplication of pseudoinverse of the centered data matrix and the indicator matrix, which makes the solution being updated without eigenanalysis. In another study, Sun L. et al.[25] investigated the relationship between the generalized eigenvalue problem and least squares problem by using the centering matrix. In [26], the application of the centering matrix in data dimension reduction algorithm of principal component analysis (PCA) [27, 28] is discussed. The application in statistical analysis of the centering matrix is addressed in [12–16]. In [21], Gutman I. and Xiao W. examined the generalized inverse of the Laplacian matrix of a connected graph. One of the conclusions obtained by them was the Laplacian matrix which commutes with its generalized inverse, and their product is the centering matrix. Recently, a MATLAB toolbox data-based key-performance-indicator oriented fault detection toolbox (DB-KIT) was developed for prognosis and fault diagnosis of the complex systems in [17] and the canonical variate dissimilarity analysis was proposed for process incipient fault detection in [18]. Both studies in which the centering matrix plays an important role greatly promoted the development of the data-driven fault diagnosis and prognosis in the modern industry.With regard to the centering matrix, more consideration and use of its properties and its matrix notation simplified role may contribute to the speed or stability improvement of some related ML algorithms. For this purpose, in this paper, we firstly discussed in detail and established some properties of the centering matrix and drew a mind map to clearly illustrate the relationship between these properties and their potential role in ML and MVA. Then, we explored the theoretical simplification applications of the centering matrix in ML and MVA theory by two examples. The research results can provide help for the proposal or speed improvement of some related ML algorithms on the regression problem or neural network algorithms.The centering matrix is defined as follows [13].Definition 1.
Cn=In−1/nJn=In−1/n1n1nT is called the centering matrix of order n, where In is the identity matrix of order n, 1n=1,1,…,1T is the column vector of n ones, Jn is a square matrix of order n with all elements unity, and 1n1nT=Jn holds.
For example,C3=2/3−1/3−1/3−1/32/3−1/3−1/3−1/32/3 is a centering matrix of order 3.
The rest of this paper is organized as follows. Section2 discusses the properties of the centering matrix. Section 3 presents the main application of the centering matrix in ML and MVA. Finally, we conclude this paper in Section 4.


## 2. Properties of the Centering Matrix

It can be proved that the centering matrix has the following important properties. In what follows,Cn will always represent the centering matrix, unless otherwise stated.Lemma 1 (centering property, see [13], and we prove it).
For any data vectorx=x1,x2,…,xnT of order n, when performing a left multiplication Cn on x, denoted by Cnx (or xC), it has the same effect as subtracting the mean of the elements of x from each element, i.e., Cnx=x1−x¯,x2−x¯,…,xn−x¯T, where x¯=1/n∑k=1nxk.Proof.
(1)Cnx=In−1nJnx=x1x2⋮xn−1n11…111…1…………11…1x1x2⋮xn=x1x2⋮xn−1n∑k=1nxk1n∑k=1nxk⋮1n∑k=1nxk=x1−x−x2−x−⋮xn−x−=x−x_1n.
This property illustrates that the data vector can be centered through left multiplication byCn. This also reveals the origin of the name centering matrix for Cn. Meanwhile, Cnx is called the centered vector of x. It is easy to know that the sum of all entries of the centered vector is 0.Corollary 1.
LetX be a data matrix of order n×m, each column represents the n observations on a particular variable and each row represents a single observation on m variables. Then, the centered matrix of X is CnX.Proof.
X can be partitioned as X=x1,x2,…,xm, and each column is the data vector xj,j=1,2,…,m. We can obtain that its centered vector is Cnxj by Lemma 1. So, the centered matrix of X is CnX.
Multiplication by the centering matrix can be used to remove not only the mean of a single vector but also multiple vectors stored in the rows or columns of a matrix. For any matrix ofn×m, the multiplication CnX removes the means from each of the m columns, while the multiplication XCm removes the means from each of the n rows [14].
It can be easily verified that the centering matrixCn is symmetric. Since(2)Cn2=In−1nJnIn−1nJn=In2−1nInJn−1nJnIn+1n2Jn2,=In−2nJn+1n2nJn=In−1nJn=Cn,the centering matrix Cn is idempotent, i.e.,Cn2=Cn. The extension form is Cnk=Cn,k=1,2,3….
It can be seen from the above two paragraphs thatCn is symmetric and idempotent. Therefore, Cn is a projection matrix. It can be easily proved that In−Cn is also a projection matrix.
Let the adjoint matrix ofCn be denoted by Cn∗. Since Cn∗T=CnT∗=Cn∗ and Cn∗2=Cn∗Cn∗=CnCn∗=Cn2∗=Cn∗,Cn∗ is also a projection matrix.Theorem 1 (quadratic form property, see [13]).
Letx=x1,x2,…,xnT be a vector with a set of data x1,x2,…,xn as entries. Then, the covariance or total sum of squares (SST) of the data can be expressed by a quadratic form whose kernel matrix is the centering matrix Cn, i.e., SST=∑i=1nxi−x−2=xTCnx.Proof.
By Lemma1, Cnx=x1−x−,x2−x−,…,xn−x−T can be obtained. Then, we calculate the inner product of the vector with itself:(3)CnxTCnx=x1−x−,x2−x−,…,xn−x−x1−x−,x2−x−,…,xn−x−T=∑i=1nxi−x−2.
Using the symmetry and idempotency ofCn, we can obtain the equation xTCnx=∑i=1nxi−x−2=SST.
The centering matrix has close relationship with the Helmert matrix which is an orthogonal matrix and has important application in the defining contrasts in general linear models of statistics. The corresponding conclusion is as follows.Proposition 1 (see [12, 29], and we prove it).
Let the Helmert matrix of ordern be partitioned as H=hTK, where h=n−1/21nT, K is a matrix consisting of the last n−1 rows of H, and each row of K is given by the following formula:(4)1λi1iT,−i,0n−i−1T,λi=ii+1,i=1,2,…,n−1,where 1iT and 0iT are row vectors of order i with all elements unity and zero, respectively. Then, SST=∑i=1nxi−x−2=xTKTKx.Proof.
SST=∑i=1nxi−x−2=xTCx can be obtained by Theorem 1. For H is orthogonal, i.e., HHT=HTH=In, whereas(5)HTH=hKThTK=hhT+KTK,=n−1/21n⋅n−1/21nT+KTK=1n1n1nT+KTK,so 1/n1n1nT+KTK=In; thus, KTK=In−1/n1n1nT=Cn. Hence, the conclusion holds.
The centering matrix also has a close association with the Laplacian matrix of a connected graph. The related property is as follows.Proposition 2.
(see [21]). The Laplacian matrix Ln of a connected graph with n vertices and its generalized inverse Ln+ satisfy the following equality: LnLn+=Ln+Ln=Cn,n≥2.Theorem 2.
The eigenvalues ofCn are 0 of multiplicity 1 and 1 of multiplicity n−1.Proof.
Owing toCn=In−1/nJn, we can find the eigenvalue of Jn at first. Assume that its eigenvalue be λ and the corresponding eigenvector be x. Then, Jnx=λx. Since Jn2x=λ2x and Jn2=nJn, we can obtain λ2x=nλx, whereas x≠0, so λ2−nλ=0; therefore, λ=0 or λ=n.
Let the multiplicity of eigenvalue 0 bek1. Since Jn is real symmetric, it can be diagonalizable; thus, R0I−Jn=n−k1. It is easy to know that RJn=1, so k1=n−1. Because there must be n eigenvalues for square matrix of order n, we can obtain that the multiplicity of eigenvalue n is 1.
The eigenvalue ofCn is equivalent to −1/nλ+1, whereas λ=n or λ=0, so the eigenvalues of Cn are 0 of multiplicity 1 and 1 of multiplicity n−1.Corollary 2 (the spectral decomposition of the centering matrix).
For the centering matrixCn, there must exist an orthogonal matrix of order nQ=y1,y2,…,yn−1,yn such that Cn=QDnQT=y1y1T+y2y2T+⋯+yn−1yn−1T+0ynynT, where y1,y2,…,yn−1 are the unit orthogonal eigenvectors corresponding to eigenvalue 1, yn is the unit orthogonal eigenvector corresponding to eigenvalue 0, and Dn=In−10.Proof.
Since the centering matrixCn is real symmetric, it can be diagonalizable. We can know that the eigenvalues of Cn are 0 of multiplicity 1 and 1 of multiplicity n−1 by Theorem 2. Suppose that the unit orthogonal eigenvectors corresponding to eigenvalue 1 are y1,y2,…,yn−1 and the unit orthogonal eigenvector corresponding to eigenvalue 0 is yn. Let Q=y1,y2,…,yn−1,yn. Then, Q is orthogonal and satisfies QTCnQ=Dn, where Dn=In−10 is a diagonal matrix. The following formula can be obtained by using the equation QTCnQ=Dn for left multiplication by Q and right multiplication by QT:(6)Cn=QDnQT=y1,y2,…,yndiag1,1,…,1,0y1T,y2T,…,yn−1T,ynTT,=y1y1T+y2y2T+⋯+yn−1yn−1T+0ynynT.Remark 1.
In the previous formula, each term at the right end is the outer product of each of the unit orthogonal eigenvector ofCn with itself multiplying the corresponding eigenvalue. Each term is a square matrix with rank 1. The above formula has simple and special structure, so it has important applications in statistics.Corollary 3.
The centering matrixCn can be decomposed as Cn=LLT, where L satisfies the relation LTL=In−1.Proof.
The equationCn=QDnQT can be obtained by the proof process of Corollary 2, where the definitions of Q and Dn are given in Corollary 2. It is easy to know that Dn=DnDnT. Therefore, Cn=QDnQT=QDnDnTQT=QDnQDnT=LLT, where L=QDn is a matrix of order n×n−1 such that LTL=QDnTQDn:(7)=y1y2…yn−1Ty1y2…yn−1,=y1Ty2T⋮yn−1Ty1y2…yn−1,=y1Ty1y1Ty2…y1Tyn−1y2Ty1y2Ty2…y2Tyn−1…………yn−1Ty1yn−1Ty2…yn−1Tyn−1=In−1.
We can obtain thatCn has 0 eigenvalue by Theorem 2; therefore, Cn=0, and thus, Cn is irreversible (singular).
The equalityCn=QIn−10QT can be obtained by Corollary 2. Hence, we have [30] Cn+=QIn−1−10QT=QIn−10QT=Cn.
It indicates that the Moore–Penrose generalized inverse ofCn is itself, i.e., Cn+=Cn.
We can know that the eigenvalues ofCn is nonnegative by Theorem 2, so it is positive semidefinite, denoted by Cn≥0; hence, it can be used as the covariance matrix of some random variable.
BecauseCn≥0, so for arbitrary nonzero column vector x,xTCnx≥0. Let x=ei=0,…,0,1,0,…,0T, where 1 is the ith element. Then, 0≤eiTCnei=cii, for i=1,2,…,n, i.e., the diagonal elements of Cn are nonnegative.
We can know thatCn is similar to diagonal matrix Dn by Corollary 2. The similar matrices have the same rank, so RCn=RDn=n−1, i.e., the rank of Cn is n−1.
We can verify that the rank ofCn∗ is 1 by the theory of linear algebra. The proof process is as follows. On the one hand, Cn=0; therefore, Cn⋅Cn∗=CnI=O, and thus, RCn+RadjCn≤n. Because RCn=n−1, so RCn∗≤1. On the other hand, since RCn=n−1, there must exist a nonzero minor of order n-1 for Cn. Then, we can obtain Cn∗≠O, and thus, RCn∗≥1. In summary, we have RCn∗=1.
We can know that the eigenvalues ofCn are 0 with multiplicity 1 and 1 with multiplicity n−1 by Theorem 2. Since the trace of a square matrix equals the sum of all eigenvalues of the matrix, trCn=n−1, i.e., the trace of the centering matrix Cn is n−1.
BecausetrCn∗=trCn∗ and trCn=n−1, so trCn∗=n−1, i.e., the trace of Cn∗ is n−1.
AlthoughCn is singular, and Cn+aJn is nonsingular for a≠0. The related conclusion is as follows:Proposition 3 (see [12], and we prove it).
The centering matrixCn and the matrix Jn satisfy the relations JnCn+aJn−1=Cn+aJn−1Jn=1/anJn and Cn+aJn−1−Cn+bJn−1=b−a/abn2Jn for a≠0,b≠0.Proof.
Forp≠0 and p+qn≠0,(8)pI+qJn1pI−qp+qnJn=pI+qJn1pI−qpp+qnJn,=pI1pI−pqpp+qnJn+qpJn−q2pp+qnJn2,=I−qp+qnJn+qpJn−q2pp+qnnJn,=I+qp−qp+qn−nq2pp+qnJn=I,so pI+qJn−1=1/pI−q/p+qnJn. Using this equation, we have Cn+aJn−1=I−1/nJn+aJn−1(9)=I+a−1nJn−1=I−a−1/n1+a−1/nnJn=I−na−1n+na−1nJn=I−1nJn+1n2aJn=Cn+1n2aJn.
Therefore,JnCn+aJn−1=JnI−1/nJn+1/n2aJn(10)=Jn−1nnJn+1n2anJn=1naJn.
In the same way, we can obtainCn+aJn−1Jn=1/naJn.
Using the equationCn+aJn−1=Cn+1/n2aJn, we get the result(11)Cn+aJn−1−Cn+bJn−1=Cn+1n2aJn−Cn+1n2bJn=b−aabn2Jn.Lemma 2 (see [12], and we prove it).
The centering matrixCn commutes with matrix Jn, and their product is zero matrix, i.e., CnJn=JnCn=O.Proof.
CnJn=In−1/nJnJn=Jn−1/nJn2=Jn−1/nnJn=O and JnCn=JnIn−1/nJn=Jn−1/nJn2=Jn−1/nnJn=O.Corollary 4.
The null space ofCn is NCn=k1n,k∈R whose dimension is 1 and a set of bases is 1n.Proof.
SincedimNCn+dimRCn=n and dimRCn=n-1,dimNCn=1, that is, the dimension of the null space is 1. The equation Cn1n=0n can be obtained by Lemma 2. So, the null space of Cn is Span1n, i.e.,NCn=k1n,k∈R. Obviously, a set of bases of NCn is 1n.Corollary 5.
For the centering matrixCn, the set of all eigenvectors corresponding to the eigenvalue 0 is k1n,k∈R/0.Proof.
Since the vector1n is the column of the matrix Jn and uses Lemma 2, the equation Cn1n=0n=01n holds. We get the result that the sum vector 1n is an eigenvector of Cn corresponding to the eigenvalue 0 according to the definition of eigenvector. Then, we have the desired result.Proposition 4 (see [14], and we prove it).
The vector resulted from the vectory=y1,y2,…,ynT by the transformation Cn and denoted by Cny, which is the projection of y to n−1 dimension subspace V1=x1,x2,…,xn|x1+x2+⋯+xn=0, where the subspace V1 is orthogonal to the subspace V0=k1n,k∈R.Proof.
We can obtain that the eigenvalues ofCn are 0 with multiplicity 1 and 1 with multiplicity n−1 by Theorem 2. Suppose that the eigenspace of eigenvalue 1 is n−1 dimension subspace V1 and the eigenspace of eigenvalue 0 is 1 dimension subspace V0; then, V1 is orthogonal to V0[22]. It is worth mentioning that the eigenspace V0 is also the null space of the centering matrix Cn.The result V0=k1n,k∈R can be obtained by Corollary 5. For Cn, assuming that the eigenvector corresponding to eigenvalue 1 is x=x1,x2,…,xnT, then we can obtain that Cnx=1⋅x by the definition of eigenvalue and eigenvector, i.e., In−1/nJnx=x. Replacing the corresponding value, we have x1x2⋮xn−1/n11…111…1…………11…1x1x2⋮xn=x1x2⋮xn; thus, x1+x2+⋯+xn=0, i.e., V1=x1,x2,…,xn|x1+x2+⋯+xn=0. Since Cn is a projection matrix, Cny is the projection of the vector y to n−1 dimension subspace V1=x1,x2,…,xn|x1+x2+⋯+xn=0.
In the following, we will discuss the Kronecker product property ofCn.
From the previous text, we can know thatCn is a positive semidefinite matrix. Suppose Bm is a positive semidefinite matrix of order m. Since the Kronecker product of two positive semidefinite matrices is also positive semidefinite, Bm⊗Cn is also a positive semidefinite matrix [31]. In particular, Cn⊗Cn is also a positive semidefinite matrix. Next, we will further discuss the other property of Cn⊗Cn.Proposition 5 (Kronecker square property).
The Kronecker square ofCn is defined as Cn2=Cn⊗Cn. It has the following characteristics:(i)
It is a projection matrix(ii)
It is irreversible(iii)
Its Moore–Penrose Generalized inverse is itself, i.e.,Cn2+=Cn2(iv)
Its rank and trace are all equal ton−12, i.e., RCn2=trCn2=n−12Proof.
(i)
On the one hand,Cn2T=Cn⊗CnT=CnT⊗CnT=Cn⊗Cn=Cn2.It shows thatCn2 is symmetric. On the one hand,(12)Cn22=Cn⊗Cn2=Cn⊗CnCn⊗Cn=CnCn⊗CnCn,Cn2⊗Cn2=Cn⊗Cn=Cn2.Hence,Cn2 is an idempotent matrix. To sum up, Cn2 is a projection matrix.(ii)
detCn2=detCn⊗Cn=detCnn⋅detCnn=0. Hence, Cn2 is irreversible.(iii)
Cn2+=Cn⊗Cn+=Cn+⊗Cn+=Cn⊗Cn=Cn2. So, the Moore–Penrose generalized inverse of Cn2 is equal to itself.(iv)
RCn2=rankCn2=rankCn⊗Cn=rankCn⋅rankCn=n−12, and trCn2=trCn⊗Cn=trCn⋅trCn=n−12.
Therefore,RCn2=trCn2=n−12.
The relationship between some previous properties such as the centering property, quadratic form property, spectral property, and their roles in ML, MVA, theoretical chemistry, and graph theory can be illustrated in Figure1.Figure 1
The relationship between some properties of the centering matrix and their roles in ML and MVA.


## 3. The Application of the Centering Matrix in ML and Statistical Analysis

### 3.1. The Application in the PCA

It is often necessary to calculate the eigenvector of the sample covariance matrix in dimensionality reduction by using the PCA, one of the ML algorithms. In this case, it is more convenient to represent the covariance matrix by the centering matrix.Example 1.
Suppose the sample data matrix is(13)X=x11x12…x1mx21x22…x2m…………xn1xn2…xnmn×m=x1Tx2T⋮xnT,=x1x2…xnT.
Then, the mean of the sample is(14)x_m×1=1n∑i=1nxi=1nx1x2…xn11⋮1,=1nXT1n.
The sample covariance matrix is as follows [26]:(15)SX=1n−1∑i=1nxi−x_xi−x_T,=1n−1x1−x_,x2−x_,…,xn−x_x1−x_,x2−x_,…,xn−x_T,=1n−1XT−x_1nTXT−x_1nTT.
Replacingx¯ by 1/nXT1n and using the symmetry and idempotency of Cn, we can obtain(16)SX=1n−1XT−1nXT1n1nTXT−1nXT1n1nTT,=1n−1XTIn−1n1n1nTXTIn−1n1n1nTT,=1n−1XTCnXTCnT=1n−1XTCnCnTX,=1n−1XTCn2X=1n−1XTCnX,where n−1SX=XTCnX is the scatter matrix.
This example shows that the sample covariance matrix and scatter matrix can be expressed succinctly by the centering matrix. Meanwhile, we can know that the sample covariance matrix is a Gramian matrix and a positive semidefinite matrix. Similarly, the correlation matrix can also be expressed simply by the centering matrix.


### 3.2. The Application in the Regression Analysis

Example 2.
The centering matrix has important simplified expression role in regression problem of supervised learning. Suppose there is a linear regression model as shown below between the random vectory and p−1 nonrandom factors x1,x2,…,xp−1.y=β0+β1x1+β2x2+⋯+βp−1xp−1+ε, where ε is the random error, and β0,β1,β2,…,βp−1 are unknown parameters. In order to determine their value, substitute the data xi1,xi2,…,xi,p−1;yi,i=1,2,…,n, by nn≥p independent observations into the above formula; then, we can obtain the following equations:(17)yi=β0+β1xi1+β2xi2+⋯+βp−1xi,p−1+εi,i=1,2,…,n.
The corresponding matrix form isy=Xβ+ε, where(18)y=y1y2⋮yn,X=1x11x12…x1,p−11x21x22…x2,p−1……………1xn1xn2…xn,p−1,β=β0β1⋮βn,ε=ε1ε2⋮εn.
It can be found that the least square estimation of the parameterβ is β^=β0^,β1^,…,βp−1^T=XTX−1XTy.
The fitted value ofy can be obtained by substituting the above formula into y=Xβ+ε and omitting error term ε:(19)y^=y1^,y2^,…,yn^T=Xβ^=XXTX−1XTy=My,where M=XXTX−1XT is a projection matrix of order n. The expression of residual vector is shown as follows:(20)ε^=ε1^,ε2^,…,εn^T=y−y^=y−Xβ^,=y−XXTX−1XTy,=I−XXTX−1XTy=I−My,where I is the unity matrix of order n, and if XTX is singular, then find its Moore–Penrose generalized inverse.
Next, we are going to express the regression sum of squares (SSR), the error sum of squares (SSE), and SST by the centering matrix and do further derivation.(i)
The following formula about SSR can be obtained by Theorem1 and the equation y^=My:(21)SSR=∑i=1nyi^−1n∑i=1nyi^2=y^TCny^=y^TI−1nJy^=yTMTI−1nJMy=yTMI−1n1n1nTMy.On the one hand, the equality∑i=1nεi^=0 holds [32]. On the other hand,∑i=1nεi^=1nTε^=1nTI−My. Combining the above two equations, we have(22)1nTy=1nTMy.Therefore,(23)SSR=yTMy−1nyTM1n1nTMy=yTMy−1nyTMT1n1nTMy=yTMy−1n1nTMyT1nTMy=yTMy−1n1nTyT1nTy=yTMy−yT1n1n1nTy=yTM−1nJy.(ii)
SSE is also called residual sum of squares and defined asSSE=∑i=1nyi−yi^2=∑i=1nεi^2. Due to ∑i=1nεi^=0, the above formula is changed into(24)SSE=∑i=1nεi^−∑i=1nεi^2=∑i=1nεi^−1n∑i=1nεi^2.We can obtain the following expression ofε^ by Theorem 1; hence,(25)SSE=ε^TCnε^=ε^TI−1nJε^=I−MyTI−1nJI−My=yTI−MTI−1nJI−My=yTI−MTI−My−1nyTI−MTJI−My=yTI−MTI−My−1nyTI−MT1n1nTI−My.We can obtain1nTI−My=0 by (i). Therefore, the preceding formula reduces to SSE=yTI−My.(iii)
According to Theorem1, SST can be expressed as(26)SST=∑i=1nyi−1n∑i=1nyi2=∑i=1nyi−y_2=yTCny.
In the above example, for the sum of squares such as SSR and SSE, we expressed it by the centering matrix firstly and then obtained the quadratic form expression ofy after a series of derivation. For the sum of squares SST, we directly obtained its quadratic form expression by Theorem 1. The Table 1 gives the quadratic form expression in which the kernel function is Cn and the quadratic form expression of y for SSR, SSE, and SST, respectively.
From Table1, we can observe that SSR, SSE, and SST are all expressed as the quadratic form with the kernel matrix Cn. Generally speaking, sum of squares all can be considered to be expressed by the centering matrix. For instance, the sum of squares of normally distribution known as χ2 distribution is such a case. From formulas (23), (25), and (26), we can easily deduce the famous decomposition equality SST=SSR+SSE which is of vital importance in regression theory and analysis of variance.
In addition, the centering matrix has important simplified role in the ranking algorithms in information retrieval [4, 19, 20]. For instance, in [19], T. Pahikkala et al. introduced the centering matrix to construct a block diagonal matrix as weighted matrix, express the loss function, obtain the matrix notation of a least squares problem, and conveniently find the linear solution of RankRLS algorithm.Table 1
Quadratic form expressions for SSR, SSE, and SST.
TypeQuadratic form expression with the kernel matrixCnQuadratic form expression ofySSRy^TCny^yTM−1/nJySSEε^TCnε^yTI−MySSTyTCnyyTCny


## 3.1. The Application in the PCA

It is often necessary to calculate the eigenvector of the sample covariance matrix in dimensionality reduction by using the PCA, one of the ML algorithms. In this case, it is more convenient to represent the covariance matrix by the centering matrix.Example 1.
Suppose the sample data matrix is(13)X=x11x12…x1mx21x22…x2m…………xn1xn2…xnmn×m=x1Tx2T⋮xnT,=x1x2…xnT.
Then, the mean of the sample is(14)x_m×1=1n∑i=1nxi=1nx1x2…xn11⋮1,=1nXT1n.
The sample covariance matrix is as follows [26]:(15)SX=1n−1∑i=1nxi−x_xi−x_T,=1n−1x1−x_,x2−x_,…,xn−x_x1−x_,x2−x_,…,xn−x_T,=1n−1XT−x_1nTXT−x_1nTT.
Replacingx¯ by 1/nXT1n and using the symmetry and idempotency of Cn, we can obtain(16)SX=1n−1XT−1nXT1n1nTXT−1nXT1n1nTT,=1n−1XTIn−1n1n1nTXTIn−1n1n1nTT,=1n−1XTCnXTCnT=1n−1XTCnCnTX,=1n−1XTCn2X=1n−1XTCnX,where n−1SX=XTCnX is the scatter matrix.
This example shows that the sample covariance matrix and scatter matrix can be expressed succinctly by the centering matrix. Meanwhile, we can know that the sample covariance matrix is a Gramian matrix and a positive semidefinite matrix. Similarly, the correlation matrix can also be expressed simply by the centering matrix.


## 3.2. The Application in the Regression Analysis

Example 2.
The centering matrix has important simplified expression role in regression problem of supervised learning. Suppose there is a linear regression model as shown below between the random vectory and p−1 nonrandom factors x1,x2,…,xp−1.y=β0+β1x1+β2x2+⋯+βp−1xp−1+ε, where ε is the random error, and β0,β1,β2,…,βp−1 are unknown parameters. In order to determine their value, substitute the data xi1,xi2,…,xi,p−1;yi,i=1,2,…,n, by nn≥p independent observations into the above formula; then, we can obtain the following equations:(17)yi=β0+β1xi1+β2xi2+⋯+βp−1xi,p−1+εi,i=1,2,…,n.
The corresponding matrix form isy=Xβ+ε, where(18)y=y1y2⋮yn,X=1x11x12…x1,p−11x21x22…x2,p−1……………1xn1xn2…xn,p−1,β=β0β1⋮βn,ε=ε1ε2⋮εn.
It can be found that the least square estimation of the parameterβ is β^=β0^,β1^,…,βp−1^T=XTX−1XTy.
The fitted value ofy can be obtained by substituting the above formula into y=Xβ+ε and omitting error term ε:(19)y^=y1^,y2^,…,yn^T=Xβ^=XXTX−1XTy=My,where M=XXTX−1XT is a projection matrix of order n. The expression of residual vector is shown as follows:(20)ε^=ε1^,ε2^,…,εn^T=y−y^=y−Xβ^,=y−XXTX−1XTy,=I−XXTX−1XTy=I−My,where I is the unity matrix of order n, and if XTX is singular, then find its Moore–Penrose generalized inverse.
Next, we are going to express the regression sum of squares (SSR), the error sum of squares (SSE), and SST by the centering matrix and do further derivation.(i)
The following formula about SSR can be obtained by Theorem1 and the equation y^=My:(21)SSR=∑i=1nyi^−1n∑i=1nyi^2=y^TCny^=y^TI−1nJy^=yTMTI−1nJMy=yTMI−1n1n1nTMy.On the one hand, the equality∑i=1nεi^=0 holds [32]. On the other hand,∑i=1nεi^=1nTε^=1nTI−My. Combining the above two equations, we have(22)1nTy=1nTMy.Therefore,(23)SSR=yTMy−1nyTM1n1nTMy=yTMy−1nyTMT1n1nTMy=yTMy−1n1nTMyT1nTMy=yTMy−1n1nTyT1nTy=yTMy−yT1n1n1nTy=yTM−1nJy.(ii)
SSE is also called residual sum of squares and defined asSSE=∑i=1nyi−yi^2=∑i=1nεi^2. Due to ∑i=1nεi^=0, the above formula is changed into(24)SSE=∑i=1nεi^−∑i=1nεi^2=∑i=1nεi^−1n∑i=1nεi^2.We can obtain the following expression ofε^ by Theorem 1; hence,(25)SSE=ε^TCnε^=ε^TI−1nJε^=I−MyTI−1nJI−My=yTI−MTI−1nJI−My=yTI−MTI−My−1nyTI−MTJI−My=yTI−MTI−My−1nyTI−MT1n1nTI−My.We can obtain1nTI−My=0 by (i). Therefore, the preceding formula reduces to SSE=yTI−My.(iii)
According to Theorem1, SST can be expressed as(26)SST=∑i=1nyi−1n∑i=1nyi2=∑i=1nyi−y_2=yTCny.
In the above example, for the sum of squares such as SSR and SSE, we expressed it by the centering matrix firstly and then obtained the quadratic form expression ofy after a series of derivation. For the sum of squares SST, we directly obtained its quadratic form expression by Theorem 1. The Table 1 gives the quadratic form expression in which the kernel function is Cn and the quadratic form expression of y for SSR, SSE, and SST, respectively.
From Table1, we can observe that SSR, SSE, and SST are all expressed as the quadratic form with the kernel matrix Cn. Generally speaking, sum of squares all can be considered to be expressed by the centering matrix. For instance, the sum of squares of normally distribution known as χ2 distribution is such a case. From formulas (23), (25), and (26), we can easily deduce the famous decomposition equality SST=SSR+SSE which is of vital importance in regression theory and analysis of variance.
In addition, the centering matrix has important simplified role in the ranking algorithms in information retrieval [4, 19, 20]. For instance, in [19], T. Pahikkala et al. introduced the centering matrix to construct a block diagonal matrix as weighted matrix, express the loss function, obtain the matrix notation of a least squares problem, and conveniently find the linear solution of RankRLS algorithm.Table 1
Quadratic form expressions for SSR, SSE, and SST.
TypeQuadratic form expression with the kernel matrixCnQuadratic form expression ofySSRy^TCny^yTM−1/nJySSEε^TCnε^yTI−MySSTyTCnyyTCny


## 4. Conclusion

As we know, the centering matrix plays an important role in ML and MVA. In this paper, we mainly carried out the theory research of the centering matrix. We discussed in detail some properties of the centering matrix, proved some previously known properties, deduced some new properties of the centering matrix, and finally drew a mind map to clearly illustrate the relationship between these properties and their potential role in ML and MVA. Then, we explore the application of the centering matrix such as PCA and regression analysis. The research results show that the centering matrix has excellent characteristics and simplified expression role by matrix notation in ML and MVA. The sample covariance matrix, scatter matrix, and sum of deviation squares all can be expressed succinctly by the centering matrix. Since the centering matrix has good properties such as symmetry and idempotency, for the algorithms based on it, the learning process can be greatly simplified and the performance such as training complexity, stability, and speed can be improved.



---

*Source: 5545061-2021-04-28.xml*