# Degrees of Freedom in Functional Principal Components Analysis

Authors: Zixin Lin
Journal: Mathematical Problems in Engineering
 (2021)

---

## Abstract

This paper develops the analytical form of the degrees of freedom in functional principal components analysis. Under the framework of unbiased risk estimation, we derive an unbiased estimator with a clear analytical formula for the degrees of freedom in the one-way penalized functional principal components analysis paradigm. Specifically, a new analytical formula incorporating binary smoothing parameters is also derived based on the singular value decomposition and half-smoothed method regarding the two-way penalized functional principal components analysis framework. The performance of our procedures is demonstrated by simulation studies.


---

## Body

## 1. Introduction

Functional principal component analysis (FPCA) is a key method for analyzing principal component from smoothing data, such as the data observed from temperature curves with sinusoidal nature. Additionally, FPCA has become a crucial research focus in many statistical fields. The author in [1] proposed the principle of the roughness penalty to deal with data curves using the smoothing spline method. The authors in [1, 2] proposed the roughness penalty to analyze functional data by decomposing variation in a two-way data table. Utilizing the rank-one approximation to the data matrix and penalizing only the right eigenvectors, the authors in [3] considered that a sample X was observed from a linear model consisting of the combination of original left singular vectors, right eigenvectors, and observed noise. Moreover, by singular value decomposition (SVD), the authors in [4] analyzed two-way functional data by penalizing left and right singular vectors of a generated covariance matrix. With functional data observed discretely, the initial model in the research of [3, 4] can be written as(1)Xn×m=∑k=1KukvkT+E,where m≤n, E is the noise matrix following N0,σ2In⊗Im, and uk and vk denote two fixed and nonobservable vectors, which have the sizes of n×1 and m×1, respectively. Based on K=1, the authors in [1, 3] considered the problem of finding the best rank-one approximation of X and estimating u^1,v^1 by(2)argminu1,v1X−u1v1T2+αv1TΩv1,where α and Ω denote the nonnegative penalized parameter and the penalized matrix, respectively, and u^1=Xv1/v1Tv1 for any fixed v1. The authors in [4] further proposed a two-way penalizing approach:(3)argminu1,v1X−u1v1T2+Pu1,v1,where Pu1,v1=u1TαuΩuu1⋅v12+u1∥2⋅v1TαvΩvv1+u1TαuΩuu1⋅v1TαvΩvv1, αu and αv are penalized parameters, and Ωu and Ωv are penalized matrices.The degrees of freedom (DoF), a conception used to quantitatively measure the complexity of a given model, is of importance in the field of FPCA and for many learning tasks. For example, the authors in [5] derived the soft thresholding DoF using Stein’s unbiased risk estimation theory [6] and demonstrated that it leads toSureShrink, a procedure of adaptive wavelet shrinkage. Besides, the author in [7] revealed that Cp, as an unbiased estimate that is used to measure the deviation from ture prediction, in some cases provides concretely more excellent accuracy than the coefficient of variation CV or related nonparametric methods when the correct DoF was used. See more details about the Bayesian information criterion BIC [8], the Generalized Cross Validation GCV [9], the Akaike Information Criterion AIC [10], the residual information criterion RIC [11], and the ‘hat’ matrix [12] in their respective papers. The author in [7] proposed that let δ be a specific fitting approach, μ^=δy denote its fit, and y be generated according to y∼μ,σ2I, where μ is the true mean vector associated with the common variance σ2. It was shown that the DoF of δ is(4)dfμ^=∑i=1ncovμ^i,xiσ2.Nonetheless, the research of [13, 14] suggested that using the method of perturbation for data to quantitatively calculate an unbiased estimator (whether the estimator is approximate or not) for dfμ^ can be computationally expensive when the analytical formula of μ^ is unavailable. It is an intriguing research issue based on theoretical and practical aspects to study strict analytical formula of the DoF of FPCA. The main purpose of this paper is to analyze the DoF using SURE for one-way and two-way penalized FPCA. The SURE theorem [15] offers a rigorous definition for the DoF under any fitting process. Particularly, under the background of FPCA, the definition of DoF is(5)dfμ^=E∑i=1m∑j=1n∂μ^ij∂xij.This definition implies that an unbiased estimator for the DoF is(6)d^fμ^=∑i=1m∑j=1n∂μ^ij∂xij.The structure of the paper is organized as follows. We show our results with respect to the setting proposed by [3, 4] and the implications of the results in section of Main Results. Numerical experiments are presented in section of Simulation. The appendix involves all technical issues and details. For ease of presentation, except in Appendix 1 and Appendix 2, we use bold letters to denote vectors.


## 2. Main Results

For a fixedv1, plugging this u^1=Xv1/v1Tv1 into (2) and transforming it as(7)argminv1v1TαΩ−XTXv1v1Tv1,it is easy to conclude that the solution to (7) is the eigenvector that corresponds to the minimum eigenvalue of the matrix αΩ−XTX. Without loss of generality, we make the following assumptions: firstly, Ω is a symmetric matrix; secondly, the eigenvalues of αΩ−XTX are ordered by λ1>⋯>λm, where λj≠λi for any i≠j; and thirdly, v˙k is the eigenvector of αΩ−XTX corresponding to the eigenvalue λk and is orthonormal with other eigenvectors. The same assumptions apply to more general situations, which are only more trivial in terms of notions. The rank-one approximation to X is written as(8)X^=Xv˙mv˙mT.Theorem 1.
For1≤k≤m, let v˙kT=vik,…,vmk, X=x1,…,xm denote the sample matrix, and Ω>0. An unbiased estimate of the DoF for (8) is(9)d^fX^=n−∑j=1m∑k=1m−12xjTxjλm2−λk2−1λkvjk2+λmvjm2−∑j=1m∑k=1m−1xjTxjλm−λk−1vjk2vjm2.Theorem1 implies that if αΩ−XTX remains positive definite, the first principal component un×1=Xv intuitively should have n free elements, but d^fX^>n in this case. Hence, the number of free elements does not measure the complexity of the model correctly. In other words, the number of free elements always underestimates the true DoF and should not be considered an accurate estimate for the DoF.LetSu1/2=I+αuΩu−1/2 and Sv1/2=I+αvΩv−1/2. Let X˜=Su1/2XSv1/2 be a data matrix with X whose rows and columns are half-smoothed. Take u¯1=Su−1/2u1 and v¯1=Sv−1/2v1; in this case, (3) can be transformed into(10)argminu¯1,v¯1X2−X˜2+X˜−u¯1v¯12.Without loss of generality, we make following notions and assumptions: firstly,X˜n×m is a half-smoothed data matrix with singular values σ1>⋯>σm>σm+1=⋯=σn=0; secondly, the SVD of X˜n×m is X˜n×m=U˜∇V˜T where V˜m×m=v˜1,…,v˜m and U˜n×n=u˜1,…,u˜n, and the elements of diagonal of ∇ are σk with nondiagonal ones taken as zero; thirdly, let Tv be a diagonal matrix such that its diagonal elements are the same as those of Sv1/2 with nondiagonal ones taken as zero, and Tu corresponds to Su1/2 under the same settings. Then, for any combination of u^1v^1T maximizing Cu¯1,v¯1, the Eckart–Young theorem suggests that next combination of u^2v^2T could be driven from minimizing(11)X2−X˜2+X˜−u^1v^1T−u¯2v¯22.Therefore, the approximation to the first-K-th principal component is(12)X^=∑l=1Ku˜lv˜lT.Theorem 2.
An unbiased estimate for the DoF of (12) is(13)d^fX^=σl−1trSv1/2u˜lTTuu˜l+σ1−1trSu1/2v˜lTTvv˜l−σl−12v˜lTTvv˜lu˜lTTuu˜l+∑l=1K∑k≠lσl2−σk2−1σkv˜lTTvv˜ku˜lTTuu˜k+∑l=1K∑k≠lσl2−σk2−1σl−1σk2v˜kTTvv˜ku˜lTTuu˜l+v˜lTTvv˜lu˜kTTuu˜k.Formula (8) categorically measures the shared weights of αu,αv. Let the norm and orthogonality of the vector space be as follows:(14)u¯kTTuu¯k=u¯k,u¯kT=1,u¯lTTuu¯k=u¯l,u¯kT=0,v¯kTTvv¯k=v¯k,v¯kT=1,v¯lTTvv¯k=v¯¯l,v¯kT=0.Theorem 3.
Under transformation of (14), formula (13) in Theorem 2 can be rewritten as(15)d^fX^=∑l=1KtrSu1/2+trSv1/2−2σl+∑l=1K∑k≠l2σk2σlσl2−σk2.As Theorem3 shows, if αu=0, under this situation, the original model (13) degrades to a one-way penalized problem but the left eigenvectors must still be considered. Combining (14) with (15), it is easy to show that under the transformation, an unbiased estimate for the DoF of a one-rank approximation with K=1 is(16)d^fX^=trSv1/2−2σ1+∑k=2m2σk2σ1σ12−σk2.


## 3. Simulation

The settings of the simulations for Theorem1 can be clarified with the following detail; firstly, the definition for the true DoF proposed by [15] under the circumstance of (6) is(17)dfX^=τ−2∑i=1n∑j=1mcovX^ij,Xij=τ−2EtrX^TX,and we use the average of 1000 sample values to estimate the EtrX^TX of the true DoF with a trivial value τ=1; secondly, compared with using 1000 samples to estimate the true DoF, we instead use the average of 50 sample values to measure the unbiased estimate suggested in Theorem 1 because when facing practical applications, we may not have a sufficient sample size. Thirdly, a naive estimate for the DoF is a constant number of free parameters which is shown in Figure 1 with the other parameters.Figure 1
The sizes of the matrix are (a)20×8, (b) 50×8, (c) 100×8, (d) 200×8. The horizontal axis represents the penalty parameter α, and the vertical axis corresponds the DoF.
(a)(b)(c)(d)Taking(18)A=100⋯0−210⋯01−21⋯001−2⋯0001⋯0⋮⋱⋯⋯1000⋯−2000⋯1,a popular choice for the penalty standard is vTΩv=∑j=2m−1vj+1−2vj+vj−12, whose principal philosophy is based on second differences of v, which means that the matrix Ω is as follows:(19)Ω=Am×m−2TAm×m−2.Figure1 shows the estimated differences in various situations by adjusting the size of the matrix.There are several pieces of information that can be obtained from Figure1. The naive estimator of the DoF does not correctly assess the true DoF of the model; furthermore, the true complexity of the model is generally greater than the number of free elements in a given eigenvector; that is, the free elements of this eigenvector underestimate the true complexity. Regardless of whether n is much greater than m, although there is a deviation due to the number of samples utilized, unbiased estimation (7) generally has excellent performance with regard to the true DoF.Many conditions that were argued for the first simulation of model (3) will continue to be utilized in the second simulation including the argument for the sample size and τ=1. A few special conditions should be clear. Under the coordinate transformation of (14), several of the crucial taken values stated in the previous part are(20)Ωv=Am×m−2TAm×m−2,Ωu=An×n−2TAn×n−2,dfX^=τ−2∑i=1n∑j=1mcovX^ij,Yij=τ−2EtrX^TY,where Y=Su1/2XSv1/2 [15]. Figure 2 shows that given the mutual constraint for the two penalty parameters, the aforementioned unbiased estimator (15) is an excellent fit for determining the true DoF.Figure 2
The sizes of the matrix are (a)50×8 and (b) 100×8. The plane coordinate axis denotes the selection of the parameters αu,αv. The z axis denotes the DoF.
(a)(b)



---

*Source: 8992841-2021-10-08.xml*