Prediction Performance of Single Index Principal Fitted 

Component Model 

 

Jia-Ern Pai 

Mathematical Statistician 

National Highway Traffic Safety Administration 

Department of Transportation 

1200 New Jersey Avenue, SE 

Washington, DC 20590 

Kofi Placid Adragni, Ph.D 

Assistant Professor 

Department of Mathematics and Statistics 

University of Maryland, Baltimore County 

ABSTRACT 

When dealing with large-p-small-n problems in regression analyses, a reduction of the 

dimensionality is always desired. A lofty goal is to reduce the dimension of the original 

design matrix without losing any regressive information. Utilizing the randomness 

property of the predictors, a sufficient reduction is obtained under a principal fitted 

component (PFC) model. Assuming that the sufficient reduction is of dimension one, we 

replace the original predictors vector with the dimension-reduced-predictors vector in a 

forward regression model to form the so called single index principal fitted component 

regression (PFCR) model. We conducted a simulation study to compare the prediction 

performances of the single index PFCR to forward dimension reduction models, 

including the partial least squares, LASSO and ridge regressions, under distinct scenarios.

1 Introduction 

Scientists in many research fields are encountering regression problems where the 

number of predictors p is larger than the number of observations n. For example, in 

genetic studies, the number of potential genes that may cause a disease is much greater 

than the number of patients in a clinical trial. Statistical analyses on data sets with p

greater than n are often referred to as “small-n-large-p” problems. To make further 

statistical inference, such as predictions, a reduction of the dimensionality of the

predictors is always desired. 

When building regression models, selecting potential variables, and making predictions, 

the conditional distribution of Y | X is traditionally applied, where Y is the response 

variable and X is a p-vector of predictors, such that 1

( ,..., )T X  X X

p

, Xi R , and 

i p 1,..., . A regression model for Y | X is called forward regression. Some forward 

regression methods, such as the partial least squares (PLS), LASSO regression, and 

principal component analysis are frequently used. When the predictors in X are fixed, the 
forward dimension reduction methods are naturally chosen for modeling . However, 

when the predictors in X are random, modeling may be a viable approach to 

attempt a dimension reduction on X. A regression model for is called an inverse 

regression. An example of inverse regression is the sliced inverse regression [14]. 

YX||XY

When dealing with large p problems, a lofty goal is to reduce the dimension of the ppredictor vector X to a d-predictor vector R( ) X , such that d p  , and use R( ) X as a 

surrogate variable. Dimension reduction methods have been developed for that purpose. 

Cook [5] has showed that when ( , ) Y X have a joint distribution, Y | X can be linked to 

X |Y through R( ) X that carries all of the regression information that X has about Y. In 

addition, Cook [5] has argued that the conditional distribution of X |Y provides more 

reductive information when encountering the large p problems. Cook [5] has proposed an 

inverse regression approach to dimension reduction in the regression context, which is 

called principal fitted components (PFC) models. They are likelihood-based approaches 

that model X |Y . 

PFC models are equipped to capture any type of associations between the predictors and 

the response variable through the use of a set of basis functions. In this research, we 

consider a special case of PFC models, where the predictors are linearly related to the 

response variable. We restrict our research scope to scenarios, where the dimension of the

sufficient reduction is one. The obtained reduction R( ) X is plugged in the forward 

regression model as the following:

Y   0 1  R( ) X  e

The prediction performances of this model, referred to as principal fitted components 

regression (PFCR), is studied through the simulations. We compare the prediction 

performances of PFCR with other traditional forward regression methods, such as the

PLS and LASSO regressions. We consider the following three scenarios: 

(a) Large n case, where n is greater than p, 

(b) Dense case, where p is larger than n and all the predictors are related to the 

response variable, 

(c) Sparse case, where p is larger than n. However, only a few predictors are related 

to the response variable. 

This thesis is organized as the following: 

In Section 1, we will present the PFCR. An exposition of PFC models will be provided. 

In Section 2, we will describe some traditional forward dimension reduction models, 

including the PLS, Ridge, and LASSO regressions. In Section 3, we will compare the 

prediction performances of the PFCR with other forward reduction models in distinct 
scenarios, including the large n, dense, and sparse cases. In Section 4, based on the 

simulation results in Section 3, the overall conclusion will be drawn. 

2 Principal Fitted Components 

2.1 Principal Components and Dimension Reduction 

Suppose 1

,..., X Xn

 represent the p-vector of predictors from n samples, Xi can be 

presented as 1 2 ( , ,..., )T X X X i i pi , where X ji R , j p 1,..., , and i n 1,2,..., . We denote 

the n p  design matrix X as 1

(( ,...,( ) ) T T T X X) X X  n

. One of the frequently applied 

methods for reducing the dimension in X is the principal component analysis. The 

principal component analysis has been studied for years, especially for its associated 

applications in linear regression models. The original idea of the principal component

analysis is to adopt the first few important components of the covariance matrix of X , so 

the dimension of the original design matrix can be reduced. While applying the principal 

component analysis in linear regressions, our purpose would be reducing the inflated 

variances of parameter estimators and providing more accurate predictions. 

Let 1 2

ˆ ˆ ˆ

, ,...,   p

be the eigenvalues of ˆ

, where ˆ

T

n

 =

X X and 1 2

ˆ ˆ ˆ

...       p

, we 

denote the associated eigenvectors as 1 2 ˆ ˆ ˆ , ,..., p

   . Applying linear transformations and 

combinations, the principal components of X are defined as 1 2 { , ,..., } ˆ ˆ ˆ

T T T T T T

p

 X X X   , 

where ˆ

i

 is called the principal component direction. In some cases, the principal 

component analysis cannot perform well in the dimension reduction. Anderson [3] has 

given an example: If all eigenvalues of ˆ

 are approximately the same, i.e. ˆ ˆ   i j  , i j , , 

such that i j  , no matter how we rotate the coordinates, we still cannot reduce the 

dimension of X . 

Depending on different application purposes, it has been discovered that by using the 

principal component analysis, the first few leading principal components

{ , ,..., } ˆ ˆ ˆ 1 2

T T T T T T

d

 X X X   , where d p  , possess many useful properties. The principal 

component analysis has also been studied for its applications in linear regressions. In a 

principal component regression (PCR), the initial set of p predictors is replaced by a set 

of d principal components. When d is less than p, a reduction of dimensionality is 

achieved. 

The most critical drawback of PCR is that it focuses on the dimension reduction in the 

design matrix without considering any regression information from the response variable. 

There is no guarantee that the first few leading components would necessarily be related 

to the response variable. It seems to be over simplified to think that the first few leading 

components contain the essential regression information regarding the response variable. 
The dimension reduction in a regression model by using the principal component analysis 

could sometimes fail, because of leaving out some major eigenvectors, which may be 

highly related to the response variable. Therefore, it may be necessary to consider both X 

and Y simultaneously when making the dimension reduction in a regression model. 

Cook [5] has claimed that given a p-vector of the predictors X, the purpose of the 

dimension reduction in a regression model is to search for a function R( ) X , whose 

dimension is less than or equal to p, such that R( ) X captures all the regression 

information that X contains regarding to Y. If Y | X has the same distribution as Y R| ( ) X

then R( ) X is called the sufficient dimension reduction. We may pursue the sufficient 

dimension reduction through the conditional distribution of Y | X , the conditional 

distribution of X |Y , or the joint distribution of ( , ) X Y . 

 

Suppose p R denote the p-dimension space. Cook [5] has further defined the dimension 

reduction :

p d R R R  , where d p  , to be sufficient, if R( ) X satisfies one of the 

following three conditions:

1. Inverse reduction, X X X X | ( , ( )) ~ | ( ) Y R R , 

2. Forward reduction, Y Y R | ~ | ( ) X X , 

3. Joint reduction, X is independent of Y given R( ) X , 

where A~B means that A and B have the same distribution. Each condition shows that if 

R( ) X is the sufficient reduction in a regression, R( ) X should contain all the regression 

information that X has in relation to Y. In the following content, we denote Y | ( ) X = x as 

Yx

, and X| ( ) Y y  as Xy. 

 

 

2.2 Principal Fitted Components 

Consider the following inverse regression of X on Y : 

X Γ y y       . (1) 

In model (1),  is a p1 vector, and Γ is a p d  semi-orthogonal matrix, such that 

d p  and T Γ Γ I  d

. The vector  y

is an unknown function of Y, which is assumed to 

have a positive definite sample covariance matrix and is centered to have the mean 0. The

sufficient dimension reduction is estimated by the first d principal components, which is 

T Γ X . Model (1) is referred to as principal component (PC) model. 

Cook [5] has stated that when Y is observed, 

y

v can be modeled as v f y y  β , where 

y

f is 

a known vector valued function of Y referred to as basis function. Thus, the PC model can 

be expressed as the following: 
 

Xy   Γβf 

y



 (2) 

 

Model (2) is called the principal fitted component (PFC) model. Suppose 2

var( )  p

 I , 

then T Γ X is still a sufficient reduction in model (2). In the PFC model, d r 

β R , where 

d r  . Thus, β has unrestricted rank d; also, r

f

y R with 0 y y  f  . When Y is 

continuous, Cook [5] has claimed that we can consider 

y

f ’s which contain a reasonably 

flexible set of basis functions, such as polynomial bases or piecewise polynomial bases.

 

In this research, we set d 1 and 

y

f  y in the PFC model, so that we can have fair and 

straightforward prediction comparisons with forward models, such as the OLS and 

LASSO regressions in the later section. Therefore, model (2) can be simplified the 

following: 

Xy     βy  . (3) 

In model (3),  is a p1 matrix, where 1

T

   , βR , and y is an observed value from 

an univariate random variable Y, such that E[ ] 0 Y  . Model (3) is referred to as singleindex-isotropic PFC model. In the following sections, we only concern isotropic PFC 

models, i.e. 2

var( )  p

 I . For the convenience, we adopt the term “signal index PFC 

model” when d = 1. 

The maximum likelihood estimates of the parameters in model (3) are provided in the 

following section. 

2.3 Estimation under Single Index PFC Model 

The parameters space is 2

( , , , )  Γ   in model (3). The log likelihood function is

2 2 ( , , , ) log(2π) log

2 2

np np L  Γ       2

1

1

( ) ( )

2

n

T

i i i i

i

y X y

 

        X  Γ Γ  . (4) 

Fixing Γ ,  , and 2  , equation (4) is a function of  , so the log likelihood function is 

maximized by 

1

ˆ /

n

i

i

n



   X X  . 

Substituting ˆ into equation (4), the log likelihood function can be expressed as the 

following: 
 

 

 

 

 

2 2 ( , , ) log(2π) log

2 2

np np

L Γ      

1

( ) ( )( ) 2 0 0 2 1

n T T T

y y i i i i

 i

       



X X Γ ΓΓ + Γ Γ X X Γ , (5) 

where 0

( , ) Γ Γ is a full rank orthogonal matrix, such that 0 0 ( , )( , )T Γ Γ Γ Γ I  p

. Holding Γ

and 2  , equation (5) is a function of  alone. Denoting 1

( ,..., )T Y  Y Yn

, 

ˆ

T

n

X Y C = , and 

2

ˆ

T

y

n

 

Y Y , equation (5) is maximized by 

ˆ

1

β ( ) 2

ˆ

T

T T T

 y



 

Γ

Γ

C X Y Y Y . 

Substituting β into equation (5), we obtain the following log likelihood function: 

2 2 ( , ) log(2π) log

2 2

np np L     Γ,  2 2

ˆ ˆ ˆ

trace( )

2 ˆ

T

y

n

 

 

PΓ CC , (6) 

where ˆ

T

n



X X  and T PΓ  ΓΓ . Fixing 2  , equation (6) is a function of Γ . 

The log likelihood function is maximized when Γ is the eigenvector of ˆ ˆ T CC

corresponding to the uniquely largest eigenvalue. Equation (6) is maximized when 

ˆ

ˆ

ˆ

Γ = C

C

and 2

ˆ

ˆβ

ˆ y



C

. 

To estimate 2  , we substitute ˆ , Γˆ

, and ˆβ into equation (6). The log likelihood 

function can be presented as 

2 2 ( ) log(2π) log

2 2

np np L      2 fit

ˆ ˆ trace( )

2

n



    , (7) 

where fit 2

ˆ ˆ ˆ

ˆ

T

 y

 

CC . Equation (7) is maximized when 2

fit

1 ˆ ˆ

ˆ trace( )

p

     . 

2.4 Signal Index Principal Fitted Components Regression 

We assume that ( , ) X Y are jointly observed, and both X and Y are random. Consider a 

forward regression model 

T

YX   β X e , (8) 
where p X R , 

p

β R , and e is the error term. Using the randomness property of X , 

we apply the PFC model. The sufficient reduction T Γ X retains all regression information 

contained in X about Y. 

 

We then replace X by T Γ X in model (8) to obtain 

( ) TYX    Γ X  , (9) 

where  R and d p  . 

We estimate Γ via the PFC model and denote ˆ

T Γ X as Z . Once Γˆ

 is obtained, equation 

(9) is equivalent to a simple linear regression 

YZ  Z + . 

It is observed that through the PFC model, ˆ

T Γ X acts like a proxy for X in a forward 

regression. The procedure of replacing X by a sufficient reduction in a forward 

regression is called the signal index principal fitted component regression (PFCR). 

The signal index PFCR is similar to the principal component regression (PCR), where the 

first few principal components of ˆ

 are used as the proxy for X in a forward regression; 

however, in the signal index PFCR, Z is called the principal fitted component. 

2.5 Prediction with Signal Index PFCR 

 

Given a new set of observations on the predictors, making predictions via the signal 

index PFCR follows the usual prediction procedure with a forward regression. Let * X be 

a new set of observations, then 

*

ˆ ˆ ˆ

* * ˆ

( | ) ( ) T

Y E Y     X

X X Γ X . 

2.5.1 Prediction Error 

The performance of the prediction is evaluated by the usual mean squared prediction 

error 2 ˆ E Y E Y [ ( | )]   X x . Consider two independent data sets ( , ) X Y and * * ( , ) X Y . 

Based on the model built in ( , ) X Y data set, we make the predictions on *

( , ) Y

* X . The 

mean squared prediction error (PE) is defined as 

* 2

1

1

PE ( ( | )) ˆ

n

i i

i

Y E Y

n 

   

* X X . 
2.5.2 Lower Bound of the Prediction Error 

 

 

 

A lower bound of the prediction error can be obtained as var( | ) Y X . Assuming ( , ) X Y

follows a joint normal distribution; a lower bound of the prediction error in the single 

index PFC model can be obtained. 

Consider the following setup of the single index PFC model 

X G G G y    βy+ β  y  , 

where G = G/ G and 2

~ (0, ) N  p

 I . This setup will be used in the later simulations. 

Let  denote the covariance matrix of X, cov(X), by applying the probability property

   var( ( | )) (var( | )) E Y E Y X X ,  can be presented as the following: 

2    var( βy) G G  I

2 2 2 2 β

T

  G GG   y

I

2 2 2 2 2

0 0 ( β ) T T       G GG G G y

. 

From  , 

1  can be obtained as 0 0

2 2 2 2 2 β

T T

y

  





G G GG

G

. 

 

 

 

Since ~ , T 2

Y y

N

Y  

       

               

X X C

C

 

, where 2 C    cov( , ) cov( β ) β X G , G G Y Y Y  y, 

we have the var( | ) Y X as 

2 1 T  y

C C  . We then derive the expression of the 

var( | ) Y X as the following: 

2 2 2 4

2 2 2 2

var( | ) β

β

T

T

y y

y

Y  

 

 

   

  

GG X G G G

G

2 2 4

2

2 2 2 2

β

β

y

y

y





 





G

=

G

2 2 2

2

2 2 2 2

β

1

β

y

y

y





 

 

   

  

G

G

2 2

2 2 2 2 β

y

y

 

 



 G

. (10) 
If 1

( ,..., )T G =   p

 in equation (10), then 2

1

p

i

i





G   . 

2.5.3 Prediction under Sparsity 

The estimate Γˆ

 of Γ in the single index PFC model can be used for variable selections. 

It is observed that the dimension reduction T Γ X is a linear combination of the p

predictors. If a predictor X j

 is inactive and not related to Y, then the th j row of Γ must 

be 0.

Numerical artifacts yield all nonzero entries of Γˆ when estimating Γ , even for the 

entries that should be zeros. This has the effect of reducing the prediction accuracy. To 

prevent this, a sparse estimation of Γ is adopted.

The sparse estimate of Γ is obtained by a hard thresholding procedure. We let Γˆ

 be a 

crude estimate of Γ and denote j

 as th j entry in Γˆ

, where j p 1,..., . Additionally, we 

denote min  and 

max

 as minimum and maximum absolute values of j

 ’s in Γˆ

. For 

min max      , we define 

ˆ ˆ ˆ I( ) Γ Γ 1 Γ     p

. 

Here, I is the indicator function; Γˆ

 stands for the elementwise absolute value of Γˆ, and 

(1,...,1)T1p  is the p1 column vector of 1’s. The indicator ˆ

I( ) p Γ 1   is a p-vector of 

0’s and1’s. If ˆ

j

   then I( ) 1 ˆ

j

    ; otherwise, I( ) 0 ˆ

j

    , where j p 1,..., . 

The appropriate value of  is determined by cross-validation. 

In the following section, we will describe other forward dimension reduction methods 

and will compare their prediction performances to the signal index PFCR later under 

various scenarios. 

3 Forward Dimension Reduction Methods

3.1 Partial Least Squares Model 

The partial least squares (PLS) model is a technique that finds a linear regression model 

by projecting the design matrix X and the response vector Y to new spaces, where 

n p  XR and n YR . Since both X and Y are projected into new spaces 

simultaneously, the PLS model is also called as bilinear factor model. Martens and Næs 

[11] have made a statement: “The PLS regression is designed to follow the declaration: 
No predictor without interpretation, no interpretation without predictive ability. A good 

interpretation property requires simplicity, such as a low dimension model. A good 

predictive ability also requires such simplicity in order to avoid overfitting. Hence, the 

intention of the PLS regression is to provide a model with as few dimensions as possible

and in such a way that these dimensions are as relevant to the response as possible”.

The PLS regression is particularly useful when the main purpose is to predict the 

response variable with a high dimension design matrix. When the predictors are full of 

multicollinearities and the number of observations is small, the parameter estimates 

would likely encounter the problem of inflated variances. The PLS model conducts the

dimension reduction procedure, so it provides more accurate parameter estimates and 

response predictions.

We aim to obtain a set of informative predictors by projecting the original design matrix

X onto a new space, where we can obtain a new lower dimension n d  design matrix 

X*

, such that d p  . Then we use X*

as a new set of predictors for Y . Unlike the 

principal component analysis, the PLS model finds components from X that are relevant 

to Y .

The set of components, which is called as latent vectors, perform a simultaneous 

decomposition on both X and Y by maximizing the covariance between X and Y . A 

number of iterative procedures are found in the literature to estimate the parameters

involved in a PLS regression. Helland [9] has provided a condensed expression of the 

PLS iterative procedures.

Applying the forward regression in model (8), the PLS estimate of β is given by 

ˆ ˆ

(PLS) 1 ˆ

( ) T T 

β F F F F   C , 

where ˆ ˆ ˆ ˆ ˆ 1

( , ,..., ) q F = C C C   , 

ˆ

T

n

C 

X Y , and ˆ

T

n

 

X X . A single index PLS 

regression is obtained by setting the value q equal to 1, so that only one latent vector is 

used. All the latent vectors are involved in a PLS regression when q p  . 

3.2 Penalized Methods for Linear Models 

When a linear regression contains a large number of predictors, parameter estimates are 

often under restrictions. By doing so, the dimension of the design matrix may be reduced. 

Several methods, such as the Ridge [8], Bridge [6], LASSO [15], and Dantzig selector [4] 

have been developed. We introduce the ridge regression first, which is one of the earliest 

penalized regression models. Then we launch into the LASSO regression, which is one of 

the most popular penalized dimension reduction methods.
3.2.1 Ridge Regression

Due to the multicollinearity problems among the predictors, the variances of the

parameter estimates can be inflated. Unlike the PFC and PLS models, the main purpose

of the ridge regression is to reduce the inflated variances. The ridge regression has 

enlightened other penalized approaches for the dimension reduction; however, the ridge 

regression itself does not reduce the dimension of X .

Consider a p-vector of predictors X and an univariate response variable Y, we regress Y

on X by applying the forward regression in model (8), where 1

(β ,...,β )Tβ  p

, β j R , 

and j p 1,..., . Hoerl [10] has argued that applying the ridge regression to minimize the 

variance of ˆβ is equivalent to minimize the square distance between ˆβ and β , i.e. 

minimizing ˆ ˆ ( ) ( ) T

β β β β   .

Tibshirani [15] has concluded that by adding some penalty or restriction term, the ridge 

regression sacrifices the unbiasedness property of parameter estimates, i.e. ˆ E( ) β β  , to 

reduce the inflated variance of ˆβ . Therefore, the prediction accuracy of the model may be 

improved.

The ridge regression estimates β by minimizing the residual sum of squares 

2

1

( β )

n

i j ij j

i

Y X



  , such that 2

β j

j   t . Equivalently, the ridge parameter estimate 

ˆ(Ridge) β is obtained as 

(Ridge) 2 2

1

ˆ arg min ( β ) β

n

i j ij j j j

i

Y X 



 

      

β β    , 

where  R . The value of  controls the amount of shrinkages of coefficient estimates. 

When  increases, the greater shrinkages will occur on ridge parameter estimates, i.e. 

some β j

’s will tend to zeros. The appropriate value of  is determined by the crossvalidation. However, the ridge regression does not set any β j

 to zero. Therefore, the 

ridge regression does not reduce the dimension of the design matrix. 

Hastie, Tibshirani, and Friedman [8] have made a connection between the ridge 

regression and the principal components analysis. By applying the singular value 

decomposition, the n p  design matrix X can be decomposed into T UDV , such that U

is a n p  orthogonal matrix, V is a p p  orthogonal matrix, and D is a p p  diagonal 

matrix. We denote the columns of the matrix U as U j, where j p 1,..., . The U j’s form 

an orthonormal basis for the space spanned by the column vectors in X . In addition, the 
columns of the matrix V form orthonormal bases for the space spanned by the row 

vectors in X . The diagonal entries of D are the eigenvalues of X , which can be 

expressed as diag( , , ..., ) d d d 1 2 p

, such that 1 2 d d d     ... 0 p. 

 

 

 

The n1 prediction vector ˆYX of the ridge regression can be presented as 

ˆ (Ridge) (Ridge) ˆ Y X X = β

1

( ) T T  p



  X X X X Y I

2 1 ( ) T

 p



  UD D I DU Y

 

2

2

1

p

j T

j j

j j

d

 d 





U U Y . 

In the geometric aspect, the ridge regression shrinks the coordinates with respect to the 

orthonormal bases formed by the principal components. The coordinates with respect to 

the principal components, which contains smaller variances will be shrunk more. 

The ridge regression identifies the most important few predictors in X from the amount 

of shrinkages. In the following section, we introduce another penalized regression, which 

reduces the dimension of X by setting some β j

’s exactly to zeros. 

3.2.2 Least Absolute Shrinkage and Selection Operator (LASSO) 

In the regression analysis, it is often to select a smaller subset of the predictors, which 

may possess enough regression information for making predictions. The prediction 

accuracy may sometimes be improved by setting some coefficients β j

’s to zeros. By 

doing so, it is equivalent to reduce the dimension of the design matrix. Similar to the 

ridge regression, the least absolute shrinkage and selection operator (LASSO) regression 

is a penalized approach, which sacrifices the unbiasedness property of the parameter 

estimates to improve the prediction accuracy.

However, unlike the ridge regression, the LASSO minimizes the residual sum of squares 

by restricting the sum of the absolute values of β j

’s to be less than a constant value. The 

restriction on 2

β j j

in the ridge regression is replaced by β j j in the LASSO 

regression.

Denote t as 

1

β

p

j j

 and 0

t as ˆ(OLS) β j j

, Tibshirani [15] has claimed that when the 

value of t is smaller than 0

t , some β j

’s should be set to zeros. Therefore, the LASSO 

regression may conduct the dimension reduction in the design matrix.
Applying the forward regression in model (8), where 1

(β ,...,β )Tβ  p

, β j R , and 

j p 1,..., , Tibshirani [15] has defined the LASSO parameter estimate ˆ(LASSO) β as

(LASSO) 2

1 1

ˆ arg min ( β ) β

n p

i j ij j j

i j

Y X 

 

 

      

β β    , 

where  R . The tuning parameter  is determined by using the cross-validation. 

In the following simulation section, we will compare the prediction performances of the

single index PFC model to the ridge regression when n p  , i.e. the large n case. Since 

both PFC model and LASSO regression conduct the dimension reduction, we will focus 

on comparing the prediction performances of these two models when n p  . 

4 Simulation Study

The purpose of the simulation study is to compare the prediction performances of the 

single index PFC model with other methods, such as the OLS, ridge, LASSO, and PLS 

regressions. We consider two main cases: n p  and n p  . In the first case, n p 

problems, we compare the prediction performances of the single index PFC model with 

PLS and LASSO regressions. In the second case, n p  , we compare the single index 

PFC model to the previous two and also the OLS and ridge regressions. In this simulation 

study, we introduce the simulation setup in the first part. Then we provide the prediction 

performances of different models in distinct scenarios to make the prediction 

comparisons. 

4.1 Data Simulation 

Before applying different models to make prediction comparisons, we start with 

generating two independent equal-sized data sets ( , ) X Y and *

( , ) * X Y . The ( , ) X Y

data set is used for model building, while the *

( , ) * X Y data set is for assessing the 

prediction performance of a regression model, which is constructed in ( , ) X Y . The way 

we generate these two data sets are identical. Here, we only introduce the data generating 

procedure of ( , ) X Y . 

We generate n observations of X’s and y’s, where p X R and yR . The response 

observations i

y ’s are independent and identically distributed from a normal distribution 

with mean 0 and a known variance 2  y

, i.e. 2

~ (0, ) i y y N  , where i n 1,..., . We denote 

the 1n vector Y as 1

( ,..., )ny y and set the n p  matrix X as 1

( ,..., )T X Xn

, where 

P X R i  , i n 1,..., . We set X to be a function of Y plus an error term  . The matrix 

X can be expressed as the following: 
 

β( )T X   ΓY  , (11) 

where p Γ R , βR and 1

( ,..., )  n

   are generated from 2

(0, ) N  p

I , such that 2  is

given. 

In many regression simulation studies, the response variable Y is generated from a linear 

function of the independent predictors Xi

Y first and make X as a linear function of . 

’s. However, based on our research purpose, we 

generate the response vector Y

From equation (11), it is noticeable that when β increases, the linear association between

X and Y becomes stronger. However, for a small value of β , the error term may 

dominate the linear relationship between X and Y . When β is small, it should be 

difficult for the single index PFC model to find the sufficient reduction, because the 

response variable Y cannot reveal sufficient regression information for X . Similarly, 

the forward dimension reduction models may not perform well either, since X cannot 

convey enough regression information on Y . 

Applying the same data generation procedure, we simulate another data set *

( , ) * X Y . 

We then substitute the model built from ( , ) X Y to *

( , ) * X Y and assess the prediction 

performance of the model. 

4.2 Simulation Setup 

By setting different values for n, p, and Γ , we consider three cases in the simulation 

study, which are the large-n, dense, and sparse cases. In each case, we use 10-fold in the 

( , ) X Y data set when applying the cross-validation. 

The single index PFC model involves a single linear combination of the predictor. 

Therefore, the single index PLS regression is applied, which is denoted as PLS (1), to 

make straightforward and fair prediction comparisons. In addition, the PLS regression 

with q latent vectors, which is denoted by PLS (q), such that q 1, is also used to make 

predictions. When applying the PLS (q) regression, the exact value of q is determined by 

the cross-validation. 

4.2.1 Large-n-case 

In the large-n-case, n p  and all predictors are active. Fixing the number of the 

predictors at 25, we set ~ (0, ) N 25  I and ~ (0,1) i

y N , where i n 1,2,..., . By changing 

the values of n and β , we create different scenarios. There are three levels of n and three 

levels of β in the large-n-case. The values of n, β, p, 

2  , and 2  y

 are summarized in the 

following table: 
Table 4.1: Large-n-case setup 

level 1 level 2 level 3 

n 100 300 600 

0.1 0.4 1 

25 

1 

1 

β, 2 

2  y

βp

2 

2 β y

2 

2  y

From equation (11), the design matrix X is a n25 matrix; with different levels of n

and β , we have 9 distinct scenarios. 

Since p = 25 and d = 1, we set 1 1 1 ( , ,..., )

25 25 25

T Γ  . Applying equation (10) and 

setting G= , with different value of β , the associated lower bounds of PE’s can be 

calculated. The lower bounds of PE’s in the large-n-case will be presented in the later 

section. 

4.2.2 Dense case

The dense case is one of the n p  problems, such that all the predictors are active. 

Fixing the number of observations at 100, we set ~ (0, ) N p

 I and ~ (0,1) i

y N , where 

i 1,2,...,100 . Different scenarios are created for prediction performance comparisons by 

changing the values of p and β . There are three levels of p and three levels of β in the 

dense case. 

The values of p, n, , and are summarized in the following table: 

Table 4.2: Dense case setup 

level 1 level 2 level 3 

p 100 200 400 

0.1 0.4 1 

n 100 

1 

1 

With different combinations of p and β , there are 9 distinct scenarios. In the dense case, 

p1 Γ R , and we set (1,1,...,1)T Γ  . Using equation (10) and setting G= , with different 

values of p and β , the associated lower bounds of PE’s can be calculated. The lower 

bounds of PE’s in the dense case will be presented in the later section. 
4.2.3 Sparse case

The sparse case is another type of the n p  problems, where only a few predictors are 

active. Since only a few predictors are relative to the response variable, the ability of 

providing the sufficient dimension reduction is critical for making accurate predictions. 

We denote the number of response-related predictors as 0

p , such that 0p p  ; 

additionally, we set the rest 0

( ) p p  predictors as response-unrelated. 

In the sparse case, the number of response-related predictor is fixed at 10, i.e. 0

p 10 , 

and the number of observations is set at 100. In addition, we set ~ (0, ) N p

 I and 

~ (0,1) iy N , where i 1,2,...,100 . By changing the values of p and β , different 

scenarios are created for comparing prediction performances. There are three levels of p

and three levels of β in the sparse case. The values of p, β, 0

p , n, 

2  , and 2  y

 are 

summarized in the following table: 

β 0

p2 

2  y

Table 4.3: Sparse case setup 

level 1 level 2 level 3 

p 100 200 400 

0.1 0.4 1 

10 

n 100 

1 

1 

Changing different values of p and β , there are 9 distinct scenarios. Since p1

R and 

0

p is fixed at 10, we let the first 10 rows of Γ to be 1’s and the rest ( 10) p  rows to be 

0’s. With this setting, Γ can be presented as (1,...,1,0,...,0)T

, and X can be expressed as 

the following: 

1 1 1,1 1,( 1) 1,

2 2 2,1 2,( 1) 2,

100 100 100,1 100,( 1) 100,

... 0 ... 0 ...

... 0 ... 0 ...

β

... ... 0 : ... : :

... 0 ... 0 ...

p p

p p

p p

Y Y

Y Y

Y Y

  

  

  







   

   

 

   

X . 

With different values of p and β , the associated lower bounds of PE’s can be calculated 

by applying equation (10) and setting G= ,. The lower bounds of PE’s in the sparse case 

will be presented in the later section. 
4.3 Simulation Results 

In the large-n, dense, and sparse cases, we apply different regression models in each 

scenario, which contains distinct values of n, β , or p. We iterate data generating, model 

buildings, and PE’s calculations 100 times in each scenario. The aforementioned 

procedure allows us to calculate the mean PE and the standard error SE(PE) from 100 

PE’s. The following simulation results in all trials are the PE ’s along with the associated 

SE(PE)’s in parenthesis. 

4.3.1 Large-n-case 

The associated lower bound of PE for each level of β is listed in Table 4 through Table 6 

on page 30, obviously, when β increases the lower bound of the PE decreases. More 

precise predictions should be obtained when β becomes larger. 

Given β 0.1  , in Table 3, the OLS and ridge regressions provide the significantly largest 

PE on each level of n. The prediction performances of the OLS and ridge regressions are 

very similar. However, by providing the significantly smallest PE on each level of n, the 

LASSO regression performs the best. In addition, it is noticeable that the single index 

PFCR, PLS (q), and PLS (1) regressions yield similar prediction performances. 

When β 0.4  , the linear association between X and the Y becomes stronger. In Table 

5, we observe that the inverse and forward dimension reduction models perform 

significantly better than the OLS and ridge regressions. When β 1  , the linear association 

between X and Y is even stronger, and the effects of the error term become less 

significant. In Table 6, the PLS (q) regression dominates other methods by providing the 

significantly smallest PE. It is noticeable that the single index PFCR and PLS (1) yield 

almost identical prediction performances. 

In the large n case, when n increases, the PE ’s do not change dramatically within any 

given regression model; however, the SE(PE) decreases quickly as n increases. X and 

Y cannot reveal sufficient regression information to each other when β is small, such as 

0.1. Thus, the dimension reduction methods, such as the single index PFC model, PLS 

(q), and PLS (1) regressions cannot precisely find the reduction to provide accurate 

predictions. However, the LASSO regression performs the best by sacrificing the 

unbiasedness property of parameter estimates. This is the advantage of the penalized 

method. 

It is easier for every model to find appropriate coefficient estimates and make predictions 

when β increases, such as 0.4 or 1. In Table 5 and Table 6, each model provides similar 

prediction performance when n is large. It is also observed that the prediction 
performances of the single index PFCR and OLS are similar when n is large from Table 4 

through Table 6. 

Table 4.4: Large n case when Beta = 0.1 

Beta = 0.1 

OLS LASSO Ridge PLS(q) PLS(1) PFC 

n = 100 1.32(0.020) 1.00(0.014) 1.31(0.020) 1.17(0.016) 1.17(0.016) 1.15(0.015) 

n = 300 1.08(0.010) 1.00(0.009) 1.08(0.010) 1.06(0.009) 1.06(0.009) 1.05(0.008) 

n = 600 1.04(0.006) 1.00(0.004) 1.03(0.006) 1.03(0.005) 1.03(0.005) 1.03(0.006) 

Lower 

Bound 

0.990099 

Table 4.5: Large n case when Beta = 0.4 

Beta = 0.4 

OLS LASSO Ridge PLS(q) PLS(1) PFC 

n = 100 1.15(0.018) 1.00(0.013) 1.14(0.018) 1.00(0.015) 1.02 (0.019) 1.03(0.021) 

n = 300 0.97(0.008) 0.94(0.008) 0.97(0.008) 0.92(0.008) 0.94(0.009) 0.95(0.009) 

n = 600 0.93(0.006) 0.91(0.006) 0.92(0.005) 0.89(0.006) 0.90(0.006) 0.91(0.005) 

Lower 

Bound 

0.862069 

Table 4.6: Large n case when Beta = 1 

 Beta = 1 

OLS LASSO Ridge PLS(q) PLS(1) PFC 

n = 100 0.66(0.018) 0.63(0.007) 0.65(0.015) 0.55(0.007) 0.60(0.010) 0.60(0.009) 

n = 300 0.56(0.006) 0.53(0.006) 0.55(0.005) 0.51(0.005) 0.53(0.006) 0.53(0.005) 

n = 600 0.53(0.003) 0.52(0.003) 0.53(0.003) 0.51(0.003) 0.52(0.003) 0.52(0.003) 

Lower 

Bound 

0.5 

4.3.2 Dense case 

All of the predictors are linearly related to the response variable in the dense case. When 

the number of predictors p increases, T Γ X accumulates more regression information 

from the response variable. Thus, the predictions of the single index PFCR may be more 

accurate when p becomes larger. Similarly, in forward dimension reduction models, the 

response variable collects more regression signals from X when p increases; thus, the 

predictions of the PLS (1) and PLS (q) regressions should be more precise. These 

phenomena can be observed from Table 7 through Table 9 on page 31 to page 32. It is 

noticeable that for a given β , when p increases, the PE ’s of the single index PFCR, PLS 

(1), and PLS (q) regressions decrease.
We can obtain equally precise predictions by using signal principal fitted component 

when the regression signals become stronger. The single index PFCR provides similar 

predictions compared to the PLS (q) regression in Table 8 and Table 9 by capturing the 

most important principal fitted component.

Unlike the single index PFCR, PLS (1), and PLS (q) regressions, the LASSO regression 

seems to be unstable in some cases. The LASSO regression shows huge values of PE

and SE(PE) when p = n in Table 7 through Table 9.

Table 4.7: Dense case when Beta = 0.1 

Beta=0.1 

LASSO PLS(q) PLS(1) PFC Lower Bound 

p = 100 5.60(2.000) 0.66(0.010) 0.73(0.010) 0.74(0.011) 0.5 

p = 200 0.82(0.015) 0.54(0.011) 0.61(0.010) 0.61(0.011) 0.3 

p = 400 0.72(0.012) 0.41(0.010) 0.49(0.009) 0.50(0.009) 0.2 

Table 4.8: Dense case when Beta = 0.4 

Beta=0.4 

 LASSO PLS(q) PLS(1) PFC Lower Bound 

p = 100 133.82(130.234) 0.06(0.001) 0.07(0.001) 0.07(0.001) 0.059 

p = 200 0.10(0.002) 0.04(0.001) 0.04(0.001) 0.04(0.001) 0.030 

p = 400 0.11(0.002) 0.02(0.000) 0.02(0.000) 0.02(0.000) 0.015 

Table 4.9: Dense case when Beta = 1 

Beta=1 

 LASSO PLS(q) PLS(1) PFC Lower Bound 

p = 100 0.70(0.152) 0.01(0.000) 0.01(0.000) 0.01(0.000) 0.0099 

p = 200 0.02(0.000) 0.01(0.000) 0.01(0.000) 0.01(0.000) 0.0050 

p = 400 0.02(0.000) 0.003(0.000) 0.003(0.000) 0.003(0.000) 0.0025 

4.3.3 Sparse case 

Since not all the predictors are active in the sparse case, when p increases, it cannot be 

guaranteed that X will accumulate more regression information from the response 

variable. Therefore, the PE’s from either inverse or forward dimension reduction models

may not be monotonically decreasing when p increases.

The sparse single index PFC model applies the hard thresholding procedure to make the 

coefficient shrinkages. The LASSO regression achieves the same goal by using the 

penalized method. In addition, the single index PFCR and LASSO regression use one 

principal component direction when making predictions. However, the PLS regression 

does not have the shrinkage procedure when estimating regression coefficients. To have a

fair and straightforward comparison, we compare the prediction performances of the 
single index PFCR and LASSO regression in the sparse case. The associated prediction 

performances are listed from Table 10 to Table 12 on page 33 to page 34.

The LASSO regression dominates the single index PFCR when β is small, such as 0.1. It 

is difficult for the sparse single index PFC model to find the most important principal 

fitted component when the linear association between X and Y is weak. It is more 

effective to use penalized method to provide accurate predictions when X and Y cannot 

reveal enough regression information to each other.

The single index PFCR is expected to improve the prediction performances when the 

linear association between X and Y becomes stronger. It is observed that the single 

index PFCR and LASSO regression provide similar prediction performances in Table 11 

and Table 12.

However, similar to the dense case, it seems that the LASSO regression is unstable in 

some scenarios. The LASSO regression provides large values of PE ’s and SE(PE)’s

when p = n in Table 10 through Table 12. 

Table 4.10: Sparse case when Beta = 0.1 and 

p0 10

 Beta=0.1 

 LASSO sparse PFC Lower Bound 

p = 100 1.41(0.281) 1.15(0.020) 0.91 

p = 200 1.05(0.020) 1.20(0.019) 0.91 

p = 400 1.04(0.017) 1.16(0.017) 0.91 

Table 4.11: Sparse case when Beta = 0.4 and 0

p 10

 Beta=0.4 

 LASSO sparse PFC Lower Bound 

p = 100 1.92(0.595) 0.48(0.008) 0.38 

p = 200 0.54(0.010) 0.53(0.009) 0.38 

p = 400 0.57(0.010) 0.62(0.010) 0.38 

Table 4.12: Sparse case when Beta = 1 and 0

p 10

 Beta=1 

 LASSO sparse PFC Lower Bound 

p = 100 0.61(0.100) 0.10(0.002) 0.09 

p = 200 0.12(0.002) 0.11(0.002) 0.09 

p = 400 0.12(0.002) 0.13(0.003) 0.09 
Conclusion 

In many real applications, we can only distinct whether the case belongs to the n p  or 

n p  problems. In a regression application with a few predictors, we can determine if all 

the predictors are relative to the response variable by plotting X versus y. However, the 

number of predictors is usually large, such as p  25, in many practical cases. It is 

difficult to determine whether all the predictors are response-related simply by plotting 

when encountering a large number of predictors. Therefore, the dense and sparse cases 

are not easily to be identified in many applications. 

In the large-n-case of this research, all the predictors are active and we assume that 

y

f  y in the single index PFC model. The single index PFCR is not specifically 

outstanding in making predictions. In some scenarios, even the OLS regression can 

provide equivalent prediction performances. But we should still take the PFC model as

another option, because not all the predictors are active in some large-n-case. The PFCR 

may provide better prediction performances by using the hard thresholding shrinkage 

procedure. 

It is observed that the single index PFCR performs similarly to the PLS (1) regression in 

the dense case. This phenomenon is based on the assumption that 

y

f  y in the single 

index PFC model. The PLS regression is set with no shrinkage procedure in this research. 

However, there are methods to put in restrictions when making parameter estimates in a 

PLS regression. By doing so, shrunk PLS coefficient estimates can be obtained. 

The LASSO regression performs even better than the single index PFCR especially when 

the regression signal is weak. However, it is noticeable that the prediction performances 

of the LASSO regression seem to be unstable when n is approximately equal to p in this 

research. Because of this reason, the PFCR is recommended. 

Prediction errors are used as criterion for model comparisons in this research. However, 

the ability of making model interpretation is also important in statistical studies. It should 

be noticed that the PFCR may not be as easily interpreted as other model, such as ridge 

regression. 

Bibliography 

[1] Abdi, H. (2007). Encyclopedia of Measurement and Statistics, Thousand Oaks, 

 California. 

[2] Adragni, K. P. and Cook, R. D. (2009). Sufficient Dimension Reduction and 

 Prediction in Regression, Philosophical Transactions of the Royal Society A, Vol. 

 367, No. 1906, 4385-4405. 
[3] Anderson, T. W. (2003). An Introduction to Multivariate Statistical Analysis, Wiley, 

 New York. 

[4] Candes, E. and Tao, T. (2007). The Dantzig selector: statistical estimation when p is 

 much larger than n, Annals of Statistics, Vol. 35, 2313-2351. 

[5] Cook, R. D. (2007). Fisher Lecture: Dimension Reduction in Regression, Statistical 

 Science, Vol. 22, No. 1, 1-26. 

[6] Cook, R. D. and Forzani, L. (2008). Principal Fitted Components for Dimension 

 Reduction in Regression, Statistical Science, Vol. 23, No. 4, 485-501. 

[7] Frank, I. E. and Friedman, J. H. (1993). A statistical view of some chemometrics 

 regression tools, Technometrics, Vol. 35, 109-148. 

[8] Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical 

 Learning: Data Mining, Inference, and Prediction, Springer, New York. 

[9] Helland, I. S. (1990). Partial Least Squares regression and Statistical models, 

 Scandinavian Journal of Statistics, Vol. 17, 97-114. 

[10] Hoerl, A. E. and Kennard, R. W. (1970). Ridge Regression: Applications to 

 Nonorthogonal Problems, Technometrics, Vol. 12, No. 1, 69-82. 

[11] Martens, H. and Næs, T. (1992). Multivariate Calibration, Wiley, New York. 

[12] Myers, R. H. (1990). Classical and Modern Regression with Applications, Duxbury 

 Press, California. 

[13] Refaeilzadeh, P., Tang, L., and Liu, H. (2009). Encyclopedia of Database Systems, 

 Springer, New York. 

[14] Li, K. (1991). Sliced Inverse Regression for Dimension Reduction, Journal of the 

 American Statistical Association, Vol. 86, No. 414, 316-327. 

[15] Tibshirani, R. (1996). Regression Shrinkage and Selection via the Lasso, Journal of 

 the Royal Statistical Society Series B (Methodological), Vol. 58, No. 1, 267-288.