Mem. Gra. Sci. Eng. Shimane Univ.
Series B: Mathematics
48 (2015), pp. 15–26

ASYMPTOTIC THEORY FOR DISCRIMINANT ANALYSIS IN
HIGH DIMENSION LOW SAMPLE SIZEMITSURU TAMATANI
Communicated by Kanta Naito(Received: March 2, 2015)
Abstract. This paper is based on the author’s thesis, “Pattern recognition
based on naive canonical correlations in high dimension low sample size”. This
paper is concerned with discriminant analysis for multi-class problems in a High
Dimension Low Sample Size (hdlss) context. The proposed discrimination
method is based on canonical correlations between the predictors and response
vector of class label. We investigate the asymptotic behavior of the discrimination method, and evaluate bounds for its misclassification rate.

1. Introduction
Discriminant analysis is a statistical method by which a new data with unknownclass label is classified into one of the previously known classes. Fisher’s well-known
linear discriminant function for K-class problems is the solution to his paradigm:
maximize the between-class variance while minimizing the within-class variances.
See Fisher [6], and Rao [10] for the general multi-class setting. Especially, for the
special case of two-class problems, we assume that the two populations, C1 and C2 ,have multivariate normal distributions which diﬀer in their means µ1 and µ2 , butshare a common covariance matrix Σ. For X from one of the two classes, Fisher’slinear discriminant function
(
)T
1
(1)
g(X) = X − (µ1 + µ2 ) Σ−1 (µ1 − µ2 )2assigns X to C1 if g(X) > 0, and to C2 otherwise. If the random vectors have equal
probability of belonging to either of the two classes, then the misclassification rateof (1) is shown to be [Φ (−∆2 /2) + {1 − Φ (∆2 /2)}] /2, where Φ is the standard
normal distribution function and ∆ is the Mahalanobis distance between the two
2000 Mathematics Subject Classification. 62H30, 62H20, 62E20,
Key words and phrases. High Dimension Low Sample Size, Canonical Correlations, Consistency, Misclassification, Multi-class Linear Discriminant Analysis.
15

16

M. TAMATANI√
populations: ∆ =(2)(µ1 − µ2 )T Σ−1 (µ1 − µ2 ). A sample version of (1) is()T1b −1 (bb 2) Σ
b 2 ),
gb(X) = X − (bµ +µµ1 − µ2 1

b are appropriate estimates for the corresponding populationb 1, µb 2 and Σ
where µparameters of the two classes. (2) can be applied to data sets in the general setting
where the sample size n is much larger than the dimension d. However, for problems
with a large dimension, it has been shown that Fisher’s linear discriminant function
performs poorly due to diverging spectra. Further for High Dimension Low Sample
Size (hdlss) problems, namely n ≪ d, it is not possible to directly define Fisher’sb in (2) becomeslinear discriminant function since the sample covariance matrix Σ
singular when the dimension d exceeds the sample size n.
Several discriminant rules have been proposed for the hdlss context which overb in diﬀerent ways: Dudoit et al. [4] proposedcome the problem of singularity of Σ
diagonal linear discriminant analysis which only uses the diagonal elements of thesample covariance matrix, Srivastava and Kubokawa [12] proposed a discriminantfunction based on the Moore-Penrose inverse, Ahn and Marron [1] constructed discriminant function based on the Maximal Data Piling (MDP) direction vectors, andAoshima and Yata [2] considered a discriminant rule based on second moments inconjunction with geometric representations of high-dimensional data. In this paper, we focus on the ‘diagonal’ approach of Dudoit et al. [4] which has special
appeal since it is conceptually simpler than the competitors.
For two-class problems in a hdlss setting, Bickel and Levina [3] and Fan andFan [5] investigated the asymptotic behavior of the naive Bayes rule, and calculated bounds for its misclassification rate. Tamatani, Koch and Naito [13] defineda modification of the canonical correlation matrix that is suitable for classification
problems, and studied the asymptotic behavior of the eigenvector and the discriminant direction of the modified canonical correlation matrix in the context of
two-class problems. Also, for multi-class problems in a hdlss context, Tamatani,
Naito and Koch [14] proposed a naive Bayes rule in a general multi-class setting
and investigated its asymptotic properties for high-dimensional data when both
the dimension d and the sample size n grow.
Throughout this paper we focus on hdlss data from K(≥ 2) classes, that is, we
assume that the dimension d of the data is much bigger than the sample size n.
Our discriminant approach is based on canonical correlations, and in particular ona modification of the canonical correlation matrix suitable for vector-valued classlabels from K classes. In this framework, we replace the covariance matrix by its
diagonal counterpart as discussed in Dudoit et al. [4], Bickel and Levina [3] andFan and Fan [5]. We call such a matrix the naive canonical correlation matrix,
and observe that this matrix plays an important role in the present theory. The
K − 1 eigenvectors belonging to K − 1 non-zero eigenvalues of the estimated naive
canonical correlation matrix yield discriminant directions which inform our choiceof a discriminant function for K classes. For this setting we study the asymptotic
behavior of the eigenvectors and associate discriminant directions.

ASYMPTOTIC THEORY FOR DISCRIMINANT ANALYSIS IN HDLSS

17

For hdlss data from K multivariate normal classes, we derive an upper boundfor the misclassification rate of the proposed multi-class discriminant function.
Our asymptotic results for the misclassification rate are divided into two disjoint
types depending on the precise growth rates of d and n. Depending on the two
distinct growth rates, we also develop hdlss asymptotic results for estimators of
the eigenvectors and discriminant directions.
The paper is organized as follows. In Section 2, we review a relationship between
multi-class discriminant function and canonical correlations for K classes. Further,
we focus on the gene expression microarray data, which is a typical example ofhdlss data. For hdlss problems, we replace Σ by diagΣ, and obtain a corresponding criterion, which leads to a natural derivation of a multi-class version of
the naive Bayes rule. Section 3 details the asymptotic behavior of the eigenvectorsof the estimated naive canonical correlation matrix and the associated discriminantdirections in a hdlss setting under general distributional assumptions. We derive
an upper bound for the asymptotic misclassification rate of the proposed multiclass discriminant function under assumptions of normal distribution. In Section
4, we summarize our results. Proofs of all mathematical results are omitted.
2. Discriminant function based on naive canonical correlation
We consider the K-class discrimination problem. Let Cℓ (ℓ = 1, . . . , K) be ddimensional populations with diﬀerent means µℓ and common covariance matrix
Σ. For a random vector X from one of the K classes, let πℓ be the probability that
X belongs to Cℓ . Let Y be the K-dimensional vector-valued class label∑ with ℓthentry 1 if X belongs to Cℓ and 0 otherwise, so P (Y = eℓ ) = πℓ , and Kℓ=1 πℓ = 1.
In canonical correlation∑ analysis of two vectors X and Y , with Y the vectorvalued labels and µ = Kℓ=1 πℓ µℓ , the matrix
[
]{ [
]}
e = Σ−1/2 E (X − µ) Y T E Y Y T −1/2
(3)C
plays an importantrole. ]From the definition of (X, Y ) it follows that E[Y Y T ] =[Π and E (X − µ)Y T = M0 Π where Π = diag(π1 , . . . , πK ) and M0 = [ µ1 −∑K
T
= M0 ΠM0T , we have
µ · · · µK − µ ]. Using M =ℓ=1 πℓ (µℓ − µ)(µℓ − µ)eCeT = Σ−1/2 M Σ−1/2 . If we put ee, where p
e is a solution to the eigenvalueCb = Σ−1/2 pTeeee = λ̃ep with λ̃ > 0, then the vector b is the maximizer of the criterion
problem C C p
T

e = b MbJ(b)bT Σbover vectors b. Note that (4) is nothing other than the criterion which yields
Fisher’s linear discriminant rule for the multi-class setting. In particular, the rankeC
eT is K − 1, so the K − 1 eigenvectors [ pe1 · · · peK−1 ] belonging to K − 1
of C
non-zero eigenvalues should be used for constructing the discriminant directionse≡[ee ··· pe
B
b1 · · · ebK−1 ] = Σ−1/2 [ p
].
(4)

1

K−1

e we want to define a discriminant function ge
Using the discriminant directions B,
for classifying new observations X whose class is unknown. For α ∈ {1, . . . , K −1},

18

M. TAMATANIT
put Zα (X) = e
bα X, and define the vector Z(X) = [Z1 (X), . . . , ZK−1 (X)]T =e T X. For Z(X) and Cℓ , define the Mahalanobis distance between Z(X) and Cℓ
B√e T µℓ and Σℓ =by ∆ℓ (Z(X)) = (Z(X) − ν ℓ )T Σ−1 (Z(X) − ν ℓ ), where ν ℓ = Bℓ

eTe Hence ∆ℓ (Z(X)) can be rewritten as
B ΣB.
(5)

2

e B
e T ΣB)e −1 Be T (X − µℓ ) .
∆ℓ (Z(X))2 = (X − µℓ )T B(

Using (5), we now derive the multi-class linear discriminant function ge as theminimizer of the Mahalanobis distance, and let(6)ge(X) = arg min ∆ℓ (Z(X))2 .
ℓ∈{1,...,K}e depends on Σ and that (6) reduces to the Fisher’s linear discriminantNote that B
function in the case of 2-class by considering the sign of ∆1 (Z(X))2 − ∆2 (Z(X))2 .
2.1. HDLSS setting. Gene expression microarray data is a form of highthroughput biological data providing relative measurements of mRNA levels forgenes in a biological sample. One important application of gene expression microarray data is the classification of new samples into known classes. In the recent years,
many laboratories have collected and analyzed microarray data, and such data havebeen appeared in public databases or on web sites. Indeed, the data collected fromgene expression microarrays consist of thousands or tens of thousands of genes that
constitute features: leukemia data (Golub et al. [7], K = 2, n = 73, d = 7129),
lung cancer data (Gordon et al. [8], K = 2, n = 184, d = 12433), prostate cancer
data (Singh et al. [11], K = 2, n = 134, d = 12600), the small round blue celltumors (SRBCT) data (Khan et al. [9], K = 4, n = 83, d = 2308), and so on.
These data, often called the High Dimension Low Sample Size (hdlss) data, arecharacterized with large number of dimensions d and a relatively small number ofsample size n, that is, we can write n ≪ d.
To establish asymptotic theory in this paper, we consider an asymptotic situationin which the dimension d and the sample size n both approach infinity in such a
way that d is much faster than n:n = o(d) as n, d −→ ∞.
2.2. Diﬃculties in HDLSS. The hdlss data involves serious problems. Let
X i (i = 1, . . . , n) be independently and identically distributed (i.i.d.) d-dimensionalrandom vectors with the covariance matrix Σ. The usual estimates of Σ is defined
as
n
1 ∑
b
b )(X i − µb )T ,(7)
Σ=(X i − µn − 1 i=1∑b = (1/n) ni=1 X i is the mean vector of the data {X i }1≤i≤n . When we
where µassume that the data satisfies n ≪ d, (7) is a singular matrix because the rank of(7) is at most n − 1. Hence, it means that the sample version of (3) is not directly
applicable in the case of n ≪ d.

ASYMPTOTIC THEORY FOR DISCRIMINANT ANALYSIS IN HDLSS

19

2.3. Naive canonical correlations. In the hdlss two-class discrimination setb is singular as was discussed in the previous
tings, the sample covariance matrix Σsubsection. For a population framework, it therefore does not make sense to definea discriminant function based on the within-class covariance matrix Σ. To overcome such a diﬃculty, we first require a suitable framework for the population. We
define the naive canonical correlation matrix C and vectors bα and pα by
[
]{
}−1/2
C = D−1/2 E (X − µ)Y T E[Y Y T ],where D = diagΣ, bα = D−1/2 pα and pα is eigenvector of the matrix CC T corresponding to the αth largest eigenvalue λ∗α . Put P = [ p1 · · · pK−1 ]. The
discriminant directions
B ≡ [ b1 · · · bK−1 ] = [ D−1/2 p1 · · · D−1/2 pK−1 ] = D−1/2 P
can now be seen to maximise the analogous naive criterionbT M b.
bT DbNote that Σ in (4) has been replaced by the diagonal matrix D in (8).
J(b) =(8)

2.4. Discriminant function in HDLSS. The corresponding discriminant function g is therefore(9)g(X) = arg min (X − µℓ )T B(B T DB)−1 B T (X − µℓ ) .
ℓ∈{1,...,K}

Note that Σ has been replaced by D both in (4) and (5) to yield (8) and (9)respectively.
It is worth noting that for K = 2, the discriminant function g reduces to thenaive Bayes discriminant function discussed in Bickel and Levina [3].
3. Asymptotic Theory
In this section, we evaluate the asymptotic behavior of the estimators of pα andbα in a hdlss setting under general assumptions about the underlying distributions.
In addition, we derive upper bounds for the misclassification rate of multi-classdiscriminant function under d-dimensional normal populations.
3.1. Preliminaries. Consider data (X ℓi , Y ℓi ) (ℓ = 1, . . . , K; i = 1, . . . , nℓ ), where
the independently distributed X ℓi are from K disjoint classes, and the Y ℓi are
independent realizations of vector labels{
1 ℓ-th component,T
Y ℓi = (Yℓi1 , . . . , YℓiK )with Yℓij =0 otherwise.
Let X and Y be matrices defined by X = [ X 11 · · · X KnK ] and Y =[ Y 11 · · · Y KnK ]. Then X is of size d × n, and Y is of size K × n, where∑n= Kℓ=1 nℓ .
Next we derive an empirical version of C and its left eigenvectors pα . Define∑ ℓb = (1/K) ∑K Sbℓ ,b of µℓ and Σ by µb ℓ and Σb ℓ = (1/nℓ ) ni=1
X ℓi and Σ
estimators µ
ℓ=1

20

M. TAMATANI∑ ℓb ℓ )(X ℓi − µb ℓ )T . Using the centering matrix,
where Sbℓ = {1/(nℓ − 1)} ni=1
(X ℓi − µa natural estimator for C isb = D
b −1/2 M
c0 N 1/2 ,(10)C
b = diagΣ,b In is the n × n identity matrix, 1n is the n-dimensional vector
where Dc0 = [ µb1 − µb ··· µbK − µb ] and N = diag(n1 /n, . . . , nK /n). Hence weof ones, M
T
−1/2
bC
b =D
b
cD
b −1/2 , where M
c=M
c0 N Mc0T .
obtain the expression C
M
bC
bT is K − 1, we use the K − 1 eigenvectors pbC
bT correbα of C
Since the rank of C
sponding to the K − 1 non-zero eigenvalues in the definition of the discriminationb −1/2 pbα (α = 1, . . . , K − 1). We note that these b
directions b
bα , and put bbα = Dbαcan be obtained as the maximizers of the function
Tcb = b Mb.
J(b)
b
bT Db
b −1/2 Pb, then gb, definedb −1/2 pb=[bb −1/2 pbK−1 ] ≡ D
b1 · · · D
Put B
b1 · · · bbK−1 ] = [ D
by
b B
bT D
b B)
b −1 Bb T (X − µb ℓ )T B(b ℓ) ,
gb(X) = arg min (X − µℓ∈{1,...,K}

is a natural estimator of g in (9), and this gb is our proposed discriminant functionin the hdlss multi-class setting. To elucidate the asymptotic behavior of gb, it isb as well as Pb in a hdlss setting.
necessary to develop first asymptotics for B
Throughout this paper, we make the assumption that the sample size of each of
the K classes satisfies c ≤ nℓ /n for some positive constant c and ℓ = 1, . . . , K.
In what follows we use the asymptotic notation:(i) an,d = O(bn,d ) to mean that an,d /bn,d → M ∈ (0, ∞) as n, d → ∞.
(ii) an,d = o(bn,d ) to mean that an,d /bn,d → 0 as n, d → ∞.
The definition of o is usually included in that of big O, however we distinguishthese cases in this paper.
bα , we need the following defiIn order to evaluate the asymptotic behavior of pnition and conditions:b , a vector ofDefinition 3.1. Let x ∈ Rd be a non-stochastic unit vector, and let x
length one, denote an estimate of x based on the sample of size n . IfP

b T x −→ 1 as n, d → ∞,xPb is hdlss consistent with x.
where −→ refers to convergence in probability, then x
Condition A. Let X = [ X 11 · · · X KnK ] be a data matrix from K classes. Eachcolumn of X can be written as X ℓi = µℓ +εℓi , where εℓi (ℓ = 1, . . . , K; i = 1, . . . , nℓ )are i.i.d. copies of an underlying random vector ε with mean 0 and covariancematrix Σ.
Condition B. (Cramér’s condition) There exist constants ν1 , ν2 , M1 and M2
such that each component of ε = (ε1 , . . . , εd )T satisfies
E[|εj |m ] ≤ m!M1m−2 ν1 /2 and E[|ε2j − σjj |m ] ≤ m!M2m−2 ν2 /2

ASYMPTOTIC THEORY FOR DISCRIMINANT ANALYSIS IN HDLSS

21

for all m ∈ N.
Condition C. Let nℓ → ∞ (for ℓ = 1, . . . , K), d → ∞, log d = o(n), n = o(d).
There exists a positive sequence Cd depending only on the dimension d such thatd/(nCd ) → ξ, where ξ ≥ 0.
Condition D. All eigenvalues λ∗α of C T C are simple (so λ∗1 > · · · > λ∗K ) andsatisfyλ∗α − λ∗α+1
λ∗α = O(Cd ) and> ξ for α = 1, . . . , K − 1,
Cd
and Cd as in Condition C.
Condition E. As d → ∞, µTℓ D−1 µℓ = O(Cd ), and there exists δ ∈ (0, 1) suchthat µTℓ D−1 µk = O(Cdδ ) for all k, ℓ ∈ {1, . . . , K}, and Cd as in Condition C.
Condition F. For k, ℓ ∈ {1, . . . , K}, and k ̸= ℓ,√ µTk D−1 µk√ µT D−1 µℓπk̸= lim πℓ ℓ
,
d→∞
d→∞
Cd
Cd
lim

and Cd as in Condition C.
We note that the positive sequence Cd will play an important role in the subsequentdiscussions since it controls the gap between d and n.
3.2. Asymptotic results for eigenvectors. To establish the asymptotic bebα , we start with an asymptotic expansion of the matrix
haviour of the estimators p
T bb
C C/Cd . In what follows, Θ denotes the parameter space for our multi-class setting:{}mink̸=ℓ (µk − µℓ )T D−1 (µk − µℓ ) ≥ Cd ,
Θ = (µ1 , . . . , µK , Σ),λmax (R) ≤ b0 , min1≤j≤d σjj > 0
where R is the correlation matrix R = D−1/2 ΣD−1/2 , λmax (R) is the largest eigenvalue of R and σjj is jth diagonal entry of Σ.
Lemma 3.2. Suppose that Condition A – Condition E hold. Then for allb d can be expanded asbT C/Cparameters θ ∈ Θ, C
b
bT C
C
CT C
=
+ ξ(IK − Π1/2 1K 1TK Π1/2 ) + 1K 1TK oP (1).
Cd
Cd
If ξ = 0, which means d = o(nCd ) by Condition C, then the above expansion
becomes very simple.
To proceed with the theoretical considerations, we need to show that the K − 1eigenvalues of C T C/Cd + ξ(IK − Π1/2 1K 1TK Π1/2 ) are simple in the case of d =O(nCd ), that is, ξ > 0.

22

M. TAMATANI

Lemma 3.3. Let λα /Cd be αth largest eigenvalue of C T C/Cd + ξ(IK −Π1/2 1K 1TK Π1/2 ). Suppose that the λα /Cd satisfy Condition D, and that d =O(nCd ). Then for all parameters θ ∈ Θ,
λα = O(Cd )

and

λα
λα+1
>
Cd
Cd

for α = 1, . . . , K − 1.

In this paper, eigenvectors have unit length. In addition, we assume that thefirst entry of each eigenvector is positive. This assumption avoids any ambiguity
about the direction of the eigenvector. Using Lemmas 3.2 and 3.3, we can now
bα as follows.
describe the asymptotic behavior of p
Theorem 3.4. Suppose that Condition A – Condition D hold. Then, for allparameters θ ∈ Θ,
bTα
p

b α
Cγ
= 1 + oP (1) for α = 1, . . . , K − 1,b α ||||Cγ

where γ α is eigenvector of C T C/Cd + ξ(IK − Π1/2 1K 1TK Π1/2 ) belonging to the nonzero eigenvalue λα .
bα , we define the vectors
To gain further insight in the behavior of the p
pα =

Cγ α
||Cγ α ||for α = 1, . . . , K − 1

by referring to Theorem 3.4. Under the assumption that d = O(nCd ), one can showbα are consistent estimators for the pα . We have the followingthat the vectors p
theorem and corollary:
Theorem 3.5. Suppose that Condition A – Condition E hold, and that d =O(nCd ). Moreover, assume that λα /Cd → κα and γ Tα Π1/2 1K → ηα . Then for allparameters θ ∈ Θ,κβ δαβ − ξ(δαβ − ηα ηβ ) P
PbT P −→  √ √,κα κβ − ξ(1 − ηβ2 )1≤α,β≤K−1

where δαβ is the Kronecker delta-function.
Corollary 3.6. Suppose that Condition A – Condition E hold, and that d =o(nCd ). Then for all parameters θ ∈ Θ,P
PbT P −→ IK−1 .

bα is hdlss consistent
Corollary 3.6 states that if d = o(nCd ) is satisfied, then p
bα is asymptotically orthogonal to pβ for α ̸= β. On thewith pα . Furthermore, pbα and pβ converges
other hand, if d = O(nCd ) is satisfied, then the angle between p
to a particular non-zero angle for all α and β.

ASYMPTOTIC THEORY FOR DISCRIMINANT ANALYSIS IN HDLSS

23

3.3. Asymptotic results for discriminant directions. Next we turn to theasymptotic behavior of the vectors b
bα . We define normalized versions of direction
vectors for discrimination byb −1/2 p∗bαD−1/2 pαDb(11), b∗α = √bα = √
pTα D−1 pαb −1 pbα
bTα D
pfor α = 1, . . . , K − 1. Then we have the following theorem and corollary:
Theorem 3.7. Suppose that Condition A – Condition E hold, and thatd = O(nCd ). Put σmax = max1≤j≤d σjj and σmin = min1≤j≤d σjj . Then for allparameters θ ∈ Θ,√b∗Tb∗β κα − ξ(1 − ηα2 )∗T ∗αbbα bβ ≤ √(1 + oP (1)),κα − ξ(1 − ηα2 ) (1 − σmin /σmax )√∗κα − ξ(1 − ηα2 )
b∗T
b
∗T ∗
α β
b
bα bβ ≥ √(1 + oP (1)).
κα − ξ(1 − ηα2 ) (1 − σmax /σmin )∗∗b∗ = [ bCorollary 3.8. Let B
b1 · · · bbK−1 ] and B ∗ = [ b∗1 · · · b∗K−1 ]. Supposethat Condition A – Condition E hold, and that d = o(nCd ). Then for allparameters θ ∈ Θ,P
b ∗T B ∗ − B ∗T B ∗ −→B
O.
∗TTheorem 3.7 states that the upper and lower bounds of bbα b∗β are determined by
the ratio of σmax and σmin . For example, if all diagonal elements of Σ are equal,
then
√
∗T ∗
ξ
P
∗T
∗
b
bα bα − bα bα 1 − (1 − ηα2 ) −→ 0.
κα∗
If d = o(nCd ) is satisfied, then the angle between bbα and b∗α converges to 0 in∗
probability, which shows that the bbα are hdlss consistent with the corresponding∗b∗ . However, bb and b∗ may not necessarily be orthogonal for α ̸= β, sinceααβpTα D−1 pβ∗T ∗b√= oP (1).
bα bβ − √pTα D−1 pα pTβ D−1 pβ3.4. Upper bound for misclassification rate. In this section, we study the
misclassification rate of our method in a multi-class setting. The misclassification
rate for two classes has been investigated in Fan and Fan [5] who derived an upperbound for the misclassification rate in a hdlss setting.
Specifically, for the results in this section we will assume that Condition A’
holds.
Condition A’. Let X = [ X 11 · · · X KnK ] be a data matrix from K classes. Eachcolumn of X can be written as X ℓi = µℓ + εℓi , where εℓi are i.i.d. random vectors

24

M. TAMATANI

distributed as Nd (0, Σ), d-dimensional normal distribution with mean vector 0 andcovariance matrix Σ.
In addition to Condition A, Condition A’ makes statements about the distribution of X.
Suppose that X belongs to class Ck . The misclassification rate of gb, an estimateof the discriminant function g in (9), for class Ck is defined as
Wk (bg , θ) = P (bg (X) ̸= k |X ℓi , ℓ = 1, . . . , K; i = 1, . . . , nℓ )()∫
1
1 T b −1√
= 1−
exp − z Σk z dz
2bkDb
|2π Σk |()
bk ; 0, Σbk ,≡ 1 − ΦK−1 D
b k is the transformed covariance matrix of size (K − 1) × (K − 1) which
where Σbk is the (K − 1)-dimensionalis defined in Tamatani, Naito and Koch [14], and D
region given by
{
}
bk =
D
z ∈ RK−1 zj < dˆkα , α ∈ {1, . . . , K − 1} ,where dˆkα = I(α < k)dˆkα + I(α ≥ k)dˆk(α+1) anddˆkα = √

b B
bT D
b B)
b −1 Bb T (bb α )/2)T B(b α)(µk − (bµk + µµk − µ
b B
bT D
b B)
b −1 Bb T ΣB(
b B
bT D
b B)
b −1 Bb T (bb α )T B(b α)(bµk − µµk − µ.

b1 results in the interval obtained in Theorem 1 in Fan
We note that the region D
and Fan [5] for their special case of K = 2.
Let Θk be the parameter space associated with the misclassification rate of gb forclass Ck :{}minℓ̸=k (µℓ − µk )T D−1 (µℓ − µk ) ≥ Cd ,Θk = (µ1 , . . . , µK , Σ).
λmax (R) ≤ b0 , min1≤j≤d σjj > 0
bk we also require the following region and quantities
In addition to the region D
in Theorem 3.9:{}
Dk,O = z ∈ RK−1 zα < dkα (1 + oP (1)), α = 1, . . . , K − 1 ,where dkα = I(α < k)dkα + I(α ≥ k)dk(α+1) ,[ {} ]−1 T T
Γ Qkα
Skα Γ ΓT C T C + (d/n)(IK − Π1/2 1K 1TK Π1/2 ) Γ√,
dkα = √
}
]−1
[ {
T
T
T
1/2
1/2
T
T
Γ Qkα
λmax (R) Qkα Γ Γ C C + (d/n)(IK − Π 1K 1K Π ) ΓASYMPTOTIC THEORY FOR DISCRIMINANT ANALYSIS IN HDLSS

25

Γ = [γ 1 , . . . , γ K−1 ], Skα = Mkα /2 + (d/n)skα Π−1/2 , Qkα = Mkα + (d/n)q kα Π−1/2 ,Mkα = (µk − µα )T D−1/2 C,
skα = [s1 , . . . , sK ] ,q kα = [q1 , . . . , qK ] ,1{I(ℓ = k) + I(ℓ = α)} ,2qℓ = I(ℓ = k) − I(ℓ = α).
s ℓ = πℓ −Furthermore, we need the following region in Corollary 3.10:{}
Dk,o = z ∈ RK−1 zα < d∗kα (1 + oP (1)), α = 1, . . . , K − 1 ,where d∗kα = I(α < k)d∗kα + I(α ≥ k)d∗k(α+1) and√T
Mkα Γ (ΓT C T CΓ)−1 ΓT Mkα
∗
√
dkα =
.
2 λmax (R)We have the following theorem and corollary using Theorem 3.4.
Theorem 3.9. Suppose that Condition A’ and Condition B – Condition F
hold, and that d = O(nCd ). Then, for all parameters θ ∈ Θk ,
(
)
b
Wk (b
g , θ) ≤ 1 − ΦK−1 Dk,O ; 0, Σk .
Corollary 3.10. Suppose that Condition A’ and Condition B – Condition
F hold, and that d = o(nCd ). Then, for all parameters θ ∈ Θk ,
(
)
bk .
Wk (b
g , θ) ≤ 1 − ΦK−1 Dk,o ; 0, Σ
Note that Theorem 3.9 and Corollary 3.10 extend Theorem 1 in Fan and Fan [5]to the general multi-class setting considered in this paper.
4. Conclusion
In this paper, we discussed the asymptotic theories of the multi-class linear
discriminant function in a hdlss context. In Section 2, we constructed the linear
discriminant function based on naive canonical correlation in the context of multiclass problem. In Section 3, we derived the asymptotic behavior of eigenvectorsof the naive canonical correlation matrix corresponding to positive eigenvalues.
In the asymptotic theory, both the dimension d and the sample size n grow, andprovided d does not grow too fast, we showed that all eigenvectors and discriminantdirections are hdlss consistent.
References[1] Ahn, J. and Marron, J. S. The maximal data piling direction for discrimination. Biometrika,
97 (2010), 254–259.
[2] Aoshima, M. and Yata, K. Two-stage procedures for high-dimensional data. Sequential analysis, 30 (2011), 356–399.
[3] Bickel, P. J. and Levina, E. Some theory for Fisher’s linear discriminant function,’naive
Bayes’, and some alternatives when there are many more variables than observations.
Bernoulli, 10 (2004), 989–1010.

26

M. TAMATANI[4] Dudoit, S., Fridlyand, J. and Speed, T. P. Comparison of discrimination methods for theclassification of tumors using gene expression data. Journal of the American statistical association, 97 (2002), 77–87.
[5] Fan, J. and Fan, Y. High dimensional classification using features annealed independence
rules. Annals of statistics, 36 (2008), 2605–2637.
[6] Fisher, R. A. The use of multiple measurements in taxonomic problems. Annals of eugenics,
7 (1936), 179–188.
[7] Golub, T. R., Slonim, D. K., Tamayo, P., Huard, C., Gaasenbeek, M., Mesirov, J. P., Coller,
H., Loh, M. L., Downing, J. R., Caligiuri, M. A., Bloomfield, C. D. and Lander, E. S.
Molecular classification of cancer: class discovery and class prediction by gene expression
monitoring. science, 286 (1999), 531–537.
[8] Gordon, G. J., Jensen, R. V., Hsiao, L., Gullans, S. R., Blumenstock, J. E., Ramaswamy,
S., Richards, W. G., Sugarbaker, D. J. and Bueno, R. Translation of microarray data into
clinically relevant cancer diagnostic tests using gene expression ratios in lung cancer andmesothelioma. Cancer Research, 62 (2002), 4963–4967.
[9] Khan, J., Wei, J., Ringnér, M., Saal, L. H., Ladanyi, M., Westermann, F., Berthold, F.,
Schwab, M., Antonescu, C. R., Peterson, C. and Meltzer, P. S. Classification and diagnostic
prediction of cancers using gene expression profiling and artificial neural networks. Nat. Med.,
7 (2001), 673–679.
[10] Rao, C. R. The utilization of multiple measurements in problems of biological classification.
Journal of the Royal Statistical Society. Series B (Methodological), 10 (1948), 159–203.
[11] Singh, D., Febbo, P. G., Ross, K., Jackson, D. G., Manola, J., Ladd, C., Tamayo, P., Renshaw, A. A., D’Amico, A. V., Richie, J. P., Lander, E. S., Loda, M., Kantoﬀ, P. W., Golub,
T. R. and Sellers, W. R. Gene expression correlates of clinical prostate cancer behavior.
Cancer cell, 1 (2002), 203–209.
[12] Srivastava, M. S. and Kubokawa, T. Comparison of discrimination methods for high dimensional data. J. Japan Statist. Soc, 37 (2007), 123–134.
[13] Tamatani, M., Koch, I. and Naito, K. Pattern recognition based on canonical correlations
in a high dimension low sample size context. Journal of Multivariate Analysis, 111 (2012),
350–367.
[14] Tamatani, M., Naito, K. and Koch, I. Multi-class discriminant function based on canonicalcorrelation in high dimension low sample size. Bulletin of Informatics and Cybernetics, 45(2013), 67-101.
Mitsuru Tamatani: Graduate school of science and engineering, Shimane University, Matsue, Shimane, 690-8504, Japan
E-mail address: tamatani@riko.shimane-u.ac.jp