Fisher information matrix: A tool for dimension reduction, projection pursuit, independent component analysis, and more

The Canadian Journal of Statistics Vol. 40, No. 4, 2012, Pages 712–730 La revue canadienne de statistique Fisher information matrix: A tool for dimension reduction, projection pursuit, independent component analysis, and more Bruce G. LINDSAY 1 and Weixin YAO 2 * 1 Department 2 Department of Statistics, The Pennsylvania State University, University Park, PA 16802, USA of Statistics, Kansas State University, Manhattan, KS 66502, USA Key words and phrases: Dimension reduction; Fisher information matrix; independent component analysis; projection pursuit; white noise matrix. MSC 2010: Primary 62-07; secondary 62H99. Abstract: Hui & Lindsay (2010) proposed a new dimension reduction method for multivariate data. It was based on the so-called white noise matrices derived from the Fisher information matrix. Their theory and empirical studies demonstrated that this method can detect interesting features from high-dimensional data even with a moderate sample size. The theoretical emphasis in that paper was the detection of non-normal projections. In this paper we show how to decompose the information matrix into non-negative deﬁnite information terms in a manner akin to a matrix analysis of variance. Appropriate information matrices will be identiﬁed for diagnostics for such important modern modelling techniques as independent component models, Markov dependence models, and spherical symmetry. The Canadian Journal of Statistics 40: 712– 730; 2012 © 2012 Statistical Society of Canada R e ´ sum e ´ : Hui et Lindsay (2010) ont propos´e une nouvelle m´ethode de r´eduction de la dimension pour les donn´ees multidimensionnelles. Elle est bas´ee sur des matrices commun´ement qualiﬁ´ees de bruits blancs obtenues a ` partir de la matrice d’information de Fisher. Leurs e ´ tudes th´eoriques et empiriques montrent que cette m´ethode peut d´etecter des caract´eristiques int´eressantes a ` partir de donn´ees de grande dimension mˆeme si les e ´ chantillons sont de taille mod´er´ee. L’emphase th´eorique de cet article e ´ tait mise sur la d´etection des projections non normales. Nous montrons ici comment d´ecomposer la matrice d’information en termes d’information d´eﬁnie non n´egative de fac¸on similaire a ` l’analyse de variance matricielle. Des matrices d’information appropri´ees peuvent e ˆ tre identiﬁ´ees comme diagnostic pour les techniques de mod´elisation modernes telles que les mod`eles en composantes ind´ependantes, les mod`eles de d´ependance markovienne et la sym´etrie sph´erique. La revue canadienne de statistique 40: 712–730; 2012 © 2012 Societe statistique du Canada 1. INTRODUCTION Let f (x) be a density function for the d-dimensional random vector X. The density f will be treated as the unknown, and will be the fundamental object of our investigation. Let f (x − θ) be the corresponding location family in θ. Assume that regularity holds, so that one can construct the d × d information matrix J X for the parameter θ. The paper Hui & Lindsay (2010) showed that J X can be reliably estimated and used for data-based dimension reduction to the most interesting linear combinations of a multivariate dataset. The selected projections were those that maximized information. The main purpose of this paper is to create a deeper understanding of how J X can be decomposed into a summation of information matrices that carry information about such features of f as spherical symmetry, independence, and Markov independence. * Author to whom correspondence may be addressed. E-mail: wxyao@ksu.edu © 2012 Statistical Society of Canada / Societe statistique du Canada