estimates of error probability for complex gaussian channels with generalized likelihood ratio detection we derive approximate expressions for the probability of error in a two class hypothesis testing problem in which the two hypotheses are characterized by zero mean complex gaussian distributions these error expressions are given in terms of the moments of the test statistic employed and we derive these moments for both the likelihood ratio test appropriate when class densities are known and the generalized likelihood ratio test appropriate when class densities must be estimated from training data these moments are functions of class distribution parameters which are generally unknown so we develop unbiased moment estimators in terms of the training data with these accurate estimates of probability of error can be calculated quickly for both the optimal and plug in rules from available training data we present a detailed example of the behavior of these estimators and demonstrate their application to common pattern recognition problems which include quantifying the incremental value of larger training data collections evaluating relative geometry in data fusion from multiple sensors and selecting a good subset of available features

feature selection based on high dimensional model representation for hyperspectral images in hyperspectral image analysis the classification task has generally been addressed jointly with dimensionality reduction due to both the high correlation between the spectral features and the noise present in spectral bands which might significantly degrade classification performance in supervised classification limited training instances in proportion with the number of spectral features have negative impacts on the classification accuracy which is known as hughes effects or curse of dimensionality in the literature in this paper we focus on dimensionality reduction problem and propose a novel feature selection algorithm which is based on the method called high dimensional model representation the proposed algorithm is tested on some toy examples and hyperspectral datasets in comparison with conventional feature selection algorithms in terms of classification accuracy stability of the selected features and computational time the results show that the proposed approach provides both high classification accuracy and robust features with a satisfactory computational time

exact sample conditioned mse performance of the bayesian mmse estimator for classification error part i representation in recent years biomedicine has been faced with difficult high throughput small sample classification problems in such settings classifier error estimation becomes a critical issue because training and testing must be done on the same data a recently proposed error estimator places the problem in a signal estimation framework in the presence of uncertainty permitting a rigorous solution optimal in a minimum mean square error sense the uncertainty in this model is relative to the parameters of the feature label distributions resulting in a bayesian approach to error estimation closed form solutions are available for two important problems discrete classification with dirichlet priors and linear classification of gaussian distributions with normal inverse wishart priors in this work part i of a two part study we introduce the theoretical mean square error mse conditioned on the observed sample of any estimate of the classifier error including the bayesian error estimator for both bayesian models thus bayesian error estimation has a unique advantage in that its mathematical framework naturally gives rise to a practical expected measure of performance given an observed sample in part ii of the study we examine consistency of the error estimator demonstrate various mse properties and apply the conditional mse to censored sampling

classifier design with incomplete knowledge abstract pattern classification is a well studied problem in which the identity of an unknown pattern is determined to be one of m classes spanning the pattern space a new criterion function the inck incomplete knowledge criterion function is proposed which approximates the error probability when the m classes do not span the entire pattern space this criterion is based on the probabilistic measures obtained from a modified version of dubuisson and masson s statistical decision rule with reject the error probability or conversely the probability of correct classification can be determined without having complete knowledge about the class distributions the attractiveness of this criterion is that it is highly correlated with the statistical decision rule with reject which can be used by the classifier thus the criterion provides a good indication of the classifier performance that can be expected with the reduced feature space a classifier design is presented which incorporates the new criterion function as well it is described how the rejection classes of the statistical decision rule with reject can be used in a multiresolution classifier design to determine the class identity of patterns which cannot be determined at a given resolution experimental evidence is provided which demonstrates the advantages of the inck criterion function used in conjunction with a multiresolution classifier design

divergence based feature selection for multimodal class densities a new feature selection procedure based on the kullback j divergence between two class conditional density functions approximated by a finite mixture of parameterized densities of a special type is presented this procedure is suitable especially for multimodal data apart from finding a feature subset of any cardinality without involving any search procedure it also simultaneously yields a pseudo bayes decision rule its performance is tested on real data

a comparison of asymptotic error rate expansions for the sample linear discriminant function abstract several asymptotic expansions for approximating the expected or unconditional probability of misclassification for the sample linear discriminant function are compared for accuracy in terms of yielding the smallest mean absolute deviation from the exact value for 104 population configurations the actual expected probabilities of misclassification are found via monte carlo simulation a simple and relatively obscure asymptotic expansion derived by raudys tech cybern 4 168 174 1972 is found to yield better approximation than the well known asymptotic expansions

factoring polynomials over algebraic number fields we show that if f x is a polynomial in z alpha x where alpha satisfies a monic irreducible polynomial over z then f x can be factored over q alpha x in polynomial time we also show that the splitting field of f x can be determined in time polynomial in splitting field of f x q log x

searchlight based multi voxel pattern analysis of fmri by cross validated manova abstract multi voxel pattern analysis mvpa is a fruitful and increasingly popular complement to traditional univariate methods of analyzing neuroimaging data we propose to replace the standard decoding approach to searchlight based mvpa measuring the performance of a classifier by its accuracy with a method based on the multivariate form of the general linear model following the well established methodology of multivariate analysis of variance manova we define a measure that directly characterizes the structure of multi voxel data the pattern distinctness d our measure is related to standard multivariate statistics but we apply cross validation to obtain an unbiased estimate of its population value independent of the amount of data or its partitioning into training and test sets the estimate d can therefore serve not only as a test statistic but also as an interpretable measure of multivariate effect size the pattern distinctness generalizes the mahalanobis distance to an arbitrary number of classes but also the case where there are no classes of trials because the design is described by parametric regressors it is defined for arbitrary estimable contrasts including main effects pattern differences and interactions pattern changes in this way our approach makes the full analytical power of complex factorial designs known from univariate fmri analyses available to mvpa studies moreover we show how the results of a factorial analysis can be used to obtain a measure of pattern stability the equivalent of cross decoding

mixture densities maximum likelihood and the em algorithm the problem of estimating the parameters which determine a mixture density has been the subject of a large diverse body of literature spanning nearly ninety years during the last two decades the

discriminative wavelet packet filter bank selection for pattern recognition this paper addresses the problem of discriminative wavelet packet wp filter bank selection for pattern recognition the problem is formulated as a complexity regularized optimization criterion where the tree indexed structure of the wp bases is explored to find conditions for reducing this criterion to a type of minimum cost tree pruning a method well understood in regression and classification trees cart for estimating the conditional mutual information adopted to compute the fidelity criterion of the minimum cost tree pruning problem a nonparametric approach based on product adaptive partitions is proposed extending the darbellay vajda data dependent partition algorithm finally experimental evaluation within an automatic speech recognition asr task shows that proposed solutions for the wp decomposition problem are consistent with well understood empirically determined acoustic features and the derived feature representations yield competitive performances with respect to standard feature extraction techniques

thresholding method for dimensionality reduction in recognition systems often recognition systems must be designed with a relatively small amount of training data plug in test statistics suffer from large estimation errors often causing the performance to degrade as the measurement vector dimension increases choosing a better test statistic or applying a method of dimensionality reduction are two possible solutions to this problem in this paper we consider a recognition problem where the data for each population are assumed to have the same parametric distribution but differ in their unknown parameters the collected vectors of data as well as their components are assumed to be independent the system is designed to implement a plug in log likelihood ratio test with maximum likelihood ml estimates of the unknown parameters instead of the true parameters because a small amount of data is available to estimate the parameters the performance of such a system is strongly degraded relative to the performance with known parameters to improve the performance of the system we define a thresholding function that when incorporated into the plug in log likelihood ratio significantly decreases the probability of error for binary and multiple hypothesis testing problems for the exponential class of populations we analyze the modified test statistic and present the results of monte carlo simulation special attention is paid to the complex gaussian model with zero mean and unknown variances

a novel hybrid linear nonlinear classifier for two class classification theory algorithm and applications classifier design for a given classification task needs to take into consideration both the complexity of the classifier and the size of the dataset that is available for training the classifier with limited training data as often is the situation in computer aided diagnosis of medical images a classifier with simple structure e g a linear classifier is more robust and therefore preferred we propose a novel two class classifier which we call a hybrid linear nonlinear classifier hlnlc that involves two stages the input features are linearly combined to form a scalar variable in the first stage and then the likelihood ratio of the scalar variable is used as the decision variable for classification we first develop the theory of hlnlc by assuming that the feature data follow normal distributions we show that the commonly used fisher s linear discriminant function is generally not the optimal linear function in the first stage of the hlnlc we formulate an optimization problem to solve for the optimal linear function in the first stage of the hlnlc i e the linear function that maximizes the area under the receiver operating characteristic roc curve of the hlnlc for practical applications we propose a robust implementation of the hlnlc by making a loose assumption that the two class feature data arise from a pair of latent rather than explicit multivariate normal distributions the novel hybrid classifier fills a gap between linear discriminant analysis lda and quadratic discriminant analysis qda in the sense that both its theoretical performance and its complexity lie between those of the lda and those of the qda simulation studies show that the hybrid linear nonlinear classifier performs better than lda without increasing the classifier complexity accordingly with a finite number of training samples the hlnlc can perform better than that of the ideal observer due to its simplicity finally we demonstrate the application of the hlnlc in computer aided diagnosis of breast lesions in ultrasound images

estimation of channelized hotelling observer performance with known class means or known difference of class means this paper concerns task based image quality assessment for the task of discriminating between two classes of images we address the problem of estimating two widely used detection performance measures snr and auc from a finite number of images assuming that the class discrimination is performed with a channelized hotelling observer in particular we investigate the advantage that can be gained when either 1 the means of the signal absent and signal present classes are both known or 2 when the difference of class means is known for these two scenarios we propose uniformly minimum variance unbiased estimators of snr2 derive the corresponding sampling distributions and provide variance expressions in addition we demonstrate how the bias and variance for the related auc estimators may be calculated numerically by using the sampling distributions for the snr2 estimators we find that for both snr2 and auc the new estimators have significantly lower bias and mean square error than the traditional estimator which assumes that the class means and their difference are unknown

simultaneous feature selection and gaussian mixture model estimation for supervised classification problems abstract a new expectation maximization em algorithm for time critical supervised classification tasks in remote sensing is proposed compared to standard em and other approaches it has the following advantages 1 no knowledge about the class distributions is needed 2 the number of components is estimated 3 it does not require careful initialization 4 singular estimates are avoided due to the ability of pruning components 5 the best discriminating features are identified simultaneously 6 the features are identified by incorporating mahalanobis distances three experiments are carried out in order to demonstrate the relevance of the method the main findings are the following 1 feature selection is very important in terms of prediction quality of models 2 in the experiments the proposed method estimates better models than other state of the art methods 3 the incorporation of mahalanobis distances is very valuable for the identification of relevant features 4 the proposed method is more robust than the compared methods 5 in case of complex data distributions the new approach is able to provide better results

information loss of the mahalanobis distance in high dimensions application to feature selection when an infinite training set is used the mahalanobis distance between a pattern measurement vector of dimensionality d and the center of the class it belongs to is distributed as a chi2 with d degrees of freedom however the distribution of mahalanobis distance becomes either fisher or beta depending on whether cross validation or resubstitution is used for parameter estimation in finite training sets the total variation between chi2 and fisher as well as between chi2 and beta allows us to measure the information loss in high dimensions the information loss is exploited then to set a lower limit for the correct classification rate achieved by the bayes classifier that is used in subset feature selection

analytic study of performance of error estimators for linear discriminant analysis we derive double asymptotic analytical expressions for the first moments second moments and cross moments with the actual error for the resubstitution and leave one out error estimators in the case of linear discriminant analysis in the multivariate gaussian model under the assumption of a common known covariance matrix and a fixed mahalanobis distance as dimensionality approaches infinity sample sizes for the two classes need not be the same they are only assumed to reach a fixed but arbitrary asymptotic ratio with the dimensionality from the asymptotic moment representations we directly obtain double asymptotic expressions for the bias variance and rms of the error estimators the asymptotic expressions presented here generally provide good small sample approximations as demonstrated via numerical experiments the applicability of the theoretical results is illustrated by finding the minimum sample size to bound the rms in gene expression classification

whitney continua of curves in this paper we prove several theorems relating shape properties of whitney continua of curves in particular we investigate the fundamental dimension and the shape type of whitney continua of curves

information theoretic bounds on target recognition performance based on degraded image data this paper derives bounds on the performance of statistical object recognition systems wherein an image of a target is observed by a remote sensor detection and recognition problems are modeled as composite hypothesis testing problems involving nuisance parameters we develop information theoretic performance bounds on target recognition based on statistical models for sensors and data and examine conditions under which these bounds are tight in particular we examine the validity of asymptotic approximations to probability of error in such imaging problems problems involving gaussian poisson and multiplicative noise and random pixel deletions are considered as well as least favorable gaussian clutter a sixth application involving compressed sensor image data is considered in some detail this study provides a systematic and computationally attractive framework for analytically characterizing target recognition performance under complicated non gaussian models and optimizing system parameters

explicit bounds for primes in residue classes let e k be an abelian extension of number fields with e q let and n denote the absolute discriminant and degree of e let denote an element of the galois group of e k we prove the following theorems assuming the extended riemann hypothesis 1 there is a degree 1 prime p of k such that p e k satisfying np 1 o 1 log 2n 2 2 there is a degree 1 prime p of k such that p e k generates the same group as satisfying np 1 o 1 log 2 3 for k q there is a prime p such that p e q satisfying p 1 o 1 log 2 in 1 and 2 we can in fact take p to be unramified in k q a special case of this result is the following 4 if gcd m q 1 the least prime p m mod q satisfies p 1 o 1 q log q 2 it follows from our proof that 1 3 also hold for arbitrary galois extensions provided we replace by its conjugacy class our theorems lead to explicit versions of 1 4 including the following the least prime p m mod q is less than 2 q log q 2

shape properties of whitney maps for hyperspaces in this paper some shape properties of whitney maps for hyperspaces are investigated in particular the following are proved 1 let x be a continuum and let k be the hyperspace 2x or c x of x with the hausdorff metric then if x is any whitney map for for any 0 s s t w x t is an approximate strong deformation retract of w s t in particular sh w 1 t sh w 1 s t 2 pointed 1 movability is a whitney property 3 for any given n oo the property of cohomological dimension n is a sequential strong whitney reversible property 4 the property of being chainable or circle like is a sequential strong whitneyreversible property 5 the property of being an far is a whitney property for 1 dimensional continua property 2 is an affirmative answer to a problem of j t rogers 16 112 properties 3 and 4 are affirmative answers to problems of s b nadler 20 14 57 and 21 0 introduction by a continuum we mean a compact connected metric space for a given continuum x by the hyperspaces of x we mean 2x a c x i a is a nonempty closed subset of x c x a e 2x i a is connected which have the hausdorff metric pg see 20 the term mapping is a continuous function let a 2xor c x a mapping w 0 o x is called a whitney map for t 20 or 31 provided that the following conditions are satisfied a if a c b and a b then w a w b b o x 0 for each x e x in 1932 h whitney 31 showed that whitney maps always exist the notion of whitney map is a convenient and important tool for hyperspace theory a topological property p is said to be a whitney property 20 or 21 provided that whenever a continuum x has property p so does w f t for each whitney map w for c x and 0 t o x it is well known that connectibility is a whitney property if x is a received by the editors april 9 1985 and in revised form october 16 1985 1980 mathematics subject classification primary 54b20 secondary 54c55 54c56 54f45 kev words anid phrases hyperspaces whitney maps whitney property sequential strong whitneyreversible property approximate strong deformation retract pointed 1 movable far anr chainable circle like e mapping indecomposable 1986 american mathematical society 0002 9947 86 1 00 25 per page

whitney levels in hyperspaces of certain peano continua let x be a peano continuum let 2x resp c x be the space of all nonempty compacta resp subcontinua of x with the hausdorff matric let xo be a whitney map defined on sc 2x or c x such that x is admissible this requires the existence of a certain type of deformation of sc if sc c x assume x contains no free arc then for any to c 0 o x it is proved that w to o to and wf to o x are hilbert cubes this is an analogue of the curtis schori theorem for k a general result for the existance of admissible whitney maps is proved which implies that these maps exist when x is starshaped in a banach space or when x is a dendrite using these results it is shown for example that being an ar an anr an lc space or an lc space is not strongly whitney reversible introduction a continuum is a nonempty compact connected metric space a peano continuum is a locally connected continuum the hyperspaces of a continuum x are the spaces 2x a c x a is nonempty and compact and c x a c 2x a is connected metrized with the hausdorff metric 21 let f1 x x x c x a whitney map for a hyperspace sc of x is a continuous function w sc 0 xo such that o x 0 for each x c f1 x and if a b c c and a c b a then a b intuitively a whitney map is a size function if w is a whitney map for sc and 0 t o x then w l t is called a whitney level if 0 t x then w l t is called a positive whitney level thus a whitney level consists of compacta or continua in x of the same size basic facts about whitney maps and whitney levels are in 21 we note that whitney levels are coverings of x which as t gets close to zero converge to l 0 and consist of compacta of small diameter note that l 0 f1 x and hence l 0 is homeomorphic to x thus as t gets close to zero the whitney levels approximate x it is of interest to obtain information about the structure of positive whitney levels and to determine those properties which are preserved by the convergence of positive whitney levels to the zero level resultsa about this could be used to determine properties of x this is in analogy with the use of nerves of covers the nerves correspond to polyhedra these polyhedra approximate the original space as the mesh of the covers approaches zero and properties of these polyhedra determine various properties of the original space

texture classification using logical operators in this paper a new algorithm for texture classification based on logical operators is presented operators constructed from logical building blocks are convolved with texture images an optimal set of six operators are selected based on their texture discrimination ability the responses are then converted to standard deviation matrices computed over a sliding window zonal sampling features are computed from these matrices a feature selection process is applied and the new set of features are used for texture classification classification of several natural and synthetic texture images are presented demonstrating the excellent performance of the logical operator method the computational superiority and classification accuracy of the algorithm is demonstrated by comparison with other popular methods experiments with different classifiers and feature normalization are also presented the euclidean distance classifier is found to perform best with this algorithm the algorithm involves only convolutions and simple arithmetic in the various stages which allows faster implementations the algorithm is applicable to different types of classification problems which is demonstrated by segmentation of remote sensing images compressed and reconstructed images and industrial images

computing automorphisms of abelian number fields let l q be an abelian number field of degree n most algorithms for computing the lattice of subfields of l require the computation of all the conjugates of this is usually achieved by factoring the minimal polynomial m x of over l in practice the existing algorithms for factoring polynomials over algebraic number fields can handle only problems of moderate size in this paper we describe a fast probabilistic algorithm for computing the conjugates of which is based on p adic techniques given m x and a rational prime p which does not divide the discriminant disc m x of m x the algorithm computes the frobenius automorphism of p in time polynomial in the size of p and in the size of m x by repeatedly applying the algorithm to randomly chosen primes it is possible to compute all the conjugates of

determination of the optimal number of features for quadratic discriminant analysis via the normal approximation to the discriminant distribution given the joint feature label distribution increasing the number of features always results in decreased classification error however this is not the case when a classifier is designed via a classification rule from sample data typically for fixed sample size the error of a designed classifier decreases and then increases as the number of features grows the problem is especially acute when sample sizes are very small and the potential number of features is very large to obtain a general understanding of the kinds of feature set sizes that provide good performance for a particular classification rule performance must be evaluated based on accurate error estimation and hence a model based setting for optimizing the number of features is needed this paper treats quadratic discriminant analysis qda in the case of unequal covariance matrices for two normal class conditional distributions the qda classifier is determined according to a discriminant the standard plug in rule estimates the discriminant from a feature label sample to obtain an estimate of the discriminant by replacing the means and covariance matrices by their respective sample means and sample covariance matrices the unbiasedness of these estimators assures good estimation for large samples but not for small samples our goal is to find an essentially analytic method to produce an error curve as a function of the number of features so that the curve can be minimized to determine an optimal number of features we use a normal approximation to the distribution of the estimated discriminant since the mean and variance of the estimated discriminant will be exact these provide insight into how the covariance matrices affect the optimal number of features we derive the mean and variance of the estimated discriminant and compare feature size optimization using the normal approximation to the estimated discriminant with optimization obtained by simulating the true distribution of the estimated discriminant optimization via the normal approximation to the estimated discriminant provides huge computational savings in comparison to optimization via simulation of the true distribution feature size optimization via the normal approximation is very accurate when the covariance matrices differ modestly the optimal number of features based on the normal approximation will exceed the actual optimal number when there is large disagreement between the covariance matrices however this difference is not important because the true misclassification error using the number of features obtained from the normal approximation and the number obtained from the true distribution differ only slightly even for significantly different covariance matrices

asymptotic expansions for the means and variances of error rates summary the problem of classifying an observation x into one of k multivariate normal distributions is considered when samples are used to estimate the population parameters the probabilities of correct classification and the associated error rates are random variables asymptotic expansions for the expected values and variances of these random variables in terms of the inverses of the sample sizes are found simulations have been performed to evaluate the expansions

texture classification by wavelet packet signatures this correspondence introduces a new approach to characterize textures at multiple scales the performance of wavelet packet spaces are measured in terms of sensitivity and selectivity for the classification of twenty five natural textures both energy and entropy metrics were computed for each wavelet packet and incorporated into distinct scale space representations where each wavelet packet channel reflected a specific scale and orientation sensitivity wavelet packet representations for twenty five natural textures were classified without error by a simple two layer network classifier an analyzing function of large regularity d sub 20 was shown to be slightly more efficient in representation and discrimination than a similar function with fewer vanishing moments d sub 6 in addition energy representations computed from the standard wavelet decomposition alone 17 features provided classification without error for the twenty five textures included in our study the reliability exhibited by texture signatures based on wavelet packets analysis suggest that the multiresolution properties of such transforms are beneficial for accomplishing segmentation classification and subtle discrimination of texture

valid population inference for information based imaging from the second level t test to prevalence inference in multivariate pattern analysis of neuroimaging data second level inference is often performed by entering classification accuracies into a t test vs chance level across subjects we argue that while the random effects analysis implemented by the t test does provide population inference if applied to activation differences it fails to do so in the case of classification accuracy or other information like measures because the true value of such measures can never be below chance level this constraint changes the meaning of the population level null hypothesis being tested which becomes equivalent to the global null hypothesis that there is no effect in any subject in the population consequently rejecting it only allows to infer that there are some subjects in which there is an information effect but not that it generalizes rendering it effectively equivalent to fixed effects analysis this statement is supported by theoretical arguments as well as simulations we review possible alternative approaches to population inference for information based imaging converging on the idea that it should not target the mean but the prevalence of the effect in the population one method to do so permutation based information prevalence inference using the minimum statistic is described in detail and applied to empirical data

exact solution of linear equations usingp adic expansions a method is described for computing the exact rational solution to a regular systemax b of linear equations with integer coefficients the method involves i computing the inverse modp ofa for some primep ii using successive refinements to compute an integer vector bar x such that a bar x equiv b modp m for a suitably large integerm and iii deducing the rational solutionx from thep adic approximation bar x for matricesa andb with entries of bounded size and dimensionsn n andn 1 this method can be implemented in timeo n 3 logn 2 which is better than methods previously used

experimental study on prototype optimisation algorithms for prototype based classification in vector spaces prototype based classification relies on the distances between the examples to be classified and carefully chosen prototypes a small set of prototypes is of interest to keep the computational complexity low while maintaining high classification accuracy an experimental study of some old and new prototype optimisation techniques is presented in which the prototypes are either selected or generated from the given data these condensing techniques are evaluated on real data represented in vector spaces by comparing their resulting reduction rates and classification performance usually the determination of prototypes is studied in relation with the nearest neighbour rule we will show that the use of more general dissimilarity based classifiers can be more beneficial an important point in our study is that the adaptive condensing schemes here discussed allow the user to choose the number of prototypes freely according to the needs if such techniques are combined with linear dissimilarity based classifiers they provide the best trade off of small condensed sets and high classification accuracy

moments and root mean square error of the bayesian mmse estimator of classification error in the gaussian model the most important aspect of any classifier is its error rate because this quantifies its predictive capacity thus the accuracy of error estimation is critical error estimation is problematic in small sample classifier design because the error must be estimated using the same data from which the classifier has been designed use of prior knowledge in the form of a prior distribution on an uncertainty class of feature label distributions to which the true but unknown feature distribution belongs can facilitate accurate error estimation in the mean square sense in circumstances where accurate completely model free error estimation is impossible this paper provides analytic asymptotically exact finite sample approximations for various performance metrics of the resulting bayesian minimum mean square error mmse error estimator in the case of linear discriminant analysis lda in the multivariate gaussian model these performance metrics include the first second and cross moments of the bayesian mmse error estimator with the true error of lda and therefore the root mean square rms error of the estimator we lay down the theoretical groundwork for kolmogorov double asymptotics in a bayesian setting which enables us to derive asymptotic expressions of the desired performance metrics from these we produce analytic finite sample approximations and demonstrate their accuracy via numerical examples various examples illustrate the behavior of these approximations and their use in determining the necessary sample size to achieve a desired rms the supplementary material contains derivations for some equations and added figures highlightsthe bayesian minimum mean square error bmmse error estimator of lda is studied the first second and cross moments of the bmmse estimator with true error are considered both conditional and unconditional performance metrics are analyzed for the first time the kolmogorov double asymptotic is used in a bayesian setting asymptotically exact finite sample approximations of performance metrics are derived

a model for nonpolynomial decrease in error rate with increasing sample size much theoretical evidence exists for an inverse proportionality between the error rate of a classifier and the number of samples used to train it cohn and tesauro 1992 have however discovered various problems which experimentally display an approximately exponential decrease in error rate we present evidence that the observed exponential decrease is caused by the finite nature of the problems studied a simple model classification problem is presented which demonstrates how the error rate approaches zero exponentially or faster when sufficiently many training samples are used

tree pruning with subadditive penalties in this paper we study the problem of pruning a binary tree by minimizing over all pruned subtrees of the given tree an objective function that combines an additive cost term with a penalty term that depends only on tree size we present algorithms for general size based penalties although our focus is on subadditive penalties roughly penalties that grow more slowly than linear penalties with increasing tree size such penalties are motivated by recent results in statistical learning theory for decision trees but may have wider application as well we show that the family of pruned subtrees induced by a subadditive penalty is a subset of the family induced by an additive penalty this implies by known results about additive penalties that the family induced by a subadditive penalty 1 is nested 2 is unique and 3 can be computed efficiently it also implies that when a single tree is to be selected by cross validation from the family of prunings subadditive penalties will never present a richer set of options than an additive penalty

on testing trained vector quantizer codebooks this paper discusses a criterion for testing a vector quantizer vq codebook that is obtained by training when a vq codebook is designed by a clustering algorithm using a training set time average distortion which is called the training set distortion tsd is usually calculated in each iteration of the algorithm since the input probability function is unknown in general and cumbersome to deal with the algorithm stops when the tsd ceases to significantly decrease in order to test the resultant codebook validating set distortion vsd is calculated on a separate validating set vs codebooks that yield small difference between the tsd and the vsd are regarded as good ones however the difference vsd tsd is not necessarily a desirable criterion for testing a trained codebook unless certain conditions are satisfied a condition that is previously assumed to be important is that the vs has to be quite large to well approximate the source distribution this condition implies greater computational burden of testing a codebook in this paper we first discuss the condition under which the difference vsd tsd is a meaningful codebook testing criterion then convergence properties of the vsd a time average quantity are investigated finally we show that for large codebooks a vs size as small as the size of the codebook is sufficient to evaluate the vsd this paper consequently presents a simple method to test trained codebooks for vq s experimental results on synthetic data and real images supporting the analysis are also provided and discussed

application of global optimization methods to model and feature selection many data mining applications involve the task of building a model for predictive classification the goal of this model is to classify data instances into classes or categories of the same type the use of variables not related to the classes can reduce the accuracy and reliability of classification or prediction model superfluous variables can also increase the costs of building a model particularly on large datasets the feature selection and hyper parameters optimization problem can be solved by either an exhaustive search over all parameter values or an optimization procedure that explores only a finite subset of the possible values the objective of this research is to simultaneously optimize the hyper parameters and feature subset without degrading the generalization performances of the induction algorithm we present a global optimization approach based on the use of cross entropy method to solve this kind of problem

effects of sample size in classifier design the effect of finite sample size on parameter estimates and their subsequent use in a family of functions are discussed general and parameter specific expressions for the expected bias and variance of the functions are derived these expressions are then applied to the bhattacharyya distance and the analysis of the linear and quadratic classifiers providing insight into the relationship between the number of features and the number of training samples because of the functional form of the expressions an empirical approach is presented to enable asymptotic performance to be accurately estimated using a very small number of samples results were experimentally verified using artificial data in controlled cases and using real high dimensional data

a hybrid feature extraction selection approach for high dimensional non gaussian data clustering this paper presents an unsupervised approach for feature selection and extraction in mixtures of generalized dirichlet gd distributions our method defines a new mixture model that is able to extract independent and non gaussian features without loss of accuracy the proposed model is learned using the expectation maximization algorithm by minimizing the message length of the data set experimental results show the merits of the proposed methodology in the categorization of object images

improvement of handwritten japanese character recognition using weighted direction code histogram several algorithms for preprocessing feature extraction pre classification and main classification are experimentally compared to improve the recognition accuracy of handwritten japanese character recognition the compared algorithms are three types of nonlinear normalization for the preprocessing the discriminant analysis and the principal component analysis for the feature extraction the minimum distance classifiers and the linear classifier for the high speed pre classification and modified bayes classifier and subspace method for the robust main classification the performance of the recognition algorithm is fully tested using the etl9b character database the recognition accuracy of 99 15 at the recognition speed of eight characters per second is achieved this accuracy is the best one ever reported for the database

integrated fisher linear discriminants this paper studies fisher linear discriminants flds based on classification accuracies for imbalanced datasets an optimal threshold is found out from a series of empirical formulas developed which is related not only to sample sizes but also to distribution regions a mixed binary decimal coding system is suggested to make the very dense datasets sparse and enlarge the class margins on condition that the neighborhood relationships of samples are nearly preserved the within class scatter matrices being or approximately singular should be moderately reduced in dimensionality but not added with tiny perturbations the weight vectors can be further updated by a kind of epoch limited three at most iterative learning strategy provided that the current training error rates come down accordingly putting the above ideas together this paper proposes a type of integrated flds the extensive experimental results over real world datasets have demonstrated that the integrated flds have obvious advantages over the conventional flds in the aspects of learning and generalization performances for the imbalanced datasets an optimal threshold is found out from a series of empirical threshold formulas developed for fisher linear discriminants based on classification accuracies weight vectors and thresholds are updated by an epoch limited iterative learning strategy the singular within class scatter matrices are reduced in dimensionality but not added with perturbations a coding system enlarges class margins and approximately preserves neighborhood relationships an integrated learning algorithm improves the learning and generalization performances of fisher linear discriminants

data compression and harmonic analysis in this paper we review some recent interactions between harmonic analysis and data compression the story goes back of course to shannon s r d theory in the case of gaussian stationary processes which says that transforming into a fourier basis followed by block coding gives an optimal lossy compression technique practical developments like transform based image compression have been inspired by this result in this paper we also discuss connections perhaps less familiar to the information theory community growing out of the field of harmonic analysis recent harmonic analysis constructions such as wavelet transforms and gabor transforms are essentially optimal transforms for transform coding in certain settings some of these transforms are under consideration for future compression standards we discuss some of the lessons of harmonic analysis in this century typically the problems and achievements of this field have involved goals that were not obviously related to practical data compression and have used a language not immediately accessible to outsiders nevertheless through an extensive generalization of what shannon called the sampling theorem harmonic analysis has succeeded in developing new forms of functional representation which turn out to have significant data compression interpretations we explain why harmonic analysis has interacted with data compression and we describe some interesting recent ideas in the field that may affect data compression in the future

on signal representations within the bayes decision framework this work presents new results in the context of minimum probability of error signal representation mpe sr within the bayes decision framework these results justify addressing the mpe sr criterion as a complexity regularized optimization problem demonstrating the empirically well understood trade off between signal representation quality and learning complexity contributions are presented in three folds first the stipulation of conditions that guarantee a formal tradeoff between approximation and estimation errors under sequence of embedded transformations are provided second the use of this tradeoff to formulate the mpe sr as a complexity regularized optimization problem and an approach to address this oracle criterion in practice is given finally formal connections are provided between the mpe sr criterion and two emblematic feature transformation techniques used in pattern recognition the optimal quantization problem of classification trees cart tree pruning algorithms and some versions of fisher linear discriminant analysis lda

deciding the nilpotency of the galois group by computing elements in the centre we present a new algorithm for computing the centre of the galois group of a given polynomial f q x along with its action on the set of roots of f without previously computing the group we show that every element in the centre is representable by a family of polynomials in q x for computing such polynomials we use quadratic newton lifting and truncated expressions of the roots of f over a p adic number field as an application we give a method for deciding the nilpotency of the galois group if f is irreducible with nilpotent galois group an algorithm for computing it is proposed

linear discriminant analysis with misallocation in training samples abstract linear discriminant analysis for a two class case is studied in the presence of misallocation in training samples a general approach to modeling of misallocation is formulated and the mean vectors and covariance matrices of the mixture distributions are derived the asymptotic distribution of the discriminant boundary is obtained and the asymptotic first two moments of the error rates are given certain numerical results for the error rates are presented by considering the random and two nonrandom misallocation models

a quadratic classifier for high dimensional periodic measurement pattern recognition problems abstract this paper presents an implementation of a quadratic classifier for pattern recognition problems with measurements which are either periodic or numerous the classifier discriminant function is derived from a second order statistical approximation of the parzen estimate of the class conditional probability density function this approach is shown to offer considerable advantage in storage and the computation time requirements during training classification and updating phases without any sacrifice of the classification accuracy

unsupervised learning of parsimonious mixtures on large spaces with integrated feature and component selection estimating the number of components the order in a mixture model is often addressed using criteria such as the bayesian information criterion bic and minimum message length however when the feature space is very large use of these criteria may grossly underestimate the order here it is suggested that this failure is not mainly attributable to the criterion e g bic but rather to the lack of structure in standard mixtures these models trade off data fitness and model complexity only by varying the order the authors of the present paper propose mixtures with a richer set of tradeoffs the proposed model allows each component its own informative feature subset with all other features explained by a common model shared by all components parameter sharing greatly reduces complexity at a given order since the space of these parsimonious modeling solutions is vast this space is searched in an efficient manner integrating the component and feature selection within the generalized expectation maximization gem learning for the mixture parameters the quality of the proposed unsupervised solutions is evaluated using both classification error and test set data likelihood on text data the proposed multinomial version learned without labeled examples without knowing the true number of topics and without feature preprocessing compares quite favorably with both alternative unsupervised methods and with a supervised naive bayes classifier a gaussian version compares favorably with a recent method introducing feature saliency in mixtures

generalization error of linear discriminant analysis in spatially correlated sensor networks generalization error the probability of error of a detection rule learned from training samples on new unseen samples is a fundamental quantity to be characterized however characterizations of generalization error in the statistical learning theory literature are often loose and practically unusable for optimizing detection systems in this work focusing on learning linear discriminant analysis detection rules from spatially correlated sensor measurements a tight generalization error approximation is developed that can be used to optimize the parameters of a sensor network detection system as such the approximation is used to optimize network settings the approximation is also used to derive a detection error exponent and select an optimal subset of deployed sensor nodes a gauss markov random field is used to model correlation and weak laws of large numbers in geometric probability are employed in the analysis

on dimensionality sample size and classification error of nonparametric linear classification algorithms this paper compares two nonparametric linear classification algorithms the zero empirical error classifier and the maximum margin classifier with parametric linear classifiers designed to classify multivariate gaussian populations formulae and a table for the mean expected probability of misclassification mep sub n are presented they show that the classification error is mainly determined by n p a learning set size dimensionality ratio however the influences of learning set size on the generalization error of parametric and nonparametric linear classifiers are quite different under certain conditions the nonparametric approach allows us to obtain reliable rules even in cases where the number of features is larger than the number of training vectors